A number of the components in your furnace may be attached to the inside of your system. Your furnace may also be attached to the wall with brackets. This mounting can become loose over time, which causes the furnace to vibrate, especially when the furnace burners and air flow suddenly shut down.
Check valves are the valves in heat pumps that control the flow of refrigerant. If your heat pump is reversing its flow or shutting down, a check valve that is not operating correctly or has become stuck can interfere with the refrigerant's flow. This can make the lines shudder as the fluid is forced through the bad valve, which leads to the vibrating sounds you're hearing.
Furnaces have blower systems that draw air in from supply vents and circulate it throughout the house once it's warmed. Vibration problems with these systems are common. Small changes in placement and the accumulation of dust over time can both result in vibration problems. The fan blades can loosen or become crooked and the ductwork itself may develop mounting issues. These all lead to vibrations in the unit which can echo through your house.
Motors in furnaces are used to drive the fans. In heat pumps, they are also used to pump refrigerant and run the compressor. When working properly these motors are typically quiet, but if the motor starts to wear out or malfunction, it can begin vibrating. This is a sign of an old motor or one that needs to be repaired before it burns out, but you should check the screws first to ensure the motor is secure, since it can become loose.