You can cut drafts from switches by sealing them, once you know where the drafty ones are. Use a stick of incense to check your switches for draftiness. Wait for a windy day, light the incense stick, hold it close to a switch and note how the smoke behaves. A drafty switch will push the smoke sideways. Or moisten the back of your hand and place it in front of the switch to feel for airflow. Pay particular attention to switches on exterior walls, but don’t forget to check the ones on interior walls. These can also be a source of air leaks.
Before doing any work on a light switch, always turn off the power to that switch. Then undo the screws holding on the cover plate and pull the plate off. Typically there will be an air gap between the sides of the electric junction box and the drywall that needs to be sealed before you do anything else.
If the gap between the junction box and drywall is small, less than a 1/4 inch, fill the gap by pressing rope caulk into the gap or filling it with latex caulk from a caulking gun. For larger gaps, use low-expansion foam sealant formulated to seal framing around doors and windows.
Junction boxes holding light switches are not airtight. The front of the box needs to be sealed. Home and hardware centers carry inexpensive foam gaskets for this purpose. They come in different styles to accommodate single and multiple switches. Place the gasket over the switch from the front. If the sealant from around the junction box interferes with setting the gasket, cut away the excess. Apply the gasket, replace the switch cover plate and tighten the retaining screws.