Insert pieces of insulation into large cracks alongside your windows. Push the bits of insulation into the cracks with a putty knife.
Finish sealing the large cracks with caulk. Insert the tip of the caulking tube into the crack. Squeeze a bead of caulk into one end of the crack to create a starting point. Continue squeezing as you draw the caulking gun down or across to fill in the remainder of the crack. Finish with a bead of caulk in the last corner and then press the caulk into the corner with your finger to ensure there are no tiny gaps.
Caulk small gaps around the window insulation to close them up. Employ the same method as you used to fill in the previous gaps with caulk.
Fill cracks too big for caulking with urethane foam. Press the nozzle of the urethane foam can into the crack. Insert enough foam to fill the crack without allowing it to overflow outside the crack.