Sunflowers, as their name implies, require at least 6 hours of sun each day, and most varieties require a lot of water. If either of these are lacking in your garden, your sunflowers will not be able to manufacture enough food for vigorous growth.
Sunflowers develop deep tap roots. Although they will grow in just about any type of soil, they prefer a type that's well-drained and fertile; compacted soil prevents their roots from adequate penetration and stunts their growth.
Sunflowers grow slowly if they don't get enough nutrition, particularly nitrogen. However, too much fertilizer results in bushy growth. Use compost, bat guano, or a 5-10-10 fertilizer.