Stand about 20 to 30 feet from the front of the house. If you can see visual damage to the siding from that distance, that is a clear indication that you need to replace the siding soon.
Check the siding up close for mildew, mold or dry rot if it is wood siding. These types of problems cannot be repaired--you need to replace the affected wooden planks. If you allow the rotted planks to remain, they could begin to affect the rest of the wood siding, making replacement an urgent matter.
Examine the siding for deep dents, holes, cracks and scratches (common for vinyl and aluminum varieties). If a significant number of the planks (about 50 percent or more) have physical damage, consider replacing the previous installation.
Test the flexibility of the material if you have vinyl siding. If it bends easily, falls off frequently or won't stay connected to the other planks at all, you have to replace the siding to resolve the issue.