There are three main types of weathering: physical weathering, chemical weathering and biological weathering. Brick buildings are primarily susceptible to physical weathering.
Bricks are naturally porous, meaning they have small holes in them. This makes it easier for water and other elements to get into the brick and break it down over time. In general, the more porous a brick is, the quicker it will break down.
Ice wedging (also known as ice crystallization) occurs when water gets into a crack or pore in a brick, and then expands as it freezes, fracturing the brick and potentially breaking pieces of it off.
Salt crystallization is much like ice wedging. This occurs when a saline solution enters a brick and expands as it crystallizes. Salt crystallization is normally only a concern in coastal areas and in hot, dry environments.
Bricks are also exposed to rain and wind. Although no single weather event significantly erodes brick, the effect of wind and rain over decades and centuries will eventually help to break it down.