Whether the soil is still good or seemingly poor, you can use it to fill low areas in the lawn or garden where water tends to pool after a hard rain. Chances are, the soil was once decent since it was used for gardening. Even if it is depleted, it is not as depleted as many varieties of fill dirt for sale, and you don't have to pay for dirt from your old raised bed.
If the soil is still fairly good, use it to revitalize an old garden or as a topsoil layer for a new garden. The top few inches of soil is where most microbial and beneficial insect life is active, and constantly recycling organic matter. Even if it hasn't had anything but weeds in it for a long time, those cycles are still active. Be sure to mulch the garden at least 3 inches after using soil that has weeds and their seeds in it to reduce future maintenance problems.
Laying clear plastic over soil to allow sunlight in but trapping heat while keeping moisture out of the soil is called solarizing. Solarizing the soil in the raised bed will kill weeds and seeds and most harmful soil microbes (and, unfortunately, the good ones as well). Then the soil can be used for potting soil. Alternately, there are many methods for cooking small amounts of soil in an oven or microwave to kill pathogens, and these smaller amounts can be used for potting soil as needed. Mix it with compost and/or sand to make it lighter for potted plants.
If the old raised bed is simply in the wrong spot, consider moving it to a more convenient location, changing its shape or revitalizing it. If moving a very deep raised bed, set aside the top 4 to 6 inches of soil, then use the bottom soil for the bottom of the new raised bed. The top few inches have important organisms and structure that would be wasted at deeper levels. Use the top few inches of the old raised bed for the top of the new raised bed. Alternately, revitalize the soil in the old raised bed by adding one or more inches of compost.