Take off the spent bloom at the ovary or the swollen part at the base of the bloom. This is where the seed forms if the flower was pollinated, and the fortnight lily seeds profusely. Unless you want little baby lily plants, it is a good idea to remove the ovary.
Prune off the ovary at the top of the stem where the two items meet. You can also simply pinch off the flower. It may need a pinch and twisting motion, depending on how thick the stem is.
Remove any swollen spent flowers even after the petals have fallen. It is not too late to remove them and doing so spurs production of more blooms. Don't cut off the flower stems until they begin to die back in late fall.
Cut back the flower stems in late fall when they are no longer producing. Leave the foliage so it can continue to collect energy for the following season's bloom. Take the flower stems off at the base of the plant with pruners.
Remove all the foliage to the ground in early winter or wait until earliest spring if the ratty foliage isn't an eyesore. This prepares the plant to grow and produce flowers the next season.