Flip the rug over to expose the rubber backing.
Remove any of the remaining rubber backing by scraping it away with a plastic putty knife. Scrape in a direction away from your body for safety.
Throw away all the rubber chunks and examine the rug for any smaller pieces that remain.
Remove any of the smaller pieces by carefully scraping them off with a razor blade.
Vacuum the back of the rug with a hand-held vac or the hose attachment of a full-sized vacuum cleaner. This will remove any rubber particles that remain.
Replace the backing, if desired. There are several options available, including liquid latex and self-adhesive rubber backing.