Although round ducts in residential HVAC systems are commonly made of fiberglass, always use sheet metal for your connection to a plenum especially if the ducts carry hot air. If you do not use sheet metal you may be violating building codes (See Reference 1).
Connecting round to square is a familiar issue to those who buy a new furnace or air conditioning unit and find that the plenum opening and existing ducts have different shapes. You can buy commercial connections that match standard square plenum openings to round ducts, both of which are commonly sold in 4-inch increments.
If you are skilled at working with sheet metal, you can make your own transition from square to round. You can buy computer aided design (CAD) software to create correct folding patterns for the proper connection. CAD programs are sold under the brand names Solid Edge, Inventor and SolidWorks among others (See Reference 1). You typically enter into the program the length and width of the plenum entrance plus diameter of your round duct and the height of the transition. The software will create a pattern for you to make a sheet metal connection yourself.
Round ducts are used mostly for horizontal branch ducts and are typically marketed in 1, 5 and 10 foot lengths (See Reference 4). An 8 by 8-inch square plenum opening contains 64 inches of space and will carry 300 cubic feet of air per minute (cfm) (See Reference 4). A round duct 8 inches in diameter has 50 square inches of area and will carry 250 cfm.