Move all your belongings out of the closet and place elsewhere.
Square the hole in the interior closet wall with a drywall saw. Make it even on all four sides, exposing the wall studs.
Measure the size of the squared hole with a tape measure.
Mark out a section of a drywall panel with a pencil equal to the size of the squared hole.
Score along the pencil lines with a utility knife on both sides of the drywall panel. Snap out the square patch from the panel.
Fit the drywall square into the closet wall hole. Fasten it to the wall studs with drywall screws using a screw gun.
Press drywall tape over the square's seams. Spread joint compound over the tape and square it with a putty knife. Let the compound dry overnight.
Sand the square patch with fine-grit sandpaper to get it smooth and flush. Wipe over with a tack cloth. Prime the patch and paint it with a matching color.