I have 5 gallons of drywall primer, would this be okay to use on the wood trim also?

Primer works on all types of dry surfaces. Just make sure it's not going to bleed thru the final coat. Try it on a piece of scrap first.