Gelt - Wizard Doomstack without special building?

About to begin a campaign as Gelt, I would like to create a doomstack of wizards, but my question is: Is it possible to have insane amount of Winds of Magic even if I return my provinces to Zhao Ming and lose access to the special building? (If I recall correctly it was called Temple of Elemental Winds or something like that).

I would like to go back to the Empire because my coop buddy is over there, but I also don't want to lose my Shadow Wizard Money Gang.