Exhaust question

Should I be using my carbon filters for exhaust all the time or only saving for flowering? If only during flower, then veg would use exhaust fans with no filter triggered by thermostat. How do I prevent odors escaping through the non-filtered fan during flower?