Reverse Isekai D&D Setting

I was toying around with the idea of a new campaign as our 5-year campaign is about to come at a close. I'm thinking of a reverse isekai D&D setting, where adventurers from various campaign settings (at Level 1) would get transported to Earth. These level 1 characters would be doing a part of the adventures they started off from, until an intervention happens and they get yeeted into Earth, which has recently been on the receiving end of attacks from D&D monsters and creatures.

I'm still trying to find a simple justification though why only items and abilities that these adventurers carry or use can harm these creatures (so that I don't modify the vanilla weapons of D&D too much). Any ideas would be greatly appreciated!