Are magical builds supposed to be this weak to physical damage?

here is the build
I'm getting absolutely shredded by those skeletons with crossbows or by anything that gets too close.

For the build, I'm following just a few general tips, but not some specific guide with planed steps.

(Additional info: I don't have any of the dlc's. I haven't finished the game yet. I'm playing on normal difficulty.)