Think of a pure melee Assassin in Diablo2. Fist of Fire, Blade of Ice, Claw of Thunder, Phoenix Strike etc. Or Vengeance for Paladin. Using Magical elements to enchant their melee weapons and attacks don't make them a caster.

Then you are saying that all the parts and upgrades shouldn't be allowed as well? Because they are interchangeable between armors just like tanks.