That would be a great build, in fact I have a guy with only one magic skill and that is destruction the rest of his major skills are blade armor block ect. ect. whatever you want, but i would suggest always taking some kind of big boom magic.
I agree with wetwired though the enemy leveling is the same as you leveling. You only get like what? 10 hitpoints? if that a level and the enemies get to be in groups too. So in essence its like fighting four or five of yourself at the same level with the same level of skills.