It all boils down to the Pagans, I think.

And there's probably some kind of Pagan roots in the whole story of Christmas, including the gifts from the Wise Men - which is where our own gift-giving might come from.

Either way, who are we to argue with it? It's not as if we dislike getting gifts at Christmas, is it?