Well, if they were different colors, then he would have to guess, correct. That is why this is a deduction answer riddle. Once, the third guy realizes that the fourth guy has to guess, he knows that his hat and the second man's are different, therefore HE (the third man) would know that his hat color is opposite of the second man's.

Get it now? The answer isn't going to be one set man, it is a combination of the third and fourth through deduction.