knowing the opponents hand is not worth much 99% of the game time (contraband and endgame is the 1%). Only a few percentage of ppl cheat. So there is a cheat possibility that only occurs rarely if ever.
On the other hand, a lot of ppl spectate better players to learn, or stream in spectating mode.
So to solve the small issue, you would do harm to a lot of people.
In cases like this, do as little harm as possible - leave it at it is.
Contraband is a very interesting edge case, tho. But in the hypothetical situation that I had to choose, I would rather ban Contraband than SpectatingMode. Reality check: Contraband is the new Scout, so the game wont loose much.