Exchanged travelers should not count towards the "gained cards" that treasure hunter refers to, but currently they do.
(I didn't think to record the game number. It wasn't much of a problem for me in-game because I had Raze to trash the excess silvers and pull out travelers.
Usually Treasure Hunter works correctly, so if there was a bug it must have been something very specific to the game. Your last game was #8530920 and it had Page so I loaded it. On your turn 5 you gained 5 silvers of TH. Your opponent gained a Magpie (playing Magpie), 3 Silvers (playing TH) and a bought Raze the previous turn. Maybe you missed the Magpie gain?
You're right. I just didn't pay enough attention to the log of his turn.