When you react to an attack by playing Caravan Guard using Way of the Otter, you draw two cards, and Caravan Guard goes into play.
But I'm seeing it get discarded immediately when your turn starts. Is this supposed to happen, or is it supposed to stay in play?
It's supposed to be discarded in your opponent's turn, when they discard their stuff (which may look like start of your turn, as there usually isn't much happening in between).
During each Clean-Up, all players discard their "no longer needed" cards in play. Usually that's only the active player, but in cases like this it would also be you. Since the Caravan Guard does not have any outstanding effects to do after this turn, it is discarded.