If you are talking a 2-player game, then the penalty is simply a loss by the resigning player (and a win by the other player). That has the usual ratings consequences. I'm not sure what more you would want.
In 3-player games, the penalty is harsher, the ratings loss by the resigning players is magnified, because it makes it impossible to get an accurate result between the remaining players.
Dota 2 is a team game, so if you quit, you are actively hurting your own team mates. In Dominion, there are no team mates, you hurt yourself only in 2-player games, and as I explained, you get a harcher penalty then just placing third in a 3-player game.