Agreed. I have a similar issue for Duchess, where my mouse pointer instinctevely goes towards the Duchess pile whenever I want to gain one via her own ability. Pilgrimage might be another case, where a crowded in-play area might make a selection physically difficult.
OTOH, player confusion might be increased when they see little '+' crosses on supply piles, thinking they are still in the buy phase. But for Changeling that is probably not a problem.