I fully agree. If you want to be able to cast subtle spells, invest in the abilities required to do so. Otherwise I say someone starting to cast a spell without warning will be treated the same as someone pulling out a knife without warning: sure they might be planning something on doing something harmless like peel an apple, but without context, you're going to assume the worst and react accordingly.