It might also just be that the person asking you just always asks. Because as you mentioned, only asking when someone “looks” trans or non-binary can be rather invalidating. So to avoid that, they just don’t assume.

For your last paragraph, I’m personally of the opinion that, short of de-gendering the language entirely, a good solution would basically just be a gender/pronoun badge, but stylised to be more easily readable from a distance. Like a bracelet or a necklace or something of that nature. That would eliminate the need to ask in the vast majority of cases, because the person would be wearing something that unambiguously signals the answer. And it would be completely detached from the presentation of their body, which might not match their gender, or their clothing, which probably shouldn’t be gendered anyway. Changing pronouns, for whatever reason like coming out or just being fluid, would just be a matter of swapping out the single symbol.

It’s not really feasible, of course, but even as a queer person I find asking and being asked quite clunky. But whenever I go into LGBT+ or geek spaces, I find that wearing a badge just sidesteps the whole issue.