Its not controversial and i also do that after hanging out with friends but it matters how u say it.

Anon would probably go “thanks for hanging out with me” which would sound kinda depressing but i guess the people in this hypothetical scenario were aware that he was a loner so it wouldnt be an issue actually.