To answer the poll, no, I would not return this man's sign. Not in public. That said...I am preparing for my EA exam, which is only 2 days away, so this is all very fresh in my mind.
The signs of an Entered Apprentice (as well as all degrees), have an allusion. The token and words are ways for you to identify yourself to other Masons. I mean, that's part of the exam, right? You need to know the signs, token, and words in order to advance. You need to be a MM in order to visit other Lodges (at least in my state), all of which being dependent on identifying yourself to others.
I don't interpret signs, tokens and words as being for Lodge use only. Part of the ritual is designed to give you the means to examine others.