It really depends on your interpretation of the wording in the UCMJ. Also it woulnt be the brother of a sr or jr military rank you would have to worry about. It would be someone whos not a member seeing you together at a social gathering. Its not unlike running in to someone at a bar, ucmj says socializing in public places is ok, as long as colonel so-in-so didnt invite privet snuffy. privately socializing is not ok.
Think like this, col smith is the. Wm, sgt snuffy is the SW, col calls sgt snuffy at lunch and tells him he.needs to see him after snuffy eats to go over the GMs official visit later that night. 1ST SGT JONES(who is snuffys superior and not a mason) sees the caller id and asks snuffy why the col is calling him..... he says its for something to do with lodge, jones then tells snuffy he is not aloud to attend meetings that the col will be at, the col overrides that and jones goes to the General (also not a mason) who decides it is fraternazation and charges them both with such.