... But the "one is enough" doesn't seem fair ...
It's not supposed to be fair. One brother is supposed to be able to exclude any candidate with or without valid reasons and that's not fair. In fact it does occasionally happen that brothers exclude candidates for improper reasons. It doesn't have to be about prejudice. A brother might reject because he's doing a veto on the current line. A brother might hold pique against his entire lodge and reject until something major changes in it (or in him). Or the reason might be valid but kept private because the secrecy of the ballot is sacrosanct.
We are told to be very careful about the men who we recommend and the possibility of a brother using the ballot for un-Masonic reasons is one of the reasons we are told to be careful.
When a brother rejects a candidate for improper reasons it reflects on all of us more than it does on that one candidate. A man rejected without knowing why can be expected to hold a low opinion of us for the rest of his life.
Whether it makes sense to change the rules to require more than one rejection, I can't judge that for other jurisdictions. To me the anonymous ballot is a landmark so the problem should be solved some other way.
As far as prejudice goes, it's our internal problem. Our principles teach us to get past them but not all brothers are able to bring themselves to do so. Our principles also teach that we are to have harmony in our assemblies but not all brothers are able to set their prejudices aside to be able to do so. Our landmarks specifically forbid the discussion of sectarian religion and partisan politics because those are not visible attributes of a man so they can be kept secret. Masonry led the world on those types of prejudice because secrecy worked for them. But visible attributes? That's our internal problem.
If I saw a rejection based on what I believed was racial prejudice I would not darken the door of that lodge again and I would demit to another lodge were I a member. But living in a racially integrated neighborhood of "Military City, USA" I have the luxury of many lodges to chose among and the certainty that I have lodges to chose among. If work required me to move to a region where I couldn't avoid the issue I'd likely start checking about American Legion posts for the moment as well as checking out job boards for another relocation. I am tolerant of tolerance and intolerant of intolerance.