I get what you're saying, but I disagree that your suggested outcome of outing someone as "bigoted and a fool" is in any way sufficient.
We cannot ignore unacceptable conditions or behavior just because it interferes with our blissful enjoyment of our favorite spectator sport. The criticism that Pryor is "politicizing" something that is directly affecting his life is what allows racist hate to be expressed and thrive in a consequence free environment. It places the responsibility of dealing with the feelings inspired by the comments on those to whom harm was intended ... which oh-by-the-way compounds the damage by reinforcing the idea that society doesn't think that he's worth protecting from this horseshit.
Also, there is an insidious gaming of the system by bad faith actors that says "debate me" and "marketplace of ideas" that really functions as cover for those shitty ideas to be expressed in public and gives the opportunity for those ideas to find a home - in the hearts of those they are intending to harm and in the minds of those they are trying to recruit. We have the ability as a society to agree that there are ideas that fall below the standard of discourse that we are willing to accept, and that we should protect those who are vulnerable.
As previously noted, freedom of speech is not freedom from consequences and some racist p.o.s. saying racist p.o.s. things does not deserve to be debated in public ... the debate's been had, either skip to the consequences or we're forced to acknowledge that the ones being protected are the racists.