What do a conservative opposed to diversity, a black woman with dreadlocks, and Asian-American high achievers dreaming of the Ivy League have in common? They all claim to be victims of discrimination. James Damore was fired from his job at Google after circulating a memo criticizing Google’s diversity efforts and suggesting that women are biologically unsuited to jobs in tech. Damore’s champions have insisted that he was a victim of discrimination against white male conservatives. Chastity Jones was offered a job as an insurance telephone call center on the condition that she comply with the company’s dress code and change her dreadlock hairstyle. She refused and sued the company for race discrimination, claiming that her hairstyle was inseparable from her race. The Asian-American plaintiffs suing Harvard College were among the close to 95 percent of applicants that were not admitted. They claimed the reason was not their failure to meet Harvard’s famously high and somewhat obscure standards but instead race discrimination against Asian Americans.

Some might see this diverse set of discrimination claims as progress, evidence of our society’s growing commitment to fairness. But the notion of fairness that informs these claims is both unrealistically broad and far too limited: too broad because ultimately it would expand the concept of invidious discrimination to include any decision that cannot be objectively justified; too narrow because, to the extent the underlying complaints have any merit as a moral or ethical matter, they attack only specific examples of much broader injustices.

Let’s look at each of these claims in turn. James Damore argues that Google’s decision to fire him was a form of discrimination against white men and conservatives. The first element of the claim is almost laughable on its face: Google is an employer with an overwhelming male and predominantly white work force and Damore can point to no evidence whatsoever of discriminatory motivation with respect to race or sex. He and his lawyers seem to realize this and so have combined the race discrimination claim with their real gripe: that Damore was fired because of his conservative views. But Google, as a private employer, isn’t obliged to guarantee freedom of expression. Still, this actually might be a legal problem if he had been fired for political activity or views expressed outside the workplace. But Damore used Google’s resources to distribute a memo that attacked its diversity efforts and arguably disparaged his female co-workers, undermining collegiality and morale in the workplace in the process; he was fired, not because of his politics but because of his insubordination and lack of sensitivity. And it shouldn’t have surprised Damore that such subjective considerations weighed heavily in the highly collaborative environment of tech, where “culture fit” is a standard criterion for employment. To be sure, there’s good reason to worry that “culture fit” can be an excuse for discrimination, especially against women and people of color who may be unfamiliar with the “tech bro” atmosphere. But Damore’s theory of discrimination takes this critique of “culture fit” to extremes, insisting that Google was obliged to ignore the workplace tensions he created because they were not objectively disruptive but only undermined the atmosphere, collegiality, and culture of the workplace. The notion that an employer can’t discriminate based on disruptive workplace behavior implies that it must limit its assessments to objective measures of productivity.

The question of workplace culture is also at the center of Chastity Jones’s lawsuit claiming that an employer’s dress code is a form of race discrimination. Her legal theory—endorsed by the EEOC and now adopted by the New York City Commission on Human Rights—is that any dress code that applies to a hairstyle associated with African-Americans is discriminatory. As is the case with workplace “culture fit,” there is certainly reason to worry that strict dress codes can become a cloak for intentional discrimination or that they can have an unjustified discriminatory impact on underrepresented groups. But these new anti-dress code theories go much further and insist that a dress code that covers any hairstyle commonly worn by African-Americans—even when it also applies to many other hairstyles worn by whites and other racial groups—is discriminatory per se. The unstated premise here is that there is no good reason to enforce a strict dress code, so it’s fair to presume its main purpose is discriminatory (and in any event there is no real harm done in forcing the employer to make exceptions or even get rid of the dress code altogether). This argument is plausible in cases where the dress code is overkill for the job in question—for instance, Jones had applied for a job in a telephone call center, where no customer would ever see her. But a blanket rule against dress codes that happen to apply to hairstyles worn by African-Americans assumes that dress codes are always unnecessary. Again, the implication is that because hair, grooming, and attire are matters of taste and opinion they can never be legitimate employment considerations: Employers should limit their evaluations to objective job criteria and those who do not are guilty of some kind of bias.

Similarly, a subtext in almost every discussion of the discrimination lawsuit against Harvard College is that objective criteria—grades and test scores—are the only acceptable bases for university admissions: Any form of subjective evaluation is suspect at best. Hence, the observation that Asian-American applicants outperformed other racial groups on grades and test scores, but underperformed other groups on subjective evaluations, was taken by Harvard’s critics as prima facie evidence that Harvard discriminated against Asian-Americans. It’s true, of course, that subjective evaluation can hide racial biases. But qualitative assessment is indispensable to creating a community with norms, ideals, and a sense of mission and history. Critics of Harvard’s admissions complained these things are too subjective—matters of opinion about which people might disagree—and suggested that Harvard’s effort to shape a college atmosphere and set of norms is illegitimate, a form of discrimination against people of a minority cultural backgrounds.

In today’s polarized and acrimonious society, it’s all too easy to conclude that a decision one happens to disagree with must be motivated by invidious prejudice. And it’s easier still when the list of traits someone could “discriminate” against keeps growing and now includes things like “culture” and “ideology.” If Google’s management insists that its employees help recruit, retain, and work harmoniously with a diverse workforce, it is discriminating against conservatives like James Damore, who are opposed to or skeptical of the goal of diversity. If an employer believes a strict, conservative dress code instills professional discipline and suggests professionalism to clients or customers, it is biased against people who prefer more adventurous or unconventional hairstyles or attire. If Harvard promotes a distinctive campus culture through its admissions decisions, it is guilty of discrimination against minority cultures. Because our society is increasingly cosmopolitan, as well as ethnically and racially diverse, it is harder and harder to find widely shared cultural norms—almost every normative or aesthetic judgment is the product of a specific, limited, exclusive community. This seems to leave us with objective facts as the only common denominator, the only legitimate criteria of evaluation in a multicultural society.

According to this line of thinking, just about any effort to encourage a common set of norms and values will be attacked as a form of discrimination against people who don’t share the values in question. Taken to its logical conclusion, this reasoning would deprive every organization and institution of its distinctiveness and personality. The expansion of anti-discrimination rights into the domains of ideology and culture protects individual expression and personality at the expense of the collective culture of organizations.

This is not always a good trade. Indeed, as the societies of the West grow more cosmopolitan, rootless and anomic people increasingly yearn for a sense of authenticity that only a common culture can offer. Nostalgia for the confident and unapologetic collective culture of the past defines our contemporary moment, all too often taking an ugly and retrograde form in ethno-nationalism and belligerent, reactionary male chauvinism. The purpose-driven organizations of civil society—businesses, universities, civic clubs—may offer the possibility of a healthier collective cultural identity based on common ideals and sensibility rather than toxic myths of blood and soil. Unlike a national government, such institutions are numerous enough that they are likely to reflect numerous different sensibilities and ideals, so people ill-suited to one will likely find another that’s a better fit. In this light “discrimination” on the basis of culture and ideas is a necessity, perhaps even a virtue.

One can acknowledge the value in culturally distinctive—and, of necessity, exclusive—groups, organizations, and institutions as a general matter, even when one dislikes some of the specific examples. I personally disapprove of an uptight corporate dress code in a telephone call center. Conservatives may hate what they will see as Google’s craven political correctness. And many people may object to the patrician sensibility underlying Ivy League admissions criteria. But would we prefer a world where every decision was based on bottom line considerations or technocratic numerical criteria? The only alternative is to tolerate cultural sensibilities one does not share. Indeed, because culture is formed and expressed by groups—institutions and organizations, not isolated individuals—the only viable form of true multiculturalism (as opposed to disguised ethno-nationalism) is one that takes the culture of such groups seriously.

This doesn’t have to mean unfettered freedom for rapacious corporations or elitist universities to dominate, control, and exclude. It only suggests the need for a more precise analysis of how, when, and if they abuse their power. Employees keep trying to expand the scope and meaning of discrimination because we’ve failed to take more comprehensive steps to make workplaces fair and humane. If employers have too much power over their employees this suggests a need for stronger labor unions and laws that improve working conditions generally. Maybe such laws should include expressive liberties for all employees: This approach that would require us to balance the legitimate interests of employers in common norms against the expressive needs of employees. Recasting the problem as discrimination wrongly suggests that no such balancing is necessary because we can and should simply outlaw the discrimination.

Similarly people keep picking at the scab of university admissions—through anti-affirmative action lawsuits and other self-righteous calls for reform—because, perversely, selective universities have come to symbolize a rare avenue of upward mobility in an increasingly unforgiving and stratified society. But step back a moment and it’s a truly strange and sad idea that elite private schools like Harvard would be our society’s answer to economic mobility. Instead, perhaps it’s time to reinvest in public education and tackle the polarization of wealth that makes the stakes of university admissions so high and the competition so cut throat. There are plenty of real injustices: We should stop dreaming up fake ones, like ideological or cultural discrimination, where what’s really at issue is not prejudice but simply judgment.