Virginia Governor Ralph Northam was undone by a medical school yearbook photo depicting a Klansman and a person in blackface. . . .which turned out to be him. Attorney General Mark Herring, third in line to be Governor, confessed that he, too, dressed in blackface while an undergraduate at UVA. Of the two, Northam may be the most vulnerable because his misdeeds are captured in an image that allows no rebuttal or equivocation. In this sense, he resembles Senator Al Franken, whose name now immediately brings to mind an image of a boor about to grope a sleeping woman.

These images exist in a depthless present; they invite evaluation in the here and now. But the transgressions underlying them happened in a different time and social context. Should we judge them according to today’s values, the norms of the era in which they misbehaved, or by some timeless moral-temporal criteria yet to be defined and articulated? To overcome the wrongs of the past, or merely their downstream inanities, must we ostracize everyone who was complicit in them? Or does the rush to call out and judge individuals for evils in which an entire society was complicit to one extent or another turn them into scapegoats for wider spread and unattributable wrongs?

It’s reasonable to think that these past misdeeds tell us something about the present-day character of the men involved. Someone who dressed in blackface is probably a racist; someone who joked about a sexual assault is probably a serial harasser. Ultimately, Franken’s case seems to have vindicated this kind of reasoning: Several other accounts of groping surfaced after the photo went viral. And it makes sense to hold people accountable for the harm their past actions have caused: People who have humiliated or exploited other people don’t deserve the honor of high office. Most of the time these two considerations lead us to the same conclusion: People of bad character do bad things, not once but often enough, that injure others.

But this is not as certain the more time that has elapsed between crime and punishment. For instance, Franken became famous at a time when crude, vaguely misogynistic humor was a staple of late-night television. Sadly, the image in the photo was representative of his usual schtick and that of hundreds of other comedians in the 1990s and early 2000s. Before other women came forward with stories of harassment, it seemed that Franken didn’t actually grope the woman in question—she was asleep when the photo was taken and Franken insisted it was just a bad joke. Franken’s defenders saw the photo as evidence of poor taste and poor judgment, but not necessarily of assault or deep-seated sexism. Withholding judgment at that point wasn’t naïve or complicit in injustice, as some have argued—it was judicious.

Similarly, Northam’s yearbook photo—while it certainly could be evidence of deep seated racial animus—was characteristic of a sort of casual racism that was still all too common in the 1980s. It could have just been a lousy joke, reflecting ignorance and insensitivity more than deliberate contempt. For perspective, recall the infamous sequence in the film Silver Streak, in which Richard Pryor convinces Gene Wilder to disguise himself by wearing blackface.

To be sure, both images are vivid examples of a widespread, thoughtless contempt that demands our condemnation. And in a sense, the fact that they were widely tolerated—even celebrated as daring taboo breaking—only makes it worse: As Lili Loufbourow writes in Slate, the type of lazy humor that involves humiliating women and minorities is really a way of celebrating the unearned privileges of patriarchy and white supremacy. Indeed, the harm caused by widespread and socially accepted moral wrongs is often greater than that caused by wrongdoing that is widely condemned.  Before heightened public awareness about its prevalence, sexual harassment left women powerless and stigmatized; by contrast, in the #MeToo era, the victims of harassment have powerful allies ready to come to their defense. Similarly, in the 1970s and 1980s, blacks were expected to laugh along with or at least tolerate blackface and the racial stereotypes and contempt it perpetuated—as Pryor’s role in Silver Streak demonstrates. Today, blackface reflects poorly not on black people, but on the white people wearing it.

But past transgressions that were widely tolerated at the time might not reflect the individual character flaws that similar transgressions would today. Over a decade ago, when it was taken, the Al Franken photo might have reflected the lazy humor of a comedian past his prime—something similar in the era of #MeToo could only be the deliberate provocation of a defiant misogynist. Similarly, blackface in the 1980s might reflect the cavalier, ill-considered racism that was pretty much in the water in the 1970s and 1980s; today it would undoubtedly reflect a belligerent assertion of white supremacy. In other words, the same behavior need not reflect the same motives or presuppositions. One need not excuse the former examples to admit that they are not as blameworthy as the former. While there is no respectable argument in favor of allowing an unrepentant bigot or sexist to occupy public office (though many do!), reasonable people may differ as to whether the crude, insensitive frat boy antics common in the recent past are grounds for exclusion from public service.

This time of evolving social norms presents a tension. We must unequivocally condemn the kind of habitual thoughtless prejudice revealed in the Franken and Northam photos, without rushing to unequivocally condemn everyone involved. But unless we get corroborating evidence that Northam is a bigot—as we did of Franken’s sexual harassment—it’s premature to demand that he resign. Since the blame for the culture of racism and sexism that these images exemplify is ultimately a collective blame, we must separate the magnitude of the social injury from our assessment of individual character. Changing deep-seated, hierarchical cultural norms requires more than just attacking individual bigots, predators, and the shamefully insensitive. Sometimes, it will also require less.

Understandably perhaps, such distinctions often get lost in the moral fervor surrounding a social justice movement, which is not always easily distinguishable from a moral panic, where such distinctions are by definition impossible. But they are indispensable to any sustainable change in social norms or durable movement for social justice. In what’s become known as “Call Out” culture, righteous indignation can give rise to an unforgiving, even cruel punitiveness. That in turn fuels a backlash that can turn the long overdue reckoning with institutionalized racism and sexism into yet another flash point of ideological polarization. If consistent and forceful condemnation of casual bigotry can be leavened with sympathy, charity, and the possibility of forgiveness, perhaps a needed change in norms can come without a culture war.