Extraordinary Evidence

"Extraordinary claims require extraordinary evidence" -Carl Sagan

 Some people don't like this quote. They say it's too subjective. There's no objective standard of what's extraordinary. And other rather silly objections.

I would generalize the saying to be something more like this: "All claims require the appropriate amount or degree of evidence.".

The ordinariness/extraordinariness of a claim is not a binary feature, but rather a sliding scale. If I said I had eggs for breakfast this morning, that's a perfectly ordinary claim. You'd probably need no more evidence than my word to believe it. If I said I saw a zebra in my backyard, that's a little bit extraordinary, since they don't live anywhere near here. You'd probably need a photograph, or a news story about a zebra that escaped from a zoo to believe it. If I said I saw a unicorn, that's even more extraordinary. You'd probably need to see the real live thing to believe that. If I said neutrinos can go faster than light, that's even more extraordinary and will require lots more evidence. And then of course, there's always God, which is about the most extraordinary claim imaginable.

Another important thing to note is that belief too is not a binary value. Rather, it's degree of certainty. Probability. Which is nice, because that means this saying can be formalized by using Bayes' Theorem.

Bayes' Theorem is a mathematical formula that lets you calculate how probable you should consider a hypothesis after seeing some evidence, given your prior probability of that hypothesis, how likely you are to see that evidence if the hypothesis were true, and how likely you are to see that evidence if the hypothesis were false.

How extraordinary a claim is, is simply how low your prior for it was. This doesn't totally eliminate claims of subjectiveness, but it's no less subjective than any other belief, and if you're doing Bayes right, it's really a lot less subjective.

And how extraordinary evidence is, is simply how much more likely that evidence is to occur if the claim were true than if the claim were not true.

Going back to my examples, the prior probability for me eating eggs for breakfast is relatively high. You already know from past experience that eggs exist and that people commonly eat them for breakfast. So the evidence doesn't need to be very strong. I'm more likely to say something if it's true than if it's false, but lying isn't unheard of.

The prior probability of me seeing a zebra in my backyard is lower. You still know that zebras exist, but you also know they don't live in the wild here. So, on just my word, it might seem more likely that I'm lying than that I actually saw a zebra. The prior probability for the unicorn is even lower, because you already know they don't exist. A photograph isn't sufficient here because it's more likely that I faked it than that unicorns actually exist.