BryanMaloney
Premium Member
This is the beginning of the world's most discouraging statistics essay. It will not teach you how to do statistics. Instead, it is designed to make you very sad about the field. This is necessary in our day and age. At one time, people could get away with dismissing statistics in daily life. Now a great deal of policy is based on statistical analysis. Most people adopt a one-sided cynicism, dismissing anything that doesn't reinforce their own biases and blindly swallowing any number that agrees with their previously held beliefs. I hope to convince at least some people that statistics do not determine truth, falsehood, or anything else at all. They're just tools and often not used properly. For example, I could "measure" the weight of something using a ruler, some references, and a bit of math. So could anyone else who knew the method. That doesn't mean it's an ideal way to measure weight. That's what statistics is like. It's a lot of methods of trying to get hold of something we just don't have the tools to really grab onto. It is always approximate, always a more-or-less "rough try".
This isn't necessarily bad. After all, most of life is an approximate rough try. You very often don't need to know how many pounds something weighs, only if it's "light enough to carry". But what if you then restrict yourself to only two categories: "light enough to carry" and "not light enough to carry", then you base all your policies on that--ignoring little details like volume, length, what the object is made of, how often you would ever actually need to move the object, if the object can be disassembled for movement, and more. But you're not allowed to look at any of that stuff. Instead, you are only allowed to look at "light enough to carry" or not. Or even worse, you're so blinded by love of "light enough to carry" as a simple and powerful concept, you automatically reject any idea that is based on taking into account any feature that isn't whether or not something is "light enough to carry". I just sneaked in a statistical mis-concept on you, the way that "significant" is (mis-)used in statistics or by people who refer to statistics.
For decades, scientists have wrestled with and squabbled over the horrible, horrible term "significant". It was never meant to mean what it is taken to mean. You see, when a scientist says that a result is "significant", what he is SUPPOSED to mean is that the given result, IF all the possible observations that could have been made of this phenomenon over the entire universe have a certain type of variation, and IF enough observations have been made to accurately guess the specific type of variation to apply, and IF there isn't something else interfering that covers up the actual type of variation that matters (yes, things can be a mixture in the real world), and IF the scientist hasn't made so many observations that it is mathematically impossible to generate a "non-significant" result (this is possible, by the way), then there is (probably) a (roughly) one in twenty chance that the result isn't just a random fart. That is what a not-otherwise-defined "significant" means in the sciences.
Coming later:
Explaining how most ways to determine "significant" are really just crude approximations of 1 divided by the sample size and how this means that, if your sample is large enough, you will get a "significant" result, no matter how tiny it is.
This isn't necessarily bad. After all, most of life is an approximate rough try. You very often don't need to know how many pounds something weighs, only if it's "light enough to carry". But what if you then restrict yourself to only two categories: "light enough to carry" and "not light enough to carry", then you base all your policies on that--ignoring little details like volume, length, what the object is made of, how often you would ever actually need to move the object, if the object can be disassembled for movement, and more. But you're not allowed to look at any of that stuff. Instead, you are only allowed to look at "light enough to carry" or not. Or even worse, you're so blinded by love of "light enough to carry" as a simple and powerful concept, you automatically reject any idea that is based on taking into account any feature that isn't whether or not something is "light enough to carry". I just sneaked in a statistical mis-concept on you, the way that "significant" is (mis-)used in statistics or by people who refer to statistics.
For decades, scientists have wrestled with and squabbled over the horrible, horrible term "significant". It was never meant to mean what it is taken to mean. You see, when a scientist says that a result is "significant", what he is SUPPOSED to mean is that the given result, IF all the possible observations that could have been made of this phenomenon over the entire universe have a certain type of variation, and IF enough observations have been made to accurately guess the specific type of variation to apply, and IF there isn't something else interfering that covers up the actual type of variation that matters (yes, things can be a mixture in the real world), and IF the scientist hasn't made so many observations that it is mathematically impossible to generate a "non-significant" result (this is possible, by the way), then there is (probably) a (roughly) one in twenty chance that the result isn't just a random fart. That is what a not-otherwise-defined "significant" means in the sciences.
Coming later:
Explaining how most ways to determine "significant" are really just crude approximations of 1 divided by the sample size and how this means that, if your sample is large enough, you will get a "significant" result, no matter how tiny it is.