Even with seemingly obvious “causation” studies, meeting the threshold of cause and effect is nearly impossible. Instead, they show that on average, the probability is extremely high that the predicted event will happen. (Disclaimer: if you use tobacco products, then you are extremely likely to die from said use.) The US Government says, “Cigarettes cause cancer.” Is that the same as, “Decapitation causes death?” No. Let’s look at the absurdity of the statement. If I don’t touch, smoke, ingest, or breathe second-hand smoke, then the cigarette is no harm to me. Further, even if I do, then it only increases the probability that I will get cancer, but does not guarantee it; whereas, decapitation will cause death. What should be said is, “Smoking cigarettes will probably cause cancer, so do NOT do it.” In this case, you should think of the correlation between cigarettes and cancer death like a game of Russian roulette. If you are an American, then the gun is loaded with one cartridge and you are playing. If you choose to smoke, then it is like putting two-to-five more cartridges in the revolver.
I taught a sophomore level Statistics course last semester and by the end of the term, 90% of the students finally understood why we must critically think about claims regarding data. With that level of success, why are so many politicians (the majority having law degrees) unable to grasp this concept? I know: “if you have a law degree, then you aren’t smart enough to differentiate between correlation and causation” (Sarcasm). With social science research, it even gets harder and ALL findings are suspect. Besides, the findings only apply to averages – not individuals. One of the test questions I gave in the Statistics was about educational performance and family income. Politicians look at the positive correlation and conclude that the root problem with education is the income gap. Huh?
The erroneous interpretation of the graph is that families with higher household incomes will produce children who will do better on the SAT. That is, “If you earn more money, then your children will do better on the SAT.” Huh, again? In reality, the proper interpretation is, “On average, students who come from wealthier families perform better on the SAT than those whose parents don’t earn very much money.” Does that mean that if I win the lottery and now have an enormous income for the rest of my life that my children will be smarter? Not necessarily, because there are too many other factors involved. For example, if I procreated with Ruth Lawrence, then the kid would probably be smart even if my only income was welfare; whereas, if I procreated with a drug addict, the probability of a child with little educational potential is very high. Simply, Darwinism plays a larger role in educational achievement than money (albeit, that someone with strong genes raised in a loving environment with sufficient resources will have a much better chance of success than the person with the same genes being raised in a home without love and without enough to eat).
In education there is always some treatment, x, that will produce a great effect on the average of some assessment, y. That is, a function is proposed where some x (e.g., posting learning objectives on the board for students, later start times for students, having students dissect clocks, etcetera) will cause the average of y (e.g., scores on the SAT, students attentiveness in class, number of STEM majors, etcetera) to increase. Many of these findings are based on a quasi-scientific method, where one class is given one treatment and another class is given a different treatment and then compare the results on a standardized assessment instrument. Sigh! The problem is, there are other factors that contribute to the variance in the results. For example, when we look at the clock-engineer correlation, what is the cause? Is it curiosity? Intelligence? Boredom? Right place and right time? Parenting? A combination of these? Something else? We will probably never know the whole answer, but for some reason we do make policy changes based on similar data.
When data are presented and conclusions drawn, we must think critically about the interpretation and always have some nagging doubt. In a court of law, the judge gives the jury directions that you may only convict if there is not “reasonable doubt.” That is a very high standard, but what is reasonable (When you can explain that well, please let me know…)? In social science, the standard is not near as high because nobody is going to jail, unjustly, if we are wrong; but maybe it should be as high, because we may hinder a generation of students based on the decisions we make. In the individual classroom, teachers should experiment and find what works for them and their students, but others must keep in mind that what works for one person at any given time is not necessarily going to work for somebody else in another place and time (i.e., not generalizable). The only thing we absolutely know about education is that it is constantly changing based on correlation research and the whims of politicians.