“Falsifiable”?

Continuing from Episode 1, we hear Wikipedia’s article, Astrology & Science say:

Where astrology has made falsifiable predictions, it has been falsified.

What does “falsifiable” mean? “Falsifiable” means that something can be proven false. 

This is a very useful concept because there are many things we cannot prove to be true, but we can be prove not to be false. For example, we cannot prove that “all swans are white” because it is impossible to ever see all the swans in the world. Still, as long as we don’t see a non-white swan we know “all swans are white” is not false. And this is functionally the same as proving it true. 

Seeing even a single black swan, however, falsifies the statement that “all swans are white.” 

Karl Popper, an outspoken critic of astrology, formulated this logical methodology. 

This method only works on statements that are not impossible to conclusively disprove. So it works best with tangible and objective things (like the color of a swan) not intangible or subjective things (like “swans often feel happy on Sundays”).

Now that we know this, let’s return to the article. It says:

Where astrology has made falsifiable predictions, it has been falsified.

What does that actually mean? It means:

“Astrologers usually make statements that are impossible to disprove, but whenever they do make a disprovable statement, it turns out to be false.”

Ironically, this statement itself is blatantly falsifiable and falsified. Even a single correct, clear predictive interpretation from even a single astrologer would be the “black swan” falsifying this claim. Astrologers certainly make inaccurate predictions and interpretations, but we also make accurate ones, at least sometimes… at least once! 

To even be a candidate for accuracy, these statement in Wikipedia needs to be edited so that it does not imply that all astrological predictions arefalse.

Shawn Carlson’s Test

Wikipedia continues:

The most famous test was headed by Shawn Carlson and included a committee of scientists and a committee of astrologers. It led to the conclusion that natal astrology performed no better than chance.

In 1985, Shawn Carlson conducted two experiments to evaluate the accuracy of astrology.

In the first experiment, the birth data of 193 test subjects was given to about 2 dozen astrologers, who then provided interpretations. Each subject was then shown three interpretations, only one of which was from their own birth data. They were asked to select which one was their own.

In the second experiment, these subjects completed a California Psychological Inventory test, three of which were given to each astrologer, along with one person’s birth data, and asked to select which test result belonged the person with that birth data. 

According to the published results, both experiments indicated that astrology performed no better than random chance. 

A Test of Falsifiability?

Wikipedia directly implies that Carlson’s test was a test of falsifiability, but was it, really?

No.

A falsifiable claim is a clear statement that can be objectively disproven from direct objective observation. Here are some examples of truly falsifiable experiments for astrology:

• Astrologers must predict the number of children born to 100 post-menopausal women.
• Astrologers must predict the height of 100 middle-aged people.
• Astrologers must predict the age of death of 100 deceased people.

What these have in common is that the astrologer’s predictions are simple, numerical facts which can be evaluated objectively. This makes them falsifiable.

Shawn Carlson’s test of astrology, on the other hand, did not deal with any simple, falsifiable empirical metric. It dealt with complex interpretations. 

Furthermore, it does not deal with objective metrics. Asking someone to selecting their own interpretation is highly subjective. I would certainly have strong biases in selecting my own interpretation. I would favor the interpretation I want to be mine. Carlson’s test might be as much an evaluation of how accurately people see themselves as it was a test of how accurately astrologers can interpret a birth chart. 

The second part of the experiment is also lacks simple empirical metrics and instead works with complex and subjective things. It might be as much an evaluation of the California Psychological Inventory as it is an evaluation of  astrology, or certainly it is mostly a test of how compatible the two were for this particular set of participants.

It cannot be denied that Carlson test results are not flattering for astrology, but they are not what Wikipedia claims they are: a well constructed controlled study that falsifies astrology in toto. 

Carlson’s test was not so much a scientific experiment as it was an open challenge to astrology.  Astrology failed this challenge, probably because the participants were not practiced enough in astrology and science to realize how poorly constructed the experiment was, and just how much variability was left uncontrolled and subject to chance.

Pseudoscience

Next, the wikipedia article states:

Astrology has not demonstrated its effectiveness in controlled studies and has no scientific validity,  and is thus regarded as pseudoscience.

Problems with Controlled Studies

Is a controlled study really an appropriate method to determine whether astrology is scientifically rational?

The first problem with any such study is that the person constructing it would need to be fluently practiced in both scientific methodology and astrological practice. Without being an actual scientist, they simply would not know how conduct a real scientific experiment. And without being an actual astrologer, they would be unaware of the variables involved in real astrological prediction. These variables are of paramount importance, because every variable possible must be accounted for and explicitly controlled in any valid scientific experiment. 

In short, it would take a scientific and astrological genius to devise an experiment that would truly and conclusively prove or disprove the validity of astrology. 

In this, is astrology different from any other science? 

No. 

In every science, the difficult part is figuring out how to conduct the experiment and derive the proof. The scientists who are praised and remembered are those who did exactly that: they figured out how to prove a hypothesis.

Why is it that, when it comes to astrology, the scientific community does a 180 and instead praises people who design experiments that fail? 

The challenges come from both sides. The scientific community simply doesn’t care enough about astrology to support anyone in devoting the time and effort to truly and conclusively prove or disprove it. The astrological community is also unwilling and unable to devote that time and effort to the endeavor simply because, overall, it does not care enough about logic and scientific methodology.

Specific Challenges in Astrological Experiments

How difficult would it be to devise a truly effective astrological experiment?

For any experiment to be scientific, its methodology must be clearly stipulated and executed identically in each iteration, without variance.  Just as significant physics experiments require masterminds of physics, it would take an astrological mastermind to derive a such a methodology for any astrological experiment. 

For example, if we want to test whether astrology can accurately predict how many children a woman will have, we first have to define the astrological method to be employed in the test.

Can it be done simply by knowing her sun-sign? I think any astrologer would immediately agree that it is impossible to do so with any efficiency better than random chance. Does this disprove astrology? Maybe, but more likely it indicates that the experiment requires much better methodology.

For example, which planet’s occupy the 5th sign? What condition are they in? Which planets influence them? What are the conditions and influences on the 5th lord? Jupiter? And, very importantly, “5th sign” from what? The Ascendant? The Moon? Should we include these factors in zodiacal subdivisions? If so, which ones?

To support any of these, we also have to explicitly define how we will evaluate “planetary conditions,” and “influences.” Deeper still, any other more basic factors also have to be stipulated: what planets are to be used? Which zodiacal and house systems? Which schools of interpretation?

Now open a new pandora’s box: Do we also need to consider these (or similar) factors in the charts of her mate(s)?

This is just a quick gloss to give you an idea of how many astrological variables are involved in making an apparently clear-cut and simple prediction.

Once all such variables are explicitly defined in mathematically calculable ways, an accurate test sample of sufficient size can be subjected to the calculations. The experimenter can then experiment with different variations in the definitions of methodology, to revise or fine tune the method until it produces the greatest accuracy.

Finally when that accuracy is compared with the accuracy of random guesses, the positive gap between the two defines the efficacy of… astrology?

How Good is “Good Enough”?

No science is 100% perfect, and many useful sciences are far from it. For example, Physics cannot tell us where an electron is at any given time. In fact, Physics cannot even tell us what an electron isat any given time. But this does not mean physics is pseudoscience, nor that we are illiterate suckers being scammed if we purchase electricity for our homes.

Another example is meteorology. It just doesn’t predict the weather very well, especially not very far in advance, and especially not very significant events like tornados and hurricanes. Geologists cannot predict earthquakes, mudslides, etc. Doctors cannot accurately diagnose or cure every disease.  Does this mean meteorology, geology, and modern medicine are pseudoscience, hoaxes? No. It simply means that the sciences and their practitioners are imperfect.

And these are fields that have gigantic financial and social backing and support for their research and development. Compare that to astrology, which is an fringe niche operating on a shoestring and some kind donations.

Suppose that somehow I managed to devise an experiment that predicted the number of children any woman would have, with exact accuracy in 20% of the subjects. Is that “good”? If not, does this prove that astrology is a hoax and a pseudoscience? How about  25%? or 50%? Is that good enough to prove… whatever it is that astrology’s critics want us to prove? 

Early in my career a Swiss nuclear physicist came to me for a reading. “Is my husband’s life in danger?” she asked. 

I informed her, “Yes, be careful of intestinal bleeding, maybe as a result of falling from a high place.”

She replied, “This confirms what I already knew: Astrology is a hoax. He is already dead. He died last month from internal bleeding while hiking in the mountains.”

…

Let’s say someone, somehow, somewhere, someday performed a miracle and devised a methodology that could accurately predict the number of children a woman would have in 90% of the cases. Would this prove that astrology is a valid science?

It would be extremely impressive, for sure. It would certainly prove the genius of that particular astrologer, and of that particular astrological method, but an honest logician and scientist would have to admit that it would still only be an indirect validation of astrology as a science.

So, if we want to conclusively know whether astrology is a science or a pseudoscience, why asking us to leap through  impractically high flaming hoops? Wouldn’t it be better to simply look at the hypothesis of astrology itself, and rationally judge whether it is plausible within the known laws of empirical science and other rational disciplines?

Yes. 

And that is exactly what Wikipedia brings up in its next sentence, and what we will explore in our next episode of Astrology vs. Science!