Tuesday, October 27, 2015

The Limitations of Scientific Experiments



            Two articles I have recently read seem to demonstrate that the scientific method is not always as reliable as we might have hoped.  Evan Horowitz discusses in his essay “Studies show many studies are false” for the Boston Globe (7/1/14) how there have been studies of scientific studies that show that most of the scientific studies under consideration can’t be replicated.  When a German pharmaceutical company tried to replicate the results of 67 published studies from academia, they were only able to do so in one quarter of the cases.  The American company Amgen tried to recreate 53 cancer studies and only got results that matched those of the original studies in 6 cases.  The results of these studies of studies are significant, because if a scientific study cannot be replicated by different investigators, then it cannot be considered to have generated conclusions that are true and accurate.  Ian Sample, the Science editor for the Guardian, wrote in his article “Study delivers bleak verdict on validity of psychology experiment results” (8/29/15) that a recent investigator has demonstrated that an attempt to replicate the results of 100 experiments written up in major psychological journals has resulted in success only 36% of the time.  More precisely, only 50% of the studies in cognitive psychology could be replicated and only a mere 25% of the studies in social psychology could be replicated.  In addition, with respect to these psychology articles, when there was success in replicating results, the average effects of the replicated results were only half as large as the first time the experiments were performed.

            None of this is very encouraging for those of us who have looked at the scientific method as the foundation for much of modern knowledge.  All kinds of explanations are given for these discrepancies.  Evan Horowitz quotes a research professor named John Ioannidis, who had himself written an article “Why Most Published Research Findings Are False” (8/29/05) in which he enumerates academic pressure, the fact that the samples in studies can be too small, and a preference for surprising experiment results as reasons for the publication of non-replicable studies.  Horowitz himself adds an unconscious selection of certain kinds of subjects for the sample and the problematic nature of early studies on a particular experimental issue.  Ian Sample points out that scientists can change the methodology in a subtle way when they repeat an experiment.  Or some of the conditions in which the experiment is carried out could be distinct from the first performance of the experiment.  Or some chance element could affect the attempt to repeat the original result.  Finally, the original experiment could have been in some way defective leading to a false positive.

            Some of these reasons hint at a line of thought I have been developing throughout my articles with regard to the way humans grasp truth and grasp reality.  Reality includes a human’s entire field of experience and includes the whole configuration of stimuli that he receives from that field of experience.  There are three kinds of stimuli in a field of experience.  There are the focused measurable defined discrete stimuli that define the boundaries of things and ideas.  There are the blurry unmeasurable flowing blendable continual stimuli that have no defined beginning and no defined end.  Like the tide of a lake or an ocean or the shifting shapes in a lava lamp.  Finally there are the endless infinity stimuli found in total darkness and total silence.  The black in total darkness and the slight hum in total silence.

Scientific truth deals only with the first of these three categories.  Science looks to discover in order to control, to manipulate and to predict within the human field of experience and thus to improve the human living condition and protect people from harm.  To do this, it needs to be able to work with controllable, manipulable and predictable phenomena.  Phenomena that it can focus on and measure and dominate.  In other words, science needs to be able to work with phenomena that, at least to a great extent, are defined discrete figures that emit defined discrete stimuli.  These are the phenomena that allow scientists to set up experiments where they can trigger certain processes with the hope of arriving at certain results.  If the scientists succeed, they can then try to apply what they learn to the physical environment, the chemical environment, the biological environment, the psychological environment or the social environment and thus generate certain improvements in the human living condition or certain protections against harm.

            But the fact is that the results of a scientific experiment represent the excision of certain elements of the total reality surrounding the experiment.  Those elements that lead to the hoped-for pliable maneuvering of at least part of the environment.  But reality is constantly shifting, even the reality of an environment that is supposed to be as sensorily controlled as a laboratory.  As much as scientists try to make laboratories as sensorily neutral as possible, each laboratory is going to have some elements that make it different from the previous lab where an experiment was performed.  The layout, the lab furniture, the lighting, the air flow, the odors – all can have an influence on the performance of the experiment.  The time of day the experiment was performed, the time of year.  And, of course, different scientists and technicians can unconsciously have their own unique influence on the result of the experiment.  No two people are alike.  The only way to get rid of the human element’s influence when performing an experiment is to somehow find a way to get rid of the human element.  Perhaps one could perform identical experiments using identical robots and that would minimize the effect of different personalities on all the different aspects of the experiment.  But the intromission of a robot, in order to create as sterile and neutral an aspect in the behavior of the conductor of the experiment as possible, could itself have an unforeseen effect on an experiment and create its own distortion in the results.  This would be particularly true in psychological experiments where the experiment relates to the impact of a total experimental presentation on human subjects.

            There are just too many elements that can’t be controlled that can impact the results of even the best constructed experiment.  These elements relate to how the people and things in the experiment are grounded in their setting and to the unmeasurable flowing blendable continual stimuli that are constantly appearing in the most isolating laboratory environments.

            So what are we to make of scientific studies given this experiential wrench that is constantly being thrown into the experimental situation.  Perhaps, first and foremost, we should understand that as seemingly neutral as science is in building a body of knowledge, it does have an ulterior motive: to help humans gain some kind of control over their living environment.  This has to be done by gaining control over those aspects of their living environment that are most amenable to control.  And this means shutting out those aspects of the living environment that are not so amenable to control.  But by definition, those aspects that are not so amenable to control are not going to be so easily subject to the control of humans trying to shut them out.  And this means that as hard as humans may try, their experiments, as, one may say, their lives in general, are always going to be impinged upon by unforeseen influences.

            In particular, experiments and lives are always going to be impinged upon by flowing blendable continual stimuli.  Modern technological humans are always going to focus in both their experiments and their lives on the defined discrete stimuli that give humans the illusion that they will someday be able to effectively control the totality of their living environment.  Scientific experiments, even with their distortions, can contribute to results that sometimes lead to a partial control over a particular aspect of the environment.  We humans should be grateful for this, even as we acknowledge that our scientific truths will never allow us to gain domination over the totality of human reality.

The topic for this article was suggested by Dr. Jorge Cappon.

© 2015 Laurence Mesirow

No comments:

Post a Comment