A TALE OF TWO STUDIES

In my previous post[1], I reviewed a study[2] about  acupressure[3]. The researchers attempted to establish acupressure as an effective therapeutic method for the management of psychophysiological insomnia[4]. The study (somehow) concluded acupressure was a “safe, effective and cost-effective therapy,” though the study design didn’t actually test acupressure, or provide a clear, descriptive methodology and data showing the effective management of insomnia in the sample group. The study instead provided presented a causative effect from the research information it collected using the Sea-Band device, even though:

• No research had even established the efficacy of the device as an acupressure therapy (nor had any science-based research established any meaningful therapeutic effects of acupressure)
• The study did not directly test the safety of the device
• The researchers concluded the effectiveness of the device using the presumed authority [5]of a standard sleep study test and quantitative data, in order to dredge for significance across 22 data points from only two data sets, one pre-treatment measurement and one post-“treatment”

The acupressure study left a sour taste in my mouth. It seemed to be an overt attempt to provide legitimacy to Eastern “medicine,” acupressure, and the Sea-Band device by masking a poorly designed, poorly implemented pseudoscientific research study as science-based clinical research.

A 2012 study[6] from the Psychiatria Danubina[7] journal, utilized a similar research design to examine a more established psychological paradigm, Cognitive-Behavioral Therapy (CBT). Researchers from the Department of Psychiatry, at the Medical University of Gdańsk (Poland), studied the effect of CBT treatments on sleep quality and hyperarousal levels in adults diagnosed with primary insomnia. While the research design presents several similar areas of concern, the researches provide a more descriptive and precise methodology, specific and replicable data collection, and a more relevant conclusion.

DIAGNOSTIC DIFFERENCES

The 2013 acupressure and the 2012 CBT study both attempted to examine the effectiveness of their respective therapies in relationship to psychophysiological insomnia[8], the primary type of insomnia and sleep disturbance under care of a psychiatrist or psychologist.

Participants in the CBT study were diagnosed by psychologist with primary insomnia, using clinical criteria from the DSM-IV TR[9] as their diagnostic basis. Psychiatrist from the acupressure study instead utilized the ICSD-2 for diagnosing participants. What the DSM-IV TR refers to as primary insomnia, the ICSM-II manual, and the updated DSM (5)[10], refer to the condition as psychophysiological insomnia. The DSM-5 was released after the publication of the CBT study and more closely resembles the ICSM-II. Despite slight differences in terminology and format, the DSM-IV TR diagnostic criteria was essentially the same.

• A complaint of difficulty in initiating, maintaining sleep or experiencing non-restful sleep for more than one month
• Significant distress or impairment in functioning during waking hours
• Sleep disturbances must exclude known biological disruptions, environmental effects, medical conditions, mental illness, or substance use

COGNITIVE BEHAVIORAL THERAPY AND INSOMNIA

Cognitive Behavioral Therapy (CBT)[11], as a psychotherapy, attempts to relieve maladaptive thinking. These negativethought processes may cause behavioral, emotional, or psychosomatic dysfunctions. CBT presents patients with a logic-based therapy: a CBT therapist:

• assists patients in identifying and assessing thought patterns (cognitions) resulting in the undesired behavior(s)
• addresses misconceptions, regarding the patient’s thoughts or behaviors
• provides patients methods to alter their thought patterns in order to modify the desired behavior.

The Gdańsk study employed an insomnia specific CBT treatment developed by Dr. Michael Perlis[12]. Three key aspects of the therapy are:

1. Sleep Restriction Therapy: the therapist instructs the patient to restrict negative sleep behaviors—such as mid-day napping, and going to bed early—in order to adjust tiredness to the desired time frame
2. Stimulus Control Instruction: the therapist provides “do’s” and “dont’s” for maintaining restful sleep—the therapist and patient examine behaviors, thought patterns prior to attempting sleep, anxieties and other negative cognitions, determining patient specific conditions that may be disrupting sleep
3.  Sleep Hygiene Education: the therapist provides customized alterations to patient specific behaviors, attempting  to produce a guide for sleep, while allowing for the patient’s input and adjustment

Using the Perlis method, therapist also teach patients how to recognize and prevent reocurrences. Unlike acupressure, and Eastern “medicine” techniques, CBT has been well established in the literature, and is endorsed by the National Institute for Health[13].

The Gdańsk CBT study, when compared to the acupressure study, provided a superior attempt utilizing the same basic research design. The Gdańsk study provided relevant targeted behaviors and precise data collection with corresponding qualitative information. However, the study still shared similar design flaws, and failed to present a clear CBT as the specific cause relieving insomnia symptoms.

• The researchers provided extensive and precise data points for comparing therapeutic effects. The data collected was relevant to testing the hypothesis, but the chosen research design still suffers from inherently  weak internal validity.
• The researchers chose a specific mechanism of effect, an insomnia specific CBT treatment developed by Dr. Michael Perlis, but failed to explain specific methods provided in treatment.
• The study shows a strong correlation between CBT treatment and the relief of insomnia symptoms in participants, but does not provide sufficient data, due to design, to infer CBT as the sole causative factor for the therapeutic effect

AN INTRODUCTION

The Gdańsk study provided a literature review relevant to insomnia and specific to their targeted behaviors. Four behaviors are specified prior to testing the research effectiveness.

1. Sleep onset latency
2. Number of sleep awakenings
3. Sleep Quality
4. Psychophysiological hyperarousal

The study examined the efficacy of CBT treatment for primary—psychophysiological—insomnia though the four specified targets behaviors, and as a secondary objective, explored differences between persons more and less vulnerable to stressors predicting insomnia.

Each targeted behavior of the Gdańsk study was established prior to testing; previous research established these aspects as key factors of psychophysiological insomnia. The researchers provide multiple studies, establishing the four targeted behaviors. DSM provides similar information. By establishing specific points for testing, with established relevance, the researchers prevented issues of data-dredging for significance, a damning weakness of the acupressure study.

For comparison: the acupressure study briefly introduced the idea of insomnia, before diving into an exploration of Eastern “medicine”, and how acupuncture and acupressure restore the flow of energy along human body’s meridians (energy channels). The researchers explained a particular point on the body, the HT-7 point according to “tradition” —FYI > not established by any scientific tradition—seems to have a therapeutic effect in patients with disorders of sleep initiation and maintenance. Regardless of the existence of these meridians on the body, specific points, the Sea-Band doesn’t even claim to utilize that point.

The study aimed to assess the efficacy and safety of the HT-7 point acupressure system—the Sea-Band?— for treating insomnia in adolescents. They specified the use of a polysomnographic evaluation—without specifying which of the 22 data points they would consider relevant to their specific study, and why those points are relevant.

The DSM-IV TR[14] reveals the ineffectiveness of the polysomnography tests in distinguishing persons with insomnia from persons experiencing typical sleep (though the test may show impairments with sleep continuity—specifically sleep onset latency and frequent sleep awakenings).

THE DESIGN

The Gdańsk study recruited 26 individuals, in their early forties—22 women and 4 men. Persons presenting with insomnias other than primary insomnia were excluded from the study. Potential participants evaluated using the Hospital Anxiety and Depression Scale (HADS-M).[15] Individuals with high HADS scores were excluded from the study.

A Ford Insomnia Response to Stress Test (FIRST) was employed to test the secondary objective of the study, and separated participants into two groups-one group with a high vulnerability to stressors resulting in insomnia, the other group with a low vulnerability to stressors resulting in insomnia[16].

The study employed the four primary tests to assess sleep quality and hyperarousal associated with insomnia:

• Athens Insomnia Scale
• Leeds Sleep Evaluation Questionnaire
• Sleep Diary
• Actimetry Test

Treatment consisted of a supervised board certified CBT therapist providing the participants the Perlis CBT program for insomnia. Sessions were 50-minute, weekly, for 8-10 weeks, with one follow-up session at 3 months post-treatment.

RESULTS

The AIS scale was assessed prior to treatment, after treatment, and during the follow-up session. High scores relate to sleep difficulties. The results presented a significant change and a negative trend that continued after treatment, as of the follow-up session.

The AIS scale is a self-assessment test, established as an effective means for research and diagnosis of insomnia. It is a holistic assessment, including questions concerning sleep quality—awakenings, induction, duration—as well as, daily functioning and sleepiness.[17]

The study used the Hyperarousal Scale to determine hyperarousal levels in the participants. A significant statistical decrease was detected in psychophysiological arousal, as measured by the the Hyperarousal Scale.

Hyperarousal—somatic and cognitive arousals, often worrying or ruminating about sleeping or sleep onset—has proven to be a key component disrupting the sleep of persons  with insomnia (Espie et

al. 2006)[18]

The Hyperarousal Scale, another self-report assessment, has been established as a valid assessment of increased behaviors and thought patterns in persons with insomnia, when compared with EEG arousal measures.[19]

The study also used the Leeds Sleep Evaluation Questionnaire (LSEQ) to evaluate sleep quality, improved quality of falling asleep, and decreased morning awakenings.

The LSEQ is another self-report, assessed: a person’s quality of sleep, ability to get to sleep, quality of awakenings from sleep, and tiredness following wakening.[20]

The sleep diaries maintained by participants during and after treatment showed a significant decrease in

• Sleep onset latency—from more than 60 minutes, to nearly 20 minutes
• Number of sleep awakenings—from an average of 2, to an average of .04
• Sleep efficiency—77% before treatment, 93% after treatment

While subjective, if effective properly maintained, the sleep diaries provide not just expanded data to assess the process of the the treatment, but a method of self-assessment and potential therapeutic benefit.

The actigraphic test night/day ratio, performed over two seven-day periods—one  prior to treatment, and one following treatment—did not show a significant change in the ratio of night/day time (sleep/wake) time.

Actigraphic testing has shown to be as effective a test as polysomnographic study, neither have proven to be specifically as an identifier of persons with psychophysiological insomnia.[21][22]

Based upon the above findings, the Gdańsk researchers concluded CBT was an effective treatment for primary-psychophysiological-insomnia. Though the above assessments showed no significant relationship between participants with high vulnerability or low vulnerability as determined by the FIRST.

The Gdańsk study, like the acupressure study, employed an inherently weak research design. Both studies utilized a one-group pretest-posttest design[13]. A single group—in the Gdańsk study, adults with insomnia—are tested prior to therapy, provided the therapy, and tested after therapy. The Gdańsk study adds a second posttest, as well as multiple points of data throughout the therapy, via the sleep diary.

Unlike the acupressure study, the Gdańsk study attempts to balance this issue with these expanded data points. Each test is a relative test of hypothesis, and an established standard test within the fields of psychology and sleep medicine.

The Gdańsk study chose the actigraphic test over a polysomnographic test, despite the assertion—non-sequitur assumption?—from the acupressure research team, test results form either test cannot establish a relationship to psychological insomnia symptoms.[23]

Regardless of non-significance, the actigraphic test was assessed over seven days, during both the pretest, and the post test. The seven day test provides adjustments for natural variations in sleep. The test performed by the acupressure, on a single day, once before and after treatment, creates too many variables and cannot be an effective measure of sleep patterns.

The other test, while qualitative by design, have established a qualitative equivalence, or relevance, in assessing the various qualities in sleep affected by insomnia. While, the acupressure study attempts to express the importance of it’s quantitative test, the Gdańsk  study employed qualitative test that may prove more relevant, especially considering the study explored three distinct, established test providing and provided similar results, at three data-intervals—pretest, posttest, and during the 3 month follow-up. These results, in conjunction with the sleep diaries also assessing the study’s four target behaviors, provide significant data in concluding a relationship between CBT to improved sleep quality and diminished hyperarousal.

“Pharmacotherapy remains the most common treatment for insomnia. However, many studies suggest CBT may be a supreme therapeutic approach resulting in better long-term outcome (Morin 2006).”[24]

While several studies have shown CBT to be an effective treatment for insomnia[24] and other disorders—such as anxiety—and may be more effective than medicine for some mental illness. It remains difficult to determine whether simply receiving an attentive psychotherapy treatment for some issues are beneficial and CBT specifically. Current evidence shows CBT to be ineffective in persons with medical illnesses—as chronic fatigue syndrome-and certain mental illness—such as severe depression, bipolar and schizophrenia—and substance abuse issues.[26] 

The study mentioned using a modified mention of the Perlis treatment. Including specific methods employed during CBT therapy would increase the ability to generalize and replicate the study, and the study’s effects.

The Gdańsk study sees the lack of polysomnographic recordings as a serious limitation. However, the lack of a control group and the limited size of the sample provide a greater hit to internal validity. The lack of a control group, and random assignment increase bias for the experimenter and with the test group. The sample, enrolled by the study, hinders external validity and generalization.

While these flaws must be considered when considering the relevance of research, the study provided relevant data for CBT treatments of insomnia, in relation to the targeted behaviors.

The study clearly defines most methods, treatment and targeted behaviors, providing a basis for future research, and replication—though as mentioned above, a greater description of the methods of CBT would increase the ability to generalize the results.

Gdańsk researcher’s concluded the efficacy of the CBT treatment; however, the research design provides too many variables, and prevents causation with any certainty. At best, the research shows a strong relationship—correlation—between CBT treatments and insomnia management.

DISCUSSION

Why do physiological test have a hard time differentiating persons with psychophysiological insomnia from persons with typical, well-adjusted sleep patterns?

Does CBT offer an effective therapy, or has the current consensus of effectiveness within psychology/psychiatry provided CBT a better mask than acupressure, alternative therapies, or other psychotherapies?

STAY TUNED! Look for NEW posts every MONDAY! For updates, follow GETPSYCHED@getpsychedblog or on google+!

Send to Kindle

NEUROPSYCHIATRIC PSEUDOSCIENCE ATTEMPTS TO LEGITIMIZE ACUPRESSURE AND THE SEA-BAND TREATMENT

Insomnia is a disruptive and sometimes debilitating condition. Nearly 30% of adults report experiencing at least one symptom of insomnia^[1]. Though many of us experience situational insomnia to a degree—especially as a grad student—only about 6% of people experience insomnia severe enough to meet clinical criteria.

There are several types of sleep disruptions classified as insomnia. Psychological research of insomnia often focuses on psychophysiological insomnia.^[2]  To be classified as psychophysiological insomnia, the sleep disruption cannot be due to substance use, mental illness, or medical conditions. Diagnosis by the DSM-5^[3] must include dissatisfied sleep and at least one of the following criteria: sleep latency—difficulty initiating sleep; frequent nocturnal waking—difficulty maintaining sleep; prolonged periods of wakefulness; or undesired early waking. The sleep disturbance must also cause significant distress and dysfunction during non-sleep activity, and must occur for at least three nights a week over the course of three months.

A new study^[4] published in the Journal of Neuropsychiatric Disease and Treatment^[5] presents a suspect treatment for psychophysiological insomnia in adolescents. Researchers at the Clinic of Child and Adolescent Neuropsychiatry in Naples, recently concluded acupressure^[6] to be an effective therapy for managing insomnia in adolescents.

How? Unfortunately, I don’t know how. I have read the article a dozen times, and the science and research design are deeply flawed. 

I believe the study intentionally attempted to provide an unsubstantiated legitimacy to acupressure as an alternative therapy^[7]—specifically assisting the interests of the manufacturer of the Sea-Band device used within the study. The researchers used pseudo-scientific conventions to convey the pretense of scientifically valid clinical research. 

I fail to see how how the authors could present the data in the study as research. It is ill structured, even to infer a loose correlation, let alone conclude significance and effect—especially when considering:

• The study employed a flawed research design, vague data collection, and data-dredging^[8] to infer statistical significance.
• The researchers did not clearly define the mechanism of effect for the therapy.
• The study is full of logical fallacies^[9], further compromising the internal validity of the study and contributing to reader confusion.

THE STUDY DESIGN

The researchers explicitly described the study as an assessment of the efficacy and safety of an HT-7 point acupressure system when used to treat insomnia in adolescents. Yet, the study more accurately assessed the Sea-Band device, and does not conclusively treat insomnia.

The study did not examine or reference any established efficacy of Sea-Band as a treatment for insomnia, nor did it provide evidence for the efficacy of Sea-Band device as an acupressure device.

Further, the study inferred—relied on—acupressure as established therapy. Yet it did not provide a clear method of effect for either acupressure or the Sea-Band device.

“Insomnia can be treated with medication, herbal therapy, and psychologic or physical therapy. Commonly used medications include hypnotic/sedative drugs, but may have adverse effects, including impairment of memory, drug resistance, dependency, and addiction. Among the nondrug therapies for insomnia, acupressure, a method used for over 5000 years in Eastern medicine, is becoming increasingly popular worldwide.”

To establish acupressure as a potential therapy for insomnia, the author referred to a single study^[14]. That study reviewed meta-analysis of various studies combining acupuncture, acupressure, and reflexology; it concluded there was no evidence supporting any of these therapies as an effective treatment for insomnia.

Even presuming acupressure is a potential therapy, and stimulating the HT-7 point is the mechanism of effect, the author does not give clear cause for the premise that use of the Sea-Band is acupressure. In fact, while providing not further exploration into how they work, the Sea-Band website FAQ^[15] states the band exerts pressure and stimulates the P6 acupressure point, not the HT-7.

It is also not clear whether the devices were given to subjects with the pretense of improvement, or as a test for therapeutic effect, but the author specifically stated:

“The Sea-Band device was given to the patients in order to improve their symptoms related to difficulty falling asleep.”

This might convey to a potential participant—or an unsuspecting reader of the study—an unwarranted assumption of the established therapeutic ability of the Sea-Band device to diminish insomnia, and may also the confirmation bias of the research team.  

Researchers recruited 25 adolescents affected by psychophysiological insomnia. Subjects were caucasian from a middle-class socioeconomic background. Initial screenings and preliminary diagnoses were made by a neuropsychiatrist, utilizing criteria from the International Classification of Sleep Disorders (ICSD-2)^[10]—criteria from the ICSD-2 closely resembles the above DSM-5 criteria. 

The small sample size and the common locality of the sample subjects might present some concern—and were the only limitations of the study admitted to provided by the researchers—however, the sample size and criteria for diagnosis are not dissimilar to other sleep studies, and seem practical for an initial therapeutic investigation.

Subjects must have reported experiencing at least two of the following symptoms for at least two years prior to the study: fragmented sleep, frequent awakenings, early morning awakenings, or feeling tired due to poor sleep quality. They were given two Sea-Bands devices, to be worn bilaterally on the wrist at the Shen Men point on the wrists during normal bedtime hours.

Two sets of two polysomnographic sleep studies^[11] were performed. The first set established a baseline before acupressure “therapy,” and the second set to establish the effects after six-months of acupressure “therapy.” Researchers did not evaluate the first night of the sleep studies, from each set, in order to remove the first night effect^[12] known to produce unrepresentative sleep data.

The basic research design of the study is weak, and presents more questions than answers.

The study utilized a one-group pretest-posttest design^[13]. A single group—in this case adolescents with insomnia—are tested prior to therapy, provided the therapy, and tested after therapy. This design is generally referred to as a pre-experimental or quasi-experimental design. The internal validity of this type of test is compromised, due to the lack of a control group, and difficulty accounting for the impact of extraneous variables.

The addition of a control group—perhaps wearing a device similar in appearance to the Sea-Band—would have strengthen the validity of the research design.

Further, only two data points were used. A more effective study might have examined the subjects for multiple days during the pretest and posttest, and might have utilized sleep journals during the intervening therapeutic period.

DATA, DATA, EVERYWHERE

The research team collected and analyzed twenty-two separate criteria from sleep studies. Criteria included: Time in bed, total sleep time, sleep latency, percentage of time in specific sleep stages, sleep efficiency percentage (sleep time/time in bed * 100), etc. The team analyzed each data point using an analysis of variance test, and significance at the P value of < 0.05. They found a “significant” change in about half of the values. The significant results included: Increased time in bed, increased total sleep time, and reduced sleep onset latency.

“A strength of the the gold standard polysomnography for assessment of sleep alterations in order to circumvent subjective reporting by patients.”

The data does not support the explicit  hypothesis and relies on non-sequiturs to infer significance.

The researchers data-dredged. There hypothesis was vague, and they did not clearly state what and how they determine a change in sleep difficulty. By examining all 22 categories, the researchers presented a ripe opportunity for false pattern-recognition. This undermines the internal validity of the study—whatever internal validity remained—and severely muddies the conclusions of the study.

The researchers presumed the the device lessened sleep difficulties due to significant differences in only 11 of the 22 categories. They completely failed to acknowledge the p-values of criteria not supporting their hypothesis and the variability within their selected sample. Persons with insomnia do not have to experience all the symptoms to meet the clinical threshold—Instead:

• The study should have provided all clear changes in data values.
• The researchers should have predetermined data criteria, and established the importance of specific criteria for improving sleep quality.
• The researchers might have Instead examined only common values experienced by subjects.

The author used a non-sequitur argument to dismiss qualitative data and increase the perceived value of their quantitative data—researchers could have utilized a well designed qualitative survey to determine treatment efficacy, especially considering the initial criteria for diagnosis of insomnia is a dissatisfaction with sleep quality and an impairment of daily function due to lack of sleep. The change in the data provided presumes the therapy manages insomnia, without explaining how the data translates to insomnia relief, or whether subjects experienced a relief from their symptoms

CONCLUSIONS

“Acupuncture and acupressure are believed to restore equilibrium. Acupressure involves using the fingers, thumbs, palms, heels of the hand, and elbows to apply pressure and stimulate specific points along the meridians (or energy channels) of the body.”

The study concluded that acupressure could be a “safe, effective, and cost-effective therapy for the management of psychophysiological insomnia.” My conclusion: The researchers cherry picked data. The data’s support of hypothesis is vague and not clearly tested by the parameters of the study. Providing such a definitive conclusion, with such shoddy science, leads me to believe the researchers initiated the study with a pre-existing agenda, and intentional fraud, providing a disservice by further adding to public confusion and concern surrounding psychological research.

If a person with insomnia is concerned about the risks associated with sleep-aid medications, alternative therapies, like acupressure, provides a risk free alternative, also free of definitive results. But,

STAY-TUNED — My next post will examine the effectiveness of Cognitive Behavioral Therapy for psychophysiological insomnia.

STAY TUNED! Look for NEW posts every MONDAY! For updates, follow GETPSYCHED@getpsychedblog or on google+!

Send to Kindle

The Importance of Replication in Psychological Science [2/2]

THE OCCURRENCE OF REPLICATION

“ONLY 1.07% of published psychology articles were replications!”

Recent accusations of scientific fraud and shoddy methodology increased public scrutiny of psychological research. This scrutiny revitalized awareness within the field. Researchers discussed and debated various solutions, hoping to reduce error and scientific fraud. These discussions frequently pointed to one solution, replication.

I previously introduced a recent study directed by Dr. Matthew Makel: Replications in Psychology Research: How Often Do They Really Occur? Dr. Makel and a team of Duke investigators researched the previously unmeasured prevalence of replication in psychology. He acknowledges the recent public condemnation, and a need to redirect the field towards more rigorous science. Consistently testing research results reveals errors and strengthens procedural rigor. Frequent replication may provide psychology the key course correction needed to reverse public opinion and repair research methods.

The study conducted an exploratory investigation. The team hoped to provide an overview and the starting point for future discussion and direction. The Duke team investigated the entire publication history of 100 psychology journals. The team examined the journals for the total number of replications, and changes in the amount of replication over time. Dr. Makel also looked for factors contributing to successful replications.

The investigation team uncovered an alarmingly low frequency of replication in psychology. They also uncovered a substantial difference in successful reproductions when the replication was conducted by a member of the original research team.

Of the 320,000 articles, only 5,000 articles used a variation of the term replication, about 1.6%. The investigation examined a randomly selected sample of 500 articles using the term. Of those 500, only 68% were actual replications. Dr. Makel adjusted the data with this  new information, it revealed only 1.07% of published psychology articles were replications! While there seems to be an increase in the last ten years, the total number of replication studies was still low, at 1.8%. Dr. Makel believes this low occurrence of replication may indicate psychology actively deters replication.

NOT ALL REPLICATIONS ARE CREATED EQUAL

Like original research, replications must provide proper methodology and an accurate analysis of the data. By expanding the original data, replications assist researchers in recognizing clumps of data perhaps leading to false-positive results. However, replication research is not without its own inherent concerns. An important aspect of replication that the research is conducted by an independent team. The Duke study provides a clear distinction between the success rate of replications involving a a member of the original research team, and  those conducted by completely independent research teams.

Replications involving an overlap of team members from the original research presents increased risk of confirmation bias. The Duke team found 92% of replications with overlapping team members successfully reproduced the findings of the original study. Only 65% of replications by independent research teams confirmed the results of the orignal study. If a key aspect of preventing bias is independent replication, using Dr. Makel’s data, only .49% of psychological studies are independent research replications!

Dr. Makel expressed other minor concerns affecting replication. Confirmation bias also affects independent researchers (Moonesinghe, et al. 2007). Researchers may confirm an expected result due to perceived consensus. Further, variations in research resources and researcher capabilities creates concern about replication equivalency, and the precision provided by research replication. However, these concerns due not provide cause to disregard replication. A little bit of replication goes a long way. Improving the frequency of independent replications provides “self-correction” for original research, as well as the replicated research.

THE POSITIVE EFFECTS OF BIAS

Why does a low occurrence replication matter?

Most psychological scientist positively regard replication. Replication seems to be a simple enough concept, and has been valued by all of science as a sort of standard candle, or what Dr. Makel refers to as the scientific “Supreme Court.” Replication expands the original data and examines research methodology. Reproducibility helps research achieve validity, diminishing concerns about false-positive results, and decreasing experimenter and measurement biases. Replication contributes to consensus.

Unfortunately, the field emphasizes a conflicting ideal for original research. Researchers must fill gaps in the literature in order to advance knowledge. Further, Researchers involved in replication are not looked on as advancing knowledge, even though many agree,“a field that replicates its work is rigorous and scientifically sound.”  Even as a graduate student contemplating my theses, I am expected to find a gap in the literature, and develop original research. Just exploring new information ignores the benefit of independent replications. Reproducing or dismissing past results serves the literature by strengthening conclusions or revealing false-positive results. It is like  filling a pot-hole with stone, and failing to pave the cavity with cement. The ideal of originality provides a greater abundance of knowledge, but also generates greater uncertainty.

The low replication rate provides few opportunities to correct false findings, and may explain the high rate of successful research in psychology.

As I discussed in my last post,  the ESP research by Stanford psychologist, Dr.Daryl Blem, exemplifies intentional scientific fraud in contemporary psychology (and exacerbated the current public scrutiny). However, not all fraud is intentional. Psychologist tend to find significance in the research more frequently than any other discipline (Fanelli 2010 and Ioannidis 2005).  A year before the Duke study, an article in the Wall Street Journal explored the difficultly various disciplines experienced in reproducing research results. When replications do occur, they often fail to reproduce results from the original study, which may lead to a file-draw effect, in which researcher will not submit findings for publication in order to maintain the status quo. Scientist are pressured to succeed. Money and recognition flows more freely towards research establishing significance. Further research is needed, but these findings may be indicative of pervasive confirmation bias, supported by Makel’s findings about researcher overlap.

The infograph below displays the amount of published positive findings throughout science. Psychology publishes more studies supporting the tested hypothesis tan any other discipline. (Faneli 2010)

CONCLUSIONS

Dr. Makel suggests researching methods for determining what research should be replicated. I believe it is important instead to attempt to alter the attitude of the field towards replication. Any scientific field ignoring replication may enter a state where most published research is false (Ioannidis 2005). Psychology must emerge from the current scrutiny with a more rigorous adherence to empirical research and replication. Psychologist acknowledge the importance of replication, yet maintain an environment overwhelmed by the pressure to publish original research, and expand the literature. Readjustment is needed alongside a redirection of resources. Replication as part of increasing rigorous adherence to the scientific method, will provide a more precise measure of psychological therapies and practices.

Psychological science seems pointed in the right direction and as a discipline is in a unique position. The Duke reported the replication rate doubled since 2000. As a young science psychological science may be more open to a revolutionary redirection emphasizing replication. The historical dismissal of psychological science by more established disciplines, and the recent scrutiny and concerns of scientific fraud, further encourages response and a rigorous redirection. Further, Dr. Makel’s investigation provides the field the initial measurement necessary to encourage and compare future results from replication. His research establishes a baseline and will allow the field to test methods supporting replication, as well as the effectiveness of replication frequency.  By establishing a baseline and an emphasis for research replication, the field establishes a foundation best able to distinguish the signals from an overwhelming cacophony of white noise.

QUESTIONS FOR DISCUSSION?

• How do the rates of replication vary throughout the different disciplines within the field?
• Should schools encourage research replication for undergrads and graduate students, as their capstone or thesis?
  • Will this remove the stigma of replication as un-original and failing to advance knowledge?

[TL;DR]

• There is increased public scrutiny of psychology
• Dr. Makel investigated 100 years of psychology publications; only 1.07% of  studies were replications
• Replication is not encouraged in psychology
• Psychological studies achieve significance more than any other discipline
• Replication diminishes coincidence and intentional/unintentional error
• Psychology requires replication to reverse public concern, and recover as a more rigorous scientific endeavor.

Interested in more information about replication…?

CHECK OUT THE SCIENCE EXCHANGE REPRODUCIBLITY PROJECT 

STAY TUNED! Look for NEW posts every MONDAY! For updates, follow GETPSYCHED @getpsychedblog or on google+!

Send to Kindle

GETPSYCHED: AN INTRODUCTION [4/5]
EXAMINING THE HUMAN CONDITION, ONE EVIDENCE BASED STUDY AT A TIME

EX UNO PLURES

Ultimately I hope to develop a community with GETPSYCHED. I want to mold a group narrative, instead of remaining a singular voice. Community support and networking will expedite information discovery, quickly and efficiently vet information, and increase the site’s ability to reach out to broader and varied audiences.

The internet is big. Really big. The information is vast. Capturing more than just bits and pieces becomes less likely each day. A once gentle stream and steady broadcast of information now threatens to push past the levees. A brief vociferous response loses tangibility, drowned and muddied by the vast, rapid stream of information. A singular bulb no longer penetrates, and instead we need a beacon.

A world increasingly connected through online services needs to exponentially expand scientific communication. The internet provides nearly free, instant access to the growing abundance of unfiltered information. While scientific communicators offer a filter, and clear and consistent voice for evidence-based information, communicators no longer need to rely upon singular voices accessed through older mediums. The answer is not a great singular beacon. The answer lies with many smaller lights illuminating en masse.

A line in the sand needs to be drawn. With sufficient voices, a small community dedicated to observable validation might provide enough volume to defend empirically based discovery, inference and practices from mere assumption, misinformation, and pseudoscience. Pseudoscience encouraging ineffective or dangerous treatments and therapies. In order to achieve a proper volume, and emerge with prominence from the banks of the online information stream, scientific communication requires a community. These communities champion science-based narrative, and assist the communicator in providing understanding.

Community support provides assistance, increases impact and improves understanding for many different areas, across many different platforms. Forums and sub-reddits provide social communities for websites. Many communities self-monitor and vet information, in a collaborative development model. Users gain rewards when they contribute and provide useful information. The collaborative process allows the r/World News sub-reddit to provide information with greater speed than traditional media, or other social medias. Twitter far outpaces traditional media and most other form of media in providing access to current events (as well as a great deal of superfluous information). Sites like Wikipedia, stackoverflow, and XDA,  present user provided information, and facilitates assistance and (hopefully) a means for developing greater independence. Many sites develop community forums to share information, gather opinion, and provide an open-exchange for topic discussion.

I see a need in the field of psychology for communicating science. This need exsists as long as the public remains rife with misinformation and misunderstanding of psychology as a science. By developing a GETPSYCHED community, I hope to establish an authoritative voice, increasing informed decisions promoting of science-based psychology.

STAY TUNED and GET PSYCHED!

Next Post: Friday, September 27th, 09:00 EST
An Introduction, Part 5: SCIENCE ISN’T MAGIC / PSYCHOLOGY IS SCIENCE

Find me @getpsychedblog on Twitter

Coming soon! The GETPSYCHED sub-reddit

Send to Kindle

GETPSYCHED: AN INTRODUCTION [3/5]
EXAMINING THE HUMAN CONDITION, ONE EVIDENCE BASED STUDY AT A TIME

SCIENTIFIC JOURNALISM

A JOURNALISTIC ENDEAVOR

Thousands of blogs serve as personal journals, mere extensions of Facebook and other social media. Others offer op-eds, or attempt to communicate information, but provide expositions with inconstant or inaccurate information. One downside of instant access though: the death of journalistic integrity, and scientific journalism.

Efforts to capture an ever distracted and fleeting audience lead to misrepresented or inaccurate information. While the intent is not always to deceive, forgiveness and correction falls on def ears, as readers follow focus on new information, and follow click bait from the original misinformation.

I hope to distinguish GETPSYCHED from the many thousands of blogging sites by providing consistent, relevant and accurate information. Posts will examine psychological topics from varied sources, though a science-based perspective. GETPSCHED begins a journalistic blogging endeavor: my attempt to provide an objective voice, and present contemporary research, issues and interests within my field.

VALIDATING THE WHOS, WHATS, WHERES, WHENS, AND WHYS?

Journalistic development and integrity declined as the internet provided platforms freely and quickly delivering information—often in real-time. In order to remain relevant and retain readership, journalists and reporters needed to adapt, they needed to be quick, be first, and to do so requires a disregard for accuracy. Haste makes waste by severely limiting fact-checking.

Scientific journalism followed suite, also restricted by staffing reductions. Once writers clearly communicated objective, detailed information of scientific research. Today, many “science journalists” will rewrite press releases for technology and scientific studies, and fail to vet information in order to save time. Quick posts prevent journalists from becoming lost in the continuous stream of information, and allows them to best capture the ever shrinking attention span of their reader-base.

Traditionally, science journalists develop a narrative of a scientific endeavor, clearly illustrating the process and information to a public audience of non-experts, and providing the source of the information and examining the validity of the topic. In addition to hasty releases, many science journalists today also ignore evidence—or their sources’ lack of evidence or evidence-based procedures. They develop a contrived  narrative. Their subject is no longer the science. To reach a wider audience of less-informed individuals, many articles present mislead or misrepresented information on their subject, and relay extraordinary claims. Journalists present unbalanced information and focus on human interest, maverick practitioners with miracle cures, and extraneous warnings. Journalists highlight practitioners relaying personal, anecdotal experiences about treatments or disease, who develop unfounded causation, and disregard current consensus or the need to develop future empirical research testing their claims and personal observations.

Steven Novella, a Yale academic clinical neurologist, science communicator and skeptic, writes about his interaction with a such a journalist on his blog, Neurologicablog . The journalist in question published an article about a newly developed therapy for brain injury. This journalist, traditionally a sports writer, presented the information with an unbalanced narrative. He explains, in depth, how the developers of the treatment believe their treatment works. He discusses the patients’ anecdotal reports of success. He fails to adequately inform readers that this potential treatment still requires extensive research, proof of effectiveness, and FDA approval.

EXTRAORDINARY CLAIMS REQUIRE EXTRAORDINARY EVIDENCE

Rather than providing a vector for public discovery of new science, mass media agencies convey validity of untested therapies or previously unrecognized “dangers” from common items or behaviors. They sensationalize perceived dangers, increasing fear, misunderstanding, and superstitions. These services often attempt to attract viewers though hyperbole.

“Tune in later! Find out what common household item could be killing you right now!”

These hyperbolic attempts to increase ratings underlay a pervasive—sometimes reckless—fear mongering and disregards journalistic integrity. This lack of integrity muddies public perception of scientific validity and research.

A few months ago, I was watching the local evening news. The health report featured the “a dangerous new trend with a common household item.” The report featured a young woman with breast cancer, and her physician. This woman was diagnosed at twenty-one, an unusually young age for a breast cancer diagnosis. Like many women her age, she used her bra to carry her cell phone. The young woman and her doctor believe carrying her cell phone in her bra contributed to her illness. A summary of the report can be found here: http://www.wrcbtv.com/story/22910787/pa-woman-convinced-keeping-cell-phone-in-bra-caused-breast-cancer

Shortly after, I noticed posts of this report on Facebook and videos on YouTube. The news presented an unbalanced narrative, and warned women of the risk of carrying a cell phone in their bra. The opinions of other experts were not discussed. They failed to mention that extensive studies presenting no relationship between cancer and cell phone use. The report failed to report other reasons a person may develop cancer at such a young age. Was this young woman genetically predisposed? Did she express genetic mutations on one or both of the BRCA genes? Was she exposed to other environmental factors previously recognized and indicated as a contributing factor in the increase or occurrence of breast cancer?

Her  doctor attributes this unusual occurrence to an extraordinary claim. His only other evidence: one other patient that also carried her cell phone in her bra. Using reason, if that same doctor has treated hundreds of other patients with breast cancer, and these patients did not keep their cell phones in their bras, perhaps instead, carrying a cell phone in your bra prevents cancer?

QUOD GRATIS ASSERITUR GRATIS NEGATUR

The focus of scientific journalism should be the science. Presenting unsupported therapies, danger, and “mavericks authorities” encourages superstition and misunderstanding. Perhaps a greater danger, inaccurate reporting provides an enhanced volume and even grounds for unsubstantiated information, ideas and arguments. It muddies public perception of science, and lifts pseudoscience to an equal footing.

Science journalists should focus on the science, and at least construct a narrative relaying accurate information. Studies should be examined for their importance to society, future research, or generalized impact. They should increase public awareness of research, examine potential applications of developing science, and question claims requiring further research.

Like science communicators, science journalists should enable public awareness of science. Journalists should fact-check, understand their subject, and be aware of speculation. They must provide a balanced report and examine areas of weakness in design and evidence. Traditional and long-form journalism provides a platform for analysis, fact-checking and independent interpretations on a subject or narrative. GETPSYCHED will provide a more traditional journalistic styling. Quality of information supersedes speed of delivery. Posts examining research will critically analyze the information provided and the information not provided. Posts will discuss the impact of future research, and examine other areas and issues from an evidence-based perspective.

STAY TUNED and GET PSYCHED!

Next Post: Thursday, September 26th, 09:00 EST
An Introduction, Part 4: EX UNO PLURE

Find me @getpsychedblog on Twitter

Coming soon! The GETPSYCHED sub-reddit

Send to Kindle