A trial called the ‘SMILE trial’ compared the Lightning Process® with specialist medical care for young patients with ME/CFS. The trial was run at Bristol Medical School by Professor Esther Crawley.
Recent studies of the Lightning Process® by the Wessely School are described here.
The information detailed in this post is drawn from the extensive detective work of David Tuller, Senior Fellow in Public Health in Journalism at the Center of Global Public Health, School of Public Health, University of California, Berkeley, California.
Unusually, the SMILE trial was published twice, first in 2018, then again, in corrected form in 2019, with an explanatory editorial and a different set of authors:
What was the SMILE trial? What were the findings? And why was it necessary to correct the trial report?
What was the SMILE trial?
I quote from the Abstract of the trial protocol:
“The Lightning Process® (LP) is a trademarked intervention derived from osteopathy, life-coaching and neuro-linguistic programming, delivered over three consecutive days as group sessions. Although over 250 children with CFS/ME attend LP courses each year, there are no reported studies on the effectiveness or cost-effectiveness.
Methods
This pragmatic randomised controlled trial is set within a specialist paediatric CFS/ME service in the south west of England. Children and young people with CFS/ME (n = 80 to 112), aged 12 to 18 years old will be randomised to specialist medical care (SMC) or SMC plus the LP. The primary outcome will be physical function (SF-36 physical function short form) and fatigue (Chalder Fatigue Scale).
Discussion
This study will tell us whether adding the LP to SMC is effective and cost-effective compared to SMC alone. This study will also provide detailed information on the implementation of the LP and SMC.”
Fears About Ethics and Safety
The ME Association and The Young ME Sufferers Trust made a joint statement saying that they “do not believe that it is ethically right to use children in trialling an unproven and controversial process such as the Lightning Process.” A survey of 4,217 people carried out by the ME Association on the management of ME/CFS found that over a fifth of those who had tried the Lightning Process were made worse (7.9% slightly worse,12.9% much worse). If any trial is to be held, it should first be on adults, who can give informed consent. The statement continued by disputing its underlying theory:
The theory upon which the Lightning Process is based, together with its claim that the prolonged nature of the illness is caused by ‘the adrenaline, nor-adrenaline and cortisol loop’ is not scientifically proven. Moreover, the Advertising Standards Authority recently ruled that an advertisement in an internet sponsored link containing claims of its effectiveness by a Lightning Process practitioner should be removed. We understand that the practitioner will be involved in this study and we find this concerning…We cannot approve of a study involving children as young as eight when no rigorous trials have first been undertaken into the safety, acceptability, long and short-term effects of the application of this controversial and unregulated ‘process’ with adults.
What were the findings?
According to the corrected report, the findings at six months from 81 of 100 participants who started the trial were as follows:
Physical function (SF-36-PFS) was better in those allocated SMC+LP (adjusted difference in means 12.5 [95% CI 4.5 to 20.5], p=0.003) and this improved further at 12 months (15.1 [95% CI 5.8 to 24.4], p=0.002). At 6 months, fatigue and anxiety were reduced and at 12 months, fatigue, anxiety, depression and school attendance had improved in the SMC+LP arm.
Conclusion The LP is effective and is probably cost-effective when provided in addition to SMC for mild/moderately affected adolescents with CFS/ME.
So, according to the report, LP is effective and is probably cost-effective. But is it safe? And…
Why was it necessary to correct the trial report?
The short answer to this question is that it was owing to the painstaking investigations of David Tuller.
I quote from a Letter dated October 8th, 2018, to Nick Brown the editor-in-chief of Archives of Disease in Childhood.
Dear Dr. Brown:
In September, Archives of Disease in Childhood published a study called “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial” [1]. Like many studies of this illness, it was an open-label trial with self-reported outcomes—a study design known to be highly vulnerable to bias.
The researchers have received criticism for their initial decision to conduct a pediatric trial of the Lightning Process–a commercial, pseudo-scientific intervention that combines elements of life-coaching, osteopathy, and neurolinguistics programming. The intervention encourages participants to report that it has made them better, another source of potential bias. Beyond the decision to study the Lightning Process, the study suffered from major cross-contamination between treatment arms and significant loss-to-follow-up, among other problems.
Methodological Concerns
The letter also mentions a “methodological anomaly” as originally reported by David Tuller:
“An earlier feasibility trial was extended into the full trial even as the researchers swapped primary and secondary outcomes following a review of the feasibility trial results. The researchers received ethical approval for these study design changes but failed to mention them in the Archives paper, although the changes enabled the findings to be presented in a much more positive light than would otherwise have been the case.”
The letter explained specific concerns as follows:
*The researchers registered the study as a prospective trial. But more than half the sample were apparently participants recruited for the feasibility trial, starting almost two years before the registration date. The Archives paper did not mention this fact or explain the discrepancy in dates.
*In the feasibility trial protocol, school attendance at six months was the primary outcome, and self-reported physical function at six months was a secondary outcome. After reviewing results from the feasibility trial, the researchers swapped these outcome measures for the full trial—school attendance at six months became a secondary outcome, and self-reported physical function at six months became the primary outcome. The Archives paper did not disclose the outcome-swapping that occurred after more than half the sample had provided data as participants in the feasibility trial.
*As its main finding, the Archives paper highlighted the reported benefits in the Lightning Process group for self-reported physical function at six months, the new primary outcome. Self-reported school attendance at six months, now a secondary outcome, yielded null results and received scant mention in the Archives paper. Thus the outcome-swapping allowed the researchers to report positive results for their primary outcome. Had they not switched the outcomes after more than half the sample had provided data, they would have had to report null primary outcome results
*Not surprisingly, the media coverage of the Archives paper focused on the reported benefits in the Lightning Process arm for the revised primary outcome of self-reported physical function at six months. The null results for school attendance at six months, the original primary outcome, were largely ignored. The outcome-swapping that occurred after more than half the sample had provided data—and that the Archives paper did not mention—ensured that the reported findings received more positive media coverage than they otherwise would have.
Faint Praise
The Science Media Centre, for whom one of the trustees is – guess who – Professor Sir Simon Wessely, gave expert reactions to the SMILE trial by four ‘experts’. IMHO, their comments on the study are damning with faint praise, almost zero praise, to be precise. In fact, these commentators chosen for their likely positive take on the study, raised issues about the biased and uncontrolled methodology.
Prof. Alastair Sutcliffe, Professor of General Paediatrics, UCL:
“A recent systematic review of neurolinguistic programming (NLP) stated “There is little evidence that NLP interventions improve health-related outcomes. This conclusion reflects the limited quantity and quality of NLP research, rather than robust evidence of no effect. There is currently insufficient evidence to support the allocation of NHS resources to NLP activities outside of research purposes.” [Br J Gen Pract. 2012 Nov; 62(604): e757–e764. Published online 2012 Oct 29. doi: 10.3399/bjgp12X658287, PMCID: PMC3481516]. But now we have this interesting study by Crawley, a well-conducted single blind clinical trial that suggests NLP, in combination with other therapies and described as the ‘Lightning Process’, is effective for some children with the very hard to treat condition of chronic fatigue syndrome (CFS).
“Although in my view the effects described show some benefit and are therefore to be welcomed, this could be due to placebo which would still be GOOD news. Costs are modest and therefore this study is to be welcomed.
If the SMILE trial is only acting as a placebo effect, then much cheaper placebos are available and so LP would therefore not be a great choice.
Prof. Dorothy Bishop, Professor of Developmental Neuropsychology, University of Oxford:
“The gains for patients in this study do seem solid, however, I am still rather uneasy because while the patient allocation and statistical analysis of the trial appear to be done to a high standard, the intervention that was assessed is commercial and associated with a number of warning signs. The Lightning Process appears based on neurolinguistic programming, which, despite its scientific-sounding name, has long been recognised as pseudoscience.
“I am sympathetic to the authors’ decision to evaluate the Lightning Process (LP), given that they had patients who had used it and reported favourably on it, and it could be argued that to fail to do so would indicate a degree of closed-mindedness. But the commercial nature of LP really creates problems. We cannot tell which aspect of LP is responsible for the gains in patients who took part.
“I noticed, for instance, that LP involves group sessions, whereas the comparison group undergoing standard medical care were treated individually. So it may be that the benefits derive from interacting with other children with chronic fatigue syndrome/ME, rather than the specific exercises and training.”
Again, the comment draws attention to the problems with LP – its warning signs of NLP and pseudoscience, badly designed study using dissimilar group conditions, inability to tell what aspects made a difference, if any.
Dr James Thompson, Honorary Senior Lecturer in Psychology, UCL:
“The treatment in this study looks like it had an effect, at least by the standard of most clinical trials. To be extra robust I would have liked to see more objective measures, but unfortunately chronic fatigue syndrome is not an objective diagnosis, it is a leftover category and fatigue is subjective.
“One limitation is that self-report scales can be subject to placebo effects, however if the patients feel better in the experimental condition in which they receive extra help, even if everyone knows it, then that is something and the pupils miss less school, which is an objective measure. In this case it may not have been the CBT element of the treatment, but it looks like it.”
Once again, a placebo effect is suggested, with subjective outcomes, so not a positive comment.
Prof. Michael Sharpe, Professor of Psychological Medicine, University of Oxford
Finally, the comments from an ‘inner circle member’:
“Chronic fatigue syndrome (CFS) is a name for an illness with symptoms of long lasting and disabling fatigue. It affects many young people and can interfere with their education. Whilst some people call it myalgic encephalomyelitis (ME) it is not clear if this is the same or a different condition.
“This trial tests the effectiveness of a commercially available brief intensive talking therapy for CFS called the Lightning Process. The treatment has similarities to cognitive behaviour therapy (CBT) and is given in groups. The treatment was found to be better than usual care in fatigue, physical function and school attendance, with benefit seen as long as a year later. It was also safe. The study does not tell us how it works however.
“This is a robust study because patient were allocated to one of the two treatments at random ensuring that any difference seen in outcome between these treatments, is not due to pre-existing differences in the patients. The main limitation is that, as it is not possible to hide which treatment they received from the patients, their self-ratings of fatigue and functioning could potentially be biased by their views on the treatment they received. However, differences in the school attendance a year later were also noted; it seems likely that these could be due to such a bias.
“Commercially available treatments like this one that are being used by patients should be rigorously tested. This is especially important for an illness like this one about which much misinformation is spread using social media. We need more studies and less polemic.”
Again, there are negative comments about the design, which Professor Sharpe believes could have biased the findings.
Implications
- The SMILE trial contained one of the major methodological problems that occurred in the PACE trial – outcome swapping. The original outcome measure, school attendance at six months, yielded a null result.
- Repeating the errors of the PACE trial, the SMILE trial used self-reported physical function, a subjective outcome measure rather than an objective one.
- Media coverage reported the positive outcome (improved self-reported physical function) but tended to not mention the null result concerning school attendance.
- In seeing a distorted view of the findings, authorities in other countries such as Norway have been persuaded to run further trials on the Lightning Process and to recommend its use for pwME/CFS.
- The safety of the LP requires in-depth study.
One thought on “ME/CFS and the SMILE trial”