Motivational Workshops are Good For You. Really. Stanford Says So
I mentioned in my last post I had been reading about academic fraud, and I was. I do. Frequently. I didn’t post anything on it, but after posting my last piece I came upon some material I had dug up a while back on a case that is, well, improbable. Not fraud, or so say the experts who know more than I, but research that makes you wonder ‘What were these people thinking?’
The story starts with an improbable figure: Tony Robbins. You may remember him, motivational speaker, was all over TV at one point, good looking guy, great teeth. This guy:
He was on OWN for a while, wrote several books, like Unlimited Power: The New Science of Personal Achievement (1986) and several others. He also went through some rough times, accused of sexual misconduct in Buzzfeed in 2019. I don’t know what came of that, and it is quite separate from his role in our current story.
A group of researchers, some from Stanford’s Genetics Department and its Stanford Health Innovation Lab (SHIL) undertook some research projects in which Robbins’s seminars figured. I don’t know how many projects there were in total, or how many papers they got published on them, here I will focus on two I do know about:
Non-traditional immersive seminar enhances learning by promoting greater physiological and psychological engagement compared to a traditional lecture format
This was published in the journal Physiology and Behavior in 2021, and you can read it here if you like. I’ll refer to this as the learning paper. The other is
Effects of an immersive psychosocial training program on depression and well-being: A randomized clinical trial
Published in The Journal of Psychiatric Research in 2022, you can read it here. I’ll refer to this as the depression paper.
Each paper lists 9 separate author-researchers, five of whom appear as authors on both papers. Michael Snyder, the Director of SHIL, is listed only on the depression paper, but other authors on the learning paper are from the Stanford Dept of Genetics.
The learning paper’s abstract starts with this sentence:
“The purpose of this study was to determine the impact of an immersive seminar, which included moderate intensities of physical activity, on learning when compared to traditional lecture format.”
In fact the researchers observed 26 people, 13 in an ‘immersive treatment’ group and 13 in a control group, all 26 of whom went through a two-day seminar. The difference in the two groups is described as follows:
“For the IMS group, participants were provided with a hard copy of lecture material which was presented at UPW with a combination of state elevation sessions (jumping, shouting, fist pumping, and high five behaviors) which were conducted approximately once every hour to raise arousal and to interrupt sedentary behavior, as well as mindfulness mediation[sic] that focused on a wide variety of awareness, affective states, thoughts and images [35] that were conducted once at the end of each day.”
The control group got the same seminar and hard copy of material, but without the meditation and ‘elevation sessions’.
What does this have to do with Tony Robbins? You will find his name in the paper, sort of, at the very end –
Funding
This study financially supported by Robbins Research International Inc.
What this does not mention is that the ‘lectures’ the research subjects attended were in fact a two-day Robbins seminar, one of those things one pays a lot for.
Still, nothing wrong that I can see with someone like Robbins sponsoring research to see if one of two different approaches leads to better learning outcomes in his seminars. However, because all the subjects are people who paid a lot to be in this environment, one can’t really claim they are representative of the general population, so one has no reason to think that the results of this study tell us much about the impact of such ‘immersive’ learning in other situations.
Beyond that, there are only 13 research subjects in each group. That is a rather small number, which makes one wonder whether the results mean much. Here’s what the researchers say they found:
“The primary findings of this study were that learning was greater in the IMS compared to the CON as the increased performance on the exam was sustained 30- days post event when compared to CON, which decreased 30-days post event.”
Ok, but the question that comes to me at this point is – why did these well-placed researchers bother with this? The paper’s Conclusion section actually notes that “Previous studies have shown that physical activity can promote learning in traditional classrooms [2,3].” They then go on to note that those previous studies utilized different types of exercise. Whoop-de-do. I would think researchers at this high level would be interested in research that could move the needle more than that. I get that Robbins (maybe only partly) funded the work, but I doubt these people are hard up for research funding. So, in the end my only question is really: why and how did these researchers get involved with Robbins at all, and on such a mediocre project?
Hang on. I came upon this in Gelman’s statistics blog, which I read regularly and mention here often. He didn’t take these guys on, all he did was reprint some material from an article in The San Francisco Chronicle which, based on the bits I’ve been able to read (paywall) was really hard on these researchers and this research.
As I noted, Robbins had been in the news not long ago for bad behavior, so if the learning paper was the extent of this Robbins/researchers interaction, I would likely chalk this up to no more than a newspaper thinking it can score points by embarrassing some Stanford researchers for their association with Robbins, and take another strip off Robbins himself in the process. Nothing (much) to see here.
But then there’s the depression paper.
Again, Robbins’ name appears only once, at the end of the paper, like this:
“This study was not funded by Robbins Research International; however, they did allow participants to participate in the DWD program at no charge. They also provided housing for two research coordinators who stayed on site during the trial.”.
DWD here refers to Date With Destiny, which is a Tony Robbins seminar that the paper describes as follows:
“…a six-day immersive training program that includes a subsequent 30-day daily psychosocial exercise follow-up period. DWD is popular with thousands of people using this intervention annually. The program combines a variety of lifestyle and psychological approaches that seek to improve well-being, including cognitive reframing, guided meditation and visualization, neurolinguistic programming, gratitude, goal setting, guided hypnosis, community belonging and engagement, and exercise. Although components of the program such as exercise, gratitude, and cognitive reframing have independently been found to improve mental health and wellness (Goyalet al., 2014; Kvam et al., 2016; Mikkelsen et al., 2017; Schuch et al., 2016), the effectiveness of the DWD program has not been investigated.”
No mention anywhere of the fact that this is an ongoing Tony Robbins (money-making) seminar series for which people (although not the subjects of this study) pay big bucks to attend, although people familiar with the Robbins operation would recognize the DWD name. Here’s what the Robbins website says about DWD:
Create life according to your terms
Dive deep into the patterns that are holding you back, ignite your motivation, and build momentum toward the life of your dreams.
I guess that’s kinda ‘mental health and wellness’, right?
The researchers describe their experiment as follows:
“A randomized clinical trial was conducted in which 45 participants were randomized at 1:1 ratio to DWD (n = 23) or a gratitude journaling control group (n = 22) (Fig. 1). Depressed individuals (n = 27), as assessed by the Patient Health Questionnaire-9 (PHQ-9; see below), and those without depression (n = 18) were recruited by email, flyers, and physician referral in the U.S.
At least there are more subjects this time, right? Ah, no. In fact there were 14 depressed subjects assigned to the Seminar and 13 assigned to the ‘gratitude journaling’ control group. The other 18 subjects, 9 in each group, were not designated as depressed according to their own original responses on the above-mentioned PHQ-9 questionnaire, and as we shall see, it is the originally depressed subjects who are the star of the show.
That’s because the big question that is being asked here is: does attending a Tony Robbins six-day DWD event reduce depression?
Can anyone guess what answer Mr. Robbins would like to hear to that question?
And indeed, that is the answer we get – in spades. Here’s the payoff, from the paper’s Abstract:
“Seventy-nine percent (11/14) of depressed participants in the intervention condition were in remission (PHQ-9 ≤ 4) by week one and 100% (14/14) were in remission at week six.”
In remission here means that on that self-reported questionnaire, the scores generated by their answers were below the threshold at which they are considered depressed. In other words, the Tony Robbins DWD seminar has a 100% cure rate for depression.
One need not even ask how the control group did, my god, all the depressed people were cured by the treatment. Huzzah!
More times than I could count, I have written in this blog, ‘If it seems too good to be true, it ain’t true’.
Reasons for skepticism are many and varied.
In clinical trials of anti-depressants, typically something like half of participants report ‘feeling better’ after six to eight weeks. A paper in the Lancet I dug up said that 62% of adults reported ‘improvement’ in depression after psychotherapy, at varying time frames.
But DWD – 100% cure rate. Go, Tony.
There is once again the small sample issue here, just as in the learning paper, but to this under-educated economist, the subjects themselves are the big question mark. These are people who self-reported being depressed, went to a DWD seminar for six days for free and then were asked again afterward about how they felt. Ya think maybe they were inclined to believe in the power of DWD?
The bits of the SF Chronicle article that are quoted on Gelman’s blog suggests that some of the SHIL researchers knew and were fans of Robbins before the research started. I can’t say anything about that, and I don’t think you need to know that to wonder about the results in the depression paper.
A final note. If you do go to download the depression paper, you will find that the journal in 2024 also published a Corrigendum about the original 2022 paper, and this corrigendum has the same original nine authors on it. These corrigenda are published by a journal when a mistake is found in a published paper, but is thought not so egregious as to warrant retracting the paper completely. It notes that there was an error in calculating the post-treatment PHQ-9 score for one of the treatment subjects, and as a result, the cure rate was ‘really’ only 93%.
Most interesting in this corrigendum, all two pages of it, is the following paragraph, which I quote:
“Finally, we note that after the article was first made available online on March 9th, 2022, Dr. Snyder became a co-founder of a startup, Marble Therapeutics, on July 12th, 2022. Mr. Robbins later invested in Marble Therapeutics on September 26th, 2022, three months after the final version of the article was published. We do not believe there was a conflict at the time this work was done, but nevertheless wish to note this relationship.”
There’s that other thing I often write: can’t make this shit up.