Landmark Psychiatry Papers

Unpublished data is still data

Unpublished results belong in the clinical picture

A study does not stop being evidence because its results never appear in a journal. People took part, outcomes were measured, and those findings may change our understanding of a treatment’s benefits and harms. Unpublished data is still data. When it can be obtained, it should be assessed alongside published findings, with the same attention to study quality.

When the results cannot be obtained, their absence still matters. We cannot assume that a missing study found no benefit, nor can we assume that it would confirm the published evidence. We should acknowledge that the available account is incomplete and that our clinical conclusions may be less secure than they appear.

This issue examines three investigations of that problem: a comparison of antidepressant publications with regulatory records, a search for unpublished psychotherapy trials, and a reanalysis of the records behind Study 329. These are studies of the evidence base, rather than three new treatment trials. Together, they help us ask a practical question: how confident should we be when some of the relevant results are missing?

1. Turner and colleagues, 2008: Published trials gave a more favourable picture

Reference: Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008;358(3):252–260. DOI. University-hosted original article.

Design and findings. The investigators matched FDA reviews of 74 short-term antidepressant trials, involving 12,564 adults and 12 agents, to journal reports. Twenty-three trials were unpublished. The journal literature presented 48 of 51 published trials as positive; FDA assessments classified 38 of all 74 as positive.

Across the full FDA dataset, the pooled standardized drug–placebo difference was 0.31 (95% CI 0.27–0.35), compared with 0.41 (0.36–0.45) from journal reports. These numbers describe the average difference between drug and placebo in standard-deviation units; they are not percentages of patients helped.

Contribution and limits. Starting with regulatory records made it possible to identify trials missing from journals and discrepancies in how results were presented. The analysis could not determine how responsibility for nonpublication was divided among investigators, sponsors, and journals. These findings concern a historical group of trials, not every antidepressant study. Methods, Table 1, and Figure 3.

2. Driessen and colleagues, 2015: Recovering psychotherapy studies outside journals

Reference: Driessen E, Hollon SD, Bockting CLH, Cuijpers P, Turner EH. Does publication bias inflate the apparent efficacy of psychological treatment for major depressive disorder? A systematic review and meta-analysis of US National Institutes of Health-funded trials. PLoS One. 2015;10(9):e0137864. Original article and DOI.

Design and findings. Starting with NIH grants, the researchers traced psychotherapy trials and sought unpublished results from investigators. Thirteen of 55 grants that started trials produced no publication. For psychotherapy versus control conditions, adding six unpublished studies to 20 published studies reduced the pooled effect-size estimate from Hedges’ g 0.52 to 0.39; the combined estimate’s 95% CI was 0.08–0.70.

Contribution and limits. Tracing funded trials showed which studies had gone unpublished. The sample covered only NIH-funded research, comparison treatments varied, and two unpublished studies remained unavailable. Missing protocols also made it difficult to check whether published reports had changed the planned outcomes. The results cannot tell us how much every psychotherapy estimate should be adjusted. Methods, Results, and Discussion.

3. Le Noury and colleagues, 2015: A published paper may leave important data out

Reference: Le Noury J, Nardo JM, Healy D, et al. Restoring Study 329: efficacy and harms of paroxetine and imipramine in treatment of major depression in adolescence. BMJ. 2015;351:h4320. DOI. University-hosted original article.

Design and findings. This independent reanalysis revisited an eight-week randomized trial of paroxetine, imipramine, and placebo in 275 adolescents. On the measures of benefit specified before the trial, it found no statistically significant advantage for either drug over placebo. Its harms review identified concerns including suicidal ideation and behaviour with paroxetine and cardiovascular problems with imipramine.

Contribution and limits. The reanalysis showed the value of checking a paper against its original study plan and detailed records of harms. Only 93 of 275 individual case report forms could be examined. Harms were described rather than compared with formal statistical tests. This short trial in a selected group cannot answer every question about adolescent depression treatment. Methods, efficacy results, and limitations.

Record update: The original 2001 Keller article has an expression of concern. That notice concerns the original publication, not this 2015 reanalysis; an expression of concern is distinct from a retraction.

What this means for clinical decisions

The editorial lesson is to account for the evidence we can read and the evidence we know is missing. If unpublished results are available, assess their methods and findings before including them; unpublished does not automatically mean unreliable, just as published does not automatically mean reliable.

If results remain unavailable, say so when describing the strength of the evidence. Missing data cannot be added to an estimate by guessing what they would show. Instead, the gap should temper confidence in that estimate and in the clinical claims built on it. There is no single percentage by which to reduce confidence whenever a study is missing.

This matters when discussing what a treatment might achieve and what remains uncertain about its harms. A useful explanation might be: “The studies we can examine suggest benefit, but we know that some results are unavailable. That limits how certain we can be about the size of the benefit and the full picture of harms.” The wording should reflect the actual evidence for the treatment being discussed.

Practical questions for the next journal club

Use one paper or treatment review and work through four questions. These are editorial discussion prompts, not additional findings from the three studies.

1. What does the paper say the treatment helps with?

Describe the main finding in plain language: who was studied, what improved, and over how long. Check harms separately. Avoid letting a broad conclusion such as “effective and well tolerated” replace those details.

2. Do we know of studies or results that are missing?

Check whether the paper or review mentions unpublished trials, unavailable results, or outcomes that were measured but not reported. Look for links to a trial register, the original study plan, or a fuller study report. For a journal-club session, identifying the gap is useful even when the group cannot recover the missing results.

3. Could the missing information change our confidence?

Discuss what is missing: a small part of one study, several whole trials, or details about harms. Do not assume the missing findings are favourable or unfavourable. Ask whether the available evidence supports a firm conclusion, a cautious conclusion, or a conclusion that needs more information.

4. What would we explain to a patient?

Prepare two sentences: one about what the available evidence suggests, and one about its main uncertainty. Then discuss what further information would help with the decision. The aim is a clearer conversation about expected benefits, possible harms, and how much confidence the evidence supports.

Discussion question: If you learned that several trials of a treatment had never reported their results, what would you change in the way you explained that treatment’s evidence to a patient?

AI and editorial disclosure

AI-assisted research and drafting. Model details were not recorded for this edition.

AI-generated content may contain errors or omissions. For education only; this does not replace individualized clinical advice or current clinical guidance.

As editor and publisher, I take editorial responsibility for published content and corrections. About this newsletter experiment and its editorial process.