Are scientific journals just parasites? (with Chris Chambers)
Spencer Greenberg and Chris Chambers discuss biases in academic publishing, like outcome and publication bias. They propose reforms such as Registered Reports, open science practices, and altering researcher evaluation to foster more robust and trustworthy scientific progress.
Deep Dive Analysis
19 Topic Outline
Introduction to Science Reform and Outcome Bias
Understanding Outcome Bias and Publication Bias
Critique of Journal Prestige and 'Flashy' Results
Introducing Registered Reports for Bias Neutralization
Mechanism and Benefits of Registered Reports
History and Adoption of Registered Reports
Registered Reports for Exploratory and Confirmatory Science
Addressing Self-Deception in Scientific Research
Critiques of Commercial Publishers and Free Peer Review
Motivations for Scientists to Conduct Peer Review
Publishers' Role: Distribution, Credibility, and Alternatives
The Peer Community Inn (PCI) Initiative
Challenges of Journal Brands as Quality Heuristics
Differentiating Scientific Record from Science Journalism
Incentives and Reforms for Academic Hiring and Promotion
The Role and Misuse of Citation Counts
Rapid Fire: Sci-Hub and Open Access
Rapid Fire: Lowering P-Values and Statistical Debates
Rapid Fire: Data Sharing and Ego Depletion Consensus
5 Key Concepts
Outcome Bias
A cognitive bias where the outcome of an intervention, policy, or decision determines whether that decision was considered good. In science, this can lead to a biased record if judgments about research quality are influenced by the results obtained.
Publication Bias
A specific form of outcome bias in scientific literature where the likelihood of a study being published is influenced by whether its results are statistically significant or conform to expectations. This can lead to a skewed scientific record where negative or inconclusive findings are underrepresented.
Registered Reports
A publishing format designed to neutralize outcome and publication bias by having journals decide whether to publish research *before* the results are known. This involves peer review of a study's protocol, methodology, and theoretical background, leading to an 'in-principle acceptance' regardless of the eventual findings.
Preprints
Scientific manuscripts that are made publicly available online prior to formal peer review and journal publication. They serve as an efficient way to disseminate research freely, allow for community feedback, and support versioning for corrections and updates.
Peer Community Inn (PCI) Initiative
A non-profit, academic-led initiative that manages the peer review process for preprints, publishing these reviews and evaluations openly. It aims to shift control of scientific credibility from commercial publishers back to the academic community, allowing journals to accept these evaluations without further peer review.
8 Questions Answered
Outcome bias is a cognitive bias where the quality of a decision or study is judged based on its outcome, rather than the rigor of the process, which can lead to a distorted scientific record if inconvenient results are suppressed.
Registered Reports tackle outcome bias by having journals conduct peer review of a study's methodology and theoretical background *before* the results are known, issuing an 'in-principle acceptance' that guarantees publication regardless of the findings.
Scientists engage in free peer review due to intrinsic motivations like contributing to public good and advancing their field, as well as more self-serving reasons such as gaining early access to research, building a reputation with editors, and career advancement.
Commercial publishers primarily manage the peer review process and maintain journal brands, but their distribution function is largely superseded by the internet, and their credibility is argued to stem from the free labor of academics rather than their own added value.
The academic community can regain control by managing peer review of preprints through non-profit initiatives like Peer Community Inn (PCI), which openly publishes reviews and evaluations, allowing journals to accept these evaluations without further peer review.
Scientific journals should focus on curating a scientific record based solely on the quality and rigor of research, regardless of outcome, while newspapers (or a separate layer) can focus on promoting 'cool' or newsworthy findings, ensuring these two functions are not blurred.
Lack of consensus in fields like psychology, exemplified by ego depletion, often stems from the low quality and rampant biases (publication, outcome, reporting) in much of the existing work, which creates unnecessary uncertainty and a distorted literature.
Preprints allow for the free and rapid dissemination of scientific research prior to formal peer review, enabling community feedback, version control for corrections, and providing a transparent record of a paper's evolution, which is valuable for learning.
14 Actionable Insights
1. Adopt Registered Reports for Research
For appropriate research, submit a detailed protocol for peer review before conducting the study to secure in-principle acceptance, ensuring publication regardless of results. This neutralizes outcome and publication bias, focusing evaluation on methodological quality and question importance.
2. Prioritize Quality Over Flashy Results
As a researcher, prioritize conducting high-quality, rigorous work over seeking exciting or novel results, as the scientific record should be determined by work quality, not outcomes. This helps prevent a lopsided literature and self-deception.
3. Embrace Transparency in Research
Be transparent about all analyses, including exploratory ones, and avoid reinventing history to present post-hoc discoveries as predictions. This ensures the scientific record is accurate and prevents misleading conclusions.
4. Publish All Rigorous Research Findings
Publish the outcomes of all rigorous research, including negative, inconclusive, or assumption-challenging findings, potentially using formats like Registered Reports. This prevents others from repeating mistakes and makes the best use of scientific resources.
5. Share Research Data Publicly
Publish data sets, even if not fully used in your research, with appropriate anonymization, especially for publicly funded work. This allows the broader scientific community to benefit from collected data and advances knowledge.
6. Shift Peer Review Control
Advocate for and participate in initiatives that shift control of the peer review process from commercial publishers to the academic community, such as using preprints and non-profit peer review entities. This reduces publishing costs and ensures the process serves scientific, not corporate, interests.
7. Reform Researcher Evaluation Criteria
As an academic, advocate for and implement changes in hiring, promotion, and assessment panels to value open science practices (e.g., data sharing, pre-registration) and the quality of research, rather than relying on crude heuristics like journal prestige or grant money.
8. Utilize Preprints for Dissemination
Publish research as preprints on public websites before formal journal submission to rapidly share findings, initiate community discussion, and allow for versioning and corrections. This makes science more accessible and dynamic, moving beyond an 18th-century fixed record model.
9. Exercise Caution with Citation Counts
View citation counts as a measure of short-term impact, not necessarily an indication of research quality. Avoid relying on them as a superficial metric for evaluating work or scientists.
10. Ensure Balanced and Ethical Citations
As an author, ensure citations are properly balanced and avoid strategically omitting contradictory work. As a reviewer, avoid coercing authors into inappropriate citations.
11. Practice Responsible Science Communication
When communicating science to the public or journalists, ensure press releases and discussions are as careful, factual, and free of spin as possible. This is crucial as many journalists may lack deep scientific training.
12. Distinguish Science Record from News
Recognize that the scientific record should publish research based solely on quality, regardless of outcome, while a separate layer can highlight ‘cool’ or newsworthy findings. This prevents blurring and distortion of scientific priorities.
13. Embrace Disagreement and Lack Consensus
Recognize that lack of consensus is normal and healthy in many areas of science, especially for newer theories, and is not necessarily a sign of failure. This fosters critical thinking and ongoing scientific inquiry.
14. Continuously Learn and Seek Understanding
Make a habit of constantly taking in information from a wide variety of sources (scientific papers, articles, books) to build broad knowledge and deeply understand the structure and models of various phenomena. This fosters intellectual growth and expertise.
6 Key Quotes
The scientific record should not be determined by the results. It should be determined by the quality of the work going into it.
Chris Chambers
You are the easiest person to fool in your own research.
Chris Chambers (attributing Richard Feynman)
Credibility does not come from publishers. They're a proxy for it. Only they're sort of a heuristic that we use.
Chris Chambers
Eliminating outcome bias in science is so much more important than deciding whether to use a Bayesian hypothesis test or a frequentist t-test.
Chris Chambers
The traditional journal-based publishing system is stuck in the 18th century of a permanent fixed record, which is there and that's that. It's just not the way science works.
Chris Chambers
If it does, it's really nothing like it's proposed at the moment in terms of theory. It's a warning, I think, to the psychological field that we need to work much harder at controlling our own biases and avoiding fooling ourselves.
Chris Chambers
1 Protocols
Registered Reports Publication Process
Chris Chambers- Authors write and propose a full theoretical background and detailed methodology (protocol).
- The protocol is submitted for peer review *before* the study is conducted.
- The journal evaluates the quality of the method, importance of the question, ethical standards, and other elements contributing to scientific quality, *without knowing the results*.
- Based on this evaluation, the journal issues an 'in-principle acceptance' (IPA) if the protocol is approved.
- Authors then conduct the research, knowing that the outcomes will not determine whether or not it gets published.
- Upon completion, authors submit the finished study.
- The same reviewers and journal assess whether the protocol was followed and if the conclusions are based upon the evidence.
- If the protocol was followed and conclusions are evidence-based, the study is published.