Tom Stafford

photo from groundlevel of a small pong. Many frogs can be seen in the water

Morning

#PondLife #Sheffield #Frogs

Pair with Kevin Munger:

https://kevinmunger.substack.com/p/things-will-have-to-change

"So, yes, the current situation with journals publishing static pdfs is indefensibly antiquated. But the current academic polyarchy is actually quite robust; the sluggishness, the institutional conservativism […]

No AI Was Used in the Writing of This Review (Unfortunately) Why the academic obsession with human-only peer review protects bias, inconsistency, and anonymous reviewer soapboxes

Matt Grawitch sketches what a peer review system which embraces AI would look like

https://mattgrawitch.substack.com/p/no-ai-was-used-in-the-writing-of

"Why the academic obsession with human-only peer review protects bias, inconsistency, and anonymous reviewer soapboxes"

Can AI Replace Social Science Researchers? No. No it can't. Come on, now.

Here's @davekarpf lucid on AI and research.

AI kills the paper and good riddance

https://davekarpf.beehiiv.com/p/can-ai-replace-social-science-researchers

Getting a complex revise and resubmit is like getting assigned to a murder trial. You call work: "Sorry, I'm out for the rest of the year"

Reviewing should be like jury service. You have a chance of being randomly selected, then the funder or journal calls up your institution and they have to give you two weeks off, or whatever, and you bash out as many reviews as you can in that time

#AcademicChatter

A large-scale randomized study of large language model feedback in peer review: www.nature.com/articles/s42...

RCT w/ 20k reviews shows that 27% of reviewers who received feedback from AI updated their reviews, and blinded eval confirmed revised feedback was more informative.

Cartoon:: a chaotic mess

Here's the plan

(Michael Leunig, https://www.leunig.com.au/)

@drjennings.bsky.social alternatively

@drjennings.bsky.social

A table of 18 types of "Disagreement Strategy," with a tag for each strategy in square brackets next to its name, and a text description of each strategy. The strategies, which are arranged in two columns with green, yellow, orange, and red color bars, include Complex Counter Argument [CCA], Dismantle [DIS], Softened Counter Argument [SCA], Regular Counter Argument [RCA], Critical Question [CQ], Invitation For Cooperation [IFC], Playing On Emotions [POE], Joking [JOK], Reasoned Direct Denial [RDD], Proposing Alternative [PRA], Deafening Silence [DES], Agree to Disagree [ATD], Breakdown of Dialogicity [BOD], Unreasoned Direct Denial [UDD], Ordering [ORD], Irrelevancy Claims [IRC], Ironic Echoing [IRE], and Blatant or Aggressive Denial [BAD].

The Hierarchical Taxonomy of Disagreement Strategies (HiTODS)

https://doi.org/10.1016/j.edurev.2026.100769

What's a multiverse good for anyway?

https://osf.io/preprints/psyarxiv/37g29_v1

"We discuss various ways in which a multiverse may be employed – as a tool for reflection and critique, as a persuasive tool, as a serious inferential tool ...it fails as a persuasive tool when researchers disagree […]

Hyperactive Minority Alter the Stability of Community Notes

https://arxiv.org/abs/2602.08970

"[We] conduct counterfactual simulations that modify the display status of notes by varying the pool of raters. Our results reveal that the system is structurally unstable: the emergence and visibility […]

This image is a statistical box plot that visualizes how different factors contribute to the variance of treatment t-statistics across various outcomes. Title: Distribution of $\eta^2$ components across outcomes (t-statistics). Y-Axis: Represents $\eta^2$ (eta-squared), ranging from 0.00 to 1.00, which measures the proportion of variance explained. X-Axis: Categorizes the variance into four components: Outlier, Missing, Transform, and Residual. This order is also the largest to smallest ordering

Results from Randomized Controlled Trials are Highly Sensitive to Data Preprocessing Decisions: A Multiverse Analysis of 97 Outcomes

https://osf.io/preprints/metaarxiv/kbgc2_v2

This is nice: take open data from RCTs, and show how defensible preprocessing […]

[Original post on mastodon.online]

Of note

https://minutes.substack.com/p/rented-virtue

"The work that lasts from this era will be no different. The companies that endure, the technologies that serve rather than consume, the institutions that hold their shape across generations when the founders are dead and the capital is […]

Sound the nerd victory klaxon!

Tax nerd throws life savings at prediction markets, after realising he could take the opposite side of bets by Musk fans. DOGE, inevitability, fails to decrease year on year federal spending. He wins big […]

Following up for this, I've written a longer piece for the new @RoRInstitute #Metascience substack

https://researchonresearchinstitute.substack.com/p/the-point-of-no-return

My minimal claim is that, contra Schweiger, it is very hard to definitely claim any scheme a waste of time and money […]

Specifically right, generally wrong Have I stumbled upon a dark pattern for generating engagement?

New newsletter!

https://tomstafford.substack.com/p/specifically-right-generally-wrong

on reactions I had to the most controversial piece I've written for a while

Photo of a book White Moss by Anna Nerkagi A silhouetted figure rides a sleigh in the arctic tundra

Recommend

@drjennings.bsky.social sure to be an attractive prospect for billionaire philanthropic dollar "For too long research has been held back by the inefficiency of having to conduct ethical studies..something something disruption ...something something red tape"

I've just an idea for a crazily unethical experiment:

- pick ~20 minor celebrities
- randomise into two halves
- half you leave alone
- half, you buy a bot army which responds with enthusiastic endorsement whenever they say anything edgy, conspiratorial or political on social media
- come back […]

Following up for this, I've written a longer piece for the new @RoRInstitute #Metascience substack

https://researchonresearchinstitute.substack.com/p/the-point-of-no-return

My minimal claim is that, contra Schweiger, it is very hard to definitely claim any scheme a waste of time and money […]

Life is pain, Highness.

#PrincessBride

@jamesjefferies organising often a thankless task. But you make the world go round

LLMs should be a private cognitive tool, like a calculator, which is not currently possible under our existing model of corporate AI. Crucially, they do not think and have no agency, again in the same way as a calculator

Some useful thinking out loud from @philipncohen on developing moderation rules to cope with AI generated content submitted to SocArxiv

https://socopen.org/2026/02/22/where-should-socarxiv-draw-the-ai-line/

Go Meta! News and Views from RoRI | Research on Research Institute | Substack By the Research on Research Institute (RoRI). We publish timely essays, reflections and signals from across metascience, research policy and reform. Expect original thinking, reactions to live debates, and practical insights from people studying research. Click to read Go Meta! News and Views from RoRI, by Research on Research Institute, a Substack publication with thousands of subscribers.

New substack from @RoRInstitute

https://researchonresearchinstitute.substack.com

#MetaScience

A heatmap with various model names for each row (Gemini Pro 3, Gemini Flash 3 etc etc on to Claude Opus 4.6) and most of the rows are coloured dark or light green (which is good)

Same Prompt, Different Outcomes: Evaluating the Reproducibility of Data Analysis by LLMs

https://arxiv.org/abs/2602.14349

Eyeballing figure 1 the ability and consistency of LLMs for data analysis looks ... pretty good. Better than humans?

Metascience seminar 1: Improve our inferences from observational studies

https://events.humanitix.com/metascience-seminar-1-harrison-hansford

The event is free for all.

Date and time: March 12 at 11AM (GMT + 11; Sydney, Melbourne, Canberra time)

Speaker: Harrison Hansford

#MetaScience […]

Trials and tribulations of responsible people trying to uphold scientific standards - Research Integrity and Peer Review Scientific misconduct threatens patient safety, progress, and trust in medicine. On October 3, 2020, Frass and colleagues published a randomized, placebo-controlled, double-blind trial in The Oncologist (published by Wiley at the time) claiming that add-on homeopathy significantly prolonged survival in advanced non-small-cell lung cancer. Since homeopathy contradicts established scientific principles, doubts about the trial’s validity quickly emerged. Concerns were first published in October 2020, followed in 2021 by a detailed analysis alleging scientific misconduct. This prompted the Medical University of Vienna, the affiliation of the study's lead author, to request an investigation by the Austrian Agency for Research Integrity (OeAWI). After conducting an in-depth review, OeAWI concluded in September 2022 with a clear recommendation for retraction. However, The Oncologist issued only an ‘Expression of Concern’ at the time, despite five co-authors formally requesting the withdrawal of their authorship— a demand that remained unaddressed as of November 2025. Repeated inquiries to the journal and its publisher, Oxford University Press (OUP), yielded only vague assurances that the matter was “under review,” with multiple deadlines passing without resolution. Finally, by November 24, 2025, The Oncologist retracted the paper. However, the retraction notice fails to address the specific concerns raised about the study’s results and conclusions, nor does it provide a clear rationale for the retraction itself.Meanwhile, the paper has been cited more than 60 times (according to Google Scholar) and is widely circulated online as “proof” that homeopathy benefits cancer patients. This highlights the harmful consequences of delayed editorial action. According to COPE guidelines, misconduct must be dealt with swiftly and transparently. Our case reveals the opposite: incomplete corrections, prolonged inaction, and even the defense of implausible claims. Against the backdrop of increasing organized scientific fraud, this experience underscores the urgent responsibility of journals and publishers to protect the scientific record and prevent harm to patients.

"Since homeopathy contradicts established scientific principles, doubts about the trial’s validity quickly emerged."

I feel like the homeopathy-reproducibility beat is a bit like shooting fish in a barrel https://link.springer.com/article/10.1186/s41073-026-00191-5