Worth a look if you're running online studies. Happy to be a part of this, great work led by Grace. Check out Oleg's thread; feedback welcome!
08.03.2026 00:18
👍 3
🔁 2
💬 0
📌 0
Worth a look if you're running online studies. Happy to be a part of this, great work led by Grace. Check out Oleg's thread; feedback welcome!
Agreed, we did not test proctoring.
Nope, not dead at all, it's actually the primary method for collecting custom survey data in social science. My personal opinion is that online data collection is an incredibly valuable resource that we need to invest in saving from obliteration by AI, your view may differ.
Ooops. Meant to say "AI checks can also mistake non-compliant human respondents for AI."
Feedback, questions are welcome!
Some caveats. Off-the-shelf AI agents can pass survey checks with human assistance and AI agents can be purpose-built to pass checks without human assistance. Detecting AI agents is a moving target: ongoing independent testing of survey platforms is needed.
Using AI checks to screen out respondents is bad, helping AI learn to evade checks. Better to just collect data and pre-register exclusions (and move to a better platform when the fail rate is too high).
espondents for AI. Survey participants ignoring instructions and copy-pasting text into open-ended responses has been a problem for a long time:
marginallysignificant.com/2019/03/18/u...
Some thoughts. Relying on just one AI check is not a good idea – AI agents differ in their capabilities. Use multiple tests, varied regularly. Not checking against a human baseline may lead to over-estimating AI agents: AI and humans are bad at some of the same things.
This can matter for survey results. Platforms with more AI failures estimated less disapproval of using AI to complete surveys. Excluding potential AI agents reduced those differences (vs. humans at Mindworks @CDR_Booth).
The results differ substantially across platforms. @joinprolific.bsky.social and @cloudresearch.bsky.social ’s Connect panel have relatively low failure rates, while Mturk (even via @cloudresearch.bsky.social) has a high failure rate.
We use five AI checks, validating that common AI agents fail our checks but in-person human respondents do not. We then collect data on 7 online platforms.
Recent work by @seanjwestwood.bsky.social in PNAS has raised a red flag about AI agents being able to complete online surveys.
www.pnas.org/doi/10.1073/...
www.pnas.org/doi/10.1073/...
When you collect data online, are the results from humans or AI? In a project led by Booth PhD student Grace Zhang, we estimate the prevalence of AI agents on commonly used survey platforms:
osf.io/preprints/ps...
🧵
“I've said to a couple of my colleagues, like, man, what would it have looked like in 2024 when we're in the thick of the campaigns, if we stopped being so defensive on immigration and we went on the offensive; had we all collectively rang the alarm when we saw it start.”
From this follow simple recommendations: as a default, meta-scientific studies of published research artefacts need to include 1) a full, identifiable list of included studies, 2) the full coding instrument and decision rules, and 3) the individual ratings together with a codebook.
Our instinct to seek confirmation leads to ‘narrow’ internet search behaviour.
Chatbots, trained to be helpful, tend to go along with this, but could be trained to help us update our beliefs.
@olegurminsky.bsky.social researched this and explains what he found:
buff.ly/0eafZ78
"It's a general issue in lots of technology that the technology is designed to try to be helpful to us and around our needs, but often it takes a simple view of what those needs are and leaves out some of the needs." - @olegurminsky.bsky.social
www.chicagobooth.edu/review/podca...
This is one of the most beautiful things I have witnessed, the craft here is impeccable.
"ChatGPT, Gemini, DeepSeek, and Grok are serving users propaganda from Russian-backed media when asked about the invasion of Ukraine, new research finds." www.wired.com/story/chatbo...
If I were running the DNC, there would be Democratic Party sponsored food relief banks across the country right now that specifically welcomed anyone regardless of party affiliation.
Along with the food, there could be voter registration and links to runforsomething.net
Today's HotFresh recommended paper is:
Pleskac, T. J., Kyung, E. J., Chapman, G. B., & Urminsky, O. (2025). Blinded versus Unblinded Review: A Field Study on the Equity of Peer-Review Processes. Management Science. doi.org/10.1287/mnsc...
The South Korean Ministry of Defense has awarded medals of merit to 11 officers for disobeying direct orders of superiors during the martial law fiasco, orders that they deemed to be contrary to the constitution and endangerment to democracy.
www.chosun.com/english/nati...
We must stand resolutely against political assassination and political violence of all kinds, and just as resolutely against everyone who exploits acts of violence as the pretext or excuse for political repression of political opponents.
Italy is doing something interesting with it's admission to medical school. Anybody with a high school diploma can enrol for the first semester (actually two months of intensive all day coursework). They take three course: Chemistry, Biology, and Physics. Mostly taught remotely.
Sen. Patty Murray (D-Wash.), top Dem on the Senate health committee, on this evening's chaos at CDC:
“We cannot let RFK Jr. burn what’s left of the CDC and our other critical health agencies to the ground—he must be fired."
🧠📝 New research from PBS Professor Tim Pleskac explores the pros and cons of author anonymity in peer review.
🔗 Learn more at the link in our bio.
#psychology #iub #iubloomington #research
Ruby Sales, who Daniels saved, went on to get degrees at Tuskegee Institute, Manhattanville College, and Princeton University. She's still alive
Daniels' murderer, who killed Daniels and wounded another activist (a Catholic priest), was never convicted and died peacefully in his late 80s
