Daniel Acuna

Great piece by jeffreybrainard.bsky.social with comments from others www.science.org/content/arti...

AI tool labels more than 1000 journals for ‘questionable,’ possibly shady practices New algorithm could help scientists avoid publishing in shady titles

Can AI help identify high-volume, low-quality, “questionable” scientific journals (which some, controversially, call #predatoryjournals )? Authors of this new study emphasize aiding not replacing human evaluators of these journals. @science.org www.science.org/content/arti...

Estimating the predictability of questionable open-access journals AI screening of journals identifies over a thousand questionable journals, helping experts review where it is needed most.

Take a look at the full paper:
- science.org/doi/10.1126/...

I started this project 5 years ago. Paper with my former students Han Zhuang and Lizhen Liang

Disclaimer: I am the founder of
ReviewerZero AI (www.reviewerzero.ai),
where we help with research integrity issues.

A few takeaways:
- the AI of course still makes mistakes, but we believe those mistakes are worth it
- predictions should be part of a triage system with experts. we make our predictions "interpretable", aligning with DOAJ guidelines
- But AI for integrity is here to stay

graph showing large grow of publications in questionable journals

Graph showing growth in citations

Graph showing distribution of countries in our prediction

Applied to journals that haven't been vetted before , we predict more than 1000 new potentially questionable journals.
They have collectively published over 500K articles, cited millions of times, and acknowledged major funders in US, China, and Japan

Table that shows good agreement between experts, DOAJ guidelines, and algorithmic predictions

We tried several methods, including simple regression, random forest and deep learning.
- Bibliometric features alone: PRC AUC ≈0.64
- Combined model: PRC AUC ≈0.79

Agreement with DOAJ guidelines and expert reviewers was strong, though false positives remain.

Overview of the system, which combines multiple kinds of signals including the website, visual features, and bibliometric features

We trained models on ~15,000 journals labeled by DOAJ (12,869 legitimate vs 2,536 removed). Features included:
- Website content (editorial boards, policies)
- Website design (HTML structure, screenshots)
- Bibliometrics (citations, author metrics)

Article front page: https://www.science.org/doi/10.1126/sciadv.adt2792

🚨New paper🚨
Open access has expanded science’s reach but also fueled the rise of "questionable" journals. Manual vetting can’t keep pace with thousands of titles and bad actors who adapt quickly. Wrong incentives too strong.
In a new Science Advances paper we ask: can AI help?

that makes sense! 🧠

Who are these hundreds of people who started following me recently? I mean, hi 👋, but how did you find me without any advertisement from my part? 😅 This place is starting to look great.

NotebookLM podcast version of the paper. Pretty good! notebooklm.google.com/notebook/d75... (need a Google account)

Incorporating costs and benefits to the evaluation of uncertain research results: applications to cancer research funding Abstract. Correctness is a key aspiration of the scientific process, yet recent studies suggest that many high-profile findings may be difficult to replicate or require considerable evidence for verif...

Excited to share a new article with Han Zhuang in Quantitative Studies of Science! "Incorporating costs and benefits to the evaluation of uncertain research results," we propose a unified framework for deciding when to continue research, even if it might be wrong 🤯. direct.mit.edu/qss/article/...

Tenure/tenure-track faculty positions in NLP in CS. Looking for people in science of science! Super interdisciplinary dept, great place to work and grow, amazing location. Learn more jobs.colorado.edu/jobs/JobDeta... Feel free to reach out and ask about the position and details.