Most benchmark papers are boring but this is a masterpiece. Authors present a comprehensive breakdown of different inference strategies on variant effect prediction (VEP) in viruses. There were two surprising results that IMO deserve more attention🧵
This line graph illustrates the percentage change in agency staff levels from the previous year for nine major U.S. federal scientific and health organizations between the fiscal years 2016 and 2025. The agencies tracked include the CDC, Department of Energy, EPA, FDA, NASA, NIH, NIST, NOAA, and NSF. For the majority of the timeline between 2016 and 2023, the agencies show relatively stable fluctuations, generally staying within a range of +5% to -5% change per year. However, there is a dramatic and uniform plummet starting in the 2024–25 period. Every agency depicted shows a sharp downward trajectory, with staffing losses ranging from approximately -15% to over -25%. The Environmental Protection Agency (EPA) shows the most significant decline, dropping to roughly -26%, while the National Institute of Standards and Technology (NIST) shows the least severe but still substantial drop at approximately -15%.
This is the most astonishing graph of what the Trump regime has done to US science. They have destroyed the federal science workforce across the board. The negative impacts on Americans will be felt for generations, and the US might never be the same again.
www.nature.com/immersive/d4...
This was fun work and a remarkable effort across the computational and wet-lab teams!
Strategies for in-silico filtering and ranking of antibody designs have been under-discussed in the literature, e.g. in most technical reports on antibody design that I've seen. Let's talk about these here! [1/n]
This complements our prior evaluation of deep mutational scans showcasing PLM underperformance, even when there is especially low sequence diversity.
There is still much room for improvement for viral modeling!
Computational variant effect predictors effectively predict viral evolution – even more so than deep mutational scans (DMS).
Yet, PLM or hybrid approaches (even with data-leakage inflating performance) provide little benefit over the best alignment-based model (EVE).
🦠📈 We ask whether models can predict mutations that actually rose in frequency in nature—providing the clearest available evaluation of their utility for pandemic preparedness.
We now evaluate protein language models (PLMs), alignment-based models, and hybrids across 30 clades from 4 well-sequenced viruses (SARS-CoV-2, HIV, H3N2, and H1N1) - the first large scale evaluation of these models on real viral evolution forecasting.
We’ve updated the EVEREST benchmark to include real-world viral evolution! www.biorxiv.org/content/10.1...
Co-led by Noor Youssef and me, along with co-authors Navami Jain, Aarushi Mehrotra, Sarrah Leung, Abigail Jackson, @deboramarks.bsky.social, and with @cepi.net @futurehousesf.bsky.social!
The fundamental problem with GISAID is this - data on the platform are neither open, nor FAIR.
Such data - key for combating infectious diseases - are, by a large margin, paid for by taxpayers and hence must be openly available to all.
With GISAID, they are not.
www.science.org/content/arti...
New preprint! We measured temperature- and pH-induced aggregation for over 18,000 natural and de novo designed protein domains!
End-to-end protein design in the browser through evedesign. Generate and interactively explore designs in 2D/3D and export them as codon-optimized DNA. The underlying open source framework (released soon) is build to easily add new methods, more on that soon.
🌐 evedesign.bio
Thrilled to announce our new preprint, “Protein Hunter: Exploiting Structure Hallucination within Diffusion for Protein Design,” in collaboration with @Griffin, @GBhardwaj8 and @sokrypton.org
🧬Code and notebooks will be released by the end of this week.
🎧Golden- Kpop Demon Hunters
Large AI models are reported to achieve high accuracy (AUROC) predicting pathogenic variants across the genome.
A preprint reports that the predictions are based on splice variants. Using only this info (no sequences, no AI) achieves AUROC=0.944 across noncoding variants.
1/2
In new study led by @ckikawa.bsky.social, we provide near real-time data on human neutralizing antibody landscape to influenza by measuring ~26,000 titers to >100 recent viral strains
Data can inform vaccine selection & evolutionary/epidemiological modeling
www.biorxiv.org/content/10.1...
There’s been a bunch of new approaches looking at deep viral evolutionary history. We’ve put together a mini review highlighting some recent advancements in structural phylogenetics and time-dependent rate models and what they could do for the field 🦠
🔗 journals.asm.org/doi/full/10....
Excited to share our new preprint co-led by @jnoms.bsky.social!
Here we reveal an exceptional diversity of viral 2H phosphodiesterases (PDEs) that enable immune evasion by selectively degrading oligonucleotide-based messengers. This 2H PDE fold has evolved striking substrate breath & specificity.
Thanks!
🦠The future of pathogen forecasting needs rigorous benchmarks and domain-specific modeling, not only bigger PLMs. EVEREST is a step in that direction.
🔗Paper: biorxiv.org/content/10.1...
💻Code + data: github.com/debbiemarksl...
12/12
🙏Amazing collaboration co-led with Noor Youssef
and Navami Jain, @deboramarks.bsky.social, and our funders @cepi.net!
11/12
This matters for:
⚠️ Future-proof vaccine and therapeutics design
⚠️ Monitoring of high-pandemic risk viruses
⚠️ Dual-use biosecurity risk assessment
Without reliable models, we risk underestimating viral evolution—and overestimating our ability to counter it.
10/12
EVEREST highlights:
✅ Where models fail—and why
✅ Which viruses are least/most predictable
✅ How to estimate per-protein, model-specific reliability
✅ Concrete steps to improve ML for viral mutation prediction
9/12
🌍Current models fail to reliably predict mutations in more than half of the high-priority viruses identified by the WHO.
8/12
💪Is bigger always better? Maybe not for other taxa but for viruses - yes! For viruses, models continue to improve with increased numbers of parameters.
7/12
🤏Why? Viruses are severely underrepresented in training datasets (<1%) and are further downsampled after common clustering approaches.
6/12
📉Despite the hype, protein language models trained across the “protein universe” are outperformed by even the simplest, site-independent alignment-based model.
5/12
💭Imagine: It’s Day 0 of an outbreak and there’s little experiment data. Computational mutational effect predictions could provide valuable information…if we could trust them. Can we?
EVEREST doesn’t just assess performance. It also quantifies reliability for new viruses.
4/12
🚀To find out, we built EVEREST: Evolutionary Variant Effect prediction with Reliability ESTimation.
We benchmark models across 45 viral deep mutational scanning datasets spanning >340,000 mutations.
3/12
🦠 Protein language models (PLMs) have shown impressive performance in predicting mutation effects. But... viruses are a different beast.
They evolve fast, cross species, and are under pressure from host immunity. Do PLMs still work here?
2/12
🚨New paper 🚨
Can protein language models help us fight viral outbreaks? Not yet. Here’s why 🧵👇
1/12
Stoked to finally have a preprint out for Phold, our tool that uses protein structural information to enhance phage genome annotation #phagesky 1/n
www.biorxiv.org/content/10.1...
