SleepLM points to a new direction for sleep AI🚀. Read all about it!
➡️Paper: arxiv.org/abs/2602.23605
Great work led by my students @ZongzheX2001, @ZitaoShuai, Eideen, and amazing collaborators @AysolaRavi and Rajesh!
More to come🌙
#AI #sleep #sensor #health #multimodal #LLMs
Finally, we wanted this to connect to real clinical workflows. 🏥
SleepLM can combine its predictions across an entire night and produce useful full-night measures, while staying stable over long sequences. This matters as real sleep analysis is about understanding the whole night in a reliable way.
We also wanted the model to be more controllable. 🎛️
Instead of always generating one broad description, SleepLM can focus on a specific part of the physiology when asked. For example, it can emphasize 🧠brain activity, 🫁breathing, ❤️heart-related signals, or 💪body movement.
SleepLM also learns when something happens, not just whether it happened. ⏱️
Our results show that the model is sensitive to timing. The strongest match appears when the text and the signal line up at the correct moment, and that match weakens as the alignment moves away.
SleepLM learns a strong link between language and physiology. 🔄
When we ask it to match text to signals, or signals to text, it performs much better than general-purpose baselines. It not only reads sleep signals well — but also learns a shared space where signal and language line up closely.
One clear takeaway: general LLMs are not enough. 📊
Even strong LLMs 🤖 are not built for dense physiology. They often work with summaries, but struggle when the task depends on subtle waveform structure.
🛌 SleepLM is designed for that setting, and it shows clear gains on zero-shot sleep tasks.
At the core is ReCoCa 🏗️, our unified training framework.
It combines three signals in one objective:
🔗 contrastive alignment
✍️ caption generation
♻️ signal reconstruction
The result is a representation that stays both language-aware and signal-grounded.
Traditional sleep scoring compresses rich signals into a small set of labels. 🧩
We built a multilevel strategy to turn sleep into layered text descriptions. This gives a much richer view of sleep, enabling us to curate the first sleep-language dataset:
🗂️100K+ hours of data from >10,000 people! 🚀
🌙 What if your sleep signals could speak?
Introducing SleepLM — sleep-language foundation models that turns raw sleep into something we can describe, query, and localize with language. 🗣️
🌐Website: yang-ai-lab.github.io/SleepLM
🕵️Code: github.com/yang-ai-lab/...
🤗Models: hf.co/yang-ai-lab/...
🧵👇
📢 My lab at UCLA is hiring PhD students and postdocs!
Please apply to UCLA CS or CompMed and mention my name if you are interested in foundation models and (Gen)AI for health / medicine / science.
More info: cs.ucla.edu/~yuzhe
Read all about it!
➡️Paper: arxiv.org/abs/2506.09108
Huge team effort! Kudos to my intern Evelyn, amazing team @kmr_ayush, @aametwally1, @Orson_Xu, @timalthoff, @pushmeet, @cecim, @xliucs, @danmcduff, and other amazing co-authors!
#AI #wearable #sensor #health #multimodal
(8/8)
Beyond its discriminative power, SensorLM showcases compelling generative capabilities. It can produce hierarchical and realistic captions from input wearable data only, offering more coherent & correct descriptions compared to LLMs like Gemini 2.0 Flash. ✍️✨
(7/8)
SensorLM also demonstrates intriguing capabilities, including interesting scaling behavior over data size, model size, and compute. 📈💡
(6/8)
Experiments across real-world tasks in human activity analysis 🏃♀️ & healthcare ⚕️ showcase its superior performance over SOTA models in:
✨ Zero-shot recognition
✨ Few-shot learning
✨ Cross-modal retrieval
(5/8)
SensorLM extends prominent multimodal pretraining architectures (e.g., contrastive, generative) unifying their principles for sensor data. It extends prior approaches, recovering them as specific configurations within a single architecture. 🏗️🔗
(4/8)
This enabled us to curate the largest sensor-language dataset to date: over 59.7 million hours of data from >103,000 people. That's orders of magnitude larger than prior studies! 🚀💾
(3/8)
Despite its pervasiveness, aligning & interpreting sensor data with language remains challenging 📈 due to the lack of richly annotated sensor-text descriptions. 🚫
Our solution? A hierarchical pipeline captures statistical📊, structural🏗️, and semantic🧠 sensor info.
(2/8)
🚨 Let your wearable data "speak" for themselves! ⌚️🗣️
Introducing *SensorLM*, a family of sensor-language foundation models, trained on ~60 million hours of data from >103K people, enabling robust wearable sensor data understanding with natural language. 🧵
🩻⚖️ AI underdiagnoses Black female patients
A new study found that expert-level vision-language models for chest X-rays systematically underdiagnose marginalised groups – especially Black women – more than radiologists.
🔗 doi.org/10.1126/sciadv.adq0305
#SciComm #AI #HealthEquity 🧪
Science News provides a great cover of our paper: www.science.org/content/arti...
Started in 2023, delayed but finally out! Huge congrats & thanks to amazing collaborators: Yujia, @xliucs, @Avanti0609, @Mastrodicasa_MD, Vivi, @ejaywang, @sahani_dushyant, Shwetak 🎉
(6/6)
#AI #health #fairness
Why the gap? These foundation models in medical imaging encode demographic info (age, sex, race) from X-rays—more than humans do! Fascinating, but a challenge for fair healthcare ⚖️.
(5/)
This fairness disparity also holds for unseen pathologies during training, as well as for differential diagnoses across 50+ pathologies. ⚕️
(4/)
While expert-level VLMs can achieve _overall_ diagnosis accuracy on par with clinicians, they show significant underdiagnosis disparity over (intersectional) subpopulations vs. Radiologists 🚨
(3/)
We tested top vision-language models like CheXzero on 5 global datasets 🌍. Result? They consistently show disparities in diagnosis based on race, sex, and age—esp. across marginalized groups—compared to certified radiologists 📷
(2/)
Do foundation models in medical imaging see everyone fairly?🤔
Excited to share our new Science Advances paper uncovering & auditing demographic biases of expert-level VLMs, and comparing to board-certified radiologists🧑⚕️
📄science.org/doi/10.1126/sciadv.adq0305
💻github.com/YyzHarry/vlm-fairness
(1/)
How multimodal A.I. of real time smartwatch data can automatically detect a person's loss of pulse—sudden cardiac death—and notify emergency services
www.nature.com/articles/s41...
@jakesunshine.bsky.social @nature.com
Just published in Nature Biomedical Engineering! Working with the incredible PhD student Wei Qiu and our brilliant collaborator Kamila Naxerova at Harvard was a great pleasure. Our deep profiling framework enables us to view 18 human cancers through the lens of AI!
www.nature.com/articles/s41...
A neurologist with 2 APOE4 copies tells us about his experience with #Alzheimers disease
washingtonpost.com/wellness/202...
Seven years ago, Scott Lundberg, presented our SHAP framework at the NeurIPS 2017 conference. Since then, SHAP has become one of the most widely used feature attribution methods, with our paper receiving approximately 30,000 citations. It's wonderful that SHAP's birthday aligns perfectly with mine!😊
I will be at #NeurIPS and #ML4H all next week — let me know if you would like to catch up in person!
📢 I am also recruiting PhD students! Drop me an email if you're attending NeurIPS and would like to chat or learn more 😀
