I have been thinking about some of the consequences of closed vs open research. Closed research can slow down scientific progress and concentrate knowledge, which results in what I call “model archaeology”. I discuss this idea in my ICLR 2026 Blogpost.
Short thread 🧵and link👇
Announcing our latest paper: CommonLID
In collaboration with @commoncrawl.bsky.social @mlcommons.org @jhu.edu we built a LID benchmark on actual Common Crawl text covering 109 languages. Existing evaluations overestimate how well LangID works on web data.
arxiv.org/abs/2601.18026
Our paper on ✨populism✨ and how LLMs struggle to detect it in political debates 🗣️ has been accepted to EACL main conference.
Meet the lead author @kiddothe2b.bsky.social in Rabat to discuss PolSci🤝NLP
A photograph of sunny Copenhagen in the summer!
📢 I am hiring a highly-motivated Ph.D student at the University of Copenhagen to work on tokenization-free NLP.
Read our previous work in this topic: aclanthology.org/2025.emnlp-m...
aclanthology.org/2023.emnlp-m...
openreview.net/forum?id=FkS...
Apply by March 8: employment.ku.dk/phd/?show=1563
New EACL paper (with @mdlhx.bsky.social)! We tested if comparing perplexity of parallel data across languages is fair. Turns out: it depends. We show the choice of test set (even with consistent meaning) can flip conclusions about which language is easier to model.
Paper: arxiv.org/abs/2601.10580
Our paper has been accepted to EACL 2026!🎉 We systematically evaluate several vision-language (VLMs) and language-only models, measuring their alignment with brain responses to concept words. Our results show that vision-language models offer a promising tool to model human concept processing
Excited to share that 📏 Cetvel benchmark has been accepted to #EACL2026! 📏 Cetvel is designed to evaluate LLMs in Turkish. Please see the thread below for more details. #NLProc
İlker
İlker and friends
This week, it was our pleasure to welcome @ilkerkesen.bsky.social, who shared some of his and his group's latest findings with our group.
It shows how multimodal learning can help models transcend the boundaries of writing systems.
We enjoyed İlker's company and look forward to his future work!
This week I’m in Utrecht! I’m visiting Utrecht University to give an invited seminar talk on pixel language models, hosted by the NLP Research Group led by Albert Gatt. Note: Don’t be fooled by the image; the weather is cloudy as expected! #NLProc
This week, I'm attending #EurIPS here in Copenhagen, where I'll present our work on pretraining a multilingual pixel language model at the #ELLIS UnConference. Find me tomorrow at 4 pm in the poster session at stand no. 59 to learn more about multilingual pixel language models.
I'm forcing GPT-5.1 to translate some English text to some target language that I don't know. It decides to use its thinking feature, and then reasons about switching to the target language for the entire output, including explanations and conversational parts. Sigh.
This week at #EMNLP2025, I'll present our research on pretraining a multilingual pixel language model. Join the multilinguality session on Friday at 10:30 in Room A301 to learn more about pixel models and their benefits in multilingual settings. (Unfortunately I’ll be on Zoom)
When is a language hard to model? Previous research has suggested that morphological complexity both does and does not play a role, but it does so by relating the performance of language models to corpus statistics of words or subword tokens in isolation.
We used to develop Knet, but after the rise of HuggingFace, we stopped using it. Before HF, it was also painful to convert each model to Julia code and array, where the models were already available in PyTorch/Tensorflow. Though, I guess you can find BERT/GPT implementations in Knet, somewhere.
For more details about 📏Cetvel, please check our preprint.
📜Paper: arxiv.org/abs/2508.16431
💻Code: github.com/KUIS-AI/cetvel
📊Leaderboard: huggingface.co/spaces/KUIS-...
Furthermore, we assessed the informativeness of each task using Gini coefficients. We found that grammatical error correction, machine translation and extractive QA (about Turkish / Islam history) are the most informative tasks for evaluating LLMs in Turkish within 📏Cetvel.
and lastly (iii) Turkish-centric 8B parameter model Cere-Llama-3-8B outperforms even 70B parameter model Llama-3.3-70B on some Turkish-centric tasks such as grammatical error correction.
We tested widely used 33 open-weight LLMs covering different modely families up to 70B parameters. We find that (i) LLMs tailored for Turkish underperform compared against general-purpose LLMs, (ii) Llama 3 models dominate other LLMs within the same parameter scale, [...]
Second, 📏Cetvel also offers NLP tasks linguistically and culturally grounded in Turkish, such as proverb understanding, circumflex-based word sense disambugiation, and extractive QA centered on Turkish and Islam history.
First, 📏Cetvel goes beyond multiple-choice QA in contrast to existing Turkish benchmarks. It spans 23 tasks across 7 categories, including grammatical error correction, machine translation, summarization, and extractive QA.
So, why another Turkish benchmark? The answer is that existing benchmarks often fall short either in limited task diversity or lack of content culturally relevant to Turkish. Unlike existing benchmarks, 📏Cetvel addresses both shortcomings adequately.
📢New preprint: We introduce 📏Cetvel, a unified benchmark for evaluating language understanding, generation, and cultural capacity of LLMs in Turkish🇹🇷 #AI #LLM #NLProc
Joint work with Abrek Er, @gozdegulsahin.bsky.social, @aykuterdem.bsky.social from KUIS AI Center.
Excited to share that our paper "Multilingual Pretraining for Pixel Language Models" has been accepted to the #EMNLP2025 main conference! Please see the thread below and the paper itself for more details.
For more details about PIXEL-M4, please check our preprint.
Paper: arxiv.org/abs/2505.21265
Model: huggingface.co/Team-PIXEL/p...
Code: github.com/ilkerkesen/p...
In collaboration with Jonas F. Lotz, Ingo Ziegler, Phillip Rust and Desmond Elliott @delliott.bsky.social
Data-efficiency analysis on the Indic NER benchmark also demonstrated that PIXEL-M4 excels at cross-lingual transfer learning in low-resource settings.
Investigations on learned multilingual hidden representations reveal a strong semantic alignment between pretraining languages in the later layers, particularly for English-Ukrainian and English-Hindi pairs.
Word-level probing analyses illustrate that PIXEL-M4 captures better linguistic features even on languages and writing systems not seen during pretraining.
Downstream experiments on text classification, dependency parsing and named entity tasks recognition show that PIXEL-M4 outperforms its English-only-pretrained counterpart PIXEL-BIGRAMS on almost all non-Latin script languages.
Announcing our recent work “Multilingual Pretraining for Pixel Language Models”! We introduce PIXEL-M4, a pixel language model pretrained on four visually & linguistically diverse scripts: English, Hindi, Ukrainian & Simplified Chinese. #NLProc
Today we are releasing Kaleidoscope 🎉
A comprehensive multimodal & multilingual benchmark for VLMs! It contains real questions from exams in different languages.
🌍 20,911 questions and 18 languages
📚 14 subjects (STEM → Humanities)
📸 55% multimodal questions
