Time to go to Vienna again! Iβll present one mechinterp work on 28th 17:00-18:30 Hall X4 X5
We have another work for multilingual instruction-following benchmark on 28th 14:00 at 1.15-16. Very honored to be involved in this oral work! Feel free to reach out & chat for mechinterp!
This work was collaborated with Chen Chen, Yuchen Hu, Yanjun Gao, Xi Wang and Prof. Emine Yilmaz.
Our paper: huggingface.co/papers/2505....
Our code: github.com/ruizheliUOA/...
This finding confirms the contribution of MLP located using ARC-JSD above, and it is reasonable because Chinese is one of main language resources used in Qwen2 pre- and post-training.
In our case study for located MLP layers in Qwen2 models, we identify several correct decoded tokens are gradually transferred from their Chinese format to the English version, such as δΈεͺ(A), ζ₯ζ(has) and ηΏ θ(wings) in the figure.
In addition, we move forward to locate relevant attention heads and MLP layers using JSD from mechinterp view. We found that JSD-based mechinterp can identify context attribution-related attention heads and MLPs, which are mainly distributed around intermediate or higher layers.
We evaluate our ARC-JSD on TyDi QA, Hotpot QA and MuSiQue datasets using Qwen2-1.5B/7B-IT and Gemma2-2B/9B-IT, which can achieve higher attribution acc than baseline.
π€Is it possible to accurately and effectively attribute RAG response to relevant context without finetuning or further training surrogate model?
π‘We propose an inference-time method called ARC-JSD using JSD for RAG context attribution, which only needs O(sent_num + 1)π
Very glad to share a good news! One main conference and one findings are accepted at #ACL2025! See you again at Vienna!
