Multimodal Instruction-Tuning Reorganizes Geometric Encoding of Identity Prompts in LLMs

Jul 14, 2026

A preprint study analyzes four open-weight transformer language models and finds that multimodal instruction-tuning causes a qualitative shift in how identity-specifying system prompts are encoded in hidden-state trajectories. Specifically, the encoding changes from being direction-based in the base model to magnitude-based after multimodal RLHF instruction-tuning, a reorganization not observed in RL distillation or SFT regimes. The study uses geometric metrics and permutation tests to support these findings.

Why it matters: This work uncovers a previously unrecognized effect of multimodal instruction-tuning on internal model representations, which could inform future approaches to model interpretability and control.

Full story at: arXiv Machine Learning ↗