Many multimodal benchmarks provide fixed vectors already extracted from text, audio, and video. A tempting route is to project each vector to a language model's hidden width and feed the result as a token. The tensors fit. But the backbone learned to process its own embedding distribution, not arbitrary coordinates from an external encoder.不少多模态基准已经提供从文本、音频和视频中提取好的固定向量。一个自然做法是把每个向量投影到语言模型的隐藏维度,再当作 token 输入。张量形状对了,但骨干学会处理的是自己的 embedding 分布,而不是外部编码器给出的任意坐标。
The problem is not one-dimensional. Is the input scale compatible with native embeddings? Can a single token carry enough evidence from a modality? Does the backbone route the unfamiliar code into useful task states? What should it see when a modality is missing? TEAL treats these as four separate interface questions rather than assuming that a learned projection answers all of them.问题不止一个:输入尺度是否接近原生 embedding?每个模态只有一个 token 时,证据带宽够不够?骨干能否把陌生编码路由成有用的任务状态?某个模态缺失时又该输入什么?TEAL 把它们作为四个独立的接口问题,而不是假设学到一个投影层就能全部解决。
Measure the evidence before the interface先测证据,再测接口
Before judging a token reader, we need to know what the original features can support. Models that read the coordinates directly establish a reference for the available supervised signal. No-backbone and randomly initialized controls then ask how much of a token reader's performance comes from the writer and task head, and how much requires pretrained computation. Missing-modality and shuffled-feature tests check whether the reader uses the evidence we think it uses.评价 token 读取器之前,得先知道原始特征本身能支持什么。直接读取特征坐标的模型提供监督信息的参考;移除骨干和随机初始化对照,再区分投影器、任务头与预训练计算各自的贡献。模态缺失和特征打乱实验,则检验读取器是否真的使用了预期的证据。
The data protocol matters as much as the architecture. Dialog-level splits keep utterances from the same conversation together, and normalization fitted only on training data keeps evaluation information out of the interface.数据协议与模型结构同样重要:按对话划分数据,避免同一段对话的语句跨训练与验证集合;标准化参数只在训练集上拟合,避免评估信息进入接口。
Build an interface the model can read构建真正可读的接口
TEAL writes each modality into multiple typed evidence slots. Modality and slot embeddings identify where the evidence came from; native-scale calibration brings the new tokens closer to the backbone's usual activation scale; learned missing tokens give absent evidence its own symbol. A frozen reader and a LoRA-adapted reader then test how much the interface alone can do and where routing needs to change.TEAL 把每种模态写入多个带类型的证据槽。模态和槽位编码标明证据来源,原生尺度校准让新 token 接近骨干熟悉的激活范围,可学习的缺失标记则为“没有证据”提供独立符号。随后用冻结读取器与 LoRA 适配读取器,检验接口本身的作用以及路由需要改变的地方。
Here the Qwen backbone reads pre-extracted features; it is not processing raw audio or video end to end. That scope makes the experiment more informative. It isolates a question that appears well beyond affective computing: when we attach a new source of information to a pretrained model, what proves the model can actually use it?在这个项目里,Qwen 骨干读取的是预提取特征,并不直接端到端处理原始音视频。这个范围让实验更清晰:它隔离出一个超越情感计算的问题——给预训练模型接上新的信息来源后,什么证据能证明模型真的用上了它?