Research→Official→arXiv Computation and Language
A new study finds that LLM judges tend to be overly generous when evaluating open-ended responses without a reference answer, often over-crediting incorrect answers. The research shows that adding a reference answer to the prompt can flip correct/incorrect decisions by up to 85% in some cases, and these changes generally align more closely with human judgments.
Why it matters: This research highlights a critical calibration issue in using LLM judges for no-reference evaluation, which is increasingly common in practice.
Jul 15, 2026
Research→Official→arXiv Computation and Language
A new preprint introduces ThReadMed-QA, a dataset of 2,437 multi-turn patient-physician threads, to evaluate large language models (LLMs) on their ability to detect and correct medical misconceptions over several conversational turns. The study finds that even advanced models like GPT-5 and Claude-Haiku drop from about 85% accuracy on initial questions to around 50% after two follow-ups, with error propagation identified as a major cause of this decline. The results highlight that LLMs' ability to maintain accurate corrections diminishes over the course of a conversation.
Why it matters: This work exposes a significant safety risk in using LLMs for medical advice, as their corrections of misconceptions degrade in multi-turn interactions, potentially leading to unsafe outcomes.
Jul 15, 2026
Research→Official→arXiv Computation and Language
Researchers introduce LakeQuest, a human-validated benchmark containing 9,846 question-answer pairs across three domains: AI/ML metadata, retail banking, and biomedical drug information. The benchmark is designed to evaluate end-to-end retrieve-and-synthesize pipelines over heterogeneous, weakly structured data lakes. Baseline evaluations show that even with high-quality retrieval, current QA systems often fail at complex reasoning tasks such as relation chaining, policy grounding, and joint tabular question answering.
Why it matters: LakeQuest reveals critical weaknesses in current QA systems when applied to real-world, weakly structured data lakes, emphasizing the need for more robust discovery and synthesis methods.
Jul 15, 2026
Research→Official→arXiv Computation and Language
Researchers introduce a scalable pipeline for generating high-quality evaluation rubrics for financial report assessment without requiring human experts in the final evaluation loop. Using 104 real-world queries and 14,450 candidate rubrics, they demonstrate that LLM-based evaluation achieves 98.67% label-level agreement with human experts on jointly unanimous items. The process yields a final set of 2,600 consensus-derived gold rubrics, enabling differentiated rankings across 10 deep research systems.
Why it matters: This work enables large-scale, automated evaluation of AI-generated financial reports, reducing reliance on human experts and facilitating more efficient system comparison and improvement.
Jul 15, 2026
Research→Official→arXiv Computation and Language
Researchers improved the Huth et al. fMRI encoding pipeline, achieving an 11% METEOR gain by expanding voxel selection and updating the proposal model. However, when using a new method that maps fMRI signals to a frozen Llama 3.2 language model, they found that decoding performance was almost entirely due to the language model's prior, as blind controls with zeroed fMRI input produced nearly identical results. This indicates that apparent decoding success may not reflect true neural decoding.
Why it matters: The study demonstrates that high-capacity language models can mask failures in fMRI decoding, emphasizing the need for rigorous blind-control evaluations in neural decoding research.
Jul 15, 2026
Research→Official→arXiv Computation and Language
A new preprint introduces Ring-Zero, a training pipeline that successfully scales reinforcement learning with verifiable rewards (zero RL) to a 1-trillion-parameter model. The authors report that scaling leads to significant improvements in sample efficiency and performance, and that the model exhibits emergent advanced reasoning behaviors such as self-verification and parallel reasoning. On seven mathematical benchmarks, the model achieves competitive results and demonstrates advantages in producing structured and concise reasoning traces.
Why it matters: This work provides evidence that scaling zero RL to trillion-parameter models can yield emergent reasoning capabilities, offering new insights into the development of large-scale reasoning systems.
Jul 15, 2026
Research→Official→arXiv Computation and Language
EcoSpec is a cost-aware speculative decoding framework designed to improve inference efficiency in large-scale Mixture-of-Experts (MoE) language models. By incorporating predicted expert activation costs into draft token selection, EcoSpec encourages reuse of already-loaded experts during speculative decoding. Evaluations on models such as DeepSeek-V3.1 (671B), Qwen3-235B-A22B, and GPT-OSS-120B show that EcoSpec reduces expert activation footprints and achieves up to 1.62× speedup in end-to-end decoding, without modifying verification rules.
Why it matters: This work offers a practical advance for accelerating inference in large MoE language models by optimizing draft selection for expert reuse, directly improving decoding speed and efficiency.
Jul 15, 2026
Research→Official→arXiv Computation and Language
A preprint study used the Iowa Gambling Task to test whether induced emotions bias large language model (LLM) decision-making. The researchers found that, unlike humans, LLMs do not show significant average bias from induced emotion. However, anger specifically made LLMs less sensitive to penalties and reduced their early-stage exploration, leading to more rigid choices. The study also validated a new paradigm for studying affective influences on LLMs.
Why it matters: Understanding how emotional context affects LLM decision-making is important for ensuring the safety and reliability of autonomous AI agents in real-world applications.
Jul 15, 2026
Research→Official→arXiv Computation and Language
A hybrid trading agent, Fin-Analyst, combining eight LLM specialists with rule-based signals, achieved first place on the FinMMEval 2026 Task 3 leaderboard for Tesla (TSLA), delivering a +13.51% return and a Sharpe ratio of 4.10. The agent uses LLMs to analyze diverse financial data sources, aggregated by a Meta-Agent, while a rule-based approach was used for Bitcoin. The study also found that memoryless agents tend to repeat errors and that fixed-threshold rules underperform LLM pipelines in sideways markets.
Why it matters: This work demonstrates a significant advance in the practical deployment of LLM-based trading agents, showing strong real-world performance and offering insights for future agent design.
Jul 15, 2026
Research→Official→arXiv Cryptography and Security
A population-scale analysis of the x402 protocol for AI agent payments on the Base blockchain reveals that, over 280 days, 21.20% of 136.7 million settlements are fictitious and 63.78% are internal to linked clusters. The study finds that genuinely independent payments are much lower, ranging between $187,861 and $20.3 million, indicating that settlement counts primarily reflect manufacturability rather than real adoption.
Why it matters: This research challenges claims of a robust AI agent economy by demonstrating that most on-chain payments can be artificially generated, undermining settlement count as a metric for genuine autonomous economic activity.
Jul 15, 2026
Research→Official→arXiv Cryptography and Security
A large-scale empirical study of 4,022 pull requests from the AIDev dataset found that 38.9% of agent-generated PRs contain at least one security smell, with supply chain integrity issues accounting for 82.3% of all detected smells. Hard-coded credentials made up 99.6% of critical-severity issues, and 81.1% of these credentials went undetected before integration. The study also found that human collaborators introduced 67.6% of genuine leaked secrets in agent-assisted workflows.
Why it matters: This research reveals that autonomous coding agents introduce significant security risks that current review processes often fail to catch, highlighting the urgent need for improved security guardrails in human-AI collaboration.
Jul 15, 2026
Research→Official→arXiv Computation and Language
Researchers introduce Malleable Prompting, a technique that transforms natural language preference expressions into interactive GUI widgets such as sliders and toggles for controlling large language model (LLM) outputs. The method includes a new decoding algorithm that adjusts token probabilities based on widget settings. A user study demonstrates that this approach enables users to achieve target preferences more precisely and perceive greater controllability and transparency compared to standard natural language prompting.
Why it matters: This work presents a novel, interactive approach that could make controlling LLM outputs more intuitive and effective for users.
Jul 15, 2026
Research→Official→arXiv Computation and Language
Researchers have introduced ESFP, a behavioral benchmark designed to assess whether large language models (LLMs) can shift their epistemic register when prompted about expert beliefs versus their own beliefs. In tests of eight frontier models, results show that epistemic flexibility is largely independent of general model capability, with a 27B open-weight model performing on par with leading proprietary systems. The study also finds that stance content density is a stronger indicator of flexibility than surface-level lexical markers.
Why it matters: This benchmark fills a key gap in evaluating conversational AI by measuring models' ability to distinguish between self-attributed and externally attributed stances, which is important for trustworthy interactions.
Jul 15, 2026
Policy Safety→Official→arXiv Cryptography and Security
Researchers introduce PVDetector, a training-free framework for detecting prompt injection attacks on purpose-specific large language model (LLM) agents. By analyzing hidden-state alignment with policy-violation concepts, PVDetector identifies attacks with less than 1% false negative rate and minimal computational overhead, outperforming existing input-output pattern-based detectors.
Why it matters: As LLMs are increasingly used for specialized tasks with unique restrictions, PVDetector provides an efficient and effective defense against prompt injection attacks by leveraging the model's internal representations of policy violations.
Jul 15, 2026
Research→Official→arXiv Cryptography and Security
A new preprint presents a reproducible, open-source pipeline that distinguishes code written by humans from that generated by large language models (LLMs) with 93% accuracy, using samples from 31 security-sensitive programming tasks across four languages. The study finds systematic differences in security patterns between human and LLM-generated code, and shows that LLMs can repair vulnerabilities in code 77% of the time, though repairs are often only partial fixes.
Why it matters: This work advances automated code provenance attribution and highlights important security differences between human and AI-generated code, informing software supply chain security and AI-assisted development practices.
Jul 15, 2026
Research→Official→arXiv Computation and Language
A new study introduces a multi-feature fusion framework that integrates static lexical (W2V) and dynamic contextual (GPT) representations using an interactive gating mechanism for semantic reconstruction from non-invasive neural recordings. The framework systematically compares linear concatenation and non-linear cross-attention fusion, finding that the cross-attention method achieves state-of-the-art performance, surpassing single-representation approaches. This demonstrates improved neural language decoding by simulating the brain's integration of contextual and lexical features.
Why it matters: The work advances non-invasive brain-to-text decoding by showing that combining multiple semantic features yields significantly better results than previous single-feature methods.
Jul 15, 2026
Research→Official→arXiv Computation and Language
Researchers introduce PCC-SQL, a method that enforces column-use policies during text-to-SQL decoding by applying per-token logits masking, which deterministically prevents policy violations. The approach achieves a 0% leakage rate and up to 88.7% coverage on the Spider-CU benchmark, with minimal increase in token usage compared to direct prompting. The method is evaluated across three benchmarks and three open-source models, and also assesses semantic alignment with execution accuracy.
Why it matters: This work provides a deterministic solution to enforcing column-level access control in text-to-SQL systems, addressing a key security concern in real-world deployments.
Jul 15, 2026
Research→Official→arXiv Cryptography and Security
Researchers introduce JADR (Jacobian Assessment of Danger Recognition), a protocol that evaluates a language model's internal representation of danger by analyzing Jacobian space activations before any response is generated. The method operates entirely locally, does not require external judge models, and enables comparison of models and quantization levels using a SafetyAUC metric with statistical significance. Applied to six models, including Qwen3 variants and Gemma 2 9B, JADR distinguishes between strong and weak internal safety mechanisms and reveals quantization effects.
Why it matters: This approach provides a novel, judge-free method for probing LLM safety by examining internal model activations, potentially uncovering hidden vulnerabilities in safety mechanisms.
Jul 15, 2026
Research→Official→arXiv Cryptography and Security
A new benchmark, VanillaBench, systematically quantifies the accuracy cost of adversarial robustness by comparing 186 robust models against vanilla baselines. The study finds that the mean clean accuracy gap ranges from -7.7 to -29.5 percentage points, with even the best robust models trailing their vanilla counterparts by 4.0-21.0 points. The authors recommend that future robustness evaluations should routinely report vanilla-referenced accuracy gaps.
Why it matters: This benchmark makes explicit the substantial accuracy trade-off of adversarial robustness, providing critical information for practitioners and decision-makers that is often missing from individual papers.
Jul 15, 2026
Research→Official→arXiv Computation and Language
Researchers introduce Knowledge-Less Language Models (KLLMs), which are pretrained on corpora with anonymized named entities to reduce the model's reliance on memorized factual knowledge. KLLMs demonstrate reduced closed-book factual recall but outperform standard models on contextual question answering, fact verification, and hallucination detection, especially in retrieval-grounded settings with imperfect evidence, achieving up to 20–25% relative gains. The models also show improved calibration and more reliable abstention behavior, indicating a shift toward evidence-grounded reasoning.
Why it matters: This work shows that controlling knowledge acquisition during pretraining can produce language models that are more robust and reliable in evidence-grounded tasks, potentially improving real-world AI applications.
Jul 15, 2026