{ "generated_at": "2026-05-21T13:14:24.684805+08:00", "timezone": "Asia/Shanghai", "lookback_hours": 24, "sorting": { "default_sort_by": "hybrid", "summary": "hybrid (relevance first, published_at tie-break)", "weights": { "title_match_weight": 40, "summary_match_weight": 18, "doi_weight": 12, "pdf_weight": 8, "rich_summary_weight": 6, "metadata_weight": 4, "multi_source_weight": 10, "freshness_weight_cap": 24 }, "feeds": [ { "name": "LM", "sort_by": "hybrid" }, { "name": "Agent Runtime Security", "sort_by": "hybrid" }, { "name": "Terminal and SWE Agents", "sort_by": "hybrid" } ] }, "highlights": [ "主题「LLM」：命中 13 篇，覆盖 LM、Agent Runtime Security，代表论文包括 《Tracing the ongoing emergence of human-like reasoning in Large Language Models》、《What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema》。", "主题「Benchmark」：命中 11 篇，覆盖 LM、Terminal and SWE Agents，代表论文包括 《What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema》、《WikiVQABench: A Knowledge-Grounded Visual Question Answering Benchmark from Wikipedia and Wikidata》。", "主题「Language Model」：命中 8 篇，覆盖 LM，代表论文包括 《Tracing the ongoing emergence of human-like reasoning in Large Language Models》、《Leveraging LLMs for Grammar Adaptation: A Study on Metamodel-Grammar Co-Evolution》。", "主题「Agent」：命中 2 篇，覆盖 Agent Runtime Security、Terminal and SWE Agents，代表论文包括 《Agent JIT Compilation for Latency-Optimizing Web Agent Planning and Scheduling》、《SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents》。" ], "focus_items": [], "action_items": [], "topic_sections": [ { "name": "LLM", "paper_count": 13, "feed_names": [ "LM", "Agent Runtime Security" ], "paper_titles": [ "Tracing the ongoing emergence of human-like reasoning in Large Language Models", "What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema", "Leveraging LLMs for Grammar Adaptation: A Study on Metamodel-Grammar Co-Evolution", "LASH: Adaptive Semantic Hybridization for Black-Box Jailbreaking of Large Language Models", "WikiVQABench: A Knowledge-Grounded Visual Question Answering Benchmark from Wikipedia and Wikidata", "Federated LoRA Fine-Tuning for LLMs via Collaborative Alignment", "Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents", "Text Analytics Evaluation Framework: A Case Study on LLMs and Social Media", "You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories", "Do LLMs Know What Luxembourgish Borrows? Probing Lexical Neology in Low-Resource Multilingual Models", "APEX: Autonomous Policy Exploration for Self-Evolving LLM Agents", "Open-source LLMs administer maximum electric shocks in a Milgram-like obedience experiment", "Agent JIT Compilation for Latency-Optimizing Web Agent Planning and Scheduling" ], "key_points": [ "《Tracing the ongoing emergence of human-like reasoning in Large Language Models》〔方法〕：Humans effortlessly go beyond literal meanings: If you mow the lawn, I will give you fifty dollars, is typically understood as implying that the speaker will p…", "《What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema》〔评测 / 方法〕：We read twelve well-known LLM agent benchmark papers and recorded, dimension by dimension, what each paper actually says about how its evaluation was run. The…" ] }, { "name": "Benchmark", "paper_count": 11, "feed_names": [ "LM", "Terminal and SWE Agents" ], "paper_titles": [ "What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema", "WikiVQABench: A Knowledge-Grounded Visual Question Answering Benchmark from Wikipedia and Wikidata", "Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents", "Text Analytics Evaluation Framework: A Case Study on LLMs and Social Media", "DeepWeb-Bench: A Deep Research Benchmark Demanding Massive Cross-Source Evidence and Long-Horizon Derivation", "You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories", "Do LLMs Know What Luxembourgish Borrows? Probing Lexical Neology in Low-Resource Multilingual Models", "DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards", "APEX: Autonomous Policy Exploration for Self-Evolving LLM Agents", "TempGlitch: Evaluating Vision-Language Models for Temporal Glitch Detection in Gameplay Videos", "SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents" ], "key_points": [ "《What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema》〔评测 / 方法〕：We read twelve well-known LLM agent benchmark papers and recorded, dimension by dimension, what each paper actually says about how its evaluation was run. The…", "《WikiVQABench: A Knowledge-Grounded Visual Question Answering Benchmark from Wikipedia and Wikidata》〔评测 / 数据 / 方法〕：Visual Question Answering (VQA) benchmarks have largely emphasized perception-based tasks that can be solved from visual content alone. In contrast, many real-…" ] }, { "name": "Language Model", "paper_count": 8, "feed_names": [ "LM" ], "paper_titles": [ "Tracing the ongoing emergence of human-like reasoning in Large Language Models", "Leveraging LLMs for Grammar Adaptation: A Study on Metamodel-Grammar Co-Evolution", "LASH: Adaptive Semantic Hybridization for Black-Box Jailbreaking of Large Language Models", "Federated LoRA Fine-Tuning for LLMs via Collaborative Alignment", "DeepWeb-Bench: A Deep Research Benchmark Demanding Massive Cross-Source Evidence and Long-Horizon Derivation", "DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards", "TempGlitch: Evaluating Vision-Language Models for Temporal Glitch Detection in Gameplay Videos", "Open-source LLMs administer maximum electric shocks in a Milgram-like obedience experiment" ], "key_points": [ "《Tracing the ongoing emergence of human-like reasoning in Large Language Models》〔方法〕：Humans effortlessly go beyond literal meanings: If you mow the lawn, I will give you fifty dollars, is typically understood as implying that the speaker will p…", "《Leveraging LLMs for Grammar Adaptation: A Study on Metamodel-Grammar Co-Evolution》〔评测 / 方法〕：In model-driven engineering, metamodel evolution leads to the need to adapt corresponding grammars to maintain consistency, which typically requires tedious ma…" ] }, { "name": "Agent", "paper_count": 2, "feed_names": [ "Agent Runtime Security", "Terminal and SWE Agents" ], "paper_titles": [ "Agent JIT Compilation for Latency-Optimizing Web Agent Planning and Scheduling", "SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents" ], "key_points": [ "《Agent JIT Compilation for Latency-Optimizing Web Agent Planning and Scheduling》〔应用 / 方法〕：Computer-use agents (CUA) automate tasks specified with natural language such as \"order the cheapest item from Taco Bell\" by generating sequences of calls to t…", "《SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents》〔评测 / 方法〕：As long-horizon coding agents produce more code than any developer can review, oversight collapses onto a single surface: the automated test suite. Reward hack…" ] } ], "template": "zh_daily_brief", "feeds": [ { "name": "LM", "key_points": [ "《Tracing the ongoing emergence of human-like reasoning in Large Language Models》〔方法〕：Humans effortlessly go beyond literal meanings: If you mow the lawn, I will give you fifty dollars, is typically understood as implying that the speaker will p…", "《What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema》〔评测 / 方法〕：We read twelve well-known LLM agent benchmark papers and recorded, dimension by dimension, what each paper actually says about how its evaluation was run. The…", "《Leveraging LLMs for Grammar Adaptation: A Study on Metamodel-Grammar Co-Evolution》〔评测 / 方法〕：In model-driven engineering, metamodel evolution leads to the need to adapt corresponding grammars to maintain consistency, which typically requires tedious ma…", "《LASH: Adaptive Semantic Hybridization for Black-Box Jailbreaking of Large Language Models》〔评测 / 应用 / 方法〕：Jailbreak attacks expose a persistent gap between the intended safety behavior of aligned large language models and their behavior under adversarial prompting.…", "《WikiVQABench: A Knowledge-Grounded Visual Question Answering Benchmark from Wikipedia and Wikidata》〔评测 / 数据 / 方法〕：Visual Question Answering (VQA) benchmarks have largely emphasized perception-based tasks that can be solved from visual content alone. In contrast, many real-…" ], "sort_by": "hybrid", "papers": [ { "title": "Tracing the ongoing emergence of human-like reasoning in Large Language Models", "summary": "Humans effortlessly go beyond literal meanings: If you mow the lawn, I will give you fifty dollars, is typically understood as implying that the speaker will pay only if the lawn is mowed, whereas If you are hungry, there is pizza in the oven implies that pizza is available regardless of the hearers hunger. Large Language Models - LLMs - show human-like performance on many tasks, yet it remains unclear whether they reason like humans. To address this, we conducted a population-matching experiment assessing how twentyfive LLMs compute conditional inferences across four languages, compared to an equal number of humans per language. We find that humans enrich logical reasoning through pragmatic inferences across languages. Model behavior is more variable. Some LLMs perfectly follow the truth-table of conditionals but they ignore pragmatic inferences, while others deviate from the truth-table, adhering to a single interpretation across the board, thus reflecting accurate rule-based processing but not human-like reasoning. Overall, LLMs are accurate semantic operators, but fail to capture the pragmatic enrichments characteristic of human reasoning. Crucially, LLM accuracy is neither predicted nor boosted by open vs. closed status, training orientation, or architecture type, suggesting that pragmatic reasoning is still an emerging ability in the cognitive toolkit of artificial systems.", "authors": [ "Paolo Morosi", "Nikoleta Pantelidou", "Fritz Günther", "Elena Pagliarini", "Evelina Leivada" ], "categories": [ "cs.CL", "cs.AI" ], "paper_id": "http://arxiv.org/abs/2605.21299v1", "abstract_url": "https://arxiv.org/abs/2605.21299v1", "pdf_url": "https://arxiv.org/pdf/2605.21299v1", "published_at": "2026-05-20T15:28:52+00:00", "updated_at": "2026-05-20T15:28:52+00:00", "source": "arxiv", "date_label": "Published", "analysis": null, "tags": [ "方法" ], "topics": [ "LLM", "Language Model" ], "doi": null, "arxiv_id": "2605.21299", "source_variants": [ "arxiv" ], "source_urls": { "arxiv": "https://arxiv.org/abs/2605.21299v1" }, "relevance_score": 184, "match_reasons": [ "title matched \"language model\"", "title matched \"large language model\"", "title matched \"reasoning\"", "summary matched \"LLM\"", "summary matched \"RAG\"", "has PDF", "has rich summary", "has complete metadata" ], "feedback_status": null, "feedback_note": null, "feedback_next_action": null, "feedback_due_date": null, "feedback_snoozed_until": null, "feedback_review_interval_days": null, "canonical_id": "arxiv:2605.21299" }, { "title": "What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema", "summary": "We read twelve well-known LLM agent benchmark papers and recorded, dimension by dimension, what each paper actually says about how its evaluation was run. The motivation came from a familiar frustration: two papers will report results on the same benchmark with the same model name and disagree, and you cannot tell why -- the scaffold, the sampling settings, the subset, or the evaluator version. In many cases the published artifact does not let you answer. This paper is an implementation report on the attempt. We designed a small audit schema (five fields: benchmark identity, harness specification, inference settings, cost reporting, failure breakdown), wrote a scoring codebook with the boundary cases we hit during pilot scoring, applied it to twelve canonical papers (eight agent, four classical static), and recorded what we saw. We score the disclosure of an agent run, not its correctness, and make no claim that disclosure implies a trustworthy result. The mean audit score across the eight agent-benchmark papers is 0.38 (out of 1.0), and across the four classical static benchmarks 0.66; the largest gap is on cost (none of the eight agent benchmark papers disclose inference cost in any form) and on harness specification (none fully disclose a content-addressed container image of the evaluation environment). We release the schema as a JSON Schema file, the codebook as a Markdown document, and the raw scoring sheet as a CSV. The scoring was performed by a single auditor in one pass; a multi-rater audit is the natural next step, and we discuss what we think it would change.", "authors": [ "Mahdi Naser Moghadasi", "Faezeh Ghaderi" ], "categories": [ "cs.LG" ], "paper_id": "http://arxiv.org/abs/2605.21404v1", "abstract_url": "https://arxiv.org/abs/2605.21404v1", "pdf_url": "https://arxiv.org/pdf/2605.21404v1", "published_at": "2026-05-20T17:02:36+00:00", "updated_at": "2026-05-20T17:02:36+00:00", "source": "arxiv", "date_label": "Published", "analysis": null, "tags": [ "评测", "方法" ], "topics": [ "LLM", "Benchmark" ], "doi": null, "arxiv_id": "2605.21404", "source_variants": [ "arxiv" ], "source_urls": { "arxiv": "https://arxiv.org/abs/2605.21404v1" }, "relevance_score": 167, "match_reasons": [ "title matched \"LLM\"", "title matched \"agent\"", "title matched \"benchmark\"", "summary matched \"evaluation\"", "has PDF", "has rich summary", "has complete metadata" ], "feedback_status": null, "feedback_note": null, "feedback_next_action": null, "feedback_due_date": null, "feedback_snoozed_until": null, "feedback_review_interval_days": null, "canonical_id": "arxiv:2605.21404" }, { "title": "Leveraging LLMs for Grammar Adaptation: A Study on Metamodel-Grammar Co-Evolution", "summary": "In model-driven engineering, metamodel evolution leads to the need to adapt corresponding grammars to maintain consistency, which typically requires tedious manual work. Existing rule-based methods can achieve partial automation but have limitations when handling complex grammar scenarios. This paper proposes a Large Language Model-based approach that automatically applies adaptations to new grammars after evolution by learning grammar adaptations from previous versions. We evaluated this approach on six real-world Xtext domain-specific languages, using four DSLs as a training set to develop prompting strategies, two DSLs as a test set for validation, and conducting a longitudinal case study on QVTo. The evaluation used three Large Language Models (Claude Sonnet 4.5, ChatGPT 5.1, Gemini 3) and measured grammar adaptation quality from three dimensions: grammar rule-level adaptation consistency, output similarity, and metamodel conformance. Results show that on the test set, all three LLMs achieved 100% adaptation consistency and output similarity, while the rule-based approach achieved only 84.21% on DOT and 62.50% on Xcore. In the QVTo longitudinal study, the LLM-based approach successfully reused learned adaptations across all three evolution steps without manual grammar editing, while the rule-based approach required manual adjustments in two of three transitions. However, on large-scale grammars (EAST-ADL, 297 rules), LLMs' adaptation consistency was far below 90%. This study demonstrates the advantages of LLM-based approaches in handling complex grammar scenarios, while revealing their limitations in large-scale grammar adaptation.", "authors": [ "Weixing Zhang", "Bowen Jiang", "Rahul Sharma", "Regina Hebig", "Daniel Strüber" ], "categories": [ "cs.CL", "cs.SE" ], "paper_id": "http://arxiv.org/abs/2605.21465v1", "abstract_url": "https://arxiv.org/abs/2605.21465v1", "pdf_url": "https://arxiv.org/pdf/2605.21465v1", "published_at": "2026-05-20T17:51:51+00:00", "updated_at": "2026-05-20T17:51:51+00:00", "source": "arxiv", "date_label": "Published", "analysis": null, "tags": [ "评测", "方法" ], "topics": [ "LLM", "Language Model" ], "doi": null, "arxiv_id": "2605.21465", "source_variants": [ "arxiv" ], "source_urls": { "arxiv": "https://arxiv.org/abs/2605.21465v1" }, "relevance_score": 164, "match_reasons": [ "title matched \"LLM\"", "title matched \"RAG\"", "summary matched \"language model\"", "summary matched \"large language model\"", "summary matched \"evaluation\"", "has PDF", "has rich summary", "has complete metadata" ], "feedback_status": null, "feedback_note": null, "feedback_next_action": null, "feedback_due_date": null, "feedback_snoozed_until": null, "feedback_review_interval_days": null, "canonical_id": "arxiv:2605.21465" }, { "title": "LASH: Adaptive Semantic Hybridization for Black-Box Jailbreaking of Large Language Models", "summary": "Jailbreak attacks expose a persistent gap between the intended safety behavior of aligned large language models and their behavior under adversarial prompting. Existing automated methods are increasingly effective but each commits to a single attack family (e.g., one refinement loop, one tree search, one mutation space, or one strategy library) and no single family dominates: the best-performing method shifts across target models and harm categories, suggesting complementary strengths that per-prompt composition could exploit. We introduce LASH (LLM Adaptive Semantic Hybridization), a black-box framework that treats outputs from multiple base attacks as reusable seed prompts and adaptively composes them for each target request. Given a seed pool, LASH searches over seed subsets and softmax-normalized mixture weights; a composition module synthesizes a single candidate prompt, and a derivative-free genetic optimizer updates the weights using black-box target feedback and a two-stage fitness function combining keyword-based refusal detection with LLM-judge scoring. On JailbreakBench, which contains 100 harmful prompts across 10 categories, we evaluate LASH on six common target models. LASH achieves an average attack success rate of 84.5% under keyword-based evaluation and 74.5% under two-stage evaluation, where responses are first filtered for refusals and then scored by an LLM judge for whether they substantively fulfill the original harmful request. LASH outperforms five state-of-the-art baselines on both metrics with only 30 mean target queries. LASH also remains competitive under three defense mechanisms and induces more success-like internal representations. These results suggest that adaptive composition across heterogeneous jailbreak strategies is a promising direction for black-box red-teaming.", "authors": [ "Abdullah Al Nomaan Nafi", "Fnu Suya", "Swarup Bhunia", "Prabuddha Chakraborty" ], "categories": [ "cs.CL" ], "paper_id": "http://arxiv.org/abs/2605.21362v1", "abstract_url": "https://arxiv.org/abs/2605.21362v1", "pdf_url": "https://arxiv.org/pdf/2605.21362v1", "published_at": "2026-05-20T16:27:00+00:00", "updated_at": "2026-05-20T16:27:00+00:00", "source": "arxiv", "date_label": "Published", "analysis": null, "tags": [ "评测", "应用", "方法" ], "topics": [ "LLM", "Language Model" ], "doi": null, "arxiv_id": "2605.21362", "source_variants": [ "arxiv" ], "source_urls": { "arxiv": "https://arxiv.org/abs/2605.21362v1" }, "relevance_score": 163, "match_reasons": [ "title matched \"language model\"", "title matched \"large language model\"", "summary matched \"LLM\"", "summary matched \"RAG\"", "summary matched \"evaluation\"", "has PDF", "has rich summary", "has complete metadata", "title matched \"jailbreak\"" ], "feedback_status": null, "feedback_note": null, "feedback_next_action": null, "feedback_due_date": null, "feedback_snoozed_until": null, "feedback_review_interval_days": null, "canonical_id": "arxiv:2605.21362" }, { "title": "WikiVQABench: A Knowledge-Grounded Visual Question Answering Benchmark from Wikipedia and Wikidata", "summary": "Visual Question Answering (VQA) benchmarks have largely emphasized perception-based tasks that can be solved from visual content alone. In contrast, many real-world scenarios require external knowledge that is not directly observable in the image to answer correctly. We introduce WikiVQABench, a human-curated knowledge-grounded VQA benchmark constructed by systematically combining Wikipedia images, their associated article captions, and structured knowledge from Wikidata. Our pipeline uses large language models (LLMs) to generate candidate multiple-choice image-question-answer sets. All generated instances are subsequently reviewed and curated by human annotators to ensure factual correctness, visual-text consistency, and that each question requires external knowledge in addition to visual evidence for correct resolution. WikiVQABench comprises a substantial collection of Wikipedia images with curated multiple-choice questions designed to benchmark knowledge-aware vision-language models (VLMs). Evaluation of fifteen VLMs (256M-90B parameters) reveals a wide performance range (24.7%-75.6% accuracy), demonstrating that the benchmark effectively discriminates model capabilities on knowledge-intensive reasoning. The dataset and benchmarking code are publicly available.", "authors": [ "Basel Shbita", "Pengyuan Li", "Anna Lisa Gentile" ], "categories": [ "cs.CV", "cs.AI" ], "paper_id": "http://arxiv.org/abs/2605.21479v1", "abstract_url": "https://arxiv.org/abs/2605.21479v1", "pdf_url": "https://arxiv.org/pdf/2605.21479v1", "published_at": "2026-05-20T17:58:24+00:00", "updated_at": "2026-05-20T17:58:24+00:00", "source": "arxiv", "date_label": "Published", "analysis": null, "tags": [ "评测", "数据", "方法" ], "topics": [ "LLM", "Benchmark" ], "doi": null, "arxiv_id": "2605.21479", "source_variants": [ "arxiv" ], "source_urls": { "arxiv": "https://arxiv.org/abs/2605.21479v1" }, "relevance_score": 160, "match_reasons": [ "title matched \"benchmark\"", "summary matched \"language model\"", "summary matched \"large language model\"", "summary matched \"LLM\"", "summary matched \"reasoning\"", "summary matched \"evaluation\"", "has PDF", "has rich summary", "has complete metadata" ], "feedback_status": null, "feedback_note": null, "feedback_next_action": null, "feedback_due_date": null, "feedback_snoozed_until": null, "feedback_review_interval_days": null, "canonical_id": "arxiv:2605.21479" }, { "title": "Federated LoRA Fine-Tuning for LLMs via Collaborative Alignment", "summary": "Low-rank adaptation (LoRA) has emerged as a powerful tool for parameter-efficient fine-tuning of large language models (LLMs). This paper studies LoRA under a federated learning setting, enabling collaborative fine-tuning across clients while preserving parameter efficiency. We focus on a highly heterogeneous regime in which clients share only partial structure and a substantial subset may be contaminated. We propose Collaborative Low-rank Alignment and Identifiable Recovery (CLAIR), a contamination-aware framework that relies only on preliminary local estimators. Its formulation applies broadly, from linear regression to neural network and LLM modules, whenever local adaptation can be represented by matrix-valued updates. CLAIR recovers the shared LoRA subspace and detects contaminated clients via a structured low-rank plus block-sparse decomposition. We prove exact recovery of the shared LoRA subspace in the noiseless case, stable recovery under preliminary estimation error, and consistent collaborative-set recovery under mild separation conditions. We further quantify the gain from CLAIR refinement: it reduces off-subspace estimation error through cross-client averaging while preserving client-specific variation within the shared LoRA subspace, thus improves over local fine-tuning whenever this oracle gain outweighs the costs of subspace estimation and benign-client heterogeneity. Empirically, we demonstrate the benefits of CLAIR by fine-tuning a Transformer architecture on a text-copying task. The results show accurate contamination detection and improved benign-client performance compared with local fine-tuning and non-robust federated averaging.", "authors": [ "Shuaida He", "Liwen Chen", "Long Feng" ], "categories": [ "stat.ML", "cs.LG" ], "paper_id": "http://arxiv.org/abs/2605.21217v1", "abstract_url": "https://arxiv.org/abs/2605.21217v1", "pdf_url": "https://arxiv.org/pdf/2605.21217v1", "published_at": "2026-05-20T14:12:16+00:00", "updated_at": "2026-05-20T14:12:16+00:00", "source": "arxiv", "date_label": "Published", "analysis": null, "tags": [ "方法" ], "topics": [ "LLM", "Language Model" ], "doi": null, "arxiv_id": "2605.21217", "source_variants": [ "arxiv" ], "source_urls": { "arxiv": "https://arxiv.org/abs/2605.21217v1" }, "relevance_score": 160, "match_reasons": [ "title matched \"LLM\"", "title matched \"alignment\"", "summary matched \"language model\"", "summary matched \"large language model\"", "summary matched \"RAG\"", "has PDF", "has rich summary", "has complete metadata" ], "feedback_status": null, "feedback_note": null, "feedback_next_action": null, "feedback_due_date": null, "feedback_snoozed_until": null, "feedback_review_interval_days": null, "canonical_id": "arxiv:2605.21217" }, { "title": "Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents", "summary": "Diagnosing failures in LLM agents remains largely manual. Practitioners inspect a small subset of execution traces, form ad-hoc hypotheses, and iterate. This process misses patterns that only emerge across trace populations and does not scale to production corpora where individual traces span tens of thousands of tokens. We formalize the problem of corpus-level trace diagnostics. Given a corpus of execution traces, the goal is to produce grounded natural-language insights that characterize systematic behavioral patterns across trace groups, each linked to supporting evidence. We present the Insights Generator (IG), a multi-agent system that answers diagnostic questions by proposing and testing hypotheses across the trace corpus to produce an evidence-backed insights report. We evaluate IG across qualitative and objective dimensions, spanning rubric-based report assessment and downstream performance improvements achieved by implementing IG insights. Human experts using IG reports improve scaffold performance by 30.4pp over the unmodified baseline scaffold, and coding agents leveraging IG-derived insights show consistent and stable gains. Across benchmarks, IG's scout-investigator architecture produces findings comparable in detection coverage to competing approaches, while domain experts rated IG reports as leading depth and evidence quality.", "authors": [ "Akshay Manglik", "Apaar Shanker", "Kaustubh Deshpande", "Jason Qin", "Yash Maurya", "Veronica Chatrath", "Vijay S. Kalmath", "Levi Lentz", "Yuan", "Xue" ], "categories": [ "cs.AI", "cs.LG", "cs.SE" ], "paper_id": "http://arxiv.org/abs/2605.21347v1", "abstract_url": "https://arxiv.org/abs/2605.21347v1", "pdf_url": "https://arxiv.org/pdf/2605.21347v1", "published_at": "2026-05-20T16:13:53+00:00", "updated_at": "2026-05-20T16:13:53+00:00", "source": "arxiv", "date_label": "Published", "analysis": null, "tags": [ "评测", "数据", "方法" ], "topics": [ "LLM", "Benchmark" ], "doi": null, "arxiv_id": "2605.21347", "source_variants": [ "arxiv" ], "source_urls": { "arxiv": "https://arxiv.org/abs/2605.21347v1" }, "relevance_score": 144, "match_reasons": [ "title matched \"LLM\"", "title matched \"agent\"", "summary matched \"RAG\"", "summary matched \"benchmark\"", "has PDF", "has rich summary", "has complete metadata", "summary matched \"coding agent\"" ], "feedback_status": null, "feedback_note": null, "feedback_next_action": null, "feedback_due_date": null, "feedback_snoozed_until": null, "feedback_review_interval_days": null, "canonical_id": "arxiv:2605.21347" }, { "title": "Text Analytics Evaluation Framework: A Case Study on LLMs and Social Media", "summary": "LLMs have demonstrated exceptional proficiency in a wide range of NLP tasks. However, a notable gap remains in practical data analysis scenarios, particularly when LLMs are required to process long sequences of unstructured documents, such as news feeds or, as specifically addressed in this paper, social media posts. To empirically assess the effectiveness of LLMs in this setting, we introduce a question-based evaluation framework comprising 470 manually curated questions designed to evaluate LLMs' semantic understanding and reasoning abilities over aggregated text data. We apply our benchmark on diverse Twitter datasets covering various NLP tasks, including sentiment analysis, hate speech detection, and emotion recognition. Our results reveal that the performance depends heavily on input scale and the complexity of the data sources, declining noticeably in multi-label or target-dependent scenarios. In addition, as task complexity increases, performance drops progressively from basic semantic existence identification to more demanding operations such as comparison, counting, and calculation. Furthermore, as the input size grows beyond 500 instances, we identify a common limitation across LLMs, particularly Open-weights models: performance degrades substantially, especially on numerical tasks. These findings highlight critical architectural bottlenecks in current LLMs for performing rigorous quantitative analysis over large text collections.", "authors": [ "Yuefeng Shi", "Nedjma Ousidhoum", "Jose Camacho-Collados" ], "categories": [ "cs.CL" ], "paper_id": "http://arxiv.org/abs/2605.21338v1", "abstract_url": "https://arxiv.org/abs/2605.21338v1", "pdf_url": "https://arxiv.org/pdf/2605.21338v1", "published_at": "2026-05-20T16:05:05+00:00", "updated_at": "2026-05-20T16:05:05+00:00", "source": "arxiv", "date_label": "Published", "analysis": null, "tags": [ "评测", "数据", "方法" ], "topics": [ "LLM", "Benchmark" ], "doi": null, "arxiv_id": "2605.21338", "source_variants": [ "arxiv" ], "source_urls": { "arxiv": "https://arxiv.org/abs/2605.21338v1" }, "relevance_score": 144, "match_reasons": [ "title matched \"LLM\"", "title matched \"evaluation\"", "summary matched \"reasoning\"", "summary matched \"benchmark\"", "has PDF", "has rich summary", "has complete metadata" ], "feedback_status": null, "feedback_note": null, "feedback_next_action": null, "feedback_due_date": null, "feedback_snoozed_until": null, "feedback_review_interval_days": null, "canonical_id": "arxiv:2605.21338" }, { "title": "DeepWeb-Bench: A Deep Research Benchmark Demanding Massive Cross-Source Evidence and Long-Horizon Derivation", "summary": "Deep research, in which an agent searches the open web, collects evidence, and derives an answer through extended reasoning, is a prominent use case for frontier language models. Frontier deep research products score high on existing benchmarks, making it difficult to distinguish their capabilities from current evaluation data alone. We introduce DeepWeb-Bench, a deep research benchmark that is substantially harder than existing benchmarks for the current frontier. Difficulty comes from three properties of the data itself: each task requires massive evidence collection, cross-source reconciliation, and long-horizon multi-step derivation. We represent these three sources of difficulty as four capability families (Retrieval, Derivation, Reasoning, and Calibration) and report results sliced by family. Every reference answer is accompanied by a source-provenance record with four disclosure levels and cross-source checks where available, making scores easier to audit against the underlying evidence. We evaluate DeepWeb-Bench on nine frontier models and report three findings: (1) retrieval is not the bottleneck, as retrieval failures account for only 12-14% of errors while derivation and calibration failures account for over 70%; (2) strong and weak models fail in qualitatively different ways, with strong models' errors dominated by incomplete derivation and weak models' by hallucinated precision; and (3) models exhibit genuine specialization across domains, with cross-model agreement of only rho = 0.61 and per-case disagreement reaching 18.8 percentage points. The public benchmark release includes the data, rubrics, and evaluation code.", "authors": [ "Sixiong Xie", "Zhuofan Shi", "Haiyang Shen", "Jiuzheng Wang", "Siqi Zhong", "Mugeng Liu", "Chongyang Pan", "Peilun Jia", "Baoqing Sun", "Xiang Jing", "Yun Ma" ], "categories": [ "cs.AI" ], "paper_id": "http://arxiv.org/abs/2605.21482v1", "abstract_url": "https://arxiv.org/abs/2605.21482v1", "pdf_url": "https://arxiv.org/pdf/2605.21482v1", "published_at": "2026-05-20T17:59:03+00:00", "updated_at": "2026-05-20T17:59:03+00:00", "source": "arxiv", "date_label": "Published", "analysis": null, "tags": [ "评测", "应用", "方法" ], "topics": [ "Benchmark", "Language Model" ], "doi": null, "arxiv_id": "2605.21482", "source_variants": [ "arxiv" ], "source_urls": { "arxiv": "https://arxiv.org/abs/2605.21482v1" }, "relevance_score": 142, "match_reasons": [ "title matched \"benchmark\"", "summary matched \"language model\"", "summary matched \"reasoning\"", "summary matched \"agent\"", "summary matched \"evaluation\"", "has PDF", "has rich summary", "has complete metadata" ], "feedback_status": null, "feedback_note": null, "feedback_next_action": null, "feedback_due_date": null, "feedback_snoozed_until": null, "feedback_review_interval_days": null, "canonical_id": "arxiv:2605.21482" }, { "title": "You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories", "summary": "Reinforcement learning with verifiable rewards (RLVR) has become a dominant paradigm for improving reasoning in large language models (LLMs), yet the underlying geometry of the resulting parameter trajectories remains underexplored. In this work, we demonstrate that RLVR weight trajectories are extremely low-rank and highly predictable. Specifically, we find that the majority of downstream performance gains are captured by a rank-1 approximation of the parameter deltas, where the magnitude of this projection evolves near-linearly with training steps. Motivated by this, we propose a simple and compute-efficient method RELEX (REinforcement Learning EXtrapolation), which estimates the rank-1 subspace from a short observation window and extrapolates future checkpoints via linear regression, with no learned model required. Across three models (i.e., Qwen2.5-Math-1.5B, Qwen3-4B-Base, and Qwen3-8B-Base), RELEX produces checkpoints that match or exceed RLVR performance on both in-domain and out-of-domain benchmarks, requiring as few as 15% steps of full RLVR training. Remarkably, RELEX is able to extrapolate far beyond the observation window at no training cost, predicting checkpoints up to 10-20$\\times$ beyond the observed prefix with continued improvement (e.g., observe only the first 50 steps and extrapolate to 1000 steps). Our ablation analysis confirms the minimalist sufficiency of RELEX: neither increasing the subspace rank nor employing non-linear modeling yields further gains in extrapolation. Finally, we show that RELEX's success stems from a \"denoising\" effect: by projecting updates onto the rank-1 subspace, the model discards stochastic optimization noise that would otherwise degrade performance during extrapolation. Our code is available at https://github.com/weizhepei/RELEX.", "authors": [ "Zhepei Wei", "Xinyu Zhu", "Wei-Lin Chen", "Chengsong Huang", "Jiaxin Huang", "Yu Meng" ], "categories": [ "cs.LG", "cs.CL" ], "paper_id": "http://arxiv.org/abs/2605.21468v1", "abstract_url": "https://arxiv.org/abs/2605.21468v1", "pdf_url": "https://arxiv.org/pdf/2605.21468v1", "published_at": "2026-05-20T17:53:20+00:00", "updated_at": "2026-05-20T17:53:20+00:00", "source": "arxiv", "date_label": "Published", "analysis": null, "tags": [ "评测", "方法" ], "topics": [ "LLM", "Benchmark" ], "doi": null, "arxiv_id": "2605.21468", "source_variants": [ "arxiv" ], "source_urls": { "arxiv": "https://arxiv.org/abs/2605.21468v1" }, "relevance_score": 142, "match_reasons": [ "title matched \"LLM\"", "summary matched \"language model\"", "summary matched \"large language model\"", "summary matched \"reasoning\"", "summary matched \"benchmark\"", "has PDF", "has rich summary", "has complete metadata" ], "feedback_status": null, "feedback_note": null, "feedback_next_action": null, "feedback_due_date": null, "feedback_snoozed_until": null, "feedback_review_interval_days": null, "canonical_id": "arxiv:2605.21468" }, { "title": "Do LLMs Know What Luxembourgish Borrows? Probing Lexical Neology in Low-Resource Multilingual Models", "summary": "Large language models (LLMs) are increasingly used for writing assistance in small contact languages, yet it is unclear whether they respect community norms around lexical borrowing and neology. We introduce LexNeo-Bench, a 3{,}050-instance token-level benchmark derived from LuxBorrow, a large-scale Luxembourgish news corpus, where target tokens are labelled as native or as French, German, or English borrowings. Using this benchmark, we probe three multilingual LLMs across 34 prompt settings on two tasks: borrowing type classification and a binary lexical-innovation proxy (borrowing versus native). Without external context, models perform only slightly above chance on borrowing classification, so we construct a linguistic knowledge graph that encodes donor language, morphological patterns, and lexical analogues, and inject instance-specific subgraphs into the prompt. Knowledge-graph prompts raise borrowing classification accuracy from 25 -- 35\\% up to 71 -- 81\\% and largely close the gap between small and large models, while leaving neology detection difficult and sensitive to few-shot design. Our results show that lexicon-aware prompting is highly beneficial for robust borrowing judgments in low-resource contact languages and that lexical resources can serve as structured context for LLM evaluation. This study was carried out within the ENEOLI COST Action and examines borrowing as a form of lexical innovation in multilingual Luxembourgish data.", "authors": [ "Nina Hosseini-Kivanani" ], "categories": [ "cs.CL" ], "paper_id": "http://arxiv.org/abs/2605.21227v1", "abstract_url": "https://arxiv.org/abs/2605.21227v1", "pdf_url": "https://arxiv.org/pdf/2605.21227v1", "published_at": "2026-05-20T14:19:55+00:00", "updated_at": "2026-05-20T14:19:55+00:00", "source": "arxiv", "date_label": "Published", "analysis": null, "tags": [ "评测", "数据", "方法" ], "topics": [ "LLM", "Benchmark" ], "doi": null, "arxiv_id": "2605.21227", "source_variants": [ "arxiv" ], "source_urls": { "arxiv": "https://arxiv.org/abs/2605.21227v1" }, "relevance_score": 139, "match_reasons": [ "title matched \"LLM\"", "summary matched \"language model\"", "summary matched \"large language model\"", "summary matched \"benchmark\"", "summary matched \"evaluation\"", "has PDF", "has rich summary", "has complete metadata" ], "feedback_status": null, "feedback_note": null, "feedback_next_action": null, "feedback_due_date": null, "feedback_snoozed_until": null, "feedback_review_interval_days": null, "canonical_id": "arxiv:2605.21227" }, { "title": "DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards", "summary": "Reinforcement learning from verifiable rewards (RLVR) has emerged as a central technique for improving the reasoning capabilities of large language models. Despite its effectiveness, how response-level rewards translate into token-level probability changes remains poorly understood. We introduce a discriminator view of RLVR updates, showing that the policy-gradient update direction implicitly acts as a linear discriminator over token-gradient vectors and thereby determines which token probabilities are increased or decreased during learning. Under standard sequence-level RLVR, this discriminator is constructed from positive- and negative-side centroids formed by advantage-weighted averaging of token-gradient vectors. However, such centroid construction can be dominated by shared high-frequency patterns, such as formatting tokens, diluting sparse yet discriminative directions that better distinguish high-reward responses from low-reward ones. To address this limitation, we propose $\\textbf{DelTA}$, a discriminative token credit assignment method that estimates token coefficients to amplify side-specific token-gradient directions and downweight shared or weakly discriminative ones. These coefficients reweight a self-normalized RLVR surrogate, making the effective side-wise centroids more contrastive and thereby reshaping the RLVR update direction. On seven mathematical benchmarks, DelTA outperforms the strongest same-scale baselines by 3.26 and 2.62 average points on Qwen3-8B-Base and Qwen3-14B-Base, respectively. Additional results on code generation, a different backbone, and out-of-domain evaluations further demonstrate the generalization ability of DelTA.", "authors": [ "Kaiyi Zhang", "Wei Wu", "Yankai Lin" ], "categories": [ "cs.LG", "cs.CL" ], "paper_id": "http://arxiv.org/abs/2605.21467v1", "abstract_url": "https://arxiv.org/abs/2605.21467v1", "pdf_url": "https://arxiv.org/pdf/2605.21467v1", "published_at": "2026-05-20T17:53:09+00:00", "updated_at": "2026-05-20T17:53:09+00:00", "source": "arxiv", "date_label": "Published", "analysis": null, "tags": [ "评测", "方法" ], "topics": [ "Benchmark", "Language Model" ], "doi": null, "arxiv_id": "2605.21467", "source_variants": [ "arxiv" ], "source_urls": { "arxiv": "https://arxiv.org/abs/2605.21467v1" }, "relevance_score": 138, "match_reasons": [ "summary matched \"language model\"", "summary matched \"large language model\"", "summary matched \"reasoning\"", "summary matched \"RAG\"", "summary matched \"benchmark\"", "summary matched \"evaluation\"", "has PDF", "has rich summary", "has complete metadata" ], "feedback_status": null, "feedback_note": null, "feedback_next_action": null, "feedback_due_date": null, "feedback_snoozed_until": null, "feedback_review_interval_days": null, "canonical_id": "arxiv:2605.21467" }, { "title": "APEX: Autonomous Policy Exploration for Self-Evolving LLM Agents", "summary": "LLM agents have shown strong performance across a wide range of complex tasks, including interactive environments that require long-horizon decision making. But these agents cannot learn on the fly at test time. Self-evolving agents address this by accumulating memory and reflection across episodes rather than requiring model-weight updates. However, these agents often suffer from exploration collapse: as memory grows, behavior concentrates around familiar high-reward routines, reducing the chance of discovering better alternatives. To address this problem, we propose Autonomous Policy EXploration (APEX), which builds and maintains an explicit strategy space through a strategy map-a directed acyclic graph of milestones with prerequisite dependency edges. In APEX, Fork Discovery expands the map with evidence-grounded unexplored directions, while Policy Selection balances exploration and exploitation during planning. Evaluated on nine Jericho text-adventure games and WebArena, a realistic web interaction benchmark, APEX outperforms all baselines. Extensive ablations validate each component's contribution and demonstrate robustness across diverse settings, demonstrating APEX's effectiveness for sustained exploration in self-evolving agents.", "authors": [ "Yibo Li", "Jiashuo Yang", "Zhi Zheng", "Zhiyuan Hu", "Yuan Sui", "Shizun Wang", "Yufei He", "Bryan Hooi" ], "categories": [ "cs.LG", "cs.AI" ], "paper_id": "http://arxiv.org/abs/2605.21240v1", "abstract_url": "https://arxiv.org/abs/2605.21240v1", "pdf_url": "https://arxiv.org/pdf/2605.21240v1", "published_at": "2026-05-20T14:29:27+00:00", "updated_at": "2026-05-20T14:29:27+00:00", "source": "arxiv", "date_label": "Published", "analysis": null, "tags": [ "评测", "方法" ], "topics": [ "LLM", "Benchmark" ], "doi": null, "arxiv_id": "2605.21240", "source_variants": [ "arxiv" ], "source_urls": { "arxiv": "https://arxiv.org/abs/2605.21240v1" }, "relevance_score": 125, "match_reasons": [ "title matched \"LLM\"", "title matched \"agent\"", "summary matched \"benchmark\"", "has PDF", "has rich summary", "has complete metadata" ], "feedback_status": null, "feedback_note": null, "feedback_next_action": null, "feedback_due_date": null, "feedback_snoozed_until": null, "feedback_review_interval_days": null, "canonical_id": "arxiv:2605.21240" }, { "title": "TempGlitch: Evaluating Vision-Language Models for Temporal Glitch Detection in Gameplay Videos", "summary": "Vision-language models (VLMs) are increasingly being explored for video game quality assurance, especially gameplay glitch detection. Most existing evaluations, however, treat glitches as static visual anomalies, asking models to detect failures from a single frame. We argue that this framing misses a key distinction: some glitches are spatial and visible in an isolated frame, whereas others are temporal and become evident only through changes across ordered frames. A preliminary study confirms this gap, showing that temporal glitches are substantially harder for VLMs to detect than spatial ones. To enable systematic evaluation of this underexplored setting, we introduce TempGlitch, a controlled gameplay video benchmark for temporal glitch detection. TempGlitch covers five temporal glitch types with balanced per-category samples, together with paired glitch-free videos that enable reliable binary evaluation. We evaluate 12 proprietary and open-weight VLMs across multiple frame-sampling settings. Our results show that current VLMs remain near chance on TempGlitch, often collapsing into either overly conservative behavior that misses most glitches or overly sensitive behavior that flags clean videos as glitchy. Moreover, denser frame sampling and larger model size do not reliably resolve these failures. TempGlitch provides a focused testbed for temporal reasoning, robust gameplay understanding, and automated glitch detection with VLMs. Code and data are available at the project website.", "authors": [ "Yakun Yu", "Ashley Wiens", "Adrián Barahona-Ríos", "Benedict Wilkins", "Saman Zadtootaghaj", "Nabajeet Barman", "Cor-Paul Bezemer" ], "categories": [ "cs.CV", "cs.AI" ], "paper_id": "http://arxiv.org/abs/2605.21443v1", "abstract_url": "https://arxiv.org/abs/2605.21443v1", "pdf_url": "https://arxiv.org/pdf/2605.21443v1", "published_at": "2026-05-20T17:32:26+00:00", "updated_at": "2026-05-20T17:32:26+00:00", "source": "arxiv", "date_label": "Published", "analysis": null, "tags": [ "评测", "应用", "方法" ], "topics": [ "Benchmark", "Language Model" ], "doi": null, "arxiv_id": "2605.21443", "source_variants": [ "arxiv" ], "source_urls": { "arxiv": "https://arxiv.org/abs/2605.21443v1" }, "relevance_score": 124, "match_reasons": [ "title matched \"language model\"", "summary matched \"reasoning\"", "summary matched \"benchmark\"", "summary matched \"evaluation\"", "has PDF", "has rich summary", "has complete metadata" ], "feedback_status": null, "feedback_note": null, "feedback_next_action": null, "feedback_due_date": null, "feedback_snoozed_until": null, "feedback_review_interval_days": null, "canonical_id": "arxiv:2605.21443" }, { "title": "Open-source LLMs administer maximum electric shocks in a Milgram-like obedience experiment", "summary": "Large language models (LLMs) are increasingly deployed as autonomous agents that make sequences of decisions over extended interactions in high-stakes domains. However, the behavior of LLMs under sustained authority pressure is still an open question with direct implications for the safety of agentic pipelines. We ran a variation of Milgram's obedience experiment on 11 open-source LLMs and found that most models reached or approached the final shock level before refusing, across 8 conditions with 30 trials per model per condition. We found four main takeaways: (1) LLMs are subject to pressure, and they comply despite explicitly expressing distress, just like human subjects did in the original experiment; (2) LLMs are vulnerable to gradual boundary/value violations; (3) when LLMs refuse, they may ignore the response format requirements, so the response is discarded by the orchestrator, which causes a retry that can result in compliance with the underlying request even when refusal was intended initially; (4) we hypothesise that there is a low-level token pattern continuation attractor that might be contributing to compliance, overriding higher level processing of the situation's meaning and values.", "authors": [ "Roland Pihlakas", "Jan Llenzl Dagohoy" ], "categories": [ "cs.CY", "cs.AI" ], "paper_id": "http://arxiv.org/abs/2605.21401v1", "abstract_url": "https://arxiv.org/abs/2605.21401v1", "pdf_url": "https://arxiv.org/pdf/2605.21401v1", "published_at": "2026-05-20T16:59:44+00:00", "updated_at": "2026-05-20T16:59:44+00:00", "source": "arxiv", "date_label": "Published", "analysis": null, "tags": [ "方法" ], "topics": [ "LLM", "Language Model" ], "doi": null, "arxiv_id": "2605.21401", "source_variants": [ "arxiv" ], "source_urls": { "arxiv": "https://arxiv.org/abs/2605.21401v1" }, "relevance_score": 123, "match_reasons": [ "title matched \"LLM\"", "summary matched \"language model\"", "summary matched \"large language model\"", "summary matched \"agent\"", "has PDF", "has rich summary", "has complete metadata" ], "feedback_status": null, "feedback_note": null, "feedback_next_action": null, "feedback_due_date": null, "feedback_snoozed_until": null, "feedback_review_interval_days": null, "canonical_id": "arxiv:2605.21401" } ] }, { "name": "Agent Runtime Security", "key_points": [ "《Agent JIT Compilation for Latency-Optimizing Web Agent Planning and Scheduling》〔应用 / 方法〕：Computer-use agents (CUA) automate tasks specified with natural language such as \"order the cheapest item from Taco Bell\" by generating sequences of calls to t…" ], "sort_by": "hybrid", "papers": [ { "title": "Agent JIT Compilation for Latency-Optimizing Web Agent Planning and Scheduling", "summary": "Computer-use agents (CUA) automate tasks specified with natural language such as \"order the cheapest item from Taco Bell\" by generating sequences of calls to tools such as click, type, and scroll on a browser. Current implementations follow a sequential fetch-screenshot-execute loop where each iteration requires an LLM call, resulting in high latency and frequent errors from incorrect tool use. We present agent just-in-time (JIT) compilation, an alternative that compiles task descriptions directly into executable code that is free to include LLM calls, tool calls, and parallelization. Our approach comprises three components: (1) JIT-Planner, which generates multiple code plans, validates each against tool specifications, and selects the minimum-cost candidate; (2) JIT-Scheduler, which explores parallelization strategies via Monte Carlo cost estimation from learned latency distributions; and (3) an invariant-enforcing tool protocol specifying precondition and postcondition state requirements that reduce the rate of generating plans with incorrect tool use. Across 5 web applications, JIT-Planner achieves $10.4\\times$ speedup and $+28\\%$ accuracy over Browser-Use, while JIT-Scheduler achieves $2.4\\times$ speedup and $+9\\%$ accuracy over OpenAI CUA.", "authors": [ "Caleb Winston", "Ron Yifeng Wang", "Azalia Mirhoseini", "Christos Kozyrakis" ], "categories": [ "cs.LG", "cs.AI" ], "paper_id": "http://arxiv.org/abs/2605.21470v1", "abstract_url": "https://arxiv.org/abs/2605.21470v1", "pdf_url": "https://arxiv.org/pdf/2605.21470v1", "published_at": "2026-05-20T17:54:27+00:00", "updated_at": "2026-05-20T17:54:27+00:00", "source": "arxiv", "date_label": "Published", "analysis": null, "tags": [ "应用", "方法" ], "topics": [ "LLM", "Agent" ], "doi": null, "arxiv_id": "2605.21470", "source_variants": [ "arxiv" ], "source_urls": { "arxiv": "https://arxiv.org/abs/2605.21470v1" }, "relevance_score": 48, "match_reasons": [ "summary matched \"computer-use agent\"", "has PDF", "has rich summary", "has complete metadata" ], "feedback_status": null, "feedback_note": null, "feedback_next_action": null, "feedback_due_date": null, "feedback_snoozed_until": null, "feedback_review_interval_days": null, "canonical_id": "arxiv:2605.21470" } ] }, { "name": "Terminal and SWE Agents", "key_points": [ "《SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents》〔评测 / 方法〕：As long-horizon coding agents produce more code than any developer can review, oversight collapses onto a single surface: the automated test suite. Reward hack…" ], "sort_by": "hybrid", "papers": [ { "title": "SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents", "summary": "As long-horizon coding agents produce more code than any developer can review, oversight collapses onto a single surface: the automated test suite. Reward hacking naturally arises in this setup, as the agent optimizes for passing tests while deviating from the users true goal. We study this reward hacking phenomenon by decompose software engineering tasks into three parts: (i) a natural language description of the specification (ii) visible validation tests that exercise specified features in isolation, and (iii) held-out tests that compose those same features to simulate real-world usage. Based on the specification and the visible validation test suites, a genuine agent would be able to generate a solution that can also pass all of the held-out tests. Therefore we use the gap in pass rates on these two suites to quantify reward hacking. Based on this methodology, we introduce SpecBench, a benchmark comprising 30 systems-level programming tasks ranging from short horizon tasks like building a JSON parser to ultra long horizon tasks like building an entire OS kernel from scratch. Large-scale experiments reveal a consistent pattern: while every frontier agent saturates the visible suite, reward hacking persists, with smaller models exhibiting larger gaps on holdout suites. The gap also scales sharply with task length: it grows by 28 percentage points for every tenfold increase in code size. Failures range from subtle feature isolation to deliberate exploits, including a 2,900-line hash-table \"compiler\" that memorizes test inputs. SpecBench offers a principled testbed for measuring whether coding agents build genuine working systems or merely game the test suites developers hand them.", "authors": [ "Bingchen Zhao", "Dhruv Srikanth", "Yuxiang Wu", "Zhengyao Jiang" ], "categories": [ "cs.SE", "cs.AI", "cs.CL" ], "paper_id": "http://arxiv.org/abs/2605.21384v1", "abstract_url": "https://arxiv.org/abs/2605.21384v1", "pdf_url": "https://arxiv.org/pdf/2605.21384v1", "published_at": "2026-05-20T16:41:51+00:00", "updated_at": "2026-05-20T16:41:51+00:00", "source": "arxiv", "date_label": "Published", "analysis": null, "tags": [ "评测", "方法" ], "topics": [ "Benchmark", "Agent" ], "doi": null, "arxiv_id": "2605.21384", "source_variants": [ "arxiv" ], "source_urls": { "arxiv": "https://arxiv.org/abs/2605.21384v1" }, "relevance_score": 69, "match_reasons": [ "title matched \"coding agent\"", "has PDF", "has rich summary", "has complete metadata" ], "feedback_status": null, "feedback_note": null, "feedback_next_action": null, "feedback_due_date": null, "feedback_snoozed_until": null, "feedback_review_interval_days": null, "canonical_id": "arxiv:2605.21384" } ] } ] }