evaluation

"lm-evaluation-harnessを使えば、60以上の学術ベンチマークを統一コマンドで実行でき、YAMLファイル1つで自作ベンチマークも追加できます。"

"Now it feels like training is just one piece, and everything around it is where most of the difficulty is."

"A model can look great on benchmarks but still fail in actual usage."

"All models consistently identify the evaluation as flawed and attribute the reported performance to non-independent data partitioning, supported by indicators such as overlapping learning curves, minimal generalization gap, and near-perfect classification results."

"MemGround introduces a three-tier hierarchical framework that evaluates Surface State Memory, Temporal Associative Memory, and Reasoning-Based Memory through specialized interactive tasks."

"Conventional evaluation protocols may therefore overestimate generalization and clinical utility, highlighting the need for strictly speaker-independent evaluation."

"This benchmark tests whether large language models can infer a specific latent theme from a few examples, use anti-examples to reject the broader but wrong pattern, and then identify the one true match among close distractors."

"Analysing these logs can help understand model capabilities, propensities, and behaviours, or assess whether an evaluation worked as intended."

"This time, we are using KokushiMD-10, a preprint released in June 2025, which organizes 10 types of Japanese national examinations in medical and related fields as an evaluation dataset for LLMs."

"Bridgebench is accusing Anthropic of last week Claude Opus 4.6 ranked #2 on the Hallucination benchmark with an accuracy of 83.3%. Today Claude Opus 4.6 was retested and it fell to #10 on the leaderboard with an accuracy of only 68.3%."

"https://www.aisi.gov.uk/blog/our-evaluation-of-claude-mythos-previews-cyber-capabilities"

"The condition for self-evolution to function is that the evaluator is independent from the generator."

"3指標の一言まとめ Recall@K → 正解が「網に入ったか」（網羅性） MRR → 正解が「何位に来たか」（順位精度） キーワードヒット率 → 取得チャンクの「中身が揃っているか」（内容充実度）"

"I've spent months building a diagnostic method for large language models. It catches what standard benchmarks miss - distributional collapse inside tensors, not just loss or perplexity."

"We additionally study Pass@1 after feedback-based repair, where a model may revise code after a runtime error or wrong answer. Across frameworks, the strongest one-shot scores reach 59.5% in Qiskit, 54.8% in Cirq, and 42.9% in PennyLane; with feedback-based repair, the best scores rise to 83.3%, 76.2%, and 66.7%, respectively."

"How to use a Held-out Test Set after 5-Fold Cross-Validation in Deep Learning?"

"We built an automated scanning agent that systematically audited eight among the most prominent AI agent benchmarks [...] and discovered that every single one can be exploited to achieve near-perfect scores without solving a single task."

"The real breakthrough is not in the model itself, but in the structure, including clear JSON functional specs, enforced testing mechanisms, progressive commits with trackable progress, and an initialization script that ensures every session starts from a working application."

"Yupp's design transforms human judgment into a renewable economic resource. As new interactions emerge, data 'expires,' creating a virtuous cycle: more use leads to more timely evaluations; more timely evaluations lead to better models; better models attract more users."

"Evaluating Netflix Show Synopses with LLM-as-a-Judge"

"Prompt engineering is evolving from "how to write a good instruction" into an engineering discipline that supports production LLM applications... prompt design is shifting its focus from "ingenuity of the prompt alone" to "design of the entire information environment.""

"Anthropic has added evaluation and benchmarking features to the 'skill-creator' tool for creating Agent Skills, allowing skill creators to measure and verify the operation of skills through code."

"If you’ve ever trained a model that performed perfectly on your dataset but failed miserably in the real world, this quick visual guide shows why it happens and how concepts like generalization, loss functions, and evaluation metrics help you build models that actually work outside your training data."

"LLMs tend to have unstable quality in riddle generation. Previous studies have also reported that riddles are a highly difficult task that simultaneously requires common sense reasoning, metaphorical understanding, and counterfactual reasoning."

"In this way, rather than a pure performance evaluation of the language model, this could potentially lead to an evaluation from the perspective of how much the language model can closely relate to humans."

"This article introduces a comparison template by use case, 12 evaluation indicators usable in budget proposals, and a roadmap from PoC to full-scale development to solve the dilemma of 'wanting to introduce Generative AI tools but not knowing how to compare them to get approval.'"

"AZARS evaluates employees on a unified standard regardless of whether they are engineers or not, measuring the company's overall AI utilization capability."

"In 4326 search samples, the proportion of accurate AI Overviews provided by Gemini 2 last October was about 85%, and after upgrading to Gemini 3 in February this year, this proportion increased to 91%."

"We find that positively- and negatively-framed pairs are significantly more likely to produce contradictory conclusions than same-framing pairs."

"To address this, I developed SUT‑XR, an external evaluation framework for AI explanations. This is not a method for improving the AI itself, but a framework for managing the quality of its explanations."