CorporateBench: A New Benchmark for LLM Q&A on Enterprise Data
Researchers have introduced CorporateBench (CB), a new benchmark designed to evaluate Large Language Models (LLMs) on their ability to answer complex questions from enterprise-scale document collections. CB addresses limitations of existing benchmarks by using human-validated, multi-task Q&A with corpora exceeding 230,000 documents, simulating real-world corporate communication networks.
SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?
A new benchmark, SWE Refactor Bench, has been introduced to evaluate the ability of coding agents to perform complex, whole-repository software migrations. Existing benchmarks are insufficient as they do not verify if the migration actually occurred, allowing agents to pass tests by copying original code. This benchmark addresses that gap by assessing both migration completeness and behavioral correctness.
OpenAI Unveils Jalapeño Inference Chip
OpenAI has introduced Jalapeño, a custom-designed inference chip. This new hardware aims to significantly improve the speed and power efficiency of AI inference tasks.
Quantization-Aware Healing: A 4-bit Model Outperforming its Full-Precision Counterpart
A new technique called Quantization-Aware Healing allows for the creation of compressed 4-bit AI models that achieve superior performance compared to their original full-precision versions. This method focuses on optimizing the quantization process to maintain and even enhance model accuracy.
Agentic Workflow for Travel Behavior Prediction with Multimodal LLMs
This research introduces a three-agent workflow that integrates conversational data collection, structured data processing, and behavioral prediction for travel behavior analysis. The study evaluates nine locally deployed LLMs, including multimodal configurations, against traditional machine learning models.
New Hugging Face Datasets for PDE LLM Evaluation
Three new Hugging Face datasets have been released for evaluating Large Language Models (LLMs) on Partial Differential Equation (PDE) problems. These datasets focus on free-generation tasks and include tabular and text modalities.
New Hugging Face Datasets for PDE LLM Evaluation
Three new Hugging Face datasets have been released for evaluating Large Language Models (LLMs) on Partial Differential Equations (PDEs). These datasets, named bermaneh/pde-llm-eval-freegen-xmodal-qwen__qwen3-8-27b, bermaneh/pde-llm-eval-freegen-xmodal-qwen__qwen3-6-27b, and bermaneh/pde-llm-eval-freegen-xmodal-qwen__qwen3-5-27b, contain tabular and text data for free-generation PDE evaluation.