Benchmarking LLM Agents on Multi-Modal Biomedical Machine Learning Tasks

NeurIPS 2026 · Evaluations & Datasets Track

Loka Li1, Duzhen Zhang1*, Xingbo Du1, Leonard Song1, Zixiao Wang1, Assanali Aukenov1, Noel Thomas1, Shakhnazar Sailaukan1, Yonghan Yang1, Feilong Chen2, Jiahua Dong1, Kun Zhang1,3, Bin Zhang1, Le Song1*

1Mohamed bin Zayed University of Artificial Intelligence    2University of Chinese Academy of Sciences    3Carnegie Mellon University

*Corresponding authors

Paper Leaderboard AutoEvaluator GitHub Hugging Face

Abstract

Large language model (LLM) agents increasingly automate machine-learning model building, yet biomedical benchmarks emphasize question answering, reasoning, and tool use or cover narrow slices of biomedical ML coding. We introduce BioXArena, a biomedical machine learning (BioML) coding benchmark for task-specific model building on heterogeneous, often multi-modal datasets. It contains 76 end-to-end tasks across 9 domains: sequence, single-cell, structure, network biology, chemical biology, perturbation dynamics, phenotype–disease, imaging, and text-integrated tasks. Each task packages primary-source data with hidden labels, held-out graders, and biology-aware metrics on a unified 0-to-100 scale; agents write runnable code, train models, and predict private test samples. BioXArena emphasizes realistic data interfaces: most tasks require agents to integrate multiple input sources, including 46 tasks that combine distinct modality families. We compare 18 harness–backbone configurations: general-purpose coding, biomedical, and ML agents plus our Minimal runner, all under the same controlled resource configuration. Codex with GPT-6 Astra achieves the highest task-level average score of 73.91 among 18 configurations, closely followed by Claude Code with Opus 5.5 (73.79). Matched-backbone comparisons show higher observed scores for mature coding systems than for Minimal; seven Minimal backends characterize model differences under a fixed runner. Beyond the main leaderboard, ablations, robustness checks, and scaling, runtime, and failure-mode analyses characterize how backbones, scaffolds, budgets, and domains affect BioML coding performance, exposing modeling and execution gaps across agent and LLM backbone families.

Figures

Figure 1: Overview of BioXArena
Figure 1: Overview of BioXArena. (a) Tasks are curated from journals, conferences, and public databases by ML and biology experts, then packaged as unified public task capsules with hidden private labels and graders. (b) The benchmark has 76 tasks across 9 biomedical ML domains. (c) The 18 configurations comprise seven Minimal runners, seven generic coding-agent configurations (OpenCode, Codex, and Claude Code), two biomedical agents, and two ML coding agents. Subscripts identify the backend LLMs. (d) All agents run under the same controlled resource configuration and submit a submission.csv to held-out task-specific graders. (e) Nine evaluation metrics feed the leaderboard, domain heatmaps, failure taxonomy, and runtime and token analysis across all 18 agents.
Figure 2: Main comparison of 18 configurations
Figure 2: Main comparison of 18 harness–backbone configurations. (a) Successful and failed runs sum to 76. Failure categories: No sub., missing output without confirmed timeout; Timeout, recorded time-limit termination; ID mm, missing or mismatched identifiers or row order; NaN/Inf, non-finite predictions; Constant, constant regression predictions with undefined correlation; Format, invalid output columns, shapes, or types. (b) Penalized domain scores, Task Avg (equal task weights), and Domain Avg (equal domain weights); failures score zero. Backbone subscripts: g56s = GPT-5.6-sol, g6a = GPT-6 Astra, o46 / o5 / o55 = Claude Opus 4.6 / 5 / 5.5, s5 = Claude Sonnet 5, ge = Gemini-3.1-Pro, glm = GLM-5.1, qw = Qwen3.6-Plus, gm = Gemma-4-31B, dv3 / dv4 = DeepSeek-V3.2 / V4-Pro. See the interactive leaderboard for per-task scores.
Figure 3: Fixed LLM backbone ablation study
Figure 3: Fixed LLM backbone ablation study over different agent scaffolds. All agents use DeepSeek-V3.2 as backbone. Panel (a) averages successful-only tasks, describing predictive quality conditional on valid evaluation; panel (b) averages all 76 tasks with failed tasks scored zero; and panel (c) shows success/failure categories. The Minimaldv3 row denotes the Minimal code-extract–run–repair baseline; the other rows use Biomni, STELLA, MLMaster2, or MLEvolve with the same backbone. Configurations share the task interface, hardware, and evaluator.

Citation

@inproceedings{li2026bioxarena,
  title={{BioXArena: Benchmarking LLM Agents on Multi-Modal Biomedical Machine Learning Tasks}},
  author={Li, Loka and Zhang, Duzhen and Du, Xingbo and Song, Leonard and Wang, Zixiao and Aukenov, Assanali and Thomas, Noel and Sailaukan, Shakhnazar and Yang, Yonghan and Chen, Feilong and Dong, Jiahua and Zhang, Kun and Zhang, Bin and Song, Le},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS), Evaluations and Datasets Track},
  year={2026},
  url={https://arxiv.org/abs/2605.15766}
}