Harbor Adds Docker Sandboxes to Run Reproducible AI Agent Evals in Isolated MicroVMs
Docker announced that Harbor, the open-source agent evaluation framework, now works with Docker Sandboxes. This change allows each evaluation task to run in its own microVM. So, instead of sharing a container, tasks are isolated for better performance By Claudio Masolo
Docker has added Docker Sandboxes to its open-source agent evaluation framework, Harbor. This update allows each evaluation task to run in its own isolated microVM, improving performance and reproducibility. Tasks can be run locally or in Docker Cloud Sandboxes, with up to 64 concurrent tasks tested in the cloud. Harbor manages the orchestration of these tasks, collecting scores and logs, while supporting benchmarks like Terminal-Bench.
Docker Sandboxes provide a consistent execution environment, with each task running in its own microVM with its own kernel, preventing interference with the host, other tasks, or external resources. The integration is currently available in the nightly release of Docker Sandboxes and will be released to stable on October 23. Teams can use the sbx extra to install Harbor, and a personal access token with Cloud Sandboxes permission is required for cloud runs.
Harbor supports Docker Compose for multi-container tasks within a single cloud sandbox. While the integration does not provide benchmark data on overhead, cost, or score variance, it does highlight the potential for higher limits on request. Teams sensitive to latency or cost will need to measure these factors themselves. Docker's addition of Docker Sandboxes to Harbor expands the range of agent runtime integrations and horizontal-scaling options, as seen with other providers like Daytona, Modal, E2B, Runloop, Tensorlake, LangSmith, Blaxel, Novita Sandbox, EC2, and Beam.
Written by urgent.news from InfoQ's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
