AutoSynthData: Generating Training Data for Enterprise Agents
Businesses require agents capable of operating within their specific environments. These agents must account for the unique systems, rules, and data structures they encounter. Even highly capable models can struggle in certain environments due to workflow limitations, improper tool usage, or unmet constraints. Improving these weaknesses requires transforming them into effective training data.
However, generating new tasks that exercise a model's capabilities in various scenarios can be challenging. To address this, ServiceNow CoreAI developed AutoSynthData, a system that leverages a target model's shortcomings and a superior teacher's successes to generate new training materials. AutoSynthData operates within an agentic environment, which defines the observable state, modifiable elements, invoked tools, and state transitions produced by the agent's actions.
A task is a specific instantiation within this environment, guided by a system specification that outlines constraints such as system instructions, environment policies, and task-specific initialization. The system specification must align with the environment's tools, state, and supported actions, while the user prompt specifies the desired outcome of the agent, along with any user-level constraints.
For a task to be valid, it must be feasible (i.e., achievable within the current environment), realistic (i.e., resembling a plausible user request), and difficult enough to provide new training signal (i.e., not already solved reliably by the agent). AutoSynthData uses the target model's failures and a stronger teacher's successes to determine the next learning objectives for the model.
The system then generates and validates new tasks that target these capabilities. As the model improves, the curriculum adapts to focus on areas where the model still struggles. The pipeline is illustrated using EnterpriseOps Gym, a dataset released by Malay et al. (2026). The pipeline begins by defining the environment in which the agent operates and what constitutes a useful training task.
An agentic environment encompasses the world in which the agent functions, including the observable state, modifiable elements, invoked tools, and state transitions generated by the agent's actions. A task is defined within this environment. The system specification establishes the constraints under which the agent operates, including system instructions, environment policies, and task-specific initialization.
The system specification must be compatible with the environment's tools, state, and supported actions, while the user prompt specifies the desired outcome of the agent, along with any user-level constraints. The generation of tasks must satisfy three key properties: feasibility (i.e., the existence of at least one trajectory in the current environment that satisfies the user prompt while adhering to the system specification), realism (i.e., the user prompt resembling a request that a user would plausibly make within the target environment), and difficulty (i.e., the task exposing a weakness in the current agent's capabilities, thereby providing new training signal).
The verifier component of AutoSynthData determines whether a generated trajectory successfully completes the task. The verifier must satisfy three properties: consistency (i.e., agreement with the user prompt, system specification, and task-specific environment state), soundness (i.e., rejecting trajectories that fail to meet the task criteria or violate relevant constraints), and completeness (i.e., accepting valid solutions rather than encoding a single reference trajectory).
By evaluating the target model and a stronger teacher in the target environment, AutoSynthData identifies which capabilities the model needs to learn next. These findings are distilled into sanitized capability specification cards, which are provided to the generator. The generator uses these cards to create new tasks with varying prompts, states, and solution paths, ensuring that a diverse set of training data is produced.
The verification process involves the stronger teacher providing successful trajectories for each task, which are then used to fine-tune the target model through supervised fine-tuning (SFT). This process involves generating a core set of training samples based on the capability specifications, with workers generating independent tasks in parallel.
Each generated task contributes to the overall training dataset, which is then expanded into novel variants. By iteratively generating, validating, and refining tasks, AutoSynthData effectively converts capability gaps into high-quality training data, allowing businesses to improve the performance of their enterprise agents within their specific environments.
Written by urgent.news from Hugging Face's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.