Multi-Turn Reinforcement Learning on Amazon SageMaker AI: A hands-on guide

Multi-Turn Reinforcement Learning (MTRL) on Amazon SageMaker AI allows training models to handle complex, multi-step tasks by providing feedback on the entire sequence of actions rather than just a single output. This technique is particularly useful for agentic tasks, where an agent must perform a series of actions that depend on previous steps. MTRL cannot utilize supervised fine-tuning labels or single-turn reinforcement learning feedback, as it focuses on rewarding the entire episode's outcome.

Amazon SageMaker AI trains models by sending prompts to an agent hosted on Amazon Bedrock AgentCore or on customer infrastructure like Amazon Elastic Kubernetes Service (Amazon EKS), Amazon Elastic Compute Cloud (Amazon EC2), or AWS Fargate via an AWS Lambda forwarder. The agent interacts with the policy model being trained, acting over multiple turns required by the task before reporting a reward.

The example provided in this post is Wordle, a game where players guess a hidden five-letter word within six attempts, receiving feedback on letter positions after each guess. This game is ideal for MTRL due to its few rules, dependency on previous feedback, and objective scoring.

The training process involves deploying the agent and reward function to AgentCore, preparing the dataset, and running the training job on SageMaker AI. The agent calls the policy model through an OpenAI-compatible endpoint, playing up to six guesses while receiving per-letter feedback and a summary of known information. When the game ends, the agent returns a reward based on the outcome.

The reward system in MTRL is outcome-dominant, with positive rewards for solving the task quickly and negative penalties for making invalid guesses or failing to solve the task.

The solution overview describes the system as having two main components: the SageMaker AI training job and the AgentCore-hosted agent that manages the Wordle environment and reward function. The training job invokes the AgentCore runtime, which plays the game using the policy model and reports the reward. GRPO (Group Relative Policy Optimization) updates a Low-Rank Adaptation (LoRA) adapter on the policy model using the spread of rewards across several rollouts of the same word.

Some key design points for MTRL include having the agent and reward function in one place, with a single Python file deployed to AgentCore holding the environment, tool, and scoring function. The dataset only contains the secret word, while Wordle lists are used only for scoring. The reward is outcome-dominant, with positive rewards for solving the task quickly, penalties for making invalid guesses or failing to solve the task, and small shaping terms to encourage good strategies within each tier.

MTRL is serverless, with the service provisioning the training and sampling infrastructure based on the chosen base model. Currently, it supports GPT-OSS-20B, GPT-OSS-20B, Gemma 4 31B (instruction-tuned), and Qwen 3.6 27B, with GPT-OSS-20B chosen for its low per-token price and controllable reasoning effort. The total cost for training was approximately $339, dominated by the train dimension, which scales with trajectory length. The AgentCore runtime also charges separately for vCPU and memory usage.