Two Dojo Tracks: How does model weight adjustment with RL differ from discipline adjustment with Harness?

Two distinct approaches exist in the world of training AI agents: Reinforcement Learning Dojo (RL Dojo) and Harness Dojo. RL Dojo focuses on adjusting the model's weight through reinforcement learning, while Harness Dojo modifies the model's behavior without altering its weight. This article examines the differences between these two methods, assisting teams in selecting the most suitable approach for their needs.

RL Dojo trains the agent's model through reinforcement learning, which involves creating an environment that rewards the agent for each step. This method, exemplified by Meta's MLE-Dojo project, adjusts the model's parameters based on the agent's performance in the environment. The process includes fine-tuning and self-play training. However, RL Dojo comes with high costs, requiring large amounts of data and specialized personnel to manage the environment setup, training runs, and evaluation.

On the other hand, Harness Dojo modifies the model itself by changing its instructions, policies, environment, and memory units when the agent makes mistakes during training. An example of this method is the copilot-agents-dojo project, which trains agents to follow specific guidelines, best practices, and rules when working on code.

In this approach, the model remains unchanged, and mistakes are treated as new rules for future training. Harness Dojo is more affordable and accessible than RL Dojo, but it may lead to a loss in model performance if the guidelines are not kept up-to-date.

When deciding between RL Dojo and Harness Dojo, teams should consider their budget, team size, and the desired performance of the trained model. Teams with substantial resources and in-house expertise should opt for RL Dojo to achieve the best performance. However, smaller teams may prefer Harness Dojo to balance cost-effectiveness and model performance.

As RL Dojo and Harness Dojo are complementary approaches, many teams combine both to create a hybrid training strategy. Upcoming articles will delve deeper into the generalization gap between these two methods, providing further insights for teams to make informed decisions.