Running Coding Agents in Parallel with Git Worktrees

Coding agents often run into issues when trying to execute multiple tasks simultaneously within the same repository. The problem arises because they are using the same working directory and checked-out branch, leading to conflicts as they both attempt to modify the same files. A solution to this issue lies within Git's worktree feature, which has been available for years.

By using git worktree, you can create multiple working directories that are backed by the same repository. Each worktree has its own branch, allowing agents to work independently without interfering with each other. For example, you can create separate worktrees for integration, login, and payments tasks, each with its own branch and directory.

To set up a worktree, you first create it using the command git worktree add, followed by specifying the path, branch name, and the main branch. For instance, git worktree add ../integration -b integration main would create a worktree named integration with the main branch. You can then repeat this process for other tasks, such as login and payments, using their respective branches.

Once you have created the worktrees, each agent can work on their respective tasks in their own folder. This way, they don't step on each other's files, and Git is aware of all the branches locally. The agents can commit their changes, merge them into the integration worktree, and run tests without the need for git push or git pull.

To clean up after a task, you can use git worktree remove to delete the worktree folder and Git's internal registry, and git branch -d to delete the branch. If you accidentally remove a worktree or branch, you can use git worktree prune to fix the bookkeeping.

GitHub comes into play when humans enter the loop, as it provides communication mechanisms like pull requests, reviews, CI, and an audit trail. You can create a pull request from the feature branch, and if you want integration to merge exactly what's on the remote branch, you can use git fetch origin and git merge origin/feature/login.

This approach separates the local development process from the remote collaboration process. Locally, agents work with worktrees, commit changes, and merge them into the integration branch. Remotely, they push their changes to GitHub, open pull requests, review code, and integrate changes through CI. The agents never need to interfere with each other's work, and the integration agent remains isolated from the other agents' folders and branches.