Turborepo in a Real Monorepo: Field Notes on Cache Misses, NEXT_PUBLIC Poisoning, and the tasks Key That Replaced pipeline

Turborepo is a task runner designed for JavaScript monorepos that builds a dependency graph of scripts, hashes their inputs, and replays cached outputs when the hash matches a previous run. This process improves efficiency by avoiding unnecessary re-execution. However, the cache's accuracy depends on the declared inputs, outputs, and environment in the turbo.json configuration file. If any of these aspects are incorrect, the Turborepo may produce wrong results, as the cache does not account for environmental changes.

The author's experience with Turborepo in a two-app, six-package monorepo highlights some key considerations. Initially, CI stopped rebuilding untouched packages, leading to faster builds. However, a production build was inadvertently pointing to the staging API, resulting in cache hits and replaying logs. This incident underscores the importance of understanding Turborepo's capabilities and limitations.

Turborepo works as a memoization layer over package.json scripts, focusing on executing tasks in an efficient manner without installing dependencies, linking workspaces, or compiling code. It relies on pnpm and tsc for dependency installation, workspaces linking, and compilation. The top-level pipeline key in turbo.json has been renamed to tasks in Turborepo 2.0, reflecting the functional change from sequential pipelines to a set of task definitions determined by dependencies.

An undeclared environment variable poses a significant risk to the cache input, as Next.js inlines NEXT_PUBLIC_* values into the client bundle at build time. This means that a cache hit can replay the incorrect environment's URLs, leading to potential issues in production environments.

Turborepo 2.0 defaults to strict environment mode, which requires tasks to see only variables listed in env or globalEnv. This change transforms silent wrong-value bugs into loud undefined errors, enhancing the visibility of configuration mistakes.

The --affected flag in Turborepo only functions as well as the git history it relies on. A shallow CI clone lacks a base commit to differentiate between runs, causing the filter to default to rebuilding everything. This highlights the necessity of maintaining a comprehensive git history to ensure the effectiveness of --affected.

Turborepo does not replace package management or workspace linking functionalities. Instead, it orchestrates tasks based on pre-existing workspace definitions, compiled code, and other tools. Understanding this division of labor is crucial for debugging, as Turborepo only manages task execution, not dependency management or compilation.

Memoization is the core principle behind Turborepo's functionality. A task is considered a memoized call to the underlying script, with the hash of source files, dependency graph, lockfile entries, and environment variables serving as its arguments. For the memoization to be reliable, the task function must be deterministic with respect to these arguments. Any incorrectly declared argument can lead to Turborepo bugs.

The renaming of the pipeline key to tasks in Turborepo 2.0 was made to clarify the configuration syntax. The old name implied a linear pipeline, which is not the case as order depends on the dependsOn attribute. Mixing up the caret (^) and regular dependency syntax can result in incorrect task scheduling, leading to deadlocks where one task waits indefinitely on another that never completes.

The most common reason for a cache miss is a change in the input hash, while a cache miss due to an empty outputs array represents a different issue. Using turbo run build --dry=json and turbo run build --summarize can help diagnose these two failure scenarios, providing clear insights into what needs to be adjusted.

A cached build shipping the wrong NEXT_PUBLIC value can occur because Next.js inlines these values into the client bundle during the build process. Since Turborepo caches based on declared inputs, if the NEXT_PUBLIC_* values change, the resulting hash will differ, causing the cache to be invalidated and the build to run again.

In summary, Turborepo is a powerful task runner for JavaScript monorepos that leverages memoization to optimize build processes. However, its effectiveness hinges on accurate configuration, careful consideration of environment variables, and understanding its limitations in relation to dependency management and compilation tasks.