How I Debugged a Broken Vertex AI Image Pipeline and Finally Made It Reliable

Last week, the author encountered significant issues with their AI infrastructure when trying to generate images through Vertex AI for a content pipeline. They experienced several classes of errors including "invalid_grant," "403 Permission denied," and "429 RESOURCE_EXHAUSTED." The author realized that most failures in AI workflows are not model issues, but rather credential, project, or policy failures. The author followed a step-by-step process to debug and ultimately fix the broken Vertex AI image pipeline.

The first step was to identify the canonical script that the team relied on, in this case, "~/clawd/ops/production/scripts/generate_panels.py." This script became the source of truth, ensuring that all subsequent debugging efforts were based on accurate information. The author discovered that the project was still hardcoded to an old project ID, and once this was corrected, the pipeline began to function properly.

To separate free-tier failures from paid-route failures, the author divided the workflow into two distinct systems: the Gemini API / AI Studio free tier and the Vertex AI paid route. This separation allowed for more focused and efficient debugging, as free-tier failures typically result in quota errors, while Vertex AI failures are usually related to IAM or service account issues.

The author then verified the service account by checking whether it could mint a token. This step was crucial in determining whether the problem lay in authentication, project permissions, or model invocation. By using Python's "google.oauth2.service_account" library and attempting to refresh the credentials, the author was able to quickly identify whether the issue was related to authentication or not.

Another significant obstacle was the organization policies that prevented the creation of a new service account's JSON key. The author discovered that the policy "iam.disableServiceAccountKeyCreation" was causing the issue, which was not immediately obvious from the user interface. To resolve this, the author created a clean personal project without inherited organization policy baggage, speeding up the debugging process.

Finally, the author set up a fresh Vertex project with a new service account, a new JSON key, and updated the canonical script to reflect the new project ID. Once this setup was successful, the author confirmed that the image pipeline was working reliably. The author emphasizes that while model performance is crucial, the real bottleneck often lies in identity, permissions, quotas, and project hygiene.

In conclusion, the author suggests creating a repeatable checklist for debugging Vertex AI image pipelines, focusing on the canonical script, the actual project being hit, the credential's ability to mint a token, free-tier vs. paid route separation, organization policies, and generating one successful image before considering the pipeline fixed.