An image API acceptance test should check pixels, not only task success

When testing an image API, it's crucial to examine not just the success of a task, but the actual pixel data and visual quality. A recent integration of Nano Banana 2.1 revealed three key checks: request acceptance, decoded output properties, and whether the resulting image is suitable for the intended layout.

The tests confirmed that the API could generate two 2K JPEG images at a 16:9 aspect ratio, four 1K WebP images in a 1:4 ratio, and one 1K PNG image with an 8:1 aspect ratio. However, it's important to note that simply accepting a request does not guarantee a useful image. The API must also verify dimensions before placing the image in a layout.

For instance, a 1K PNG with an 8:1 ratio was generated, but the actual ratio was approximately 8.32:1. This discrepancy does not necessarily mean the image is unusable, but it does highlight the need for manual review. The tests also emphasized the importance of tracking request IDs, separate counts for decoded files and tasks, and distinct records for credits shown to users, credits reserved/refunded, and the provider's invoice.

It's essential to understand that a technical success, such as an API returning a success status, does not automatically mean the image is usable. Providers like Nano Banana 2.1 must be reviewed before any submission. Always keep the invoice and quality fields unknown until verified, as an advertised API price does not equal an actual invoice.

In summary, when testing an image API, it's vital to verify both the technical success of a task and the visual quality of the resulting image. This includes checking dimensions, ratios, and layout suitability, as well as keeping separate records of credits and invoices. Only through thorough testing can we ensure that the generated images meet our expectations and requirements.