Can an AI Impact Receipt Be Standardized?
Here's the problem with building a useful developer tool: If only one company can use it, it's a product feature. If different systems can produce and consume the same information, it has the potential to become infrastructure. That's the question I've been thinking about while developing the Impact Receipt concept at CarbonLayer. In Part 3, What Should an AI Impact Receipt Actually Contain?, we…
The Impact Receipt concept, developed by CarbonLayer, aims to standardize AI impact data across different systems. This includes workload information, model identity, latency, environmental metrics, methodology, provenance, and uncertainty. However, identifying useful fields is just the beginning. The challenge lies in handling differences in how AI providers describe the same concept, different carbon values calculated using various boundaries, and cases where providers lack enough evidence to calculate a value.
The key issue here is standardizing the interface of an AI Impact Receipt without standardizing misleading assumptions. It's not about forcing every provider to calculate environmental impact in the same way, as different providers have different hardware, workloads, measurement systems, data sources, and accounting boundaries. The goal should be to agree on how to describe what is known, what is unknown, and how the knowledge was obtained.
A proposed structure for an Impact Receipt includes a status, value, unit, evidence type, and methodology. For instance, a value of 14.7 grams of CO2e could be represented as: { "carbon": { "status": "available", "value": 14.7, "unit": "gCO2e", "evidenceType": "modeled", "methodology": { "name": "example-method", "version": "1.0" } } }. This structure allows receiving applications to distinguish the value, unit, evidence type, and methodology.
However, standardizing JSON doesn't guarantee comparable results. Different providers might return data in different formats, such as Provider A returning 14.7, Provider B returning 0.0147, and Provider C returning { "co2e_g": 14.7 }. Even if all providers use the same JSON structure, the underlying accounting boundaries might differ, leading to potentially non-comparable results.
A critical aspect of an Impact Receipt is acknowledging when there's insufficient evidence to calculate a value. Instead of returning a precise-looking number, the system could return a status indicating that the value is unavailable. This approach ensures that zero is not treated as a value, but rather as an evidence state.
Furthermore, it's important to distinguish between different types of evidence in an Impact Receipt. Measured values are supported by documented measurement processes, modeled values are estimated using assumptions or models, derived values are calculated from other inputs, and unavailable values indicate a lack of sufficient support. Each of these categories should be treated as distinct and not interchangeable.
Finally, methodology and versioning play a crucial role in interpreting trends in environmental impact data. Changes in methodology can significantly affect the results, and without versioning, it can be challenging to determine whether improvements or declines are due to changes in the workload, infrastructure, accounting boundaries, or methodology itself. Including methodology versioning in an Impact Receipt can help clarify these nuances and provide a clearer picture of environmental impact trends.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
