Audit your AI forecast dataset before calling it a benchmark

An AI consensus dataset can easily be mistaken for a benchmark because it contains scores, multiple advisor perspectives, forecast horizons, and enough rows to create an impressive chart. However, before determining which forecast performed best, it is crucial to ask a fundamental engineering question: what does a single row represent, and what evidence does it actually contain?

This small audit of a public release from iPulse AI, an Open Agentic Investment Research Platform developed by Future Edge Group, provides a step-by-step guide to conducting such an audit.

To begin, identify the unit of observation. In this example, each row describes an asset's stored consensus output for a specific snapshot and forecast horizon. It is essential to understand that this is not an individual advisor's forecast, a realized return, or an independent trading experiment. After auditing all 746 records in the historical consensus snapshot dataset, the audit found several important results.

First, there were exactly 746 records and 746 distinct record IDs. Next, there were 373 records for one-year and five-year horizons, but no missing values in the six required metadata fields. Additionally, no records had a model_count of 1, and all records were marked as "not_applicable_not_evaluated."

These findings matter more than a polished leaderboard, as the release records forecasts without providing a completed performance evaluation. Moreover, the model count should raise caution when treating different advisor perspectives as independent models. This public release is a bounded metadata audit and does not validate the source data, forecast quality, or every field in its schema.

To perform this audit, you can use the following Python example, which utilizes only the standard library and the public Hugging Face viewer API. The script reads the snapshots configuration's train split in pages, starting with the dataset name, 'train' split name, and a few other parameters. By executing this script, you can check the presence of checksum fields, record distinct voices as independent evidence, and count the number of perspectives.

However, it is crucial to remember that shared models, evidence, prompts, and aggregation rules can create correlated errors. Therefore, it is essential to declare the experimental unit first and preserve those identities throughout the evaluation.