Nethobench: Forecasting design and conditioning shape the structural realism of generated neural dynamics

Forecasting models are being utilized to model neural population activity, however, current benchmarks generally assess them based on the pointwise predictive accuracy when compared to unseen observations. In systems where there is a degree of randomness and incomplete observation, aligning with a single observed scenario does not guarantee the maintenance of the statistical and dynamic properties of the system.

To address this, we present Nethobench, a benchmark designed to assess structural compatibility across five categories of metrics: distributional, temporal-spectral, relational, geometric, and state-dynamical. To verify the effectiveness of Nethobench, we employed targeted modifications and stochastic systems with known dynamics.

Furthermore, we utilized empirical references to aid in the interpretation of results. The benchmark was applied to datasets derived from mouse calcium imaging, mouse Neuropixels recordings, and human intracranial recordings, with empirical references used to calibrate the interpretation of the findings. Nethobench was able to identify specific structural deficiencies and assign high scores to independent samples from the correct synthetic process, even when the trajectories did not match.

It also monitored structural alterations during the training phase and when transitioning between different domains. Model comparisons revealed the impact of various factors such as forecasting approach, network architecture, training methodology, and loss function on the structural characteristics of the models. Additionally, improvements in performance were observed when additional behavioral information was integrated into the video data during a single held-out session.

In the Brain Treebank dataset, the retrieval of analog data exhibited better structural alignment, while a multistep GRU model demonstrated superior performance in predicting the observed data. These findings emphasize the importance of evaluating neural forecasts using calibrated structural metrics, in conjunction with pointwise accuracy, probabilistic scores, and task-specific endpoints, when comparing models across different species and recording techniques.