Black-box technologies could undermine confidence in scientific findings

Scientists are increasingly relying on complex data sources and tools, but these often operate as "black boxes" - systems where the processes behind the results are largely hidden. This lack of transparency is a growing concern according to a new study by an international team of scientists. Technologies such as artificial intelligence (AI), satellite imagery, online data, and digital sensors are transforming scientific research, enabling the analysis of massive data sets and revealing patterns that were once unattainable.

However, many of these tools are controlled by private companies that limit access to information about how they operate or process data, often driven by commercial interests. The study identifies several types of black boxes prevalent in ecology and conservation, including large language models and other AI technologies, which are used to analyze data, interpret satellite imagery, and model ecosystems.

These tools often lack transparency in terms of the data used to train them, the underlying algorithms, direct system testing, or an understanding of how they generate specific outputs. This issue extends beyond AI to include remote sensing products and wildlife tracking devices, which may provide processed information while withholding raw data.

Additionally, online platforms that study biodiversity and human interactions with nature rely on hidden algorithms and policies that can introduce unknown biases. The study also highlights how social surveys are affected by private companies handling participant recruitment and survey management, often without providing information on selection processes, data quality, or potential interference by AI agents.

The growing dependence on these black-box technologies is further exacerbated by pressures in the scientific community to increase productivity and adapt to the abundance of data and urgent environmental challenges. This trend could undermine the reproducibility of scientific findings, as key analytical steps may become unverifiable if they cannot be inspected or repeated.

The authors of the study recommend several solutions to address these issues, such as prioritizing open-source software and hardware, benchmarking proprietary tools against transparent data sets, comparing results across multiple methods, carefully documenting training data, pipelines, versions, settings, and tool limitations, and enhancing awareness of the problem.

They stress that human oversight should remain central throughout the research process, as scientists must take responsibility for any errors and uncertainties arising from the use of black-box tools. The researchers also advocate for intensified efforts toward open science, including regulations to improve researchers' access to digital platforms and their underlying data.

However, they caution that some black boxes may remain resistant to these solutions, and scientists should remain cautious about the potential trade-offs and risks associated with their uncritical adoption.