Quantum computing may be facing a replication crisis

A new analysis suggests that the majority of scientific papers reporting advancements in quantum computing may not be replicable, potentially undermining the credibility of the industry's achievements. Quantum computers have the potential to solve specific problems that conventional computers cannot, but their practical applications remain uncertain.

The field is rapidly evolving, with researchers exploring various use cases, such as simulating molecules and optimizing logistics. However, for these applications to be widely used, quantum computers must reliably perform the same tasks across different machines, which necessitates reproducibility.

Researchers from the Technical University of Applied Sciences Regensburg in Germany evaluated thousands of papers on quantum computing, finding that only 24.4% of the 127 manually analyzed papers included executable code, and only 64.5% of that code successfully ran. The automated analysis of 4,966 papers yielded similar results, with only 26.8% providing enough information for potential replication.

Compared to traditional computer science studies, this low reproducibility rate is concerning, according to Wolfgang Mauerer, one of the researchers.

The variability and inconsistency of current quantum computing hardware may be a significant factor contributing to this issue. Unlike conventional computers, where researchers can abstractly write code without concerns about hardware changes, quantum computers are prone to variability, such as fluctuations in physical characteristics. Mauerer emphasizes that this inherent variability in quantum computers makes reproducibility more challenging.

William Zeng, from the Unitary Foundation, suggests that the issue may be partly due to the lack of community standards and the need for better software maintainers. He believes that community efforts and the use of AI agents for coding could improve the situation in the future. Fred Chong, from the University of Chicago, remains relatively unfazed by the analysis, attributing the lack of reproducibility to the field's infancy and rapid changes. He suggests that reproducibility will become increasingly important as the field matures.

Despite the negative findings, researchers involved in the study remain optimistic and hopeful. They emphasize the importance of starting to consider reproducibility from the beginning of each experiment and provide a template for creating a reproducibility package to help researchers in this endeavor.