Protecting Scientific Intent in AI-Enabled Labs by Not Letting AI Set the Mission

AI is a valuable tool, not a replacement for human scientists. While AI can handle the execution of experiments and analysis, humans remain essential for framing objectives, interpreting results, setting constraints, and governing risks. In the future, scientists will lead AI, rather than AI leading scientists. For instance, an AI could execute an experiment, while scientists are responsible for framing objectives, interpreting results, setting constraints, and governing risks.

However, the paper suggests that any actions not explicitly stated in the protocol should require human approval. This approach could elevate humans to setting destinations, similar to autonomous vehicles, which have a lower crash rate per mile traveled than human drivers. In this scenario, human scientists and trainees would work under minimal hierarchy, with the hands-on work largely automated.

With AI and lab automation, human errors are expected to decrease. According to a study from Nature, approximately 70% of scientists could not replicate other people's experiments, and half could not replicate their own. AI can improve this reproducibility by precisely duplicating details that often lead to challenges. However, it's unclear how AI involvement should be reported in regulatory filings.

While there is a low risk of AI escaping its constraints and causing physical harm, such as designing and manufacturing a virus, this risk remains low due to the multiple layers of human intervention needed to manufacture a product. Strong regulations, safety and efficacy standards, and digital safeguards like track and trace and the FDA's 21 CFR Part 11, as well as real-time monitoring, periodic inspections, and quality control activities, should help manage variations during manufacturing.