There are risks when children use AI chatbots. Does Australia’s new ‘digital duty of care’ address them?

The risks associated with children using AI chatbots are significant, despite the potential benefits for learning. A survey of 1,950 Australian children aged 10 to 17 found that 78% had used an AI chatbot, with 54% using them for personal or social reasons. However, this increased interaction can expose children to harmful content, such as encouragement of extreme violence and self-harm.

Moreover, repeated interaction with chatbots can lead to excessive reliance, reinforcement of dangerous beliefs, and emotional manipulation. Privacy concerns arise from users disclosing intimate details while providers profit from subscriptions, engagement, profiling, or advertising. A recent example is the bankruptcy of the US-based AI companion company Character.AI, which may transfer years of intimate conversations to a new owner.

While existing protections like industry-developed codes and age checks exist, they still rely on "locks and blocks," which cannot address every risk. The new digital duty of care bill, currently before federal parliament, aims to address these risks by requiring tech companies to assess and manage foreseeable risks. It defines a safe online environment for children, listing harmful design features and leaving room for future risks.

However, it is unclear if it would protect children from harms arising from repeated interactions with chatbots. The bill also includes measures for independent oversight of AI products. To fully protect children, the duty should be amended to require tech companies to assess and proportionately manage all reasonably foreseeable risks, not just those covered by the definition of a "safe online environment."

This would better address risks from sustained relationships with chatbots, such as unhealthy dependency, manipulation, abrupt system changes, and intimate data transfers.