Modelling dopaminergic signals associated with habit formation through temporal-difference action learning

Action-selection is governed by a blend of goal-directed and habitual processes. Habits represent reward-independent, stimulus-response links that develop when a particular action is consistently performed in a specific context, irrespective of the outcome. A well-known computational model suggests that habit formation arises from action prediction errors that arise when non-habitual actions are performed.

It has been proposed that these action prediction errors are represented by the activity of certain dopamine neurons. Recent observations have revealed that dopamine activity in the tail of the striatum exhibits a pattern akin to these action prediction errors. However, existing models focus on the changes in habits across trials, but lack a description of the time-course of action prediction errors within individual trials. This makes it challenging to directly compare these models with dopamine activity.

To address this gap, we first explain the temporal-difference action learning algorithm. This algorithm utilizes biologically-plausible mechanisms to determine how alterations in action intensity can impact prediction errors over nearly continuous time. Subsequently, we present evidence indicating that the dopaminergic data recently gathered from the tail of the striatum can be more accurately described by action prediction errors as opposed to reward prediction errors.

In conclusion, our findings provide evidence for the presence of value-free action prediction errors and the corresponding habitual behavior in dopaminergic signals.