DOI: 10.3724/SP.J.1042.2018.00667

Advances in Psychological Science (心理科学进展) 2018/26:4 PP.667-677

The neural mechanisms of goal-direct and habitual learning system

The dual system model of behavior learning recognizes two distinct systems:goal-directed system and habitual system. The goal-directed system, encoding the action-outcome behavior, is associated with caudate, ventromedial prefrontal cortex (vmPFC) and orbitofrontal cortex (OFC). The habitual system, concerning stimulus-response behaviors, is associated with the putamen and supplementary motor area (SMA). Research paradigms for these two behavioral learning systems mainly include outcome devaluation and contingency degradation paradigms. A large number of studies have found a dynamic defect of goal-direct learning system and habitual learning system in obsessive-compulsive disorder, autism, addiction and other mental disorders.

Key words:goal-direct learning system,habitual learning system,caudate,putamen,stereotyped behavior

ReleaseDate:2018-05-04 05:07:49

Alvares, G. A., Balleine, B. W., & Guastella, A. J. (2014). Impairments in goal-directed actions predict treatment response to cognitive-behavioral therapy in social anxiety disorder. PLoS One, 9(4), e94778.

Alvares, G. A., Balleine, B. W., Whittle, L., & Guastella, A. J. (2016). Reduced goal-directed action control in autism spectrum disorder. Autism Research, 9(12), 1285-1293.

Apergis-Schoute, A. M., Gillan, C. M., Fineberg, N. A., Fernandez-Egea, E., Sahakian, B. J., & Robbins, T. W. (2017). Neural basis of impaired safety signaling in obsessive compulsive disorder. Proceedings of the National Academy of Sciences of the United States of America, 114(12), 3216-3221.

Ashby, F. G., Turner, B. O., & Horvitz, J. C. (2010). Cortical and basal ganglia contributions to habit learning and automaticity. Trends in Cognitive Sciences, 14(5), 208-215.

Balleine, B. W., Killcross, A. S., & Dickinson, A. (2003). The effect of lesions of the basolateral amygdala on instrumental conditioning. Journal of Neuroscience, 23(2), 666-675.

Balleine, B. W., & O'Doherty, J. P. (2010). Human and rodent homologies in action control:corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology, 35(1), 48-69.

Banca, P., Voon, V., Vestergaard, M. D., Philipiak, G., Almeida, I., Pocinho, F.,... Castelo-Branco, M. (2015). Imbalance in habitual versus goal directed neural systems during symptom provocation in obsessive-compulsive disorder. Brain, 138(3), 798-811.

Barker, J. M., & Taylor, J. R. (2014). Habitual alcohol seeking:modeling the transition from casual drinking to addiction. Neuroscience & Biobehavioral Reviews, 47, 281-294.

Boorman, E. D., O'Doherty, J. P., Adolphs, R., & Rangel, A. (2013). The behavioral and neural mechanisms underlying the tracking of expertise. Neuron, 80(6), 1558-1571.

Brovelli, A., Nazarian, B., Meunier, M., & Boussaoud, D. (2011). Differential roles of caudate nucleus and putamen during instrumental learning. NeuroImage, 57(4), 1580-1590.

Corbit, L. H., Nie, H., & Janak, P. H. (2012). Habitual alcohol seeking:Time course and the contribution of subregions of the dorsal striatum. Biological Psychiatry, 72(5), 389-395.

Culbreth, A. J., Westbrook, A., Daw, N. D., Botvinick, M., & Barch, D. M. (2016). Reduced model-based decision-making in schizophrenia. Journal of Abnormal Psychology, 125(6), 777-787.

Cushman, F., & Morris, A. (2015). Habitual control of goal selection in humans. Proceedings of the National Academy of Sciences of the United States of America, 112(45), 13817-13822.

Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011). Model-based influences on humans' choices and striatal prediction errors. Neuron, 69(6), 1204-1215

Daw, N. D., Niv, Y., & Dayan, P. (2006). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience, 8(12), 1704-1711.

De Wit, S., Barker, R. A., Dickinson, A. D., & Cools, R. (2011). Habitual versus goal-directed action control in Parkinson disease. Journal of Cognitive Neuroscience, 23(5), 1218-1229.

De Wit, S., Corlett, P. R., Aitken, M. R., Dickinson, A., & Fletcher, P. C. (2009). Differential engagement of the ventromedial prefrontal cortex by goal-directed and habitual behavior toward food pictures in humans. Journal of Neuroscience, 29(36), 11330-113308.

De Wit, S., van de Vijver, I., & Ridderinkhof, K. R. (2014). Impaired acquisition of goal-directed action in healthy aging. Cognitive, Affective, & Behavioral Neuroscience, 14(2), 647-658.

De Wit, S., Watson, P., Harsay, H. A., Cohen, M. X., van de Vijver, I., & Ridderinkhof, K. R. (2012). Corticostriatal connectivity underlies individual differences in the balance between habitual and goal-directed action control. Journal of Neuroscience, 32(35), 12066-12075.

Delmonte, S., Gallagher, L., O'Hanlon, E., Mcgrath, J., & Balsters, J. H. (2013). Functional and structural connectivity of frontostriatal circuitry in autism spectrum disorder. Frontiers in Human Neuroscience, 7, 430.

Dickinson, A., Wood, N., & Smith, J. W. (2002). Alcohol seeking by rats:Action or habit? The Quarterly Journal of Experimental Psychology B Comparative and Physiological Psychology, 55(4), 331-348.

Dolan, R. J., & Dayan, P. (2013). Goals and habits in the brain. Neuron, 80(2), 312-325.

Everitt, B. J., & Robbins, T. W. (2005). Neural systems of reinforcement for drug addiction:From actions to habits to compulsion. Nature Neuroscience, 8(11), 1481-1489.

Everitt, B. J., & Robbins, T. W. (2013). From the ventral to the dorsal striatum:Devolving views of their roles in drug addiction. Neuroscience & Biobehavioral Reviews, 37(9), 1946-1954.

Fettes, P., Schulze, L., & Downar, J. (2017). Cortico-striatal-thalamic loop circuits of the orbitofrontal cortex:Promising therapeutic targets in psychiatric illness. Frontiers in Systems Neuroscience, 11, 25.

Gęsiarz, F., & Crockett, M. J. (2015). Goal-directed, habitual and pavlovian prosocial behavior. Frontiers in Behavioral Neuroscience, 9, 135.

Geurts, H. M., & de Wit, S. (2014). Goal-directed action control in children with autism spectrum disorders. Autism, 18(4), 409-418.

Gillan, C. M., Apergis-Schoute, A. M., Morein-Zamir, S., Urcelay, G. P., Sule, A., Fineberg, N. A.,... Robbins, T. W. (2015). Functional neuroimaging of avoidance habits in obsessive-compulsive disorder. American Journal of Psychiatry, 172(3), 284-293.

Gillan, C. M., Michal, K., Robert, W., Phelps, E. A., & Daw, N. D. (2016). Characterizing a psychiatric symptom dimension related to deficits in goal-directed control. eLife, 5, e11305.

Gillan, C. M., Morein-Zamir, S., Kaser, M., Fineberg, N. A., Sule, A., Sahakian, B. J.,... Robbins, T. W. (2014a). Counterfactual processing of economic action-outcome alternatives in obsessive-compulsive disorder:further evidence of impaired goal-directed behavior. Biological Psychiatry, 75(8), 639-646.

Gillan, C. M., Morein-Zamir, S., Urcelay, G. P., Sule, A., Voon, V., Apergis-Schoute, A. M.,... Robbins, T. W. (2014b). Enhanced avoidance habits in obsessive-compulsive disorder. Biological Psychiatry, 75(8), 631-638.

Gillan, C. M., Papmeyer, M., Moreinzamir, S., Sahakian, B. J., Fineberg, N. A., Robbins, T. W., & De Wit, S. (2011). Disruption in the balance between goal-directed behavior and habit learning in obsessive-compulsive disorder. American Journal of Psychiatry, 168(7), 718-726.

Gillan, C. M., & Robbins, T. W. (2014). Goal-directed learning and obsessive-compulsive disorder. Philosophical Transactions of the Royal Society B:Biological Sciences, 369(1655), 20130475.

Gremel, C. M., & Rui, M. C. (2013). Orbitofrontal and striatal circuits dynamically encode the shift between goal-directed and habitual actions. Nature Communications, 4, 2264.

Haber, S. N. (2016). Corticostriatal circuitry. Dialogues in Clinical Neuroscience, 18(1), 7-21.

Hadj-Bouziane, F., Benatru, I., Brovelli, A., Klinger, H., Thobois, S., Broussolle, E.,... Meunier, M. (2013). Advanced Parkinson's disease effect on goal-directed and habitual processes involved in visuomotor associative learning. Frontiers in Human Neuroscience, 6, 351.

Hogarth, L., Attwood, A. S., Bate, H. A., & Munafò, M. R. (2012). Acute alcohol impairs human goal-directed action. Biological Psychology, 90(2), 154-160.

Hogarth, L., He, Z., Chase, H. W., Wills, A. J., Troisi, J., Leventhal, A. M.,... Hitsman, B. (2015). Negative mood reverses devaluation of goal-directed drug-seeking favouring an incentive learning account of drug dependence. Psychopharmacology, 232(17), 3235-3247.

Isoda, M., & Hikosaka, O. (2011). Cortico-basal ganglia mechanisms for overcoming innate, habitual and motivational behaviors. European Journal of Neuroscience, 33(11), 2058-2069.

Ito, M., & Doya, K. (2015). Parallel representation of value-based and finite state-based strategies in the ventral and dorsal striatum. PLOS Computational Biology, 11(11), e1004540.

Jahanshahi, M., Obeso, I., Rothwell, J. C., & Obeso, J. A. (2015). A fronto-striato-subthalamic-pallidal network for goal-directed and habitual inhibition. Nature Reviews Neuroscience, 16(12), 719-732.

Jonker, F. A., Jonker, C., Scheltens, P., & Scherder, E. J. (2015). The role of the orbitofrontal cortex in cognition and behavior. Reviews in the Neurosciences, 26(1), 1-11.

Just, M. A., Keller, T. A., Malave, V. L., Kana, R. K., & Varma, S. (2012). Autism as a neural systems disorder:A theory of frontal-posterior underconnectivity. Neuroscience & Biobehavioral Reviews, 36(4), 1292-1313.

Langen, M., Leemans, A., Johnston, P., Ecker, C., Daly, E., Murphy, C. M.,... Murphy, D. G. (2012). Fronto-striatal circuitry and inhibitory control in autism:findings from diffusion tensor imaging tractography. Cortex, 48(2), 183-193.

Lee, S. W., Shimojo, S., & O'Doherty, J. P. (2014). Neural computations underlying arbitration between model-based and model-free learning. Neuron, 81(3), 687-699.

Maier, S. U., Makwana, A. B., & Hare, T. A. (2015). Acute stress impairs self-control in goal-directed choice by altering multiple functional connections within the brain's decision circuits. Neuron, 87(3), 621-631.

McDannald, M. A., Lucantonio, F., Burke, K. A., Niv, Y., & Schoenbaum, G. (2011). Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning. Journal of Neuroscience, 31(7), 2700-2705.

Mole, T. B., Irvine, M. A., Worbe, Y., Collins, P., Mitchell, S. P., Bolton, S.,... Voon, V. (2015). Impulsivity in disorders of food and drug misuse. Psychological Medicine, 45(4), 771-782.

Morris, L. S., Kundu, P., Dowell, N., Mechelmans, D. J., Favre, P., Irvine, M. A.,... Harrison, N. A. (2016). Fronto-striatal organization:Defining functional and microstructural substrates of behavioural flexibility. Cortex, 74, 118-133.

Morris, R. W., Quail, S., Griffiths, K. R., Green, M. J., & Balleine, B. W. (2015). Corticostriatal control of goal-directed action is impaired in schizophrenia. Biological Psychiatry, 77(2), 187-195.

Nebe, S., Kroemer, N. B., Schad, D. J., Bernhardt, N., Sebold, M., Müller, D. K.,... Smolka, M. N. (2017). No association of goal-directed and habitual control with alcohol consumption in young adults. Addiction Biology, doi:10.1111/adb.12490.

O'Doherty, J. P. (2011). Contributions of the ventromedial prefrontal cortex to goal-directed action selection. Annals of the New York Academy of Sciences, 1239(1), 118-129.

Ostlund, S. B., & Balleine, B. W. (2005). Lesions of medial prefrontal cortex disrupt the acquisition but not the expression of goal-directed learning. Journal of Neuroscience, 25(34), 7763-7770.

Ostlund, S. B., Maidment, N. T., & Balleine, B. W. (2010). Alcohol-paired contextual cues produce an immediate and selective loss of goal-directed action in rats. Frontiers in Integrative Neuroscience, 4, 19.

Pauls, D. L., Abramovitch, A., Rauch, S. L., & Geller, D. A. (2014). Obsessive-compulsive disorder:An integrative genetic and neurobiological perspective. Nature Reviews Neuroscience, 15(6), 410-424.

Plassmann, H., O'Doherty, J. P., & Rangel, A. (2010). Appetitive and aversive goal values are encoded in the medial orbitofrontal cortex at the time of decision making. Journal of Neuroscience, 30(32), 10799-10808.

Plessow, F., Kiesel, A., & Kirschbaum, C. (2012). The stressed prefrontal cortex and goal-directed behaviour:Acute psychosocial stress impairs the flexible implementation of task goals. Experimental Brain Research, 216(3), 397-408.

Postuma, R. B., & Dagher, A. (2006). Basal ganglia functional connectivity based on a meta-analysis of 126 positron emission tomography and functional magnetic resonance imaging publications. Cereb Cortex, 16(10), 1508-1521.

Reiter, A. M. F., Deserno, L., Wilbertz, T., Heinze, H. J., & Schlagenhauf, F. (2016). Risk factors for addiction and their association with model-based behavioral control. Frontiers in Behavioral Neuroscience, 10, 26.

Rose, A. K., Brown, K., Field, M., & Hogarth, L. (2013). The contributions of value-based decision-making and attentional bias to alcohol-seeking following devaluation. Addiction, 108(7), 1241-1249.

Rostami, M., Hosseini, S. M., Takahashi, M., Sugiura, M., & Kawashima, R. (2009). Neural bases of goal-directed implicit learning. NeuroImage, 48(1), 303-310.

Ruble, L. A. (2001). Analysis of social interactions as goal-directed behaviors in children with autism. Journal of Autism and Developmental Disorders, 31(5), 471-482.

Salowitz, N. M. G., Dolan, B., Remmel, R., van Hecke, A. V., Mosier, K., Simo, L., & Scheidt, R. A. (2013). Functional magnetic resonance imaging of goal-directed reaching children with autism spectrum disorders:A feasibility study. 17th World Multi-Conference on Systemics, Cybernetics and Informatics, Orlando, FL.

Schiltz, C. A. (2006). Habitual responding and the dorsal Striatum. Journal of Neuroscience, 26(7), 1891-1892.

Schwabe, L., & Wolf, O. T. (2010). Socially evaluated cold pressor stress after instrumental learning favors habits over goal-directed action. Psychoneuroendocrinology, 35(7), 977-986.

Schwabe, L., & Wolf, O. T. (2011). Stress-induced modulation of instrumental behavior:From goal-directed to habitual control of action. Behavioural Brain Research, 219(2), 321-328.

Sebold, M., Deserno, L., Nebe, S., Schad, D. J., Garbusow, M., Hägele, C.,... Huys, Q. J. M. (2014). Model-based and model-free decisions in alcohol dependence. Neuropsychobiology, 70(2), 122-131.

Sjoerds, Z., Luigjes, J., van den Brink, W., Denys, D., & Yücel, M. (2014). The role of habits and motivation in human drug addiction:A reflection. Frontiers in Psychiatry, 5, 8.

Smittenaar, P., FitzGerald, T. H. B., Romei, V., Wright, N. D., & Dolan, R. J. (2013). Disruption of dorsolateral prefrontal cortex decreases model-based in favor of model-free control in humans. Neuron, 80(4), 914-919.

Spreng, R. N., Stevens, W. D., Chamberlain, J. P., Gilmore, A. W., & Schacter, D. L. (2010). Default network activity, coupled with the frontoparietal control network, supports goal-directed cognition. NeuroImage, 53(1), 303-317.

Tanaka, S. C., Balleine, B. W., & O'Doherty, J. P. (2008). Calculating consequences:Brain systems that encode the causal effects of actions. Journal of Neuroscience the Official Journal of the Society for Neuroscience, 28(26), 6750-6755.

Tricomi, E.M., Delgado, M.R., and Fiez, J.A. (2004). Modulation of caudate activity by action contingency. Neuron, 41, 281-292.

Tricomi, E., Balleine, B. W., & O'Doherty, J. P. (2009). A specific role for posterior dorsolateral striatum in human habit learning. European Journal of Neuroscience, 29(11), 2225-2232.

Vaghi, M. M., Vértes, P. E., Kitzbichler, M. G., Apergis-Schoute, A. M., van der Flier, F. E., Fineberg, N. A.,... Robbins, T. W. (2017). Specific frontostriatal circuits for impaired cognitive flexibility and goal-directed planning in Obsessive-Compulsive Disorder:Evidence from resting-state functional connectivity. Biological Psychiatry, 81(8), 708-717.

Valentin, V. V., Dickinson, A., & O'Doherty, J. P. (2007). Determining the neural substrates of goal-directed learning in the human brain. Journal of Neuroscience, 27(15), 4019-4026.

Voon, V., Derbyshire, K., Rück, C., Irvine, M. A., Worbe, Y., Enander, J.,... Bullmore, E. T. (2015). Disorders of compulsivity:A common bias towards learning habits. Molecular Psychiatry, 20(3), 345-352.

Wunderlich, K., Dayan, P., & Dolan, R. J. (2012). Mapping value based planning and extensively trained choice in the human brain. Nature Neuroscience, 15(5), 786-791.

Yasumura, A., Kokubo, N., Yamamoto, H., Yasumura, Y., Moriguchi, Y., Nakagawa, E.,... Hiraki, K. (2012).. Neurobehavioral and hemodynamic evaluation of cognitive shifting in children with Autism Spectrum Disorder. Journal of Behavioral and Brain Science, 2(4), 463-470.

Yin, H. H., Knowlton, B. J., & Balleine, B. W. (2004). Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. European Journal of Neuroscience, 19(1), 181-189.

Zalla, T., Labruyere, N., & Georgieff, N. (2006). Goal-directed action representation in autism. Journal of Autism and Developmental Disorders, 36(4), 527-540.