Publications 2018

  • doi pdf Prefrontal glutamate levels predict altered amygdala-prefrontal connectivity in traumatized youths
  • Ousdal OT, Milde AM, Craven AR, Ersland L, Endestad T, Melinder A, Huys QJM, Hugdahl K
  • Psychol Med (2018) In Press
  • Background: Neurobiological models of stress and stress-related mental illness, including post-traumatic stress disorder (PTSD), converge on the amygdala and the prefrontal cortex (PFC). While a surge of research has reported altered structural and functional connectivity between amygdala and the medial PFC following severe stress, few have addressed the underlying neurochemistry. Methods: We combined resting-state functional magnetic resonance imaging (rsfMRI) measures of amygdala connectivity with in vivo MR-spectroscopy (1H-MRS) measurements of glutamate in 26 survivors from the 2011 Norwegian terror attack and 34 control subjects. Results: Traumatized youths showed altered amygdala - anterior midcingulate cortex (aMCC) and amygdala - ventromedial prefrontal cortex (vmPFC) connectivity. Moreover, the trauma survivors exhibited reduced levels of glutamate in the vmPFC which fits with previous findings of reduced levels of Glx (glutamate + glutamine) in the aMCC (Ousdal et al. 2017) and together suggest long-term impact of a traumatic experience on glutamatergic pathways. Importantly, local glutamatergic metabolite levels predicted the individual amygdala - aMCC and amygdala - vmPFC functional connectivity, and also mediated the observed group difference in amygdala - aMCC connectivity. Conclusions: Our findings suggest that traumatic stress may influence amygdala-prefrontal neuronal connectivity through an effect on prefrontal glutamate and its compounds. Understanding the neurochemical underpinning of altered amygdala connectivity after trauma may ultimately lead to the discovery of new pharmacological agents which can prevent or treat stress-related mental illness.

  • bioarchiv Major Depression Impairs the Use of Reward Values for Decision-Making
  • Rupprechter S, Stankevicius A, Huys QJM, Steele JD and Seriès P
  • Scientific Reports In Press
  • Depression is a debilitating condition with a high prevalence. Depressed patients have been shown to be diminished in their ability to integrate their reinforcement history to adjust future behaviour during instrumental reward learning tasks. Here, we tested whether such impairments could also be observed in a Pavlovian conditioning task. We recruited and analysed 32 subjects, 15 with depression and 17 healthy controls, to study behavioural group differences in learning and decision-making. Participants had to estimate the probability of some fractal stimuli to be associated with a binary reward, based on a few passive observations. They then had to make a choice between one of the observed fractals and another target for which the reward probability was explicitly given. Computational modelling was used to succinctly describe participants' behaviour. Patients performed worse than controls at the task. Computational modelling revealed that this was caused by behavioural impairments during both learning and decision phases. Depressed subjects showed lower memory of observed rewards and had an impaired ability to use internal value estimations to guide decision-making in our task.
  • Depression is associated with enhanced aversive Pavlovian control over instrumental behaviour
  • Nord C, Lawson RP, Huys QJM, Pilling S, Roiser JP
  • Scientific Reports, In Press
  • The dynamic modulation of instrumental behaviour by conditioned Pavlovian cues is an important process in decision-making. Patients with major depressive disorder (MDD) are known to exhibit mood-congruent biases in information processing, which may occur due to Pavlovian influences, but this hypothesis has never been tested directly in an unmedicated sample. To address this we tested unmedicated MDD patients and healthy volunteers on a computerized Pavlovian-Instrumental Transfer (PIT) task designed to separately examine instrumental approach and withdrawal actions in the context of Pavlovian appetitive and aversive cues. This design allowed us to directly measure the degree to which Pavlovian cues influence instrumental responding. Depressed patients were profoundly influenced by aversive Pavlovian stimuli, to a significantly greater degree than healthy volunteers. This was the case for instrumental behaviour both in the approach condition (in which aversive Pavlovian cues inhibited `go' responses), and in the withdrawal condition (in which aversive Pavlovian cues facilitated `go' responses). Exaggerated aversive PIT provides a potential cognitive mechanism for biased emotion processing in major depression. This finding also has wider significance for the understanding of disrupted motivational processing in neuropsychiatric disorders.
  • Deficits in context-dependent adaptive coding in early psychosis and healthy individuals with schizotypal personality traits
  • Kirschner M, Haugg A, Manoliu A, Simon JJ, Huys QJM, Seifritz E, Tobler PN and Kaiser S
  • Brain (2018). In Press.
  • Objective:Adaptive coding of information is a fundamental principle of brain functioning. It allows for efficient representation over a large range of inputs and thereby alleviates the limited coding range of neurons. In the present study, we investigated for the first time potential alterations in context-dependent reward adaptation and its association with symptom dimensions in the schizophrenia spectrum.

    Method: We studied 27 patients with first-episode psychosis, 26 individuals with schizotypal personality traits and 25 healthy controls. We used functional magnetic resonance imaging in combination with a variant of the monetary incentive delay task and assessed adaptive reward coding in two reward conditions with different reward ranges.

    Results: Compared to healthy controls, patients with first-episode psychosis and healthy individuals with schizotypal personality traits showed a deficit in increasing the blood-oxygen-level dependent response slope in the right caudate for the low reward range compared to the high reward range. In other words, the two groups showed inefficient neural adaptation to the current reward context. In addition, we found impaired adaptive coding of reward in the caudate nucleus and putamen to be associated with total symptom severity across the schizophrenia spectrum. Symptom severity was more strongly associated with neural deficits in adaptive coding than with the neural coding of absolute reward outcomes.

    Conclusions: Deficits in adaptive coding were prominent across the schizophrenia spectrum and even detectable in unmedicated (healthy) individuals with schizotypal personality traits. Furthermore, the association between total symptom severity and impaired adaptive coding in the right caudate and putamen suggests a dimensional mechanism underlying imprecise neural adaptation. Our findings support the idea that impaired adaptive coding may be a general information-processing deficit explaining disturbances within the schizophrenia spectrum over and above a simple model of blunted absolute reward signals.

  • Personalized prediction of antidepressant versus placebo response: Evidence from the EMBARC study.
  • Webb CA, Trivedi MD, Cohen C, Dillon DG, Fournier F, Goer F., Fava M, McGrath PJ, Weissman M, Parsey R, Adams P, Trombello JM, Cooper C, Deldin P, Oquendo MA, McInnis MG, Huys Q, Bruder G, Kurian B, Jha M, DeRubeis RJ, and Pizzagalli DA
  • Psychological Medicine (2018). In Press.
  • Background: Major Depressive Disorder (MDD) is a highly heterogeneous condition in terms of symptom presentation and, likely, underlying pathophysiology. Accordingly, it is possible that only certain individuals with MDD are well-suited to antidepressants. A potentially fruitful approach to parsing this heterogeneity is to focus on promising endophenotypes of depression, such as neuroticism, anhedonia and cognitive control deficits. Methods: Within an eight-week multisite trial of sertraline vs. placebo for depressed adults (n =216), we examined whether the combination of machine learning with a Personalized Advantage Index (PAI) can generate individualized treatment recommendations on the basis of endophenotype profiles coupled with clinical and demographic characteristics. Results: Five pre-treatment variables moderated treatment response. Higher depression severity and neuroticism, older age, less impairment in cognitive control and being employed were each associated with better outcomes to sertraline than placebo. Across 1000 iterations of a 10-fold cross-validation, the PAI model predicted that 31% of the sample would exhibit a clinically meaningful advantage (post-treatment Hamilton Rating Scale for Depression [HRSD] difference # 3) with sertraline relative to placebo. Although there were no overall outcome differences between treatment groups (d =.15), those identified as optimally suited to sertraline at pre- treatment had better week 8 HRSD scores if randomized to sertraline (10.7) than placebo (14.7)(d =.58). Conclusions: A subset of MDD patients optimally suited to sertraline can be identified on the basis of pre-treatment characteristics. This model must be tested prospectively before it can be used to inform treatment selection. However, findings demonstrate the potential to improve individual outcomes through algorithm-guided treatment recommendations.
  • doi Drunk decisions: Alcohol shifts choice from habitual towards goal-directed control in adolescent intermediate-risk drinkers
  • Obst E, Schad DJ, Huys QJM, Sebold M,, Nebe S, Sommer C, Smolka MN and Zimmermann US.
  • J Psychopharmacol (2018) ePub
  • Background: Studies in humans and animals suggest a shift from goal-directed to habitual decision-making in addiction. We therefore tested whether acute alcohol administration reduces goal-directed and promotes habitual decision-making, and whether these effects are moderated by self-reported drinking problems. Methods: Fifty-three socially drinking males completed the two-step task in a randomised crossover design while receiving an intravenous infusion of ethanol (blood alcohol level=80 mg%), or placebo. To minimise potential bias by long-standing heavy drinking and subsequent neuropsychological impairment, we tested 18- to 19-year-old adolescents. Results: Alcohol administration consistently reduced habitual, model-free decisions, while its effects on goal-directed, model-based behaviour varied as a function of drinking problems measured with the Alcohol Use Disorders Identification Test. While adolescents with low risk for drinking problems (scoring <8) exhibited an alcohol-induced numerical reduction in goal-directed choices, intermediate-risk drinkers showed a shift away from habitual towards goal-directed decision-making, such that alcohol possibly even improved their performance. Conclusions: We assume that alcohol disrupted basic cognitive functions underlying habitual and goal-directed decisions in low-risk drinkers, thereby enhancing hasty choices. Further, we speculate that intermediate-risk drinkers benefited from alcohol as a negative reinforcer that reduced unpleasant emotional states, possibly displaying a novel risk factor for drinking in adolescence.
  • doi Advancing Clinical Improvements for Patients Using the Theory-Driven and Data-Driven Branches of Computational Psychiatry
  • Huys QJM
  • JAMA Psychiatry (2018). Epub ahead of print.
  • This viewpoint is part of a series on pragmatic evidence-based psychiatry that arises from the tangible disillusionment with the speed at which explanatory advances in our understanding of the brain have been translated into clinical improvements for patients. Here, I argue that both theory-driven and data-driven branches of computational psychiatry can advance this pragmatic agenda.

Key Publications

  • Special Issue on Computational Psychiatry in Biologial Psychiatry
  • Edited by Tiago Maia, Quentin Huys and Michael Frank.
  • Special Issue on Computational Psychiatry in Biologial Psychiatry:CNNI
  • Edited by Martin Paulus, Quentin Huys and Tiago Maia.
  • Computational Psychiatry
  • MIT Press book in the Strüngmann Forum Series is out.
  • doi Advancing Clinical Improvements for Patients Using the Theory-Driven and Data-Driven Branches of Computational Psychiatry
  • Huys QJM
  • JAMA Psychiatry (2018). Epub ahead of print.
  • This viewpoint is part of a series on pragmatic evidence-based psychiatry that arises from the tangible disillusionment with the speed at which explanatory advances in our understanding of the brain have been translated into clinical improvements for patients. Here, I argue that both theory-driven and data-driven branches of computational psychiatry can advance this pragmatic agenda.
  • doi pdf The neural basis of aversive Pavlovian guidance during planning
  • Lally* N, Huys* QJM, Eshel N, Faulkner P, Dayan P and Roiser JP
  • J. Neurosci. (2017) 37(42):10215-10229
  • Important real-world decisions are often arduous as they frequently involve sequences of choices, with initial selections affecting future options. Evaluating every possible combination of choices is computationally intractable, particularly for longer multi-step decisions. Therefore, humans frequently employ heuristics to reduce the complexity of decisions. We recently used a goal-directed planning task to demonstrate the profound behavioral influence and ubiquity of one such shortcut, namely aversive pruning, a reflexive Pavlovian process that involves neglecting parts of the decision space residing beyond salient negative outcomes. However, how the brain implements this important decision heuristic, and what underlies individual differences in its strength have hitherto remained unanswered. Therefore, we administered an adapted version of the same planning task to healthy volunteers undergoing functional magnetic resonance imaging (fMRI) to determine the neural basis of aversive pruning. Through both computational and standard categorical fMRI analyses, we show that when planning was influenced by aversive pruning, the subgenual cingulate cortex was robustly recruited. This neural signature was distinct from those associated with general planning and valuation, two fundamental cognitive components elicited by our task but which are complementary to aversive pruning. Furthermore, we found that individual variation in levels of aversive pruning were associated with the responses of insula and dorsolateral prefrontal cortex to the receipt of large monetary losses, and also with sub-clinical levels of anxiety. In summary, our data reveal the neural signatures of an important reflexive Pavlovian processes that shapes goal-directed evaluations, and thereby determine the outcome of high-level sequential cognitive processes.
  • doi pdf A formal valuation framework for emotions and their control
  • Huys QJM and Renz D
  • Biological Psychiatry (2017) 82:413--420
  • Computational psychiatry attempts to apply mathematical and computational techniques to help improve psychiatric care. Here, we consider formal valuation accounts of emotion. The flexibility of emotional responses and the nature of appraisals suggest the need for a model-based framework for emotions. Resource limitations make plain model-based valuation impossible, and require strategies to apportion cognitive resources adaptively. We argue that emotions can implement such approximations by restricting the range of behaviours and states considered. We consider the processes that guide the deployment of the emotional approximations, discerning between innate, model-free, heuristic and model-based controllers. A focus on complex model-based decisions reveals the necessity for strategies to deal with the complexity of the problems. Emotions may provide such approximations, and this framework may provide a principled approach to examining them.
  • doi pdf When habits are dangerous: alcohol expectancies and habitual decision-making in alcohol dependence
  • Sebold M, Nebe S, Garbusow M, Schad DJ, Beck A, Kuitunen-Paul S, Sommer C, Neu P, Zimmermann US, Rapp MA, Smolka MN, Huys QJM, Schlagenhauf F, Heinz A
  • Biological Psychiatry (2017) 82(11):847-856
    Commentary by T. Duka
  • Addiction is supposed to be characterized by a shift from goal-directed to habitual decision-making, thus facilitating automatic drug intake. The two-step task allows distinguishing between these mechanisms by computationally modelling goal-directed and habitual behavior as model-based and model-free control. In addicted patients, decision-making may also strongly depend upon drug-associated expectations. Therefore, we investigated model-based vs. model-free decision-making and its neural correlates as well as alcohol expectancies in alcohol-dependent patients and healthy controls and assessed treatment outcome in patients. Ninety detoxified, medication-free alcohol-dependent patients and 96 age- and gender-matched control underwent functional magnetic resonance imaging during the two-step task. Alcohol expectancies were measured with the Alcohol Expectancy Questionnaire.. Over a follow-up period of 48 weeks, 37 patients remained abstinent whereas 53 patients. Patients who relapsed displayed reduced medial prefrontal cortex (mPFC) activation during model-based decision-making. Furthermore high alcohol expectancies were associated with low model-based control in relapsers, while the opposite was observed in abstainers and healthy controls. However, reduced model-based control per se was not associated with subsequent relapse. These findings suggest that poor treatment outcome in addicted patients does not simply result from reduced model-based control but is rather dependent on the interaction between high drug expectancies and low model-based decision-making. Reduced model-based mPFC signatures in prospective relapsers point to a neural correlate of relapse risk. These observations suggest that therapeutic interventions should target subjective alcohol expectancies.
  • doi pdf A Roadmap for the Development of Applied Computational Psychiatry
  • Paulus MP, Huys QJM and Maia T
  • Biological Psychiatry: Cognitive Neuroscience and Neuroimaging (2016) 1(5):386-392
  • Background: Computational psychiatry is a burgeoning field that utilizes mathematical approaches to investigate psychiatric disorders, derive quantitative predictions, and integrate data across multiple levels of description. Computational psychiatry has already led to many new insights into the neurobehavioral mechanisms that underlie several psychiatric disorders, but its usefulness from a clinical standpoint is only now starting to be considered. Methods: Examples of computational psychiatry are highlighted, and a phase-based pipeline for the development of clinical computational- psychiatry applications is proposed, similar to the phase-based pipeline used in drug development. It is proposed that each phase has unique endpoints and deliverables, which will be important milestones to move tasks, procedures, computational models, and algorithms from the laboratory to clinical practice. Results: Application of computational approaches should be tested on healthy volunteers in Phase I, transitioned to target populations in Phase IB and Phase IIA, and thoroughly evaluated using randomized clinical trials in Phase IIB and Phase III. Successful completion of these phases should be the basis of determining whether computational models are useful tools for prognosis, diagnosis, or treatment of psychiatric patients. Conclusions: A new type of infrastructure will be necessary to implement the proposed pipeline. This infrastructure should consist of groups of investigators with diverse backgrounds collaborating to make computational psychiatry relevant for the clinic.

  • doi pdf Computational psychiatry as a bridge from neuroscience to clinical applications
  • Huys QJM*, Maia T* and Frank MJ
  • Nature Neuroscience (2016) 19(3):404--413
    Commentary in Scientific American
  • Translating advances in neuroscience into benefits for patients with mental illness presents enormous challenges because it involves both the most complex organ----the brain---and its interaction with a similarly complex environment. Dealing with such complexities demands powerful techniques. Computational psychiatry combines multiple levels and types of computation with multiple types of data in an effort to improve understanding, prediction, and treatment of mental illness. Computational psychiatry, broadly defined, encompasses two complementary approaches: data-driven and theory-driven. Data-driven approaches apply machine-learning methods to high-dimensional data to improve classification of disease, predict treatment outcomes, or improve treatment selection. These approaches are generally agnostic as to the underlying mechanisms. Theory-driven approaches, in contrast, use models that instantiate prior knowledge of, or explicit hypotheses about, such mechanisms, possibly at multiple levels of analysis and abstraction. We review recent advances in both approaches, with an emphasis on clinical applications, and highlight the utility of combining them.

  • doi pdf The specificity of Pavlovian regulation is associated with recovery from depression
  • Huys QJM, Gölzer M, Friedel E, Heinz A, Cools R, Dayan P and Dolan RJ
  • Psychological Medicine (2016) 46(5):1027-1035
  • Background: Changes in reflexive emotional responses are hallmarks of depression, but how emotional reflexes impact on adaptive decision-making in depression has not been examined formally. Using a Pavlovian-Instrumental Transfer (PIT) task, we compared the influence of affectively valenced stimuli on decision-making in depression and generalized anxiety disorder compared with healthy controls; and related this to the longitudinal course of the illness.

    Methods: Fourty subjects with a current DSM-IV-TR diagnosis of Major Depressive Disorder, Dysthymia, Generalised Anxiety Disorder or a combination thereof, and 40 matched healthy controls performed a Pavlovian-instrumental transfer (PIT) task that assesses how instrumental approach and withdrawal behaviours are influenced by appetitive and aversive Pavlovian conditioned stimuli. Patients were followed up after 4-6 months. Analyses focussed on patients with depression alone (n=25).

    Results: In healthy controls, Pavlovian conditioned stimuli (CSs) exerted action-specific effects, with appetitive CSs boosting active approach and aversive CSs withdrawal. This action-specificity was absent in currently depressed subjects. Greater action-specificity in patients was associated with better recovery over the follow-up period.

    Conclusions: Depression is associated with abnormal influence of emotional reactions on decision-making in a way that is predictive of recovery.

  • doi pdf Pavlovian-to-instrumental transfer effects in the nucleus accumbens relate to relapse in alcohol dependence
  • Garbusow M*, Schad D*, Sebold M, Friedel E, Bernhardt N, Koch SP, Steinacher B, Kathmann N, Geurts DE, Sommer C, Müller DK, Nebe S, Paul S, Wittchen H-U, Zimmermann US, Walter H, Smolka MN, Sterzer P, Rapp MA, Huys QJM*, Schlagenhauf F* and Heinz A*
  • Addiction Biology (2016) 21(3):719-31
  • In detoxified alcohol-dependent patients, alcohol-related stimuli can promote relapse. However, to date, the mechanisms by which contextual stimuli promote relapse have not been elucidated in detail. One hypothesis is that such contextual stimuli directly stimulate the motivation to drink via associated brain regions like the ventral striatum and thus promote alcohol seeking, intake and relapse. Pavlovian-to-Instrumental-Transfer (PIT) may be one of those behavioral phenomena contributing to relapse, capturing how Pavlovian conditioned (contextual) cues determine instrumental behavior (e.g. alcohol seeking and intake). We used a PIT paradigm during functional magnetic resonance imaging to examine the effects of classically conditioned Pavlovian stimuli on instrumental choices in n=31 detoxified patients diagnosed with alcohol dependence and n=24 healthy controls matched for age and gender. Patients were followed up over a period of 3 months. We observed that (1) there was a significant behavioral PIT effect for all participants, which was significantly more pronounced in alcohol-dependent patients; (2) PIT was significantly associated with blood oxygen level-dependent (BOLD) signals in the nucleus accumbens (NAcc) in subsequent relapsers only; and (3) PIT-related NAcc activation was associated with, and predictive of, critical outcomes (amount of alcohol intake and relapse during a 3 months follow-up period) in alcohol-dependent patients. These observations show for the first time that PIT-related BOLD signals, as a measure of the influence of Pavlovian cues on instrumental behavior, predict alcohol intake and relapse in alcohol dependence.
  • doi pdf Decision-theoretic psychiatry
  • Huys QJM, Guitart-Masip M, Dolan RJ and Dayan P
  • Clin Psychol Sci (2015) 3(3):400-421
  • Psychiatric disorders profoundly impair many aspects of decision-making. Poor choices have negative consequences in the moment, and also make it very hard to navigate complex social environments. Computational neuroscience provides normative, neurobiologically informed, descriptions of the components of decision-making which serve as a platform for a principled exploration of dysfunctions. Here, we identify and discuss three classes of failure-modes arising in these formalisms. They stem from abnormalities in the framing of problems or tasks, from the mechanisms of cognition used to solve the tasks, or from the historical data available from the environment.
  • doi pdf Depression: A Decision-Theoretic Analysis
  • Huys QJM, Daw ND and Dayan P
  • Annu Rev Neurosci (2015), 38:1-23
  • The manifold symptoms of depression are a common and often transient feature of healthy life that are likely to be adaptive in difficult circumstances. It is when these symptoms enter a seemingly self-propelling spiral that the maladaptive features of a disorder emerge. We examine this malignant transformation from the perspective of the computational neuroscience of decision making, investigating how dysfunction of the brain's mechanisms of evaluation might lie at its heart. We start by considering the behavioural implications of pessimistic evaluations of decision variables. We then provide a selective review of work suggesting how such pessimism might arise via specific failures of the mechanisms of evaluation or state estimation. Finally, we analyze ways that miscalibration between the subject and environment may be self-perpetuating. We employ the formal framework of Bayesian decision theory as a foundation for this study, showing how most of the problems arise from one of its broad algorithmic facets, namely model-based reasoning.
  • doi pdf The interplay of approximate planning strategies
  • Huys QJM, Lally N, Faulkner P, Eshel N, Seifritz E, Gershman SJ, Dayan P and Roiser JP
  • PNAS (2015), 112(10):3098-3103
    Commentary by Daniel, Schuck & Niv
  • Humans routinely formulate plans in domains so complex that even the most powerful computers are taxed. To do so, they seem to avail themselves of many strategies and heuristics that efficiently simplify, approximate, and hierarchically decompose hard tasks into simpler subtasks. Theoretical and cognitive research has revealed several such strategies; however, little is known about their establishment, interaction, and efficiency. Here, we use model-based behavioral analysis to provide a detailed examination of the performance of human subjects in a moderately deep planning task. We find that subjects exploit the structure of the domain to establish subgoals in a way that achieves a nearly maximal reduction in the cost of computing values of choices, but then combine partial searches with greedy local steps to solve subtasks, and maladaptively prune the decision trees of subtasks in a reflexive manner upon encountering salient losses. Subjects come idiosyncratically to favor particular sequences of actions to achieve subgoals, creating novel complex actions or "options."
  • doi pdf The role of learning-related dopamine signals in addiction vulnerability
  • Huys QJM, Tobler PT, Hasler G, Flagel SB
  • Progress in Neurobiology (2014): 211:31-77
  • Dopaminergic signals play a mathematically precise role in reward-related learning, and variations in dopaminergic signalling have been implicated in vulnerability to addiction. Here, we provide a detailed overview of the relationship between theoretical, mathematical and experimental accounts of phasic dopamine signalling, with implications for the role of learning-related dopamine signalling in addiction and related disorders. We describe the theoretical and behavioural characteristics of model-free learning based on errors in the prediction of reward, including step-by-step explanations of the underlying equations. We then use recent insights from an animal model that highlights individual variation in learning during a Pavlovian conditioning paradigm to describe overlapping aspects of incentive salience attribution and model-free learning. We argue that this provides a computationally coherent account of some features of addiction.
  • doi pdf
  • Striatal dysfunction during reversal learning in unmedicated schizophrenia patients
  • Schlagenhauf F*, Huys QJM*, Deserno M, Rapp MA, Beck A and Heinz A
  • Neuroimage (2014) 89:171-180
  • Subjects with schizophrenia are impaired at reinforcement-driven reversal learning from as early as their first episode. The neurobiological basis of this deficit is unknown. We obtained behavioral and fMRI data in 24 unmedicated, primarily first episode, schizophrenia patients and 24 age-, IQ- and gender-matched healthy controls during a reversal learning task. We supplemented our fMRI analysis, focusing on learning from prediction errors, with detailed computational modeling to probe task solving strategy including an ability to deploy an internal goal directed model of the task. Patients displayed reduced functional activation in the ventral striatum (VS) elicited by prediction errors. However, modeling task performance revealed that a subgroup did not adjust their behavior according to an accurate internal model of the task structure, and these were also the more severely psychotic patients. In patients who could adapt their behavior, as well as in controls, task solving was best described by cognitive strategies according to a Hidden Markov Model. When we compared patients and controls who acted according to this strategy, patients still displayed a significant reduction in VS activation elicited by informative errors that precede salient changes of behavior (reversals). Thus, our study shows that VS dysfunction in schizophrenia patients during reward-related reversal learning remains a core deficit even when controlling for task solving strategies. This result highlights VS dysfunction is tightly linked to a reward- related reversal learning deficit in early, unmedicated schizophrenia patients.
  • doi pdf Mapping anhedonia onto reinforcement learning. A behavioural meta-analysis.
  • Huys QJM, Pizzagalli DA, Bogdan R and Dayan P
  • Biology of Mood & Anxiety Disorders (2013) 3:12
  • Background: Depression is characterised partly by blunted reactions to reward. However, tasks probing this deficiency have not distinguished insensitivity to reward from insensitivity to the prediction errors for reward that determine learning and are putatively reported by the phasic activity of dopamine neurons. We attempted to disentangle these factors with respect to anhedonia in the context of stress, Major Depressive Disorder (MDD), Bipolar Disorder (BPD) and a dopaminergic challenge.

    Methods: Six behavioural datasets involving 392 experimental sessions were subjected to a model-based, Bayesian meta-analysis. Participants across all six studies performed a probabilistic reward task that used an asymmetric reinforcement schedule to assess reward learning. Healthy controls were tested under baseline conditions, stress or after receiving the dopamine D2 agonist pramipexole. In addition, participants with current or past MDD or BPD were evaluated. Reinforcement learning models isolated the contributions of variation in reward sensitivity and learning rate.

    Results: MDD and anhedonia reduced reward sensitivity more than they affected the learning rate, while a low dose of the dopamine D2 agonist pramipexole showed the opposite pattern. Stress led to a pattern consistent with a mixed effect on reward sensitivity and learning rate.

    Conclusion: Reward-related learning reflected at least two partially separable contributions. The first related to phasic prediction error signalling, and was preferentially modulated by a low dose of the dopamine agonist pramipexole. The second related directly to reward sensitivity, and was preferentially reduced in MDD and anhedonia. Stress altered both components. Collectively, these findings highlight the contribution of model-based reinforcement learning meta-analysis for dissecting anhedonic behavior.

  • doi pdf Go and nogo learning in reward and punishment: Interactions between affect and effect
  • Guitart-Masip M*, Huys QJM*, Fuentemilla L, Dayan P, Düzel E and Dolan RJ
  • Neuroimage (2012) 62(1):154-66
  • Decision-making invokes two fundamental axes of control: affect or valence, spanning reward and punishment, and effect or action, spanning invigoration and inhibition. We studied the acquisition of instrumental responding in healthy human volunteers in a task in which we orthogonalized action requirements and outcome valence. Subjects were much more successful in learning active choices in rewarded conditions, and passive choices in punished conditions. Using computational reinforcement-learning models, we teased apart contributions from putatively instrumental and Pavlovian components in the generation of the observed asymmetry during learning. Moreover, using model-based fMRI, we showed that BOLD signals in striatum and substantia nigra/ventral tegmental area (SN/VTA) correlated with instrumentally learnt action values, but with opposite signs for go and no-go choices. Finally, we showed that successful instrumental learning depends on engagement of bilateral inferior frontal gyrus. Our behavioral and computational data showed that instrumental learning is contingent on overcoming inherent and plastic Pavlovian biases, while our neuronal data showed this learning is linked to unique patterns of brain activity in regions implicated in action and inhibition respectively.
  • doi pdf Bonsai trees in your head: How the Pavlovian system sculpts goal-directed choices by pruning decision trees
  • Huys QJM*, Eshel N*, O'Lions E, Sheridan L, Dayan P and Roiser JP
  • PLoS Comp Biol (2012) 8(3): e1002410
  • When planning a series of actions, it is usually infeasible to consider all potential future sequences; instead, one must prune the decision tree. Provably optimal pruning is, however, still computationally ruinous and the specific approximations humans employ remain unknown. We designed a new sequential reinforcement-based task and showed that human subjects adopted a simple pruning strategy: during mental evaluation of a sequence of choices, they curtailed any further evaluation of a sequence as soon as they encountered a large loss. This pruning strategy was Pavlovian: it was reflexively evoked by large losses and persisted even when overwhelmingly counterproductive. It was also evident above and beyond loss aversion. We found that the tendency towards Pavlovian pruning was selectively predicted by the degree to which subjects exhibited sub-clinical mood disturbance, in accordance with theories that ascribe Pavlovian behavioural inhibition, via serotonin, a role in mood disorders. We conclude that Pavlovian behavioural inhibition shapes highly flexible, goal- directed choices in a manner that may be important for theories of decision-making in mood disorders.
  • doi pdf Disentangling the roles of approach, activation and valence in instrumental and Pavlovian responding
  • Huys QJM, Cools R, Gölzer M, Friedel E, Heinz A, Dolan RJ and Dayan P
  • PLoS Comp Biol (2011) 7(4): e1002028
  • Hard-wired, Pavlovian, responses elicited by predictions of rewards and punishments exert significant benevolent and malevolent influences over instrumentally-appropriate actions. These influences come in two main groups, defined along anatomical, pharmacological, behavioural and functional lines. Investigations of the influences have so far concentrated on the groups as a whole; here we take the critical step of looking inside each group, using a detailed reinforcement learning model to distinguish effects to do with value, specific actions, and general activation or inhibition. We show a high degree of sophistication in Pavlovian influences, with appetitive Pavlovian stimuli specifically promoting approach and inhibiting withdrawal, and aversive Pavlovian stimuli promoting withdrawal and inhibiting approach. These influences account for differences in the instrumental performance of approach and withdrawal behaviours. Finally, although losses are as informative as gains, we find that subjects neglect losses in their instrumental learning. Our findings argue for a view of the Pavlovian system as a constraint or prior, facilitating learning by alleviating computational costs that come with increased flexibility.
  • doi pdf A Bayesian formulation of behavioral control
  • Huys QJM and Dayan P
  • Cognition (2009) Special Issue
  • Helplessness, a belief that the world is not subject to behavioral control, has long been central to our understanding of depression, and has influenced cognitive theories, animal models and behavioral treatments. However, despite its importance, there is no fully accepted definition of helplessness or behavioral control in psychology or psychiatry, and the formal treatments in engineering appear to capture only limited aspects of the intuitive concepts. Here, we formalize controllability in terms of characteristics of prior distributions over affectively charged environments. We explore the relevance of this notion of control to reinforcement learning methods of optimising behavior in such environments and consider how apparently maladaptive beliefs can result from normative inference processes. These results are discussed with reference to depression and animal models thereof.