Deficient goal-directed control in a population characterized by extreme goal pursuit

Post-Test Assessment of Task Transition Structure

HC and AN did not differ significantly in their recall of the transition structure for the monetary or food outcomes task at Time 1 (Money: χ²(1, N = 87) = 0.274, p = 0.60; Food: χ²(1, N = 84) = 0.925, p = 0.34) or Time 2 (Money: χ²(1, N = 52) = 0.006, p = 0.94; Food: χ²(1, N = 52) = 0.650, p = 0.42). HC and AN also did not differ significantly on the average discrepancy between their estimates of the transition probabilities and the actual probabilities at Time 1 (Money: t(82) = -0.91, p = 0.37; Food: t(82) = -0.59, p = 0.56) or Time 2 (Money: t(51) = -1.73, p = 0.09; Food: t(51) = -1.34, p = 0.18).

Correlations with clinical variables

Model-based behavior was not significantly associated with BMI (over and above the group difference) or duration of illness in the patient group (BMI: Est: 0.003, SE = 0.03, z = 0.09, p = 0.93; Duration: Est: 0.05, SE = 0.07, z = 0.80, p = 0.42). In AN at Time 1, higher scores on the YBC-EDS were associated with less model-based learning for food outcomes relative to monetary outcomes (Est: 0.09, SE = 0.05, z = 1.95, p = 0.051), a trend mainly driven by (and significant for) the Preoccupations subscale (Est: 0.13, SE = 0.04, z = 3.07, p = 0.002) rather than the Rituals subscale (Est: 0.04, SE = 0.05, z = 0.77, p = 0.44). (Data were not collected from HC, for whom scores would be expected to be zero, or from AN after weight restoration.)

AN subtypes

Individuals with AN can be classified as either restricting (AN-R) or binge eating/purging (AN-BP) subtype and the current sample was split approximately equally between subtypes (19 AN-R, 22 AN-BP). Exploratory analyses suggested that the impairment in model-based learning was largely driven by individuals with AN meeting criteria for binge-eating/purging, rather than restricting subtype (see supplementary material and supplementary Fig. S1).

How does the extreme, yet inflexible, “self-control” that characterizes AN relate to model-based learning?

AN poses a fascinating conceptual challenge for the classic psychological dichotomy between goal-directed, controlled behavior and inflexible, habitual responses. Selection of a low-fat, low calorie diet with limited variety is a stereotyped feature of illness in AN (Mayer, Schebendach, Bodell, Shingleton, & Walsh, 2012; Schebendach et al., 2008; Sysko, Walsh, Schebendach, & Wilson, 2005). Yet nearly half of patients also engage in eating that is accompanied by a sense of loss of control (binge eating). The coexistence of these behaviors, as well as other indicators that behavior is actually quite inflexible in AN, makes this illness particularly intriguing as an opportunity to better understand the relative contributions of goal-directed and habitual behavior. Maladaptive behaviors in AN are commonly understood to reflect heightened self-control, which might be expected to relate to enhanced dominance of goal-directed (or model-based) behavior. Yet these same behaviors are also rigid and difficult to change, which might suggest the opposite result: relatively weakened model-based control. This study allowed us to test existing hypotheses that impairments in goal-directed, or model-based, behavior are a transdiagnostic mechanism underlying compulsive psychopathology (Gillan et al., 2016; Voon et al., 2015).

Our results suggest that, compared to HC, AN rely significantly less on the model-based approach, similar to individuals with other disorders involving compulsion such as OCD or drug abuse (Gillan et al., 2016; Voon et al., 2015). The pathology of AN involves complex, multi-step phenomena to avoid food intake. The findings in this study suggest that these computational mechanisms may be relevant even to producing behaviors, like pathological food avoidance, that extend beyond the traditional notion from animal behavioral psychology of habits as simple stimulus-response motor programs. Indeed, a recent study in a large general population sample found that model-based behavior in the Two-Step task, used herein, correlated with a set of psychiatric symptoms that included not just simple compulsive actions (such as repetitive checking or morning drinking) that extend easily from a basic notion of habits but also broader, more cognitive symptoms such as intrusive thoughts (Gillan et al., 2016). In contrast, this study found no significant difference between AN and HC in the model-free approach in the task, and groups also differed significantly in a composite index of relative reliance on the two systems, supporting the specificity of the effect. That said, as with previous studies using this task (Gillan et al., 2016), we focused our hypotheses and analyses on the modulation of model-based behavior, as this has proven to be the measure from this task that has most reliably tracked manipulations or individual differences likely to be relevant to the goal-habit balance (Otto, Raio, et al., 2013; Otto, Skatova, Madlon-Kay, & Daw, 2015).

Does specific psychopathology reflect a domain-general failure of model-based learning?

Our data indicate that the deficit in model-based learning in AN was not significantly remediated by using food outcomes. Rather, it was consistent across both domain-general (money) and illness-specific (food) outcomes, with no significant task differences in the group effect. This finding cuts against one potential explanation for the current results on the money task and other similar results showing reduced model-based learning for money in psychiatric populations (Voon et al., 2015): that generic monetary outcomes are less motivating in psychiatric populations than they are for HC participants. Interestingly, it has recently been shown that incentives can reliably boost model-based performance–even in the more severe end of the spectrum for a range of psychopathology constructs (Patzelt, Kool, Millner, & Gershman, 2019). By using an incentive-compatible food outcome (success on the Two-Step task led to actual snack choices after the task), the present study included a condition in which motivation would be expected to be more salient for individuals with AN than for HC. Yet, even in the food context, patients with AN showed less model-based behavior than HC. Of course, the failure to detect a Group-by-Task interaction is a negative result, and as with any such result, the possibility remains that there does exist some interaction smaller than our study was powered to detect. However, this is mitigated by a positive result: that AN were significantly impaired at model-based learning, relative to HC, even in the food task considered alone (Est=0.17, SE=0.067, z=2.6, p=0.01).

Exploratory analyses of the relationship between symptom severity and task performance in AN did suggest one result that was selective for the food task: higher scores on the YBC-EDS, especially the Preoccupation subscale, were associated with less model-based learning for food outcomes relative to monetary outcomes. Such a graded deficit seems plausible, and its specificity to food intriguing, but it is of note that the direction of the effect (worse learning with food outcomes) is opposite that predicted by a motivational account. Also, given the exploratory nature of the analysis and the smaller number of data points underlying it, there is a danger of false positives and replication will be required to confirm this finding.

Are model-based deficits secondary to psychopathology, or potentially primary?

The idea that compulsive psychopathology might relate to impaired model-based learning is appealing in part because it hints at a causal mechanism by which pathological habits might emerge. Of course we are not in a position to address causality, nor even whether the cognitive deficits precede the symptoms. However, we were able to address whether the cognitive differences between AN and HC persisted after acute weight restoration. That they did indicates they were not due to starvation alone. Prior research has shown that while psychological measures improve with weight restoration, eating behavior—namely, the pursuit of low-fat low-calorie diets—does not improve substantially with weight restoration (Mayer et al., 2012; Sysko et al., 2005). Additionally, in a previous study we found that deficits in feedback-based learning remained after weight restoration (Foerde & Steinglass, 2017). The effect on model-based learning here shows a similar persistence, consistent with model-based deficits as a relatively stable characteristic in AN. This result also parallels a recent report, using the same task, that cognitive behavioral therapy for OCD also does not improve goal-directed deficits in that population (Wheaton et al., 2018). In the current dataset, the absence of an effect of weight restoration is a negative result and subject to similar interpretational caveats as the money vs. food contrast. However, the confidence interval on the effect suggests that any undetected effect of weight restoration on model-based behavior is likely to be quite small relative to the overall deficit.

Participants

Participants were 41 women with AN and 53 healthy comparison women (HC). Individuals were eligible if they were between the ages of 16 and 46 years (Table 1), with an estimated IQ > 80 (measured by Wechsler Abbreviated Scale of Intelligence, WASI-II, (Wechsler, 1999)). Eligible patients met DSM-5 (American Psychiatric Association, 2013) criteria for AN—restricting (AN-R, n=19) or binge eating/purging (AN-BP, n=22) subtype—and were receiving inpatient treatment at the New York State Psychiatric Institute (NYSPI) specialized Eating Disorders Unit. Patients with AN were not eligible if they had a history of a psychotic disorder, were at imminent risk of suicide, or met criteria for substance use disorder. Anxiety and depressive disorders were not exclusionary, as these commonly co-occur with AN (Hudson, Hiripi, Pope, & Kessler, 2007). Three individuals with AN were taking antidepressant medications (SSRIs). HC were recruited through the community and were compensated $50 for their time. HC were group-matched for age and ethnicity and were included if they had no current or past psychiatric illness, including any history of an eating disorder, and had a body mass index (BMI) in the normal range (18–25 kg/m²). The HC group included 19 individuals who endorsed a history of dieting behavior. This study was approved by the NYSPI Institutional Review Board; after complete description of the study to the participants, adult participants provided written informed consent and adolescents provided written assent with parental consent.

Procedure

Psychiatric diagnoses were established using the Semi Structured Interview for DSM-IV (SCID; (First, Spitzer, Gibbon, & Williams, 2002) and the Eating Disorders Assessment for DSM5 (EDA-5, (Sysko et al., 2012)). Height and weight were obtained on a beam balance scale (Detecto, Webb, MO). Estimated IQ was assessed with the WASI-II (Wechsler, 1999). Severity of eating disorder psychopathology was measured by the Eating Disorder Examination Questionnaire (EDE-Q; (Fairburn, 2008; Fairburn & Beglin, 1994)), a 36-item self-report assessment of eating disorder symptoms which has established community norms for adolescents and adults. The four symptom subscales (Restraint, Eating Concern, Shape Concern, and Weight Concern) are scored on a scale of 0-6 and averaged to obtain a “global” score. In addition, participants with AN completed the Yale-Brown-Cornell Eating Disorder Scale (YBC-EDS; (Mazure, Halmi, Sunday, Romano, & Einhorn, 1994)), an interview measure of eating disorder symptoms with separate subscales related to Preoccupations and Rituals. Each subscale is scored on a scale from 0-16 and the subscales are summed to obtain the total score (ranging from 0-32). Higher scores indicate greater symptom severity on each measure.

Participants were assessed at two points: Individuals with AN were tested within 1 week of hospital admission (Time 1) and again after inpatient weight restoration treatment to at least 90% ideal body weight (Metropolitan Life Insurance, 1959), corresponding to a body mass index (BMI) of approximately 19.5-20.0 kg/m² (Time 2). HC individuals were tested twice at an interval group matched to AN (M_HC = 51 +/- 23 days, M_AN = 59 +/- 22 days, t(45) = 1.17, p = 0.25). Of 41 AN participants, 18 completed both time points, 16 completed only Time 1, and seven completed only Time 2; of 53 HC participants, 29 completed both time points and 24 completed only Time 1. For one AN participant, data from the food outcome task at Time 2 was lost due to computer malfunction.

Two-Step Decision Task

Each participant completed two versions of the Two-Step Decision Task (Daw et al., 2011; Decker et al., 2016; Sharp et al., 2016)—one that involved playing for money and one that involved playing for a food snack to be consumed after the task (order counterbalanced across participants). The tasks were structurally identical but included distinct cover stories and task environments that matched the outcomes used (see Fig. 1AB). For both tasks each trial proceeded in two stages (Fig. 1C). In the first stage (Stage 1), participants chose between two spaceships (or cafés), revealing a second-stage (Stage 2) choice between two aliens (or food trays). Each second-stage alien (or food tray) had a slowly changing chance of delivering space treasure (or a food token) vs. nothing, necessitating continuous learning by trial and error. The four Stage 2 options were determined by independently drifting Gaussian random walks with a standard deviation = 0.025 and bounded by 0.25 and 0.75 probability of reward, such that the reward probability associated with each Stage 2 option changed slowly from trial to trial (Fig. 1D). The response window at each stage was three seconds. Participants completed 200 trials of each task.

A key design feature of the task was the probabilistic association between first- and second-stage choices: Choosing the blue spaceship (or blue café) lead to the purple planet (or green kitchen) 70% of the time—a common transition, and the red planet (yellow kitchen) 30% of the time—a rare transition. The contingencies were reversed for the green spaceship (pink café). The transition structure between stages allows dissociation of model-free vs. model-based learning strategies: Model-free learning is ignorant of transition structure and favors repeating a first-step choice that ultimately results in reward, even if it does so via a low probability transition. By contrast, model-based learning is sensitive to the transition contingencies and uses them to infer the first-stage choice most likely to lead to the preferred second-stage environment.

In the monetary version of the task, each piece of space treasure collected was worth $0.10 and the total earned was paid out in cash at the end of the experiment. In the food version of the task, the accrued tokens bought access to a selection of foods from which to choose a snack item. Prior to completing the task, participants were given a “menu” (supplementary Fig. S2) with pictures of 15 food items and were asked to rank them in order of preference, but were not told that the ratings were related to later snack options. An increasing number of earned food tokens granted access to an increasing number of preferred food items (e.g., 92 tokens provided access to items ranked 9-15, 114 tokens or more provided access to all items, see supplementary Table S1).

A subset of participants (HC: n=9, AN: n=18) completed a different version of the food decision task (the monetary task was the same for all participants). In this version, rather than receiving a food token vs. nothing as the second-stage outcome, participants saw a picture of a preferred food item vs. a non-preferred food item. The specific food items were individually determined per participant: prior to the decision task participants completed a rating task in which 9 food items were rated on a scale from 1-9 (1= highly preferred). Participants were not instructed that their ratings would relate to their later snack options. The food image participants received on most trials determined which snack they would receive after the task. In this task, participants had 2 seconds to make choices in both stages. Results were qualitatively similar for both food tasks, and they were therefore combined in all analysis below.

After completing the Two-step tasks, participants’ knowledge of the transition structure between Stage 1 and Stage 2 (e.g., If you picked the blue spaceship, which planet would you most likely land on?) and their estimates of the transition probabilities were assessed (e.g., If you picked the blue spaceship, how likely would you be to see the purple planet?). Responses were not collected from 3 HC for the monetary task at Time 1 and from 1 AN on both the monetary and food tasks at Time 2.

Assessment of Model-based and Model-free learning (Computational Model)

In order to capture the influence of incremental learning across many trials, we fit subjects’ choices using a reinforcement learning model in which choices are modeled as arising due to the weighted combination of model-free and model-based reinforcement learning. The model is equivalent to the one used by Gillan et al. (Gillan et al., 2016) and Otto et al. (Otto, Raio, et al., 2013), except that following Sharp et al. (Sharp et al., 2016) we have eliminated one free parameter (by setting β^MF0 in the GIllan model to 0, or equivalently λ in the Otto model to 1). It reflects a simplification and a change of variables over the model from Daw et al. (Daw et al., 2011).

On each trial, t, participants make a Stage 1 choice c_1,t, leading to a transition to a Stage 2 state s_t where another choice, c_2,t is made, followed by reward r_t. At Stage 2, it is assumed that participants learn a value function over states and choices, Q^stage2(s, c), whose value for the chosen action is updated based on the reward received at each trial according to a delta rule, . Here, α is a free learning rate parameter. (In this, and analogous update equations, a factor of α is omitted from the last term of the update, equivalent to rescaling the rewards and Qs by and the corresponding weighting parameters β by α (Otto, Raio, et al., 2013)). The probability of a particular Stage 2 choice is modeled as governed by these values according to a logistic softmax, with free inverse temperature parameter , normalized over both options c.

Stage 1 choices are modeled as determined by the weighted combination of both model-free and model-based value predictions about the ultimate, Stage 2 value of each Stage 1 choice. Model-based values Q^ME are given by the learned values of the corresponding Stage 2 state, maximized over the two actions: , where s is the Stage 2 state predominantly produced by Stage 1 choice c. Model-free values are learned by TD(1), where . The Stage 1 choice probabilities are given by a logistic softmax, with a contribution from each value estimate weighted by its own free inverse temperature parameter: . Here, I(c = c_1,t−1) is a binary indicator of whether a choice repeats the previous trial’s choice, so the weight β^stick measures the general tendency to perseverate or switch regardless of feedback.

At the end of each trial, all value estimates Q for unchosen actions and unvisited states are decayed by multiplying by (1 − α).

The model has a total of five free parameters: four weights β (β^stick2, β^MB, β^MF, β^stick and a learning rate α. Our main measures of interest are β^MB and β^MF, measuring the contribution of model-based and model-free learning.

The free parameters of the model were estimated by maximizing the likelihood of each participant’s sequence of choices, using a distinct set of parameters for each game (i.e., each combination of participant, session, and task type: up to four games per participant, two task types at two timepoints). These were estimated jointly with group-level distributions over the entire population using an Expectation Maximization procedure (Huys et al., 2011) implemented in the Julia language (Bezanson, Karpinski, Shah, & Edelman, 2012). The per-game model-based and model-free weightings β^MB and β^MF, indicating the strength of each type of learning, were extracted for further analysis. (We also computed a standard measure of their relative fractional reliance, ; Daw et al, 2011.) These estimates were used as dependent variables in a series of regression analyses with group as the main explanatory variable of interest. All analyses controlled for task type, session, IQ, and age as additional independent variables. In followup analyses, we included interactions of group by session or task, and measures of disease severity (BMI, duration illness, and YBC-EDS); all covariates were z-scored. The regressions were conducted using mixed-effects logistic regression, and estimated using Julia’s MixedModels package. All within-subject parameters (e.g. task and session) were taken as random effects per participant, so as the to capture the repeated-measure structure of the data (e.g., tasks repeated at two timepoints) and also the imbalanced data (e.g., not all subjects completed both timepoints).