Improving Upper-limb Prosthesis Usability: Cognitive Workload Measures Quantify Task Difficulty

DRT

The DRT is a secondary task in which a visual, auditory, or tactile stimulus prompts the user to respond by pressing a button. As the primary task increases in difficulty, the response time typically increases, and stimulus detection rate typically decreases (i.e., the user does not respond). An ISO standard of the DRT (ISO 17488:2016, 2016) has been used extensively in driving contexts (Ranney, T. A., Baldwin, G. H. S., Smith, L. A., Mazzae, E. N., & Pierce, 2014; Chang et al., 2017; Stojmenova et al., 2017; Strayer et al., 2017; Stojmenova and Sodnik, 2018), in which the stimulus is presented at random intervals of 3-5 s. More broadly, secondary tasks have been applied to prosthesis tasks with promising results, such as auditory discrimination tasks (Witteveen et al., 2012), memory tasks (Valle et al., 2020), and games (Raveh et al., 2018b, 2018a). In contrast with the referenced uses of secondary tasks, we find the DRT attractive because trials are collected rapidly (every few seconds) and the response times are nearly continuous. The DRT has not been applied previously to prosthesis research.

EEG and Event-Related Potentials

EEG is the measure of electrical potentials produced by the brain at the scalp surface. Alpha waves (8-12 Hz) in parietal regions indicate cortical idling and alpha power decreases with increased task demands (Keil et al., 2006). Theta waves (4 - 7 Hz) in frontal midline regions arise when cognitive control is required for a task (i.e., the task cannot be completed through an automatic strategy). Two studies have measured alpha waves (Gonzalez et al., 2012; Parr et al., 2019), but no study has analyzed theta waves during prosthesis use.

Event-related potentials (ERPs) are the brain’s electrophysiological response to a particular sensory, cognitive, or motor event (Luck, 2005). ERPs contain several components that represent various stages in neural processing of an event. When used to measure cognitive workload, ERPs are usually elicited through a secondary task, such as a DRT (Strayer et al., 2014). The P3 component, a positive potential arising roughly 300-ms post-stimulus, decreases in amplitude as the primary (in our case, prosthesis) task increases in difficulty and requires more resource allocation (Luck, 2005). Only one previous study has used ERPs as a cognitive workload measure during prosthesis use (Deeny et al., 2014).

Pupillometry

The eyes have been described as the “visible part of the brain” (Hess and Janisse, 1978). Pupillometry is the continuous measure of pupil size over the course of a task. Pupil size increases with cognitive demands, demonstrated as early as the 1960s (Kahneman and Beatty, 1966). Because pupil size changes due to several environmental, neurological, and psychological factors, trial averaging is often used to produce a “task-evoked pupillary response” (Beatty, 1982). Measuring the percentage of pupil dilation provides a measure that is robust to inter-individual and inter-trial baseline pupil size differences (Payne et al., 1968). Pupillometry has been used only rarely in the prosthesis domain (White et al., 2017; Zahabi et al., 2019).

Electrocardiography

Electrocardiography (ECG) is the measure of electrical potentials produced from the heart. Several time-domain and frequency-domain measures are sensitive to cognitive workload (Charles and Nixon, 2019). For our study, we used the low frequency (0.02 – 0.06 Hz) to high frequency (0.15 – 0.5 Hz) ratio (LF/HF ratio) because it showed the greatest sensitivity in our pilot experiments. One other prosthesis study has used ECG to measure cognitive workload (Gonzalez et al., 2012).

NASA TLX Survey

The NASA TLX is a subjective survey designed to measure perceived workload (Hart and Staveland, 1988) through six different categories on a 100-point scale: mental demand, physical demand, temporal demand, performance, effort, and frustration. Participants compare categories pairwise based on perceived importance in the task, and individual weightings from these comparisons are used to produce a composite score. The TLX has been used widely across many domains, including prostheses (Gonzalez et al., 2012; Markovic et al., 2018, 2020; Shaw et al., 2019; Thomas et al., 2019).

Prosthesis Control

The prosthesis control methodology used in this study has been described previously (George et al., 2020a). In brief, sEMG was collected from an sEMG sleeve (George et al., 2020b) with the Grapevine System (Ripple Neuro LLC, Salt Lake City, UT). Thirty-two single-ended channels were acquired at 1 kHz and band-pass filtered between 15 Hz and 375 Hz with 4^th-order Butterworth filters, and 60, 120, and 180 Hz 2^nd-order Butterworth notch filters. After the sEMG sleeve was connected to the acquisition device, channels were manually inspected and removed if broken channels were detected (generally less than two channels). The differential pairs of all monopolar channels were calculated, and features (single-ended and differential) were created at 30 Hz using the mean absolute value of a 300-ms buffer (i.e., 528 features from an overlapping 300-ms boxcar filter). At 30 Hz, the buffer is updated every 33 ms. This update rate and buffer length has been used by our group extensively (George et al., 2018, 2020a).

sEMG was collected as participants mimicked preprogrammed movements of the virtual MSMS hand (Davoodi and Loeb, 2011). The preprogrammed movements consisted of index, middle, and ring finger flexions. Each movement consisted of a 0.7-s transition to flexed position, 4-s hold, and 0.7-s return to rest position. Participants completed two trials of each flexion as practice to gain familiarity with the virtual environment. After familiarization, participants completed five trials of each movement. Using a Gram-Schmidt forward selection algorithm (Nieveen et al., 2017), 48 sEMG features were selected as inputs to the decoder, a modified Kalman filter (George et al., 2020a). The 48 features and virtual hand kinematics were used to fit the parameters of the modified Kalman filter. After fitting the modified Kalman filter, users were given control over the virtual hand. We let the participants spend a few minutes exploring the control; in cases where the participants struggled to fully flex the fingers, participants repeated the five-trial mimicry and the modified Kalman filter was refitted.

DRT

We made a custom DRT system that interfaced with the Ripple Grapevine Digital I/O board. This system turned the tactile buzzer, a 10 mm x 2 mm vibration motor on a 4.5 V power supply, on or off when an output of the Digital I/O was set to high or low, respectively. The response button, when depressed, was recorded by the Ripple Grapevine system. The DRT vibrations were set to 1 s and turned off if the user pressed the response button before the 1 s had ended. Timestamps for the Digital I/O board are recorded at 30-kHz resolution. The DRT system was placed on the table near the participant and two separate cables for the vibration motor and response button were routed to the participant. We attached the DRT vibration motor to the collarbone with medical tape, opposite the hand used for the prosthesis task. We attached the response button to the index finger using a hook and loop fastener.

EEG & ERP Recordings

EEG was recorded based on the standard 10-20 system using a 34-electrode cap (Ripple Neuro LLC, Salt Lake City, UT). Electrode locations were: FP1, FP2, F7, F3, Fz, F4, F8, AFz, FT7, FT8, FC3, FCz, FC4, T3, C3, Cz, C4, T4, CP3, CPz, CP4, T5, P3, Pz, P4, T6, O1, Oz, O2, A1, A2, VEOL, HEOR, HEOL. The online reference was on electrode CPz, and the ground was AFz. We used Electro-Gel™ to bridge the connection between the electrodes and the scalp. Impedances of the electrodes were brought below 10 kOhm, typically close to 5 kOhm, using gentle scalp abrasion. We recorded the scalp EEG at 1 kHz and band-pass filtered between 1 Hz and 125 Hz with 4^th-order Butterworth filters.

Pupillometry Recordings

Pupil diameter was recorded using the Pupil Labs’ Pupil Core head-mounted pupillometry device. We recorded pupil diameter with both pupil cameras at 120 Hz and 400×400 resolution. The 2D diameter output from the Pupil Labs’ software was used for analysis, which contains a measured diameter and the measurement confidence, ranging from zero to one. Room lighting was kept constant at approximately 100 lux, as measured by an Urceri MT-912 light meter.

ECG Recordings

ECG was recorded with the five-wire, four-lead Shimmer3 ECG unit. The unit recorded at 512 Hz. The Vx electrode was placed at V5, as suggested in the Shimmer3 ECG user manual, and the remaining electrodes were placed on the chest in the direction of the right arm, left arm, right leg, and left leg. ECG recordings were programmatically started and stopped when a target set was started or finished, respectively. The Shimmer3 logs the data onto an internal SD card, which was later extracted using Shimmer3 Consensys software.

Virtual Target Task

Participants completed a virtual target task in the MSMS virtual environment (Davoodi and Loeb, 2011) for easy and hard difficulties. In the virtual target task, a spherical target indicates the desired position of each degree-of-freedom. When a degree-of-freedom is within a specified radius of the target, the target is green; outside the allowable radius, the target is red. For our target tasks, the target was placed halfway through the movement window, with a target size (i.e., allowable radius) of 35% and 15% of the movement window for the easy and hard difficulties, respectively.

Participants were instructed to focus most of their attention on the active target, which was only one degree-of-freedom at a time. Participants were instructed that their objective was to keep the target green, not to keep the active degree-of-freedom in the middle of the target. Participants were encouraged to stay focused on the task and to avoid talking during the task in order to reduce cognitive demands beyond the task itself.

The subsequent sections describe the target task paradigm for each group. Because the different cognitive load measures have differing recording requirements, the experimental paradigms were slightly different for the two groups. The difficulty of the task (i.e., target size) was identical for both groups.

DRT

We analyzed the DRT according to the ISO standard (ISO 17488:2016, 2016). Responses (button presses after vibrotactile stimuli) less than 100 ms or greater than 2500 ms were counted as a miss. Response times more than three scaled median absolute deviations from the median were excluded from the analysis. We measured the hit rates and response times during each target size.

EEG & ERP

EEG was analyzed using EEGLAB v2021.0. The data were first resampled to 250 Hz. We re-referenced the electrodes to electrodes A1 and A2. The data were filtered from 0.1 Hz to 30 Hz using a second-order Butterworth filter. Artificial blink and horizontal eye movement channels were created by subtracting VEOL from FP1, and HEOL from HEOR, respectively.

For the frequency analysis, 15-s bins were created for the duration of the active target and separated by target size. Artifacts were detected and removed if the blink or horizontal movement channels exceeded a 100-µV threshold within a 200-ms sliding window. The 200-ms window passed across the 15-s bin in 50 ms increments. Individual bins were Hann-windowed prior to calculating the power-spectral density of each trial to avoid edge effects. Power-spectral densities of each trial were averaged together. The power for the alpha band (8-12 Hz) on electrode Pz and theta (4-7 Hz) band on electrode Fz were calculated. The percentages of power in the alpha and theta bands were calculated by dividing the power in the selected bands by the total power.

For the ERP analysis, bins were created from 200 ms prior to the buzzer onset to 1100 ms after the onset and separated by target size. Artifacts were detected and removed if the blink or horizontal movement channels exceeded a roughly 60-µV threshold within a 200-ms sliding window. The threshold was slightly adjusted when blinks or horizontal eye movements were not detected by the initial 60-µV threshold. The 200-ms window passed across the 1300 ms bin in 50 ms increments. Non-artifact trials were averaged together to produce an averaged ERP for each participant. The signed area was calculated from 200 ms to 650 ms (Strayer et al., 2014) to calculate the P3 ERP size. Averaged ERPs for each participant were averaged across participants to produce grand-averaged ERPs.

Pupillometry

Pupil recordings were aggregated by target size. We removed outliers defined as measurements greater than three scaled mean absolute deviations from the median of 60 samples (a 0.5-s window). We removed measurements with measurement confidence less than 0.8. Removed measurements were replaced with linearly interpolated values. Target trials with more than 20% low confidence measurements were removed from the aggregated set. The pre-trial baseline diameter, 1 s before the target became active, was subtracted from each trial. The percentage change in pupil size was calculated by dividing the response by the average size of the pupil during the 1-s pre-trial baseline. The baseline-subtracted pupillary responses of both eyes were combined and averaged to find the average pupillary response to the target task. The averaged pupillary response from the mimicry target (where the computer controlled the virtual hand) was subtracted from the averaged pupillary response to the active target (where the user controlled the virtual hand) to create a difference wave that would mitigate target-size dependent luminance effects in the response. We calculated the average value of the difference wave during the 5 s the target was active.

ECG

We obtained the LF/HF heart-rate variability ratio using the standard settings of PhysioZoo version 1.2.0 (Behar et al., 2018). The ECG was band-pass filtered from 3 Hz to 100 Hz with second and fifth-order Butterworth filters, respectively. Peaks in the ECG were detected using an energy-based QRS detector (Behar et al., 2014). The heart rate variability (intervals between normal heart beats) was calculated after removing outliers in the R-R peak intervals. Outliers were defined as intervals above or below 20% of the average of the moving window, which was 21 intervals. Power spectral density of the heart rate variability was calculated with Welch’s method. Finally, the LF/HF ratio was calculated by dividing the power in the low frequency region (0.04 Hz to 0.15 Hz) by the power in the high frequency region (0.15 Hz to 0.4 Hz).

Target Task Performance

We calculated the average percentage of time spent within the target window for each target size for each participant.

Statistical Procedures: Across-Subject

We tested the paired values derived from each measure for normality using the Shapiro-Wilk test. If the paired values were normally distributed, we used a paired t-test to show differences between the responses to the large and small targets. If the paired values were nonparametric, we used Wilcoxon’s signed-rank test. Because only one amputee participant completed each experiment, we did not include amputee participant results in our across-subject statistical measures and instead overlay results from amputee participants with the results of non-amputee participants.

Statistical Procedures: Within-Subject

Different measures may work well for some individuals, but not others. Additionally, due to the costs associated with implanting neural and electromyographic interfaces, it is common for studies to be completed with only a few subjects. We therefore were interested in the within-subject reliability of the cognitive workload measures. We completed within-subject analyses for each subject for each measure as appropriate for the measure and experimental paradigm. For the DRT, we conducted a two-sample t-test for all the DRT trials in an experiment. For the EEG & ERP measures, we conducted paired-sample t-tests with the average response for each of the eight rounds of the target task. For the pupil & ECG measures, we conducted paired-sample t-tests with the average response for each of the four rounds of the target task. We calculated the p-value from the statistical test and the absolute effect size using Cohen’s D for each participant. We report the median and first and third quartiles of p and D across non-amputee participants and the individual outcomes for the amputee participant. We report the number of participants for whom p < 0.05 by measure.