Giovanni Pignoni

Cognitive Workload Tool

An open-source tool that estimates cognitive workload from pupil size outside the lab. It works out how large the pupil should be for the light reaching the eye and keeps only what the light does not explain.

Why

The pupil dilates with cognitive workload, which makes pupillometry an attractive measure in field studies. But the pupil reacts far more strongly to light than to mental effort. In a laboratory the lighting is held constant. On a ship's bridge, in a vehicle or inside a VR headset it changes all the time, and a change in brightness can hide or imitate a change in workload.

I wanted a measure that can be used where the light is not controlled.

The approach

The tool estimates the pupil size that the light alone should produce and reports what is left over, the ΔPD:

ΔPD(t) = measured pupil(t) − expected pupil(luminance(t))
  1. Luminance at the eye. The eye tracker's scene video and gaze give the brightness of what the participant is looking at: a circle around the gaze point, weighted against the rest of the view. On a VR headset the absolute level comes from the display's photometry. With glasses-type trackers, whose scene camera adjusts its exposure automatically, it comes from a small lux sensor I mounted on the tracker.
  2. Expected pupil. The luminance goes into Watson and Yellott's unified formula for light-adapted pupil size, with the participant's response latency and pupil dynamics.
  3. ΔPD. The expected pupil is subtracted from the measured one. What remains is attributed to factors other than light, such as workload.

A short participant calibration, a full-field colour sequence of about five minutes, fits each person's light sensitivity, colour weights, latency and pupil dynamics. These parameters are then reused for that participant's other recordings with the same device.

Implementation

The analysis is a Python package that also runs without a user interface, for batch processing. Around it there are three tools.

Analysis app. You open a recording folder; the scene video is analysed in the background and cached. Every parameter change then updates the measured pupil, the expected pupil and the ΔPD plots at once. A video preview shows the analysed areas on the scene frame under the cursor. An export writes one package per participant: pupil, ΔPD, per-event means, sensor data and the parameters used.

Analysis app showing measured and expected pupil diameter, the ΔPD rising during two task phases, an ECG with detected heartbeats, skin conductance and a moving RMSSD trace, next to a heart rate variability table

The analysis app on synthetic data: the light changes every few seconds while a task dilates the pupil twice, and the ΔPD isolates the two task phases.

Sensor logger. It records the lux sensor, a Shimmer (ECG) and an EmotiBit (skin conductance) side by side on the computer's clock, together with the phases of the experiment. The ECG gives heart rate variability, which the analysis app shows and exports next to the ΔPD.

Sensor logger during a recording, with live plots of illuminance, ECG and skin conductance and a protocol at phase 2 of 4

Calibration presenter. A web page, with nothing to install, that plays the calibration sequence full-screen on a monitor or inside a VR headset and logs every run.

Calibration presenter with a timeline of grey, black and blue full-field steps and the run settings below it

The tool reads recordings from the Varjo XR-4, Pupil Core, Pupil Neon and Tobii Pro Glasses 3. All screenshots use synthetic data.

Status

The tool started in 2018–2019 as part of my master's thesis in Interaction Design (MIXD) at NTNU Gjøvik, and it was used in field studies, including a maritime usability study. The lux sensor method was published in IEEE Sensors Journal in 2021.

Version 2.0 is a rewrite and is still in development. The Varjo XR-4 and Pupil Core paths are tested on real recordings. The Pupil Neon and Tobii Pro Glasses 3 readers are experimental: they were written from the manufacturers' documentation, and their results carry a warning. The method used for the Varjo headset is described in a manuscript that is in preparation and not yet published. The 1.x tools are kept for archive on separate branches of the repository.

Links

← All projects