Judging 'argumentative texts'

Two types of processes?

http://www.youtube.com/watch?v=z2f_Ue45KWM

So...

Gaze duration data might result from different cognitive processes:

$\to$ shorter visits in both texts (reading for building a first 'mental model' / scanning)

$\to$ longer visits in one of both texts (reading for text comprehension)

But...

How to statistically model the resulting duration data?

... without categorisation (setting tresholds to distinguish scanning from text comprehension)

... and avoiding aggregation

Goal of this research:

build and test statistical models that acknowledge the data generating cognitive processes and that do not make use of 'trimming', 'setting tresholds' or 'aggregation'

Procedure

• 26 high school teachers (Dutch) voluntary participated

• each did 10 comparisons of 2 argumentative texts from 10th graders

• 3 batches of comparisons with random allocation of judges to one of the batches

• all batches similar composition of comparisons regarding the characteristics of the pairs; the pairs, however not the same

• Tobii TX300 dark pupil eye-tracker with a 23-inch TFT monitor (max. resolution of 1920 x 1080 pixels)

• data sampled binocularly at 300 Hz

AOI's

The data

Statistical model 1

Simple mixed effects model

= 'Gaze Event duration' of AOI visits are nested within the combination of juges and comparisons

$$y_{i\left(jk\right)}={\beta }_0+{\mu }_{0j}+{\nu }_{0k}+{ϵ}_{i\left(jk\right)}$$

with:

• $y_{i\left(jk\right)}$ = gaze event duration of a visit to an AOI (text);
• ${\beta }_0$ = the intercept (overall average duration);
• ${\mu }_{0j}$ = unique effect of judge $j$;
• ${\nu }_{0k}$ = unique effect of comparison $k$;
• ${ϵ}_{i\left(jk\right)}$ = residual;

Statistical model 2

first mixed effects finite mixture model

Model1 + assuming two data generating processes + gaze event durations from 'text comprehension' differ for judges and comparisons

$$\begin{array}{cc}{y}_{i\left(jk\right)}=& \theta \times ({\beta }_1+{ϵ}_{1i\left(jk\right)})+\\ & (1-\theta )\times ({\beta }_2+{\mu }_{2j}+{\nu }_{2k}+{ϵ}_{2i\left(jk\right)})\end{array}$$

with:

• ${\beta }_1$ = intercept 1, overall average duration process 1 $\to$ 'scanning';
• ${\beta }_2$ = intercept 2, overall average duration process 2 $\to$ 'text comprehension reading';
• ${\mu }_{2j}$ = unique effect of judge $j$ on 'text comprehension reading';
• ${\nu }_{2k}$ = unique effect of comparison $k$ on 'text comprehension reading';
• ${ϵ}_{1i\left(jk\right)}$ & ${ϵ}_{2i\left(jk\right)}$ = residuals;
• $\theta$ = mixing proportion (weight)

Statistical model 3

Model 1 + assuming two data generating processes + gaze event durations from 'scanning' differ for judges and comparisons

$$\begin{array}{cc}{y}_{i\left(jk\right)}=& \theta \times ({\beta }_1+{\mu }_{1j}+{\nu }_{1k}+{ϵ}_{1i\left(jk\right)})+\\ & (1-\theta )\times ({\beta }_2+{ϵ}_{2i\left(jk\right)})\end{array}$$

with:

• ${\beta }_1$ = intercept 1, overall average duration process 1 $\to$ 'scanning';
• ${\beta }_2$ = intercept 2, overall average duration process 2 $\to$ 'text comprehension reading';
• ${\mu }_{1j}$ = unique effect of judge $j$ on 'scanning';
• ${\nu }_{1k}$ = unique effect of comparison $k$ on 'scanning';
• ${ϵ}_{1i\left(jk\right)}$ & ${ϵ}_{2i\left(jk\right)}$ = residuals;
• $\theta$ = mixing proportion (weight)

Statistical model 4

Model 1 + two data generating processes + gaze event durations from both processes differ for judges and comparisons

$$\begin{array}{cc}{y}_{i\left(jk\right)}=& \theta \times ({\beta }_1+{\mu }_{1j}+{\nu }_{1k}+{ϵ}_{1i\left(jk\right)})+\\ & (1-\theta )\times ({\beta }_2+{\mu }_{2j}+{\nu }_{2k}+{ϵ}_{2i\left(jk\right)})\end{array}$$

With:

• ${\beta }_1$ = intercept 1, overall average duration process 1 $\to$ 'scanning';
• ${\beta }_2$ = intercept 2, overall average duration process 2 $\to$ 'text comprehension reading';
• ${\mu }_{1j}$ & ${\mu }_{2j}$ = unique effect of judge $j$ on 'scanning' & 'text comprehension reading';
• ${\nu }_{1k}$ & ${\nu }_{2k}$ = unique effect of comparison $k$ on 'scanning' & 'text comprehension reading';
• ${ϵ}_{1i\left(jk\right)}$ & ${ϵ}_{2i\left(jk\right)}$ = residuals;
• $\theta$ = mixing proportion (weight)

Analysis (Bayesian estimation)

• brms (wrapper around Stan) to estimate the models making use of MCMC in R

• flat (uninformed) priors

• 4 chains of 15000 iterations (with 1000 burn-in it.)

• compare the models with 'leave-one-out cross-validation' approach

• summarize best model (interpret the posterior distribution)

Comparison of the models

$\to$ Model 4 fits best

• 2 data generating processes

• differences between judges AND comparisons

Posterior distribution of fixed effects

Posterior distribution of theta & residual variances

Posterior distribution of random effects

Diff. between judges in reading for text comprehension

Diff. between comparisons in reading for text comprehension

Conclusion

• model that acknowledges gaze event durations come from different cognitive processes is most likely

• judges and comparisons result in different gaze event durations

• mixed effects finite mixture models are promising for this kind of data

  • no need to decide which fixation duration point to scanning and which to text comprehension

  • opens up many possibilities for follow-up research questions and analyses

Discussion

• what about more than 2 processes?

• more informed priors

• a need for triangulation to understand the two types of processes

• what if representations are not texts; how task specific are these models?

• other applications of (mixed effects) finite mixture models when modelling process data?