In general, the experimental method assumes that all respondents were gazing only at the center of the 20% viewport width image, during the last 4 seconds of each stimulus. The method also assumes that respondents adhere to the experimental procedure and don’t significantly change environment during the experiment; for example, no excessive head movement or changing from an indoor to an outdoor setting. Gaze points where the respondents occasionally were looking elsewhere or sessions where the participant did not fully follow behaviour instructions have not been removed from the dataset. Those gaze points of course will have impact on the mean errors.

Therefore, the conclusion can be made that the potential accuracy of the algorithm is higher than reported; however, this white paper reports on the effective accuracy likely to be seen when using the product.

One of the main motivations of the white paper is to recommend minimum AOI sizes within experiments; however, this is difficult due to device viewport sizes and resolutions changing significantly between participants. To simplify recommendations, this white paper focuses on minimum AOI size recommendations as a percentage of the smaller screen dimension (which for portrait orientations is the screen width). In live usage we aim to accurately register approximately 80% of all respondents gazing at a particular area of interest and capture approximately half of the theoretical maximal time spent on that area. To achieve this with Sticky’s algorithm, the minimum area of interest in a mobile portrait experiment should be at least 20% of the screen width.

Due to the reasons discussed in the methodology section above, in an experiment we can expect the accuracy of several sessions to be decreased by participant behavior. This behavior is more pronounced in a mobile environment compared to desktop. We therefore recommend using a minimum of 40 participant sessions. Further details of why 40 participants are recommended can be found in the appendix (Recommended number of experiment participants).

Portrait

The average radial gaze error in Sticky’s algorithm is 6% of the screen, with a 5% standard deviation in the sample (89 sessions).

Furthermore, at 40 sessions, there is a confidence interval in the mean radial error of ± 1.5%. Thus, if this test is repeated with 40 respondents or more, there is a 95% chance that the resulting average radial error is 6% ± 1.5%.

Landscape

The average radial gaze error in Sticky’s algorithm is 7% of the screen, with a 6% standard deviation in the sample (88 sessions).

Furthermore, at 40 sessions, there is a confidence interval in the mean radial error of ± 1.8%. Thus, if this test is repeated with 40 respondents or more, there is a 95% chance that the resulting average radial error is 7% ± 1.8%.

Sticky’s algorithm automatically estimates the quality of each session and removes sessions not suitable for eye tracking. The most common reasons for this are:

The largest weakness of Sticky’s algorithm is the low robustness compared to a stationary eye tracker. However, it could be argued that a stationary eye tracker on a controlled environment is not a good representation of normal behavior when consuming mobile media. This issue of low robustness can be solved with oversampling as the cost of recording sessions is much cheaper using webcams and online panels.

The Sticky platform does not apply any fixation filter, to resolve gaze data. This means that we deliver raw gaze data. Fixation filters used and evaluated in the past have shown to be of limited use. Front facing cameras provide too few data points (on average 15 Hz). However, the use of a relevant sample size (app. 40 usable) has shown that the technology does give normalized data distribution where the error can be decided by an average deviation (a deviation normally measured in degrees but also as percentage of the screen or as pixels). We do use a median filter, a type of noise reduction filtering, which we apply to our predicted raw gaze output.

The Sticky platform only counts AOI hits if the gaze data is within the AOI boundaries. If the gaze is outside the AOI it won’t be allocated to the AOI during the analysis.

The data provided by the platform is based on sessions meeting a minimum quality threshold, i.e., all sessions have passed a data validation check where significant outliers are identified and removed (classified as “unusable”).

The following section contains data from the portrait orientation mobile experiment, though similar considerations apply to landscape orientations. Providing recommendations on participant sample size is complicated as increasing the sample size will increase the statistical significance; while the accuracy requirements of a specific experiment and the quality of the recorded sessions means one size does not fit all. Here we have based out general recommendation on two things.

1. Accuracy requirements we have placed on our sample metrics.

2. Quantitative evidence from this white paper on the benefits of increasing sample size.

A minimum threshold was put on the confidence the system should give us across two different sample statistics, the radial error in pixels and the time viewed in milliseconds. The confidence interval in the radial error was required to be less than 1.5% of the screen and the confidence interval in the time viewed to be less than 10% of the total 5 second viewable time. These requirements were met at 40 sessions with a 95% confidence interval in mean radial error of 6.2% ± 1.47% and a 95% confidence interval in the time viewed of 1.7± 0.46 seconds.

By plotting the confidence in the above sample statistics for different sample sizes, we can see the relative benefit of increasing the sample size below. If an individual experiment requires higher confidence, then you may want to increase the sample size, however Sticky’s recommendation is that given the increase cost of larger sample sizes there are diminishing returns after 40 participants for most studies.