TY - JOUR
T1 - Performance analysis of a video quality ruler methodology for subjective quality assessment
AU - Freitas, Pedro Garcia
AU - Silva, Alexandre Fieno
AU - Redi, Judith A.
AU - Farias, Mylène C.Q.
PY - 2018
Y1 - 2018
N2 - Objective video quality metrics are designed to be as reliable as the subjective quality assessments on which they are calibrated and validated. However, existing standard methodologies for subjective video quality assessment provide low reliable results for some conditions. We investigate whether an extension of the quality ruler experimental methodology, originally defined for images and shown to be more reliable than, e.g., standard single stimulus (SS) methods, can be adapted to reliably assess the quality of videos. The video quality ruler methodology allows subjects to assess video quality using a set of reference anchor images (the ruler), spanning a wide range of quality altogether, but closely spaced in function of quality one from the other. Subjects are asked to compare the quality of the displayed test video with the quality of these anchor images, displayed on a tablet, and indicate which of the reference images matches in quality the test video. As a result, the video quality assessment task is reduced to a set of visual comparisons between video and reference image quality. We describe how to adapt the original quality ruler methodology to video quality assessment, and we compare the proposed methodology with two other, widely used experimental methodologies: the single stimulus (SS) and the double stimulus (DS) method. Our results show that video quality ruler is a reliable method to assess video quality according to a multitude of criteria.
AB - Objective video quality metrics are designed to be as reliable as the subjective quality assessments on which they are calibrated and validated. However, existing standard methodologies for subjective video quality assessment provide low reliable results for some conditions. We investigate whether an extension of the quality ruler experimental methodology, originally defined for images and shown to be more reliable than, e.g., standard single stimulus (SS) methods, can be adapted to reliably assess the quality of videos. The video quality ruler methodology allows subjects to assess video quality using a set of reference anchor images (the ruler), spanning a wide range of quality altogether, but closely spaced in function of quality one from the other. Subjects are asked to compare the quality of the displayed test video with the quality of these anchor images, displayed on a tablet, and indicate which of the reference images matches in quality the test video. As a result, the video quality assessment task is reduced to a set of visual comparisons between video and reference image quality. We describe how to adapt the original quality ruler methodology to video quality assessment, and we compare the proposed methodology with two other, widely used experimental methodologies: the single stimulus (SS) and the double stimulus (DS) method. Our results show that video quality ruler is a reliable method to assess video quality according to a multitude of criteria.
KW - quality ruler
KW - subjective experiment
KW - subjective quality assessment
KW - testing conditions and methods
UR - http://www.scopus.com/inward/record.url?scp=85054299865&partnerID=8YFLogxK
U2 - 10.1117/1.JEI.27.5.053020
DO - 10.1117/1.JEI.27.5.053020
M3 - Article
AN - SCOPUS:85054299865
VL - 27
SP - 1
EP - 10
JO - Journal of Electronic Imaging
JF - Journal of Electronic Imaging
SN - 1017-9909
IS - 5
M1 - 053020
ER -