TY - GEN
T1 - Assessing the Safety and Reliability of Autonomous Vehicles from Road Testing
AU - Zhao, Xingyu
AU - Robu, Valentin
AU - Flynn, David
AU - Salako, Kizito
AU - Strigini, Lorenzo
PY - 2019
Y1 - 2019
N2 - There is an urgent societal need to assess whether autonomous vehicles (AVs) are safe enough. From published quantitative safety and reliability assessments of AVs, we know that, given the goal of predicting very low rates of accidents, road testing alone requires infeasible numbers of miles to be driven. However, previous analyses do not consider any knowledge prior to road testing - knowledge which could bring substantial advantages if the AV design allows strong expectations of safety before road testing. We present the advantages of a new variant of Conservative Bayesian Inference (CBI), which uses prior knowledge while avoiding optimistic biases. We then study the trend of disengagements (take-overs by human drivers) by applying Software Reliability Growth Models (SRGMs) to data from Waymo's public road testing over 51 months, in view of the practice of software updates during this testing. Our approach is to not trust any specific SRGM, but to assess forecast accuracy and then improve forecasts. We show that, coupled with accuracy assessment and recalibration techniques, SRGMs could be a valuable test planning aid.
AB - There is an urgent societal need to assess whether autonomous vehicles (AVs) are safe enough. From published quantitative safety and reliability assessments of AVs, we know that, given the goal of predicting very low rates of accidents, road testing alone requires infeasible numbers of miles to be driven. However, previous analyses do not consider any knowledge prior to road testing - knowledge which could bring substantial advantages if the AV design allows strong expectations of safety before road testing. We present the advantages of a new variant of Conservative Bayesian Inference (CBI), which uses prior knowledge while avoiding optimistic biases. We then study the trend of disengagements (take-overs by human drivers) by applying Software Reliability Growth Models (SRGMs) to data from Waymo's public road testing over 51 months, in view of the practice of software updates during this testing. Our approach is to not trust any specific SRGM, but to assess forecast accuracy and then improve forecasts. We show that, coupled with accuracy assessment and recalibration techniques, SRGMs could be a valuable test planning aid.
KW - Autonomous vehicles
KW - Conservative Bayesian inference
KW - Reliability claims
KW - Safety-critical systems
KW - Software reliability growth model
KW - Statistical testing
KW - Ultra-high reliability
UR - http://www.scopus.com/inward/record.url?scp=85077947557&partnerID=8YFLogxK
U2 - 10.1109/ISSRE.2019.00012
DO - 10.1109/ISSRE.2019.00012
M3 - Conference contribution
AN - SCOPUS:85077947557
T3 - Proceedings - International Symposium on Software Reliability Engineering, ISSRE
SP - 13
EP - 23
BT - Proceedings - 2019 IEEE 30th International Symposium on Software Reliability Engineering, ISSRE 2019
A2 - Wolter, Katinka
A2 - Schieferdecker, Ina
A2 - Gallina, Barbara
A2 - Cukier, Michel
A2 - Natella, Roberto
A2 - Ivaki, Naghmeh
A2 - Laranjeiro, Nuno
PB - IEEE
T2 - 30th IEEE International Symposium on Software Reliability Engineering, ISSRE 2019
Y2 - 28 October 2019 through 31 October 2019
ER -