A Safety Framework for Critical Systems Utilising Deep Neural Networks

Xingyu Zhao, Alec Banks, James Sharp, Valentin Robu, David Flynn, Michael Fisher, Xiaowei Huang

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

1 Citation (Scopus)

Abstract

Increasingly sophisticated mathematical modelling processes from Machine Learning are being used to analyse complex data. However, the performance and explainability of these models within practical critical systems requires a rigorous and continuous verification of their safe utilisation. Working towards addressing this challenge, this paper presents a principled novel safety argument framework for critical systems that utilise deep neural networks. The approach allows various forms of predictions, e.g., future reliability of passing some demands, or confidence on a required reliability level. It is supported by a Bayesian analysis using operational data and the recent verification and validation techniques for deep learning. The prediction is conservative – it starts with partial prior knowledge obtained from lifecycle activities and then determines the worst-case prediction. Open challenges are also identified.

Original languageEnglish
Title of host publicationComputer Safety, Reliability, and Security - 39th International Conference, SAFECOMP 2020, Proceedings
EditorsAntónio Casimiro, Pedro Ferreira, Frank Ortmeier, Friedemann Bitsch
PublisherSpringer Open
Pages244-259
Number of pages16
ISBN (Print)9783030545482
DOIs
Publication statusPublished - 2020
Externally publishedYes
Event39th International Conference on Computer Safety, Reliability and Security, SAFECOMP 2020 - Lisbon, Portugal
Duration: 16 Sep 202018 Sep 2020

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12234 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference39th International Conference on Computer Safety, Reliability and Security, SAFECOMP 2020
CountryPortugal
CityLisbon
Period16/09/2018/09/20

Keywords

  • Assurance arguments
  • Bayesian inference
  • Deep learning verification
  • Quantitative claims
  • Reliability claims
  • Safe AI
  • Safety cases

Fingerprint Dive into the research topics of 'A Safety Framework for Critical Systems Utilising Deep Neural Networks'. Together they form a unique fingerprint.

Cite this