Abstract
Interval Markov Decision Processes (IMDPs) are finitestate uncertain Markov models, where the transition probabilities belong to intervals. Recently, there has been a surge of research on employing IMDPs as abstractions of stochastic systems for control synthesis. However, due to the absence of algorithms for synthesis over IMDPs with continuous actionspaces, the actionspace is assumed discrete apriori, which is a restrictive assumption for many applications. Motivated by this, we introduce continuousaction IMDPs (caIMDPs), where the bounds on transition probabilities are functions of the action variables, and study value iteration for maximizing expected cumulative rewards. Specifically, we decompose the maxmin problem associated to value iteration to Q max problems, where Q is the number of states of the caIMDP. Then, exploiting the simple form of these max problems, we identify cases where value iteration over caIMDPs can be solved efficiently (e.g., with linear or convex programming). We also gain other interesting insights: e.g., in certain cases where the action set A is a polytope, synthesis over a discreteaction IMDP, where the actions are the vertices of A, is sufficient for optimality. We demonstrate our results on a numerical example. Finally, we include a short discussion on employing caIMDPs as abstractions for control synthesis.
Original language  English 

Title of host publication  HSCC 2023  Proceedings of the 26th ACM International Conference on Hybrid Systems 
Subtitle of host publication  Computation and Control, Part of CPSIoT Week 
Publisher  Association for Computing Machinery (ACM) 
Number of pages  19 
ISBN (Electronic)  9798400700330 
DOIs  
Publication status  Published  2023 
Event  26th ACM International Conference on Hybrid Systems: Computation and Control, HSCC 2023, Part of CPSIoT Week 2023  San Antonio, United States Duration: 10 May 2023 → 12 May 2023 
Conference
Conference  26th ACM International Conference on Hybrid Systems: Computation and Control, HSCC 2023, Part of CPSIoT Week 2023 

Country/Territory  United States 
City  San Antonio 
Period  10/05/23 → 12/05/23 
Keywords
 boundedparameter Markov decision processes
 control synthesis
 planning under uncertainty
 uncertain Markov decision processes
 value iteration
Fingerprint
Dive into the research topics of 'Interval Markov Decision Processes with Continuous ActionSpaces'. Together they form a unique fingerprint.Datasets

IntervalMDP.jl
Mathiesen, F. B. (Creator), GitHub, 5 Jan 2024
https://github.com/Zinoex/IMDP.jl
Dataset/Software: Software