On attitude representations for optimization-based Bayesian smoothing

This paper presents a study of the behavior of seven different attitude representations in a batchwise orientation smoothing problem. The representations include the well-known unit quaternions and rotation vectors(/axis-angle), as well as modified Rodrigues parameters (MRPs). We consider error states as well as direct orientation formulations for the orientation and propose two methods to handle the singularity of MRPs in the latter case. The Bayesian smoothing problem is posed as a maximum a posteriori estimate with Gaussian noise, which results in a non-linear weighted least squares problem. With this we estimate the trajectory for a single inertial measurement unit with only sparse magnetometer samples for two challenging scenarios. In the evaluation we mainly focus on the convergence but also consider the estimation errors. Monte Carlo simulations show that orientation error states, with rotation vectors or MRPs, in general converge faster than other representations. However, with an initialization of the optimization problem up to deviations of 90 degrees MRPs with one of the proposed singularity handlings performs similar, but with a much smaller memory consumption. The source code for this study is available online.