Hydrogen is playing a key role in the transition to a sustainable economy and in a variety of industrial processes. For its safe handling, the detection of hydrogen in a fast, reliable and accurate manner is crucial. Thin film metal hydride based optical hydrogen sensors provide an attractive option to sense hydrogen in a variety of conditions and have an attractive safety benefit over other methods of detection: They do not require the presence of electrical leads near the sensing area. These sensors rely on a change of the optical properties arising from a change in the hydrogenation of the metal hydride sensing layer in response to a different partial hydrogen pressure in the environment of the sensor. In this paper, we review how the performance and characteristics of an optical hydrogen sensor are related to the material properties of the metal hydride sensing layer and we discuss previously considered materials and challenges and opportunities left for the future.