Although the damage development in the railway steels is relatively well documented in the scientific literature, its appearance in railway crossings remains a quite complicated issue because of the extreme loading conditions during the train passage. The exact local conditions cannot be determined with sufficient accuracy, but the steel microstructure contains the traces of the mechanisms of microstructure and damage evolution. In this work, we discuss the microstructural evolution and damage in a field-loaded railway crossing made of cast austenitic Hadfield steel and subjected to impact and rolling contact fatigue (RCF) loading. A nanoscale twinning substructure surrounded by dislocation cells and a high dislocation density are identified using Electron Channeling Contrast Imaging (ECCI) and Transmission Electron Microscopy (TEM) in the deformed microstructure of Hadfield steels. The effect of these substructures as well as of the non-metallic inclusions and other casting defects on the damage development in the austenitic Hadfield steels is also discussed. Additionally, the strain-induced transformation of the austenite into martensite in the deformed crossing surface is studied by X-ray diffraction and magnetometer measurements. The results do not show evidence of strain-induced austenite-to-martensite transformation under impact and RCF loading of the railway crossings. ECCI in controlled diffraction conditions is used to study the fatigue crack growth in undeformed cast Hadfield steel specimens when subjected to laboratory scale fatigue testing. Furthermore, the role of twins and grain boundaries on the fatigue crack growth is discussed.