It is well known that the quasinormal modes (or resonant states) of photonic structures can be associated with the poles of the scattering matrix of the system in the complex-frequency plane. In this work, the inverse problem, i.e., the reconstruction of the scattering matrix from the knowledge of the quasinormal modes, is addressed. We develop a general and scalable quasinormal-mode expansion of the scattering matrix, requiring only the complex eigenfrequencies and the far-field properties of the eigenmodes. The theory is validated by applying it to illustrative nanophotonic systems with multiple overlapping electromagnetic modes. The examples demonstrate that our theory provides an accurate first-principles prediction of the scattering properties, without the need for postulating ad hoc nonresonant channels.