Brillouin light scattering is an established technique to study magnons, the elementary excitations of a magnet. Its efficiency can be enhanced by cavities that concentrate the light intensity. Here, we theoretically study inelastic scattering of photons by a magnetic sphere that supports optical whispering gallery modes in a plane normal to the magnetization. Magnons with low angular momenta scatter the light in the forward direction with a pronounced asymmetry in the Stokes and the anti-Stokes scattering strength, consistent with earlier studies. Magnons with large angular momenta constitute Damon-Eschbach modes which are shown to inelastically reflect light. The reflection spectrum contains either a Stokes or anti-Stokes peak, depending on the direction of the magnetization, a selection rule that can be explained by the chirality of the Damon-Eshbach magnons. The controllable energy transfer can be used to manage the thermodynamics of the magnet by light.