Atomic-scale disorder at the top interface of a Si quantum well is known to suppress valley splitting. Such disorder may be inherited from the underlying substrate and relaxed buffer growth, but can also arise at the top quantum well interface due to the random SiGe alloy. Here, we perform activation energy (transport) measurements in the quantum Hall regime to determine the source of the disorder affecting the valley splitting. We consider three Si/SiGe heterostructures with nominally identical substrates but different barriers at the top of the quantum well, including two samples with pure-Ge interfaces. For all three samples, we observe a surprisingly strong and universal dependence of the valley splitting on the electron density (Ev ∼ n2.7) over the entire experimental range (Ev = 30-200 μeV). We interpret these results via tight binding theory, arguing that the underlying valley physics is determined mainly by disorder arising from the substrate and relaxed buffer growth.