Micromechanics, which can be used to relate the properties of a composite to the properties of individual constituents, is considered a good approach to understanding the fundamental mechanisms behind the behavior of asphalt materials. Compared with the semi-empirical and numerical micromechanical models, analytical micromechanical models do not need calibration factors. In addition, they can provide analytical solutions on the basis of a series of assumptions. Using these models, researchers have separated the effects of different stiffening mechanisms (i.e., the volume-filling reinforcement, the physicochemical reinforcement, and the particle-contact reinforcement) for mastic. However, similar research work has not been conducted for asphalt mixtures and, moreover, the characteristics of the particle-contact reinforcement have not been deeply analyzed by researchers. Therefore, this paper aims to understand the stiffness of asphalt mixture through micromechanics. The focus of this study was on porous asphalt mixture where particle-contact reinforcement plays an important role in its behavior. The stiffening effects of different mechanisms were separated using analytical micromechanical models. The effects of temperature/frequency and the properties of the matrix phase on the stiffening effect of the particle-contact reinforcement were analyzed.