Transiently crosslinked actin filament networks allow cells to combine elastic rigidity with the ability to deform viscoelastically. Theoretical models of semiflexible polymer networks predict that the crosslinker unbinding rate governs the timescale beyond which viscoelastic flow occurs. However a direct comparison between network and crosslinker dynamics is lacking. Here we measure the network's stress relaxation timescale using rheology and the lifetime of bound crosslinkers using fluorescence recovery after photobleaching (FRAP). Intriguingly, we observe that the crosslinker unbinding rate measured by FRAP is more than an order of magnitude slower than the rate measured by rheology. We rationalize this difference with a three-state model where crosslinkers are bound to either 0, 1 or 2 filaments, which allows us to extract crosslinker transition rates that are otherwise difficult to access. We find that the unbinding rate of singly bound crosslinkers is nearly two orders of magnitude slower than for doubly bound ones. We attribute the increased unbinding rate of doubly bound crosslinkers to the high stiffness of biopolymers, which frustrates crosslinker binding.