The vacuum referred binding energy (VRBE)-guided design of Bi3+-based storage and afterglow materials together with charge carrier trapping processes is explored with a study on bismuth- and lanthanide-doped rare earth ortho-phosphates. By combining Bi3+ with the shallow hole trap of Tb3+ or Pr3+, Bi3+ appears to act as a deep electron trap and as a hole recombination center in YPO4. Combining Bi3+ with the deep electron trap of Tm3+, Sm3+, Yb3+, or Eu3+, Bi3+ appears to act as a shallow hole trap in YPO4. Here recombination is also realized by means of hole release instead of more commonly reported electron release. Holes are released from Bi4+ and then recombine through the valence band with the electrons trapped at Ln2+ to produce Ln3+ 4f-4f emission. Lu3+ was introduced into YPO4 to engineer the valence band (VB) energy and to tailor the hole trap depth of Bi3+ in Y1-xLuxPO4 solid solutions. The results show that with increasing x the VRBE at the valence band top moves downward and the hole trap depth of Bi3+ increases. With a deep understanding of the Bi2+ and Bi3+ trap level locations and on the charge carrier trapping process, this work broadens the avenue to explore new persistent luminescence and storage materials using Bi3+ both as electron and hole traps.