Ok, so let's think about what has to happen for inversion to occur. As discodermolide said, you have to go through some sort of a quasi planar intermediate (e.g. trigonal planar for N/P) to get to the other enantiomer - can you draw molecular orbital diagrams for both cases (i.e. tetrahedral and trigonal planar). In a nutshell, my reasoning would be this: as you go down a group, the s/p gap increases and s/p mixing and hence hybridization decrease in extent. For nitrogen it is fairly easy to change shape as the s/p mixing is strong and it can go from sp3 to sp2 and again to sp3 with little effort. However, bonding in P is determined more by the intrinsic angles between px, py and pz orbitals than by tetrahedral sp3 mixing (see the experimental bond angles trends as you go down a group for confirmation of this). Hence a change of shape would actually require some s/p mixing to occur, as deviation from the atomic inter-p angles is not possible (or very highly energetically unfavorable) otherwise. This however is much harder than in N, because of the s/p gap. Does this make sense?