From an orbital perspective, I think you can go by the same reasoning as why in general SN2 reactions don't occur on double bonds: there is no constructive overlap possible with the antibonding pi orbital, so there is no gain in energy when the nucleophile approaches the alkene.
So, the energy of the antibonding pi orbital may be lower than of the antibonding sigma orbital between the allylic carbon and Cl, but there is no gain in using it to form a new bond with the nucleophile.
The general rule is that electrons flow from the HOMO of the nucleophile to the LUMO of the electrophile. It doesn't specify that electrons from from the (highest) HOMO to the lowest LUMO.