Two things could be happening. One is that acetone might be extracting plasticizers which changes the mechanical properties of the o-rings, making them brittle. Plasticizers are put into polymers to make them more pliable.
The other things that could happen is that the fluoropolymers could be "swelling" with acetone. Swelling with solvent is something really only polymers do. A material isn't dissolving, but a solvent is penetrating the polymer matrix. This too can greatly modify the mechanical properties of the o-rings, and it also can really change the volume they take up if they swell enough, causes whatever the rest of the seal is to abrade them.