Physical chemistry can be very math intensive. To be honest, the skills you gained in Calc 2 will not be nearly enough to really have a firm understanding of the subject. IMO, an understanding in linear algebra, calculus of several variable and differential equations are essential to really getting what is going on. It is possible to learn the math through a pchem course, but it will obviously be more difficult than having those skills under your belt prior. I'm not sure you can get a degree in chemistry without taking a pchem course.
When it comes down to it, assess how much you like the subject. If you really love it, then the work might not be that difficult.
I don't think any of us are trying to tell you not to pursuit it, but since most of us have degrees in chemistry, we understand how hard it can be at times.