The way I learned physical chemistry was by going ot the basics first using a general chemistry book. Then, I spent days and days reading a physical chemistry book. Time consuming, but it worked in the end and I think it was worth it. The main thing is, just don't give up and don't let complicated looking derivations intimidate you. Be able to understand the text without the math, then go back to the math after you feel you understand the text.
EDIT: I used Physical Chemistry 6e by Ira Levine