Speaking only for myself, I don't use the word polarity to mean exactly the same thing as the size of the dipole moment. Polarity is a more multifaceted and complex concept. It is related to relative solubility in different solvents and mobility in silica chromatography, among other things. In my experience, the presence or absence of hydrogen bond donor groups on a molecule is a major contributor to its overall polarity.