Maybe you had the right idea but what you wrote wasn't quite correct.
What it means that polarities cancel out is you have two atoms of equal, or near equal electronegativity, covalently bonded to eachother.
For example, the simplest case, H2. H-H, Both hydrogens share the same electrons but they 'want' the electrons equally so their electronegativities cancel out, making a nonpolar bond.
What it means when you say dipoles cancel out is like in the case of CO2.
Each C=O bond is polar in the direct of oxygen, but there are two equal bonds on opposite sides of each other so they completely cancel out. This is the case of dipoles canceling out.
Now lets look at carbon monoxide.
Anyways, more directly on the problem.
To explain the small dipole moment you'll have to incorporate two concepts - resonance and inductive effects.