Welcome, AllAmericanBreakfast!
Part of the answer is that the surfaces of the crystals don't match an other due to irregular shape, so just very few chemical bonds can establish simultaneously.
An other part of the answer is that no surface is clean because they're too reactive for that. Within our atmosphere, a new surface (made for instance by breaking a crystal) reacts in nanoseconds and covers itself with a complicated and ill-defined layer, several molecules thick, of oxides, hydroxides, carbonates and more exotic things. When you approach two crystals, it's already too late to establish the strong chemical bonds: the surfaces are "passivated".
I must forget a few dozen reasons more.