It basically has to do with potential energy. Consider a ball on the top of a hill. Because the ball will roll down the hill but not up the hill, one could say that the ball "wants" to move downward. Now consider two hills next to eachother. The ball would move down the hill, but it would not move up the second hill. In this case, the ball "wants" to occupy a position between the two hills.
Now we all know the reason that the ball wants to occupy these positions is because of gravity. The gravitational potential energy of the ball is lower when the ball is at a lower position than when it is at a higher position. The same applies to atoms. A configuration with 8 valence electrons is more stable because it is a state of lower potential energy. As for why it has a lower potential energy it has to do with reducing the number of unpaired electrons and lowering something known as degeneracy.