There is friction from interplanetary dust. Other planets and satellites (i.e. moon) gravity can effect orbits.
But our planets gravity is also acting on the sun, causing it to slow its rotation. From this effect, we are actually moving away from the sun by ~1 micrometer a year from this effect.
Also, keep in mind the sun is continuously producing energy, and losing mass. For the main sequence of a star like or sun, that is still only a fraction of a percent of its mass. But, if we assume a constant energy output, that equates also to a moving away distance of ~1.5cm a year.
At the same time, as the solar system moves around the galaxy, and as the galaxy moves around the universe, all kinds of other gravitational effects can and dooccur. Some of these changing gravitational forces can and do cause the orbit to increase or decrease.
And while distance from the sun is important (why our planet supports life so well), there are other important things. For instance, for the norther hemisphere, when we are in the summer, we are actually the furtherest away from the sun, and in the winter for the norther hemisphere we are the closest to the sun. Yet the temperatures appear to go in the opposite direction one would predict. That is because the angle of the sun is far more crucial to the temperature of earth and the energy the earth receives.
There are obviously catastrophic things that can happen to dramatically change things. And they do not have to happen just to earth. If Mars got hit by a giant asteroid and changed its orbit, it could have an impact on our climate, it could even theoretically kill of the human race by making such drastic changes.
Anyway, we are currently moving away from the sun, but this can change.