Yes, technically the potential is not stable, but it is perfectly stable in practical terms. In typical case electrode (Ag wire covered with AgCl) is immersed in 1 M (or comparable concentration) solution of KCl. As long as the electrode is used only as a reference currents flowing through it are so low, changes in the concentration of Cl- are completely negligible.