I have ordered from United Nuclear (uranium and potassium metals). They can be a bit slow, but their selection is good and their prices seem reasonable on some things. It looks like they can provide you with a good start, but you might also want some common chemicals that they don't carry. Try Hi-Valley chemical - http://www.hvchemical.com/
- if you live in the U.S. They have many of the most popular chemicals, including the strong acids and bases that have numerous uses. You could also try eBay, soap making suppliers, pottery suppliers, or photography suppliers.
There are other places you might look for suggestions. First, the contents of old-fashioned chemistry sets would be a good guide on how to start out. You might find some listings on eBay or elsewhere. Try searching "antique chemistry set". Or there still appears to be a chemistry set that has some of what used to be commonly included. It is the Chem C3000 set. I haven't bought it (I already have many of the common chemicals and a decent selection of glassware, so I don't really need it at this point), but from the listing of what is in it, it looks like a good one (perhaps the only reasonably useful one currently available). You could look at the list of chemicals in it as an idea of what might be good to have.http://www.discoverthis.com/chem-c3000.html
Another good guide of what is useful is the now-infamous "Golden Book of Chemistry Experiments". If you do a search on it, you'll eventually run across a copy available for download; that is how I got mine. (Since the radioactive boy scout made the book infamous, original copies are now rather expensive.) The book has a good list of useful chemicals, and it has some good basic experiments (including a few potentially dangerous ones). I strongly recommend downloading a copy. Here is the list of common chemicals from that book.
acetic acid, ammonium chloride, ammonium hydroxide, calcium carbonate, calcium hydroxide, calcium oxide, calcium sulfate, carbon tetrachloride, copper sulfate, ferrous sulfate, glucose, hydrochloric acid, hydrogen peroxide, iron (powder), magnesium, magnesium sulfate, manganese dioxide, naphthalene, phenolphthalein, potassium aluminum sulfate, potassium ferrocyanide, potassium iodide, potassium nitrate, potassium permanganate, salicylic acid, silver nitrate, sodium bicarbonate, sodium bisulfate, sodium carbonate, sodium chloride, sodium hydroxide, sodium hypochlorite, sodium potassium tartrate, sodium silicate, sodium tetraborate (borax), sodium thiosulfate, sucrose, sulfur, zinc, zinc chloride
Most of these chemicals are still reasonably easy to obtain, but a few (i.e. carbon tetrachloride) could be difficult these days. I have a number of them, and I intend to collect as many of them as I can.
Nearly all of the chemicals listed in the "Golden Book" would be useful to have. I have found some others to be good for a few things; you might want some of them, depending on your interests.
calcium chloride, isopropyl alcohol, lithium chloride, methyl alcohol, nitric acid, potassium hydroxide, sodium nitrate, sulfuric acid.
(Be careful with some of these. The strong acids and bases - hydrochloric acid, nitric acid, sulfuric acid, sodium hydroxide, potassium hydroxide - must be treated with the utmost respect. A beginner might want to start with dilute solutions of these. Some others are dangerous in concentrated form - acetic acid (glacial), ammonium hydroxide, hydrogen peroxide.)
The ones that you have listed seem to be good choices. If you add a few others, you will have a good beginning collection.