There are some relatively simple rules that govern how broad a patent can be. In terms of Markush structures, the following rules tend to be the ones that cause problems:
1) the invention must be enabled by the patent specification. That means that a skilled person in your field must be able to practice your invention relying only on your patent and his basic knowledge of the field without having to carry out more than a modest amount of experimentation. If you claim new compounds that are useful for new treatments, you also have to describe how to make those compounds and how to use them for treatments. In general, the more compounds you claim, the more specific examples you have to provide to demonstrate that your synthetic techniques are sufficient to enable a good synthesist to make any compound that you claim.
2) the claims must specifically describe the limits of the invention. This means that a skilled person in your field must be able to read your patent and know exactly which compounds are being claimed. With Markush claims, if you write them broadly enough, the number of compounds covered becomes enormous, and the examiner at some point will decide that it wouldn't really be possible to determine what compounds were not covered.
And of course the two most people are familiar with:
3) the invention must be novel. This means that none of the possible variants of the compounds you have put into your Markush structure can already exist, if you are claiming those structures as your invention. This can require some very careful patent writing to be sure that as many novel compounds are included without including anything in the prior art.
4) the invention must be non-obvious. This is a really tough one. Is simple replacement of a methyl group with a chloro group obvious? In some cases, maybe yes, in some cases, maybe no. If your patent is written as broadly as it should be, there will almost always be some back-and-forth with the examiner on this point. The examiner's job is to act on behalf of the American people. In return for you making your magnificent new invention public, the American people will give you a short term exclusive right to the invention, but the examiner tries to ensure that you aren't getting exclusive rights to something the public already owns.
Another rule that can sometimes be a nightmare to work around -
5) You only get one invention per patent. That doesn't necessarily mean only one compound, nor does it mean that the invention can't be used in different ways, but if your claims start to get too out of hand, the examiner may limit the amount of material you can include in a single patent by issuing a restriction requirement. The examiner divides your material up into the different inventions he sees in the material, then forces you to select one invention that will be examined for the patent. If you want to patent the other inventions, you are allowed to file additional patent applications (called Divisional Applications, or divisionals) to cover those additional inventions.
One that has been alluded to previously on this thread:
6) the invention has to be useful. That means it has to actually work! If you can't demonstrate to the examiner that your invention is good for something, it can't get patented. That doesn't mean that finding one example in a broad claim that doesn't work is enough to invalidate the patent, but if you are trying to patent a process to produce aspirin and it doesn't in fact produce aspirin, then it will not receive a patent. If you are trying to patent a perpetual motion machine or a free energy machine, you will be required to actually build one and prove that it works before you will get a patent. (Incidentally, treatments for baldness used to be in this list, until one was actually found!)
To Orgopete's point about the generic claims covering prior art - a good patent application will be written with claims going from very broad to very narrow. You try to include as much in your invention as you can in the very broad claims, and in the very narrow claims you include as exact a description as possible of the commercial embodiment of your invention, if you know what that will look like. Then the examiner will decide that your broadest claim is anticipated by the prior art (meaning that somewhere in the prior art, compounds which are exactly like the ones you claim have already been made), that some of your narrower claims are made obvious by the prior art (meaning that any skilled worker would have known to make the compounds without seeing your patent, just based on the prior art), and possibly that your narrowest claims are allowable, (meaning that they are novel, non-obvious, useful, enabled by your specification, and clear as to the limits of the invention). There then follows a series of negotiations with the patent examiner in the form of Office Actions and Responses by which you try to prove that your rejected claims are in fact patentable and the examiner describes in great detail why they are not, and if you can come to an agreement, you get a patent. If you can't, there is an appeal board within the patent office where you can go to try to get the examiner overruled, and if that doesn't work, you go to the Federal Court system.
The point of that last paragraph is that if you read published patent applications rather than actual patents, you will see claims that are far broader than anything which can possibly be allowed. In drug patents especially, you may see claims that would cover millions of individual chemical structures in the broadest claim. Even though there may be a couple of hundred individual examples of compounds listed in the specification, that is usually only sufficient to claim a couple of thousand compounds in an allowed patent unless it is completely virgin territory synthetically. In many countries outside the United States, it is even tighter - in some places, you actually have to prove that you have synthesized every compound that you claim. That means you need to show an example for every possible structure in your Markush space. These makes it almost impossible to prevent a competitor in those countries from making very close analogues to your compounds - it just isn't possible to make and characterize every compound in any chemical class.
Wow, this got long, didn't it? Let me just make one more point. Even if the very broad generic claims in a patent are not allowed, they are still prior art for any other patent that comes along. You may not get the patent yourself on your broadest claim, but if it is sufficiently enable, you will prevent anybody else from getting a patent based on your prior publication as a patent application.
I hope some of this is useful.