You have it right - it all depends on the reaction conditions. Your three scenarios are exactly right, let's just flesh them out a bit.
Before we begin, we need to start by recognizing that every step in the formation of an acetal under acidic conditions is reversible. Carbonyl + 2 eq. alcohol + cat. acid --><-- acetal + water. Every step in that mechanism is reversible. The reaction can go in either direction with the same reaction conditions. Second, every step in the formation of an enol under acidic conditions is also reversible. Carbonyl + cat. acid --><-- enol. Every step in that mechanism is reversible. Sounds trivial, but it is not.
Ok, so. how to predict what's going to happen. Reactions only occur one way (in this chapter, at least)... nucleophiles attack electrophiles. That's it. So, lets examine the reactions: formation of an (hemi)acetal from a carbonyl, and formation of an enol from a carbonyl.
Case 1: Formation of a hemiacetal under basic conditions.
Let's first examine formation of a hemiacetal under basic conditions. Carbonyl + sodium alkoxide. For example, acetaldehyde + sodium methoxide. We know the carbonyl carbon is a good electrophile. We know the methoxide oxygen anion is a good nucleophile. (the carbonyl oxygen is also a nucleophile, but a much weaker nucleophile than the methoxide ion). Nucleophiles attack electrophiles. Based on the limited reaction conditions, the only plausible first step is nucleophilic attack of the alkoxide on the carbonyl carbon atom, and the pi electrons in the C=O double bond go up on to oxygen to form the anionic intermediate (see schemes for pictorial representations throughout).
From here, there are really only two options. The electrons on the newly formed O- atom can collapse back down to reform the carbonyl and kick off a good leaving group. The only good leaving group is CH3O-, and that proves the first step in the mechanism is reversible. The only other thing that can happen is a proton transfer. The O- atom is nucleophilic now and can act as a nucleophile. But to do so, it would need an electrophile. The only electrophile around is the acidic proton on solvent (methanol in this example). The O- atom can deprotonate a molecule of solvent to form the neutral hemiacetal. The reasons for both of these steps take into consideration the possible nucleophiles and electrophiles in solution and proceeding accordingly. This is what happens (and why) under basic conditions.
Case 2: Formation of an enol under acidic conditions.
To start, we should again examine the reaction conditions for possible nucleophiles and electrophiles. Let's say the reaction conditions are acetaldehyde and H3O+. Under acidic conditions, the best electrophile by far is the acidic proton. The carbonyl oxygen, as wehave mentioned, is a nucleophile (but a weak one). The oxygen atom can pick up a proton from the acid (i.e. the carbonyl oxygen can deprotonate the acid), and the carbonyl becomes protonated.
Now, we have made the carbonyl a much better electrophile (we have activated the carbonyl). The carbonyl is pretty unhappy with the positive charge on oxygen and would like to relieve itself of that charge. The conjugate base of the acid (water in this case) can take a proton from the alpha carbon. The electrons in the former C-H single bond can form the C=C double bond, and the electrons in the C=O pi bond can go up on to oxygen to relieve the positive charge. This is how the enol is formed. Again, it occurs because of the nature of the compounds... are they nucleophiles? Are the electrophiles? If they're nucleophiles, what electrophiles might they react with?
The enol is itself a nucleophile. It is nucleophilic at the alpha carbon atom (a resonance structure puts a full minus charge on the alpha carbon atom). There aren't really any electrophiles to react with. The only good electrophile is the acidic proton on the acid. The enol can react with the acidic proton on the acid to regenerate the protonated carbonyl compound, proving that the enol-forming step is reversible.
Case 3: Formation of an acetal under acidic conditions.
The reaction conditions here look remarkably similar to the conditions for the formation of the enol. the only difference is the presence of the alcohol. Now, the alcohol is not a good enough nucleophile to attack the carbonyl straight away (it's nowhere near as good a nucleophile as the alkoxide from before). So we need to make the carbonyl a better electrophile before the nucleophile can attack. That's why we activate the carbonyl by protonating the carbonyl oxygen atom. The carbonyl oxygen atom deprotonates the acid to form the protonated carbonyl compound.
This is the same first step as the formation of the enol. what gives? Why couldn't it just form the enol under these conditions, too? Here's the key: it does. There's nothing stopping the formation of the enol. It probably happens a lot under these conditions. But why do we recognize these conditions as formation of an acetal... when the acetal mechanism doesn't involve an enol? Here's where it's important to remember that all these steps are reversible. The enol is a nucleophile. The only electrophile it can react with is an acidic proton on the acid. If (and when) the enol forms (and it probably does), the enol doesn't have anything useful to do except react with an acidic proton to reform the protonated carbonyl compound. To the extent that it occurs, the enol formation leads to an unproductive intermediate that funnels back into the productive pathway. In essence, we don't see the enol in the mechanistic pathway, so we don't concern ourselves with drawing it... but it does form.
Ok, so we know the enol does form under the conditions needed to make an acetal. But we know the enol will decompose back into the protonated carbonyl. Now, we also know the protonated carbonyl is activated and a much better electrophile than the neutral, unprotonated carbonyl. Now that it's such a hot electrophile, the weakly nucleophilic alcohol molecule can attack the protonated carbonyl carbon to form the protonated hemiacetal. From here, I'll assume you know the rest of the steps in the formation of the acetal under acidic conditions.
...
so, to answer your question. how does the reaction know it's supposed to form an acetal instead of an enol? it doesn't. Reactions don't sit around thinking, "ok... There's acid nearby. but wait! I also sense an alcohol nearby. I'd better not form an enol, because I'm supposed to form an acetal." No. The react to whatever reaction conditions are put in front of them. In the acetal case, the reaction continues on the acetal pathway because the presence of the the alcohol molecule provides the reaction with a nucleophle to attack the activated carbonyl to begin our journey toward the acetal.
That was long winded and a bit rambling. Did I help any or just make it more confusing?