@Typhon,
Coumarin itself isn't used as a fluorophore but there is a whole family of coumarin derivatives very frequently used as laser dyes in the blue-green region of the spectrum (e.g., coumarin 1:
http://omlc.ogi.edu/spectra/PhotochemCAD/html/045.html). I'm not aware of any that have yellow fluorescence in ethanol (these are usually more typical of xanthenes) but I'd have to check my catalog to be sure. Anyway, many of the coumarin and xanthene dyes (the latter of which fluorescein is a member) exhibit strong solvatochromism and/or polarity dependent fluorescence coloring or intensity, particularly those that have free amino groups. Though not directly related to your question, I found an old slide I presented for one of my lab's group meetings based on a photograph I took that demonstrates this kind of effect in some of the coumarin dyes, which you might find interesting (the numbers are quantum yield values I extracted from the noted citation):
No color change here for the most part (although the one on the right starts to take on a yellowish cast in water), but you can see the one with the free amino group has much higher susceptibility to fluorescence quenching in polar solvents due to what's called a twisted intramolecular charge transfer (TWICT or TICT) state. Color shifts are usually due to solvent reorganization around the excited state. If you're seeing yellow in water but blue in alcohol, most likely something like this is occurring.
Another possibility is concentration effects. Some highly fluorescent dyes form excimers and undergo very different luminescent color as a function of concentration. Pyrene is a classic example.
Without knowing the identity of the dye in your marker, it's impossible to say why the color change happens. Most likely it is polarity related, which either affects the intrinsic excited state relaxation properties of the molecule, or impacts the way they clump together.