What are some cases and how to recognize the reactions where steady-state and rate-determining step can't be used to derive the overall rate equation? What tool is used then?
There's no fundamental reason why there ought to be a neat, closed form expression in the first place. You might have to stay content with a full model that you solve numerically.
In most practical kinetics models these days that is exactly what is done.