The reaction is through the intermediate product that is alken bromonium.
The things I want to ask here, why do we choose -OCH3 but not -OH though they also nucleophilic reagent
I told you, think about what you have in the mixture.
You don't 'choose' your nucleophile, you either have it or don't have it in there.
You have CH3OH, Br2 and the alkene, plus the product when it's formed and the intermediates like the bromonium, bromide ion, etc.
So where would OH- come from? If you are you thinking of something like:
CH3OH --> CH
3+ + OH
-I strongly advise you to check your basics of org.chem.