It is based on electronegativities.
The allen scale electronegativities of the elements are as follows.
For simple analysis, we can say that the greater the difference in electronegativity, the weaker the bond.
The difference in the H-B bond is only 0.16, so it is said to be covalent and strong.
The difference in the H-Al bond is 0.39, which is covalent, but not as strong as above.
Furthermore, going down and to the left on the per. table orbital size increases. As the orbital size increases, the bond will have a lower energy and therefore will be easier to break.
Not sure what you meant by orbital overlap affecting bond strength. H-Br and H-Al are theoretical bonds in a random theoretical molecule, so you wouldnt have to factor in orbital shape, and if you did you wouldnt be using allen scale electonegativities.