It all depends on the s character of the bond. Think about the molecular orbitals. The electrons in the s orbital are more tightly bound to the nucleus, while electrons in the p orbitals are more inclined to be shared between atoms (e.g. in pi bonds). Therefore, electrons in molecular orbitals with a high s character are less inclined to be shared between atoms. If electrons are less inclined to be shared, the bond formed by those electrons is weaker and more easily broken. So, in a terminal alkene, which contains a bond between an sp-hybridized carbon (50% s character) and a hydrogen, the bond is easily broken because of the carbon's high s character, resulting in a relatively high acidity (pKa ~20). However, in a bond between an sp3-hybridized carbon (25% s character) and a hydrogen, the bond is much stronger, resulting in very low acidity (pKa ~50-60).