Could be a few things. First I might want to check condition of cabin filter (If so equipped usually accessed via door in glove compartment) Could also be blower motor resister assembly , typical symptoms would be losing speed one first, then two ETC. (does the sound change netween speeds?)
The fact that it sorta works sometime but not others might have me thinking about HVAC cluster issue, perhaps bad connection or has seen better days. Of course it could also be issue with blower motor itself.