I'm in the same boat as you, mainly ride on-road and have a great pair of Gaerne's for that. My offroad is seldom, and usually confined to a couple months in the summertime.
For that, I wear a pair of Thor Quadrant's. I bought them years ago, but they have held up well, buckles still work as they should and they have solid ankle support and protection. I don't think the Quadrant is still available, but I would venture that any Thor boot will do the job for just under $150.
I also bought a pair of AXO's for my son...Boxer model. Again, no longer made, but the replacement for that boot is also solid and I wouldn't hesitate to purchase and wear it myself, either.
So yes, you can get good protection from a boot that costs less than $180 (esp. if you find them on closeout or discounts).