Take a look at Dainese TRQ-Tour boots. I have them for road, not dirt, but it may be what you're looking for. It has the ankle protection you want without being bulky. They're comfortable to walk if you have socks that are as high as the boot (the labels at shin-height are scratchy).
The bad news is they're $330 and over your budget by $30. They were worth it for me because they have the right combination of comfort, protection, and looks that I didn't find in any other boot.