Get a smaller dirt oriented ds bike.
If your goal is to travel a day, then ride trails for a couple of days...well, I'd rather have fun off-road and suffer a little to get there. Any 250/350 will travel highway speed for a day (it might be screaming, but it'll do it), then when it comes time to turn onto a trail...you have a manageable sized bike to throw around.
There are used bikes showing up all the time. Patience will pay off.
I ride a KLX250, bought it to commute, but ride the snot out of it in the dirt. No matter how good a rider I become in the dirt, I will never try and muscle a KLR on a trail, it'll never be as fun as a little bike. Maybe the KLX650 but not a KLR.
Almost bought a DR350 a couple of times. Good bikes.
Fun on the trails is where you should be focusing your bike search.
As someone else said - a 125 will get you there. CC's in the dirt are not as important to the fun factor as on the road. And bigger is definitely not better as a beginner in the dirt. I ride a 900 racebike on the street if I feel the need to go fast. My 250 goes plenty fast in the dirt. And is way too heavy after a long trail day with multiple dirt naps.
My KLX650 will be interesting to try in the dirt(if I get it running before the snow).
If it can have a kick start, you have a million used options.
Cruise all the bike specific threads, look at the flea market.
All of this is flavored with the belief that a KLR is not a DS bike, nor is any BMW. Round the world, while carrying the kitchen sink, you bet. Can you ride them like a DS, you bet. Singletrack, sure. Fun in the woods, not as much.
And I know nothing about the DRZ, but that seems the best bet from your list.
Weight on the trails is only a penalty when you drop it, not if, when. If I had to, I could haul my bike out of almost any situation I might get it into. A KLR, not w/o a winch. I look forward to being confident enough to take the 650 instead of the 250 onto singletrack, but will I? Only if I start hitting the weights.