Thanks for the responses guys.
I picked up a 2006 klx250s cheap to see if I would miss the 650's power and I have to say that I don't miss the power all that much. I've had the klx on some super slab and it had no problem keeping up with traffic but passing is an issue. I imagine that it would have been worst if I was loaded down but as it stands, 1.5 hr on the highway is not a problem for the bike. In fact, it was pretty smooth. A little more horsepower would make the bike just about perfect, maybe I'll do an overbore when the 250 wears out but I can't really complain how it rides now.
The bike is head and shoulders better offroad than the klr650 or dr650. It's almost like riding a mtn bike offroad. I'm sure the better suspension helps alot.
I think when we (or at least I) buy a dual sport for the first time, there is an expectation of doing long epic travels. The reality for me is that I am lucky to do 1 or 2 multiday trips a year between family and work obligations. Most of my "adventures" are close to home and the 250 is a great fit. All I need is a bigger tank so I'm not stopping for gas a couple of times a ride!
I don't have enough experience to give anyone advice but I will say this to anyone contemplating a smaller bike: Give a lot of thought to how many long trips you really need a packmule of a bike for. Chances are you will be happier loading up your lighter more responsive bike for couple of trips and traveling at a more relaxed pace than wrestling a heavier bike on your day to day "adventures."