FWIW, I have a 2001 KTM 640E right now. I would have liked a 400cc bike that's 100lbs lighter, but there was so much prep (re: expense and time) to "adventurize" it. The 640 is MUCH more capable of a bike than I am, and it's clear to me that a more experienced rider would be able to take advantage of it more off road and whip it around like a 400cc.
But, as someone mentioned earlier in the thread, the big bikes are great road/dirt bikes as long as you don't get off into tight singletracks much. I've taken this big bike with limited skills in places it surprised me to be that it got out of.
My next adventure bike is probably going to be a 400EXEC setup with more oil capacity, increased valve reliability and a reinforced subframe, including all the usual suspects like bigger tank, misc luggage things, etc. I would like to take the 400 in places more easily (re: more fun) than the big bike. That's a long ways off though, as I'm really enjoying the 640 and it's capabilities.