Here's my perspective: I'm new to DS (DR650) but have liter street bikes (two FZ1s) and a 250cc trail bike (WR250F) - I would not consider using my DR for serious trail/dirt use with friends but would use it, say, if the whole family went on an easy trail ride. On the other hand, my WR feels faster than the DR but obviously could not run 70 on the highway for an hour. I'd love to ride the 250 around town but it's not street legal.
So, it really depends on how far and fast you need to go on the street, and the capability of the specific 250 you're considering. The WR250R, for example, can run at highway speeds no problem. Ultimately, I think I'd be as happy on that, but I couldn't justify the cost..