Well, to start: The frame is designed to handle higher-speed road stresses, not dirt. I don't know all of the details, but street frames are quite a bit heavier than enduro frames (and have real rear subframes and pegs for a passenger - even if you never carry one). Then, street wheels are wider, heavier alloy; the rotors and calipers are heavier, too, to resist fade at highway speed. Gas tanks are metal, electrical systems, batteries, and lighting systems are larger, more sophisticated and add weight.
Lot's of this stuff is mandated by DOT regulations. Overall weights are up quite a bit. KTM 350-EXC-F weighs 255 lbs. wet; Duke 390 weighs 324 lbs. wet. Difference is ~70 lbs.
Many of these same issues apply to the weight differences between an enduro and a dual-sport. For example, lots of folks complain about the weight of the WR250R at 298 lbs. wet, or the WR250X at 302 lbs. wet. (The DRZ falls into this category, with low-tech (e.g. heavier) components.) These dual sports meet all of the DOT regs for street bikes. KTM street-legal enduros meet the minimum regs too, but without the electrical systems and the subframe support. Call them race bikes or call them "dual sport light". It is all a continuum. Pick your desired performance envelope and pony up!
1990 Honda NT-650 Hawk-GT
Never doubt that a small group of thoughtful, committed citizens can change the world;
indeed, it's the only thing that ever has. - Margaret Mead