Yup. The hiker who shouts/glares at you for riding your dirtbike down what they wrongly believe to be a "footpath" probably drove to their start point in a car. You almost certainly burnt less fuel in the same trip, despite using the bike for the 'hiking' part too. We could argue about the specifics of the damage you do DSing and even conclusively prove that it was less harmful than sitting at home in your house on Saturday instead, but it'd wouldn't make a difference. Let's be frank; some hikers simply don't like sharing the trails and will use any argument, however spurious, to make us look bad and to try and get vehicles banned from "their" rights of way.
That said, you get way fewer negative interaction bimbling along on an adventure bike, stopping, taking off your lid and talking to people, than nailing it on a race-ready 2T with loud pipes...
Me? I'm honest about it. I simply do not care about the CO2 emissions or whatever of my bike, or the damage my knobblies supposedly do to the trails. I don't believe the people who have a problem with it will be convinced otherwise, so why waste my time researching it, or my breath arguing with them? I derive enough enjoyment from doing so and live cleanly enough in other aspects of my life that my conscience doesn't bother me in the slightest.
I like my bike because I can overtake 4x4s down farm tracks with a week's worth of shopping on the back.