I have two sets of wheels and think it is worth doing despite any accelerated wear.
Right now I can slot in my EXC wheels which have a 1.6" front, non-cush-drive, bigger sprocket and rim locks. All worth having in the dirt.
In the past I had DR-Z250 and I used to change tyres on that, as I commuted on it during the week. It would take me about an hour to change 2 tyres and remove all the road gear. I'm reasonably proficient at changing tyres now!
although I still get lots of practice. It was quite a relief to get the second set of wheels with a bigger sprocket and rim locks, made the swap time 30 mins, much less effort and less stress on the tyre beads.