From a practical standpoint I see where you are coming from. I went riding on a pretty hot day last summer and thought, I'll just wear my bids and jersey, it will be WAY more comfortable. I usually wear "baggy" shorts over padded underwear.
I SHREDDED them. Totally ruined a set of $100 bibs..... I did not fall, it was from my hips bouncing off of trees and rock outcroppings. I can't imagined what would have happened if I had fallen.
That said, I would die in pants. We don't get to go down nearly enough to get the airflow needed.
2007 Factory Husaberg 380FE
1984 Husqvarna 500AE