Just like a lot of sports
I do a lot of white water kayaking and snowboarding/skiing and have found in those sports when you get more than 4 things just get too, well crowded. Too much keeping track of each other.
I do most of my riding solo, mostly cause I have a lot of weekdays off, but when I do ride with others the same thing seems to apply. I like small groups 3 ppl, 4 tops. I guess one of my favorite things about riding is you don't need anyone else to ride with, at least street riding. I figure trail riding is right back to "better to have someone with you, looking out for ya Just in case."