If you are mostly road and smooth off road, then I can't imagine not having the rollers will make much difference. However, any more aggressive off road and you might want to reconsider. Without the rollers you will have a potentially much larger variance in chain angle and could experience derailing.
Do you set your preload higher with a passenger? If not, setting it much higher when two-up might take care of the issue with the roller being in contact.
ps. love the picture......