I commend you for being a responsible and unselfish person first of all.
I have seen (or been on a horse) several times when people passed way too close and gave NO consideration at all.
If possible depending on positioning - Ask the riders "how should I pass you?" or " is it safe for me to pass you now?" I probably wouldn't ask a question with a Yes or No answer like"May I pass you." Some people (especially younger ones) may just reply Yes or No and not give you any direction.
The times I have been asked I have dismounted and held the horse's reins when I was on a horse that wasn't "bombproof." And I really appreciated the consideration.