Ask them to back up. I do this all the time, especially if the peripheral distraction is on my right side, I don't know why the left side doesn't bother me as much. Ask them every single time if they continue to do it and back off from your lie until they do. If you go ahead and putt knowing full and well they are bugging you then it's your fault, not theirs.
This is a pet peeve of mine and I've seen so many players do this, and sadly it is sometimes players who should know better. We all do it from time to time being distracted, but it's on you to let the person know.