In my settings I turned off: Send me email notifications even when I am active on the sites (description: "We'll only email you if we haven't seen you in the last 10 minutes.").
But I got emails anyway, even for my own replies. I consider this a bug.
And yes, this is annoying.