If its something that's happened a couple times, probably something along the lines of, "Well it's too late now, dingus." And then not go.

If its becoming a bad habit, that's probably break up worthy.

But if its not a constant thing, I'd probably go anyways and complain a little about it to him.