Presumptuous boss

Tallk about gall. My boss (well, actually the overall manager of the department) wants to schedule a dinner to celebrate a project. Fine and good. But this afternoon he wants to know if tomorrow evening's good. And he has the nerve to act shocked when I tell him I won't be able to be there, and to ask why I can't clear my schedule. Well, technically I could rearrange things, yes. But I have a rule: there are only a few people who get to ask me to clear my schedule on 24 hours notice. Family. A handful of close friends. A manager at work is... well, not in either of those categories. So even if I could theoretically clear my schedule, on general principles I'm not going to. On 2 days notice, maybe, if I don't have anything critical already scheduled. But 24 hours notice for a non-emergency? Not happening.