I think the reason it's so much worse than an audio-only call is that there's inevitably a couple of seconds where you're trying to find the "end call" button, and you can see each other looking for it, and you can't be waving or making eye contact or whatever