...
I was able to work around this by recompiling open-vm-tools without all those dependencies. The resulting open-vm-toolbox package is then broke, but I don't need that anyway. The open-vm-tools package seems to work just fine.
At least my VSphere is happy with it and recognize it. (pics here)
It looks like using the server install ISO also install the whole X11 stuff - which is not what you'd expect on a server. I managed to get a bare install by using the alternate ISO, then using F4 from the boot menu (install command line system).