...
The trick here is to use --no-install-recommends. The GUI tools are in the Recommends of the CLI tools (don't ask me), so installing the CLI tools with apt-get install open-vm-tools will also pull down the X stuff.
This way everything seems to work just fine:
