Hi,
to login to your ESXi Hosts by ssh using public/private key authentication you need to place your public key into the authorized_keys file of the correspondenting user.
Other than on Linux/UNIX systems the location of the authorized_keys files is not located in the home directory of the user.
Place your public key into the /etc/ssh/keys-$USER/authorized_keys file.
For example for the User root
/etc/ssh/keys-root/authorized_keys
Michael