I would suggest to put the port number on the inventory file. Like the following example.
[linux-servers]
xcpng5.homelab.com ansible_port=3511
xcpng2.homelab.com ansible_port=3522
xcpng1.homelab.com ansible_port=3523
I was trying to achieve the same and that helped me to set up different ssh port.