Client for hyper-v won't connect to remote server

I have two servers set up, one at our office and one offsite at our datacenter. The one at the datacenter backs up devices at the office and vice versa. We have a virtual host at the office that we want to backup at the datacenter but no matter what I do it always connects to the local server.

I have removed the client from the local server, ran cleanup, and waited and it just never connects to the remote server. It will “connect” if I shut down the local server but it will not recognize any of the virtual machines and as soon as I turn on the local server it will switch connections and no longer be “connected” to the remote server.

Any suggestions?

Both servers are running version 2.5.31 and are working with other regular clients as expected.