CPU requirements? - Linux 32 vs 64

I have a now rather ancient server (at home) with early Opterons, which support SSE2 but not 3 or 4. This hosts about 20 LXC containers, almost all of which are 32 bit because they’re as old as the physical server. The 32 bit client appears to run just fine, until it hits a limit (is it PIDs?) that is present on 32 bit clients. I solved this at my work by simply changing the LXC containers to amd64, and reinstalling the client (which installed the 64 bit version and ran fine).

However when I tried the same trick at home today, the client won’t run and gives various errors from “Invalid instruction” to a segfault. From some searching, it sounds like this is related to lack of SSE4, and I might potentially solve it through a custom build…

Am I looking the right way? Does the 32 bit build run because there’s no SSE4 requirement there and the 64 bit does require it?