Tried capturing UDP packets with tcpdump, this is the result:
10:06:49.198607 IP (tos 0x0, ttl 64, id 45771, offset 0, flags [DF], proto UDP (17), length 29)
192.168.15.183.35623 > 255.255.255.255.35622: [udp sum ok] UDP, length 1
10:06:49.298680 IP (tos 0x0, ttl 64, id 14576, offset 0, flags [DF], proto UDP (17), length 36)
192.168.15.132.35622 > 192.168.15.183.35623: [bad udp cksum 0xa0ad -> 0xfd81!] UDP, length 8
10:07:39.921399 IP (tos 0x0, ttl 64, id 58342, offset 0, flags [DF], proto UDP (17), length 29)
192.168.15.183.35623 > 255.255.255.255.35622: [udp sum ok] UDP, length 1
10:07:40.199207 IP (tos 0x0, ttl 64, id 23291, offset 0, flags [DF], proto UDP (17), length 36)
192.168.15.132.35622 > 192.168.15.183.35623: [bad udp cksum 0xa0ad -> 0xfd81!] UDP, length 8
After disabling TCO (tcp offloading) for TX/RX on the NIC the problem is gone
$ sudo ethtool -K eth0 tx off rx off
10:26:27.497545 IP (tos 0x0, ttl 64, id 2614, offset 0, flags [DF], proto UDP (17), length 29)
192.168.15.183.35623 > 255.255.255.255.35622: [udp sum ok] UDP, length 1
10:26:27.910184 IP (tos 0x0, ttl 64, id 5001, offset 0, flags [DF], proto UDP (17), length 36)
192.168.15.132.35622 > 192.168.15.183.35623: [udp sum ok] UDP, length 8
However, client still cannot connect to server