Holepunching failing with linux iptables

Question

I am trying to test a custom made hole punching library. I want to setup a simulator using Docker. There are 3 subnets:

• 2 nated ones (172.30.0.0/16 and 172.31.0.0/16), with one Debian machine each playing the role of NAT instances. They are running the following iptable config.
• 1 other "public" subnet (172.29.0.0/16) where both NAT instances have an IP (172.29.0.3 and 172.29.0.4).

A node in one NATed subnet (172.30.0.10) is trying to communicate with one in the other one (172.31.0.10)

The iptables rules for the subnets are are defined by the following script:

#!/bin/bash

# Required env:
# NATED_SUBNET the source subnet
# OUTPUT_IP the ip of the output ("public" network) interface
# INPUT_IP the ip of the input (NATed network) interface
# OUTPUT_GATEWAY_IP the ip of the gateway in the output network

set -e -o xtrace

OUTPUT_IFNAME=$(ip -br addr show | grep $OUTPUT_IP | cut -d "@" -f 1)

# enable access to public internet
ip route replace default via $OUTPUT_GATEWAY_IP dev $OUTPUT_IFNAME

# setup syslogs
iptables -I FORWARD -s $NATED_SUBNET -d 0/0 -j NFLOG --nflog-prefix "[nated|FORWARD|OUT]:" --nflog-group 1
iptables -I FORWARD -s 0/0 -d $NATED_SUBNET -j NFLOG --nflog-prefix "[nated|FORWARD|IN]:" --nflog-group 1
iptables -N LOG_DROP 
iptables -A LOG_DROP -j NFLOG --nflog-prefix "[nated|DROP]:" --nflog-group 1
iptables -A LOG_DROP -j DROP

iptables -t nat -A POSTROUTING -s $NATED_SUBNET -o $OUTPUT_IFNAME -j MASQUERADE

iptables -A INPUT -j LOG_DROP

I get the following logs

docker-router2-1  | Feb  5 18:21:11 69e0e4683741 [nated|FORWARD|OUT]: IN=eth0 OUT=eth1 MAC=02:42:ac:1f:00:03:02:42:ac:1f:00:0a:08:00 SRC=172.31.0.10 DST=172.29.0.3 LEN=60 TOS=00 PREC=0x00 TTL=63 ID=35566 DF PROTO=TCP SPT=8080 DPT=56943 SEQ=3939548479 ACK=0 WINDOW=64240 SYN URGP=0 MARK=0 
docker-router1-1  | Feb  5 18:21:11 86ed5150c6b9 [nated|DROP]: IN=eth1 OUT= MAC=02:42:ac:1d:00:03:02:42:ac:1d:00:04:08:00 SRC=172.29.0.4 DST=172.29.0.3 LEN=60 TOS=00 PREC=0x00 TTL=63 ID=35566 DF PROTO=TCP SPT=8080 DPT=56943 SEQ=3939548479 ACK=0 WINDOW=64240 SYN URGP=0 MARK=0 
docker-router1-1  | Feb  5 18:21:12 86ed5150c6b9 [nated|FORWARD|OUT]: IN=eth0 OUT=eth1 MAC=02:42:ac:1e:00:03:02:42:ac:1e:00:0a:08:00 SRC=172.30.0.10 DST=172.29.0.4 LEN=60 TOS=00 PREC=0x00 TTL=63 ID=7623 DF PROTO=TCP SPT=56943 DPT=8080 SEQ=491450156 ACK=0 WINDOW=64240 SYN URGP=0 MARK=0 
docker-router1-1  | Feb  5 18:21:12 86ed5150c6b9 [nated|DROP]: IN=eth1 OUT= MAC=02:42:ac:1d:00:03:02:42:ac:1d:00:04:08:00 SRC=172.31.0.10 DST=172.29.0.3 LEN=60 TOS=00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=8080 DPT=56943 SEQ=3955213225 ACK=491450157 WINDOW=65160 ACK SYN URGP=0 MARK=0 
docker-router2-1  | Feb  5 18:21:12 69e0e4683741 [nated|FORWARD|IN]: IN=eth1 OUT=eth0 MAC=02:42:ac:1d:00:04:02:42:ac:1d:00:03:08:00 SRC=172.29.0.3 DST=172.31.0.10 LEN=60 TOS=00 PREC=0x00 TTL=62 ID=7623 DF PROTO=TCP SPT=56943 DPT=8080 SEQ=491450156 ACK=0 WINDOW=64240 SYN URGP=0 MARK=0 
docker-router2-1  | Feb  5 18:21:12 69e0e4683741 [nated|FORWARD|OUT]: IN=eth0 OUT=eth1 MAC=02:42:ac:1f:00:03:02:42:ac:1f:00:0a:08:00 SRC=172.31.0.10 DST=172.29.0.3 LEN=60 TOS=00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=8080 DPT=56943 SEQ=3955213225 ACK=491450157 WINDOW=65160 ACK SYN URGP=0 MARK=0 

Note: The order of logs is not perfect because logs from different hosts are gathered by Docker Compose in a somewhat arbitrary order.

What we can see though is that:

• the first SYN that is supposed to do the punching of docker-router2-1 goes through it and is dropped by docker-router1-1 as expected
• the connection from the client behind docker-router1-1 then seems to go through docker-router2-1 thanks to the punching
• the server behind docker-router2-1 returns SYN-ACK, still as expected
• but docker-router2-1 does not seem to replace the source IP of the SYN-ACK properly -> docker-router1-1 receives it with SRC=172.31.0.10 instead of SRC=172.29.0.4 and hence drops it

What is wrong with the setup?

Edit: I recorded the conntrack output on the routers:

simulator-router2-1 | tcp 6 118 SYN_SENT2 src=172.31.0.10 dst=172.29.0.3 sport=8080 dport=42773 src=172.29.0.3 dst=172.29.0.4 sport=42773 dport=8080 mark=0 use=1
simulator-router1-1 | tcp 6 110 SYN_SENT src=172.30.0.10 dst=172.29.0.4 sport=42773 dport=8080 [UNREPLIED] src=172.29.0.4 dst=172.29.0.3 sport=8080 dport=42773 mark=0 use=1