I'm trying to test out running a Docker container via Hashicorp Nomad that can have open ports on multiple networks. A use case would be a traefik load balancer with an 'outside' and an 'inside' interface. I am using RHEL 8 as my container host/Nomad client.
It's important that the traffic goes back out the interface it came in on - I'm not doing any forwarding across this host or containers. Since you can only have one default gateway, I've done some reading/testing on getting this working right. Using NetworkManager, I have (from what I can tell) correctly setup the two interfaces such that they will respond out the interface from which the packet came in on. Here's the output of ip rule:
ip rule
0: from all lookup local
10: from all iif ens224 lookup 10 proto static
10: from 10.10.40.2 lookup 10 proto static
32766: from all lookup main
32767: from all lookup default
I have the second (non-default) interface set to lookup route table 10 for incoming traffic.
ip route show table 10
default via 10.10.40.1 dev ens224 proto static metric 101
10.10.40.1 dev ens224 proto static scope link metric 101
ip route show table main
default via 10.10.51.1 dev ens192 proto static metric 100
10.10.40.0/24 dev ens224 proto kernel scope link src 10.10.40.2 metric 101
10.10.51.0/24 dev ens192 proto kernel scope link src 10.10.51.2 metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
This works great - if I run httpd, for example, directly on the host (not in a container) I can talk to it on either the 10.10.40.2 or 10.10.51.2 interface on port 80. However, if I run a docker container via Nomad, I can only talk to it on the 10.10.51.2 interface. Even though I can see it bound to other interface and showing in a netstat, I cannot talk to it on 10.10.40.2. I'm thinking it has to do with the way my ip rules are setup and/or needing to do some kind of marking of packets with iptables?
This is all quite new to me and I appreciate any guidance anyone can share. I'm not using CNI or anything fancy - just Nomad's multi interface support. Please let me know what I haven't articulated well/what questions you have.
You would define two network blocks in your nomad client configuration to different interfaces - jnternal and external. I typically have localhost added. See https://developer.hashicorp.com/nomad/docs/configuration/client#host_network-block .
Then, in the job group specification you would request two ports from both networks. See https://developer.hashicorp.com/nomad/docs/job-specification/network#host-networks .
There is no iptables or networkmanager involved, you can remove all the changes, nomad picks port dynamically on the interfaces by itself. Networkmanager is needed to configure networking. The only routing is done by docker just like with docker run -p . You should run a firewall on external interface tho.