Try to create tun interface inside a pod in k8s but "ioctl(tunsetiff): device or resource busy"

234 views Asked by At

I would like to create tuntap interace inside a pod because the main objetive of this is communicate two pods via tun interface (the apps that run there expect to communicate via tun interface). I have the following Dockerfile as an example to try this: Dockerfile

# Defino la imagen base que usará el constructor
FROM ubuntu:focal as builder
# admito una instalación de forma no interactiva
ENV DEBIAN_FRONTEND=noninteractive
# Versionamiento
LABEL Name=ueransim Version=0.0.2
#Instalación de dependencias 
RUN apt-get update && apt-get install -y wget \
    make \
    gcc \
    g++ \
    libsctp-dev \
    lksctp-tools \
    tcpdump \
    git \
    nano \
    iproute2 \
    iptables \
    net-tools \
    ifupdown \
    iputils-ping \
    libssl-dev
# con snap instalamos make
RUN mkdir ~/temp && \
    cd ~/temp && \
    wget https://cmake.org/files/v3.20/cmake-3.20.0.tar.gz && \
    tar -xzvf cmake-3.20.0.tar.gz && \
    cd cmake-3.20.0/ && \
    ./bootstrap && \
    make -j `nproc` && \
    make install && ldconfig && \
    cmake --version
# clonamos el repositorio de UERANSIM
RUN git clone https://github.com/aligungr/UERANSIM && \
    cd UERANSIM && \
    make -j `nproc`
# Construimos la imagen final
FROM ubuntu:focal 
# admito una instalación de forma no interactiva
ENV DEBIAN_FRONTEND=noninteractive
# Mantenemos las dependencias en nuestra imagen final
RUN apt-get update && apt-get install -y --no-install-recommends \
    sudo \
    libsctp-dev \
    lksctp-tools \
    nano \
    netbase \
    iproute2 \
    iptables \
    net-tools \
    ifupdown \
    iputils-ping \
    iperf3 \
    libssl-dev \
    net-tools \
    systemd \
    netbase \
    pkg-config \ 
    tcpdump \
    openssh-server && apt-get autoremove -y && apt-get autoclean
# Copiamos los archivos compilados del constructor a la imagen final
COPY --from=builder /UERANSIM/build /UERANSIM/build
COPY --from=builder /UERANSIM/config /UERANSIM/config
# Creamos el usuario SSH
RUN useradd -rm -d /home/ubuntu -s /bin/bash -g root -G sudo -u 1000 ubuntu
RUN echo 'ubuntu:ubuntu' | chpasswd && adduser ubuntu sudo 
RUN mkdir /var/run/sshd
EXPOSE 22
# Movemos los directorios de UERANSIM a /home/ubuntu
RUN mv /UERANSIM /home/ubuntu
# Terminamos con una instancia de bash que ejecute el container con el servicio SSH
CMD ["/usr/sbin/sshd", "-D"]

After that using helm I deployed the image as a pod with following manifest: Deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "netnode.fullname" . }}
  namespace: {{.Release.Namespace | quote}}
  labels:
    {{- include "netnode.labels" . | nindent 4 }}
spec:
  selector:
    matchLabels:
      {{- include "netnode.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      {{- with .Values.podAnnotations }}
      annotations:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      labels:
        {{- include "netnode.selectorLabels" . | nindent 8 }}
    spec:
      {{- if .Values.nodeSelector }}
      ## Para especificar a qué nodo el POD será asignado
      nodeSelector: {{- include "common.tplvalues.render" ( dict "value" .Values.nodeSelector "context" $) | nindent 8 }}
      {{- end }}
      imagePullSecrets:
        {{- toYaml .Values.image.pullSecrets | nindent 8 }}
      containers:
        - name: ue
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag}}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          securityContext: 
            privileged: true
            capabilities:
              add: ["NET_ADMIN"]
          command: ["/bin/bash"]
          args: ["/entrypoint.sh"]
          ports:
            - name: mgm-connection
              containerPort: 22
              protocol: TCP
            - name: gnb-ue
              containerPort: {{ .Values.service.port}}
              protocol: UDP
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
          volumeMounts:
            - name: config
              mountPath: /home/ubuntu/UERANSIM/config/ue.yaml
              subPath: "ue.yaml"
            - name: config
              mountPath: /entrypoint.sh
              subPath: "entrypoint.sh"
      volumes:
        - name: config
          configMap:
            name: {{ include "netnode.fullname" . }}-configmap
            defaultMode: 0777

The "entrypoint.sh" file consist of this:

entrypoint.sh

#!/bin/bash
ip tuntap add name demotun mode tun 
ip link set demotun up
echo "Setting IP to device"
ip add add {{ .Values.ip_netbase}} dev demotun;
sysctl -w net.ipv4.ip_forward=1;

And after launching the pod using helm I get this using "kubectl logs ": Logs for pod Pod Output

ioctl(TUNSETIFF): Device or resource busy
Setting IP to device
RTNETLINK answers: File exists
net.ipv4.ip_forward = 1

And the following pod description: Describe output

Name:             nodo1-netnode-57675cffb8-qqzbh
Namespace:        default
Priority:         0
Service Account:  default
Node:             worker1/10.0.0.3
Start Time:       Mon, 02 Oct 2023 17:02:47 +0000
Labels:           app.kubernetes.io/instance=nodo1
                  app.kubernetes.io/name=netnode
                  pod-template-hash=57675cffb8
Annotations:      cni.projectcalico.org/containerID: f751d2656368d97144b89f4b49c64013dba12e24526a65a3f59bef73501c540f
                  cni.projectcalico.org/podIP: 172.16.235.137/32
                  cni.projectcalico.org/podIPs: 172.16.235.137/32
Status:           Running
IP:               172.16.235.137
IPs:
  IP:           172.16.235.137
Controlled By:  ReplicaSet/nodo1-netnode-57675cffb8
Containers:
  ue:
    Container ID:  containerd://3201af2d1915052953893e7416d3d778d839da3658d64fffce8c1b8192b0263e
    Image:         agustinvl/gira:ueransim
    Image ID:      docker.io/agustinvl/gira@sha256:a9780969e6967976a0b58af11a05dd91ea323d64603b9f394ed45d4bd2e9beb7
    Ports:         22/TCP, 4455/UDP
    Host Ports:    0/TCP, 0/UDP
    Command:
      /bin/bash
    Args:
      /entrypoint.sh
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 02 Oct 2023 19:07:20 +0000
      Finished:     Mon, 02 Oct 2023 19:07:20 +0000
    Ready:          False
    Restart Count:  29
    Environment:    <none>
    Mounts:
      /entrypoint.sh from config (rw,path="entrypoint.sh")
      /home/ubuntu/UERANSIM/config/ue.yaml from config (rw,path="ue.yaml")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rqtrc (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      nodo1-netnode-configmap
    Optional:  false
  kube-api-access-rqtrc:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason   Age                     From     Message
  ----     ------   ----                    ----     -------
  Warning  BackOff  4m19s (x576 over 129m)  kubelet  Back-off restarting failed container ue in pod nodo1-netnode-57675cffb8-qqzbh_default(06d69b13-b861-428b-9d26-d78d95db1d65)

I already tried the following actions unsuccessfully:

  1. Check if some tun device is on the worker but my workers only have the default k8s configuration: Similar Output for worker2 (2 nodes cluster)
  2. If I don't run the command during the installation after that the TUN device doesn't work (executing the bash script after the pod was created)
  3. I check if my workers resources was full but I already saw that no other pods was launched in the cluster.

I hope you can help me, I would be infinitely grateful :(

0

There are 0 answers