How can I pod exec (and run a command) using Kubernetes API?

Question

While using Kubernetes API, I came across this issue where I am not able to pod exec into another pod using the API.

I made sure I am running a pod with a serviceAccount that has correct RBAC permissions (and both pods are in the same namespace). I tried with curl (7.74.0) but I stumbled upon this SO answer that curl may not be the right utility to do this. I have already tried other possible options (this and this) mentioned in the comments in the same post, but no luck.

TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
NAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)    

curl -k -H "Upgrade: SPDY/3.1" -H "Authorization: Bearer $TOKEN" -H "Connection: Upgrade" -H "Sec-WebSocket-Version: 13" -H "Accept: application/json" "https://kuberne
    tes.default.svc/api/v1/namespaces/$NAMESPACE/pods/nginx-deployment-5c547569b4-jgm85/exec?command=ls"
    {
      "kind": "Status",
      "apiVersion": "v1",
      "metadata": {},
      "status": "Failure",
      "message": "Upgrade request required",
      "reason": "BadRequest",
      "code": 400
    }

Have already tried adding/removing few of those headers.

The next thing I tried was wscat (5.2.0). Tried a couple of variations, but needless to say, "no luck" again.

wscat -n -H "Authorization: Bearer $TOKEN" -c wss://kubernetes.default.svc/api/v1/namespaces/$NAMESPACE/pods/nginx-deployment-5c547569b4-jgm85/exec?command=ls

error: Unexpected server response: 403

This command outputs nothing (at all):

wscat -c wss://kubernetes.default.svc/api/v1/namespaces/$NAMESPACE/pods/nginx-deployment-5c547569b4-jgm85/exec?command=ls&stdin=true&stdout=true&tty=true&token=$TOKEN&namespace=$NAMESPACE 

I looked at this SO post, but it doesn't work for me.

wscat  -n -H "Authorization: Bearer $TOKEN" -c 'https://kubernetes.default.svc/api/v1/namespaces/$NAMESPACE/pods/nginx-deployment-5c547569b4-jgm85/exec?command=ls&stdin=true&stdout=true&tty=true'

...
SyntaxError: The URL's protocol must be one of "ws:", "wss:", or "ws+unix:"
...

Not sure how the fella even managed to use http(s) in the first place.

Can someone kindly advise if there's something wrong with what I doing or an alternative (can't use kubectl) to do the same? TIA

Answer 1

Curl Magic and trial and error. Make sure that the user has the right permissions so you don't get the 403.

curl -k \
  --http1.1 \
  -H "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" \
  -H "Sec-WebSocket-Version: 13" \
  -i \
  -N \
  -L \
  -k \
  -H "Connection: Upgrade" \
  -H "Upgrade: websocket" \
  -H "Authorization: Bearer $TOKEN" \
  "https://127.0.0.1:53329/api/v1/namespaces/default/pods/single-wrong-arch/exec?command=ls&command=-l&stdin=true&stdout=true&stderr=true"

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: qGEgH3En71di5rrssAZTmtRTyFk=
Sec-WebSocket-Protocol:


total 56
~drwxr-xr-x    2 root     root          4096 Mar 29 14:45 bin
drwxr-xr-x    5 root     root           360 Apr 27 11:03 dev
drwxr-xr-x    1 root     root          4096 Apr 27 11:03 etc
drwxr-xr-x    2 root     root          4096 Mar 29 14:45 home
drwxr-xr-x    7 root     root          4096 Mar 29 14:45 lib
drwxr-xr-x    5 root     root          4096 Mar 29 14:45 media
drwxr-xr-x    2 root     root          4096 Mar 29 14:45 mnt
drwxr-xr-x    2 root     root          4096 Mar 29 14:45 opt
dr-xr-xr-x  248 root     root             0 Apr 27 11:03 proc
drwx------    2 root     root          4096 Mar 29 14:45 root
drwxr-xr-x    1 root     root          4096 Apr 27 11:03 run
drwxr-xr-x    2 root     root          4096 Mar 29 14:45 sbin
drwxr-xr-x    2 root     root          4096 Mar 29 14:45 srv
dr-xr-xr-x   13 root     root             0 Apr 27 11:03 sys
drwxrwxrwt    2 root     root          4096 Mar 29 14:45 tmp
drwxr-xr-x    7 root     root          4096 Mar 29 14:45 usr
drwxr-xr-x   12 root     root          4096 Mar 29 14:45 var

That's against a minikube cluster, which should explain the port choice and here's the rest of the environemt:

$ curl --version
curl 7.87.0 (x86_64-apple-darwin22.0) libcurl/7.87.0 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.11 nghttp2/1.51.0
Release-Date: 2022-12-21
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL threadsafe UnixSockets

$ minikube version
minikube version: v1.30.1
commit: 08896fd1dc362c097c925146c4a0d0dac715ace0

$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.0", GitCommit:"1b4df30b3cdfeaba6024e81e559a6cd09a089d65", GitTreeState:"clean", BuildDate:"2023-04-11T17:04:23Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:33:12Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/arm64"}

$ uname -a
Darwin 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar  6 21:00:41 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T8103 arm64