LoadBalancig kube-apiserver using L7 nginx

Question

maybe someone has encountered or done something similar. Basically, I am running standard auditing in k8s(bare-metal on a closed network), my installation contains 3 masters, and usually requests between them need to be balanced(using HAproxy or Nginx). These methods are good, but if I need to see the real ip address of the client in the audit log file (as an example of myself when I try to do something through kubectl), then I see the ip address of the balancer, and then the whole point of the audit is lost, you can't track who performed what actions. Now I stopped on nginx and here is an example of my configuration,

server {
    listen 6443 ssl;
    ssl_certificate /usr/local/nginx/ssl/apiserver.crt;               # kube-apiserver cert
    ssl_certificate_key /usr/local/nginx/ssl/apiserver.key;              # kube-apiserver key
    ssl_trusted_certificate /usr/local/nginx/ssl/ca.crt;                         # ca.pem
    location / {
        proxy_ssl_certificate /usr/local/nginx/ssl/admin.crt;                    # kubectl cert
        proxy_ssl_certificate_key /usr/local/nginx/ssl/admin.key;            # kubectl key
        proxy_ssl_trusted_certificate /usr/local/nginx/ssl/ca.crt;               # ca.pem
        proxy_pass https://control_plain_6443/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-Ip $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
    }
  }
}

Everything seems to be ok and this way you can get the real ip address of the client, but because of two lines, namely

        proxy_ssl_certificate /usr/local/nginx/ssl/admin.crt; # kubectl cert
        proxy_ssl_certificate_key /usr/local/nginx/ssl/admin.key; # kubectl key

The sense in this is also lost, because the cluster can be accessed by those who have certificates from this kubeconfig, and the same kube-controller-manager and kube-scheduler have other configs with other certificates and obviously I will get an error like "kube-controller-manager" is forbidden: User "system:kube-controller-manager" cannot get resource, and so on, for all similar configurations and SAs that have certificate. In general, how can I solve this problem? Maybe someone has had such experience or there is a completely