Read/Copy files between pods via kubernetes API or using ruby client (kubeclient package) - 400 BAD Request Error

Question

I am using ruby kubeclient library and Kubernetes APIs to read file contents from different pod from the current pod. I have set up RBAC and am able to authenticate my Kubernetes API. Followed: https://kubernetes.io/docs/tasks/run-application/access-api-from-pod/

To read the file from the pod from the host I was able to try the below options:

using kubectl exec with cat command to get file contents ( able to see file contents on console )
using kubectl cp to copy the file from container to host ( able to copy the file in the host )

I want to replicate either of these options to automate this process to run from within the container.

From the container when I run :

curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET "https://kubernetes.default.svc/api/v1/namespaces/mynamespace/pods/mypod"

I am able to get the details of my pod.

But when I tried the above URL with exec I am getting 400 Bad request errors.

curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET "https://kubernetes.default.svc/api/v1/namespaces/mynamespace/pods/mypod/exec?command=ls"
or
curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET "https://kubernetes.default.svc/api/v1/namespaces/mynamespace/pods/mypod/exec?command=cat&command=/tmp/myfile.txt"

The error I am getting is:

  {
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "Upgrade request required",
  "reason": "BadRequest",
  "code": 400

NOTE: I found StackOverflow thread for this above error: Kubernetes pod exec API - Upgrade request required

The recommendation was to use the CLI tool which supports WebSockets ( like wscat or wssh ).

I tired replacing curl with ruby rest client:

RestClient::Request.execute( :url => url, :method => :get, :verify_ssl => true, :ssl_ca_file => "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt", :payload => {}.to_json,
    :headers =>  {:Authorization => "Bearer %s" % [access_token]})

But I am getting the same 400 error.

My rbac configuration:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: {{ .Chart.Name }}.{{ .Release.Namespace}}
rules:
  - apiGroups: [ "" ]
    resources: [ "services", "pods"]
    verbs: [ "get", "list" , "watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]

Is there any equivalent for kubectl exec or kubectl cp which I can use via KUBERNETES API ? or is there any sample code that shows how to use kubeclient (ruby client) to copy files in and out of containers?

github.com/kubernetes-client/python/blob/master/examples/… — Harsh Manvar
@WytrzymałyWiktor no I could not found the resolution/workaround for this. — Akhilesh Joshi
@AkhileshJoshi have you considered using a sidecar/init container to achieve this? — Jakub Siemaszko

Jakub Siemaszko Jakub Siemaszko · Accepted Answer · 2021-09-20T10:14:07

In my opinion kubectl exec and kubectl cp by design should be used for debugging/troubleshooting/testing scenarios however you can use it in an init or sidecar container (for debugging/troubleshooting/testing). In production environment you should avoid giving access to pod/exec resource to your workload.

To achieve your goal I would recommend to use one of those methods:

An external storage bucket such as Google Cloud storage bucket.
A Persistent Volume with fitting access modes from one of the providers mentioned in the official documentation.

If you are interested in copying files from your host machine in Kubernetes you can also find hostPath volume concept however this is not recommended way.

Warning:

HostPath volumes present many security risks, and it is a best practice to avoid the use of HostPaths when possible. When a HostPath volume must be used, it should be scoped to only the required file or directory, and mounted as ReadOnly

Read/Copy files between pods via kubernetes API or using ruby client (kubeclient package) - 400 BAD Request Error

1 Answers