How to debug a nodeJS application inside a Kubernetes Pod?

2
votes

Pretty much the title of the question.

I am working with an Ubuntu system that has a k8s deployment with multiple nodes and with multiple pods that run Docker containers. A few of the pods are nodeJS microservices which run the following command at initiation:

node app.js

Sometimes I need to debug the microservice by adding logs, changing logic inside, etc.

Working with the same microservices in Windows I could just change the source code and restart the node.exe process. How would I achieve doing the same in Linux with a Kubernetes deployment?

I attempted to run a shell:

user@node1:~$ kubectl exec my-microservice-XXXX -it -- sh

Change source code and save: nano app.js

Find the node process: ps aux

PID   USER     TIME  COMMAND
    1 root      0:00 npm
   22 root      0:00 npm
   42 root      0:27 node --max-http-header-size=65000 app.js

Then send SIGTERM to PID 42:

kill SIGTERM 42

And this results in me being booted out of the pod:

/usr/src/app # kill SIGTERM 42
sh: invalid number 'SIGTERM'
/usr/src/app # command terminated with exit code 137
test@node1:~$

And a new pod starts automatically:

my-microservice-XXXX                       0/1     Completed   1          19h
my-microservice-XXXX                       1/1     Running     2          19h
2
node.jsdockerkubernetes
One note: You missuse kill command. Btw. SIGTERM is a default signal so you don't have to specify its type and you could actually use only kill 42. If you want to specify the type of the signal, you should use following syntax: kill -<signal> <pid>, -s <signal> or --signal <signal> e.g. kill -SIGTERM 42. – mario
And what do you actually want to obtain by killing this process and restarting the pod ? You shouldn't do any changes inside a running pod. You should rather build an image that uses newer code version and update your deployment. – mario
@maro - I want to kill the process in order for the changes I made in the source code to take effect. It's meant to debug customer production environments where we can't recompile the code change and deploy it as it's part of larger installation. – Kobbi Gal
I would say that in Kubernetes it isn't meant to be done like that. Rather than for debugging it should be used for running working images. And an image should be immutable i.e. if you change the code of your application, you're supposed to swap the old image with the new one which contains new code. It's just not the way you're supposed to use Kubernetes. See also answer provided by @Sagar Chilukuri. As he mentioned, application code should be the integral part of the image and shouldn't be stored in volumes. – mario

2 Answers

1
votes

This is not possible in Kubernetes as a straight forward way, as we do not manage the container (Creation, termination, etc). This is done by Kubernetes and hence the process is ephemeral.

If you don't want to lose your changes on container restart, then you can use volume mount of the directory where you're making the changes. (This completely defeats the purpose of docker containerisation and a not a recommended for production (any) environment to store the code in volume).

1
votes

With the following two steps, you can debug a Node app running inside a Docker container in a kubernetes Pod:

  1. Log into the container and run the Node app in the debug mode:
kubectl exec -it <pod-name> bash
node --inspect-brk index.js
  1. Forward connections to a local port to a port on the Pod
kubectl port-forward <pod-name> 9229

Note: 9229 is the default port number that the debugger listens on, and you don't need to expose this port in your Kubernetes configuration yaml file.

That is it.

Now you can open you Chrome browser with address chrome://inspect, click the remote target, and start debugging.