How do you debug python code with kubernetes and skaffold?

Question

I am currently running a django app under python3 through kubernetes by going through skaffold dev. I have hot reload working with the Python source code. Is it currently possible to do interactive debugging with python on kubernetes?

For example,

def index(request):
    import pdb; pdb.set_trace()
    return render(request, 'index.html', {})

Usually, outside a container, hitting the endpoint will drop me in the (pdb) shell.

In the current setup, I have set stdin and tty to true in the Deployment file. The code does stop at the breakpoint but it doesn't give me access to the (pdb) shell.

Answer 1

There is a kubectl command that allows you to attach to a running container in a pod:

kubectl attach <pod-name> -c <container-name> [-n namespace] -i -t

-i  (default:false) Pass stdin to the container
-t  (default:false) Stdin is a TTY

It should allow you to interact with the debugger in the container. Probably you may need to adjust your pod to use a debugger, so the following article might be helpful:

• How to use PDB inside a docker container.

There is also telepresence tool that helps you to use different approach of application debugging:

Using telepresence allows you to use custom tools, such as a debugger and IDE, for a local service and provides the service full access to ConfigMap, secrets, and the services running on the remote cluster.

Use the --swap-deployment option to swap an existing deployment with the Telepresence proxy. Swapping allows you to run a service locally and connect to the remote Kubernetes cluster. The services in the remote cluster can now access the locally running instance.

Answer 2

It might be worth looking into Rookout which allows in-prod live debugging of Python on Kubernetes pods without restarts or redeploys. You lose path-forcing etc but you gain loads of flexibility for effectively simulating breakpoint-type stack traces on the fly.