[Kubernetes] How to Debug Programs That Use the Kubernetes Client SDK in VSCode (Golang Example)

When you first work on a Go project that uses the Kubernetes client SDK, you might feel lost about how to debug it on your local environment. Don’t panic. It’s actually very simple once you understand what’s happening behind the scenes. I’ll walk you through how to make it work using VSCode as an example.

The Common Confusion

When your program is designed to run inside a Kubernetes cluster, it usually connects to the cluster using rest.InClusterConfig(). This configuration relies on the ServiceAccount token mounted at /var/run/secrets/kubernetes.io/serviceaccount/token. When you run the same program locally, that path doesn’t exist, and the SDK will fail to initialize with an error like:

open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory

Many developers think this means you can’t debug locally. But actually, you can.

The Docker Analogy

Think of it like debugging a program that talks to the Docker API. As long as you have Docker installed and running, you can connect to docker.sock locally, and the Docker client SDK will work.

Kubernetes works the same way. The difference is that instead of connecting through a Unix socket (docker.sock), it connects through the Kubernetes API server using your kubeconfig file. So, as long as you have a working Kubernetes cluster (like kind, minikube, or orbstack) and a valid ~/.kube/config, your program can run and debug locally just fine.

Setting Up the Debugger

You can just use VSCode debugger. If you don’t know how to setup the debugger in VSCode, you can check out the following post. It will walk you through.
[Golang/VSCode] Master Golang Debugging in VSCode: Step-by-Step Guide.

To start debugging, ensure you have:

1. A running Kubernetes cluster on your local environment
2. A valid ~/.kube/config configured correctly. Make sure your kubeconfig works by running:

kubectl get nodes

You may see few issues during this process. It varies based on how your project interacts with the Kubernetes client SDK. I picked some common scenarios below. Even though they might not exactly match your case, they should give you a good idea of how to proceed.

Scenario 1: Missing In-Cluster Token

Error:

open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory

Explanation:
This happens because the program uses rest.InClusterConfig(), which expects the ServiceAccount token that only exists inside a Kubernetes pod. When you run the program locally, there is no such file.

Solution:
Do not use in-cluster configuration when debugging locally. Replace it with a kubeconfig-based configuration that connects to your local cluster.
Let’s say you have kubeconfig at the default location ~/.kube/config, your code that initializes the Kubernetes client looks like this:

func NewKubernetesClient() (*kubernetes.Clientset, error) {
	config, err := rest.InClusterConfig()
	if err != nil {
		return nil, err
	}

	return kubernetes.NewForConfig(config)
}

Instead of removing the existing code, you can modify the code to load the kubeconfig in the error handling block when in-cluster config fails like below in line 4-12:

 1func NewKubernetesClient() (*kubernetes.Clientset, error) {
 2	config, err := rest.InClusterConfig()
 3	if err != nil {
 4      home, err := os.UserHomeDir()              
 5      if err != nil {
 6          panic(err)
 7      }
 8      kubeconfig := filepath.Join(home, ".kube", "config")
 9      config, err := clientcmd.BuildConfigFromFlags("", kubeconfig)
10      if err != nil {
11          panic(err)
12      }
13      // return nil, err
14	}
15
16	return kubernetes.NewForConfig(config)
17}

Even if your production code uses in-cluster configuration, you can temporarily switch to this for local debugging. This allows the debugger to connect to the cluster through the same credentials used by kubectl.

Scenario 2: File Path Expansion Failed

Error:

Op = "stat"
Path = "~/.kube/config"
Err = error(syscall.Errno) golang.org/x/sys/unix.ENOENT (2)

Explanation:
Go does not automatically expand ~ to your home directory. When the code passes "~/.kube/config" directly, it fails because Go treats it as a literal path. It happens when you set environment variable KUBECONFIG to ~/.kube/config

Solution:
Manually expand the home directory before joining the kubeconfig path.

home, _ := os.UserHomeDir()
kubeconfig := filepath.Join(home, ".kube", "config")

Or you can directly set environment variable KUBECONFIG="/home/{user}/.kube/config"

This simple fix ensures the program finds your real kubeconfig file.

Scenario 3: Failed to Get Pod IP

Issue:
Sometimes, you might want to get the pod IP address through the KUBERNETES_POD_IP environment variable. If it’s empty, your code might return on error.

Solution:
Since the purpose was only to let the program continue running, I solved it by injecting mock data. I set an environment variable KUBERNETES_POD_IP to 127.0.0.1, which represents the local machine.

export KUBERNETES_POD_IP=127.0.0.1

This makes the logic work during debugging without changing much of the production code.

Scenario 4: Failed to Call Pod API

Issue:
It’s likely you need to communicate with real Kubernetes resources, such as fetching pod logs.

cli.CoreV1().Pods(namespace).GetLogs(podID, &corev1.PodLogOptions{
    TailLines: ptr.Int64(10),
    Timestamps: true,
}).Stream(context.TODO())

Explanation:
The log streaming API requires a real pod name (podID) and namespace. When running locally, there’s no such pod, so the request fails.

Solution:
Use a real pod from your local cluster. You can create one easily with kubectl:

kubectl create namespace debug-ns
kubectl run debug-pod --image=busybox -n debug-ns -- sleep 3600

Then, set the following environment variables before launching the debugger:

export HOSTNAME=debug-pod
export NAMESPACE=debug-ns

Now when the program calls GetLogs, it will successfully retrieve logs from the running pod in your local cluster.

The image busybox is just an example. When you debug, you definitely want to debug your own application pod. So deploy your application pod to the local cluster and use its name and namespace.

This step is where having a real Kubernetes cluster matters. It provides the live API resources (namespaces, pods, etc.) that your client can interact with during debugging.

Configure VSCode for Debugging

In VSCode, open your .vscode/launch.json and add:

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Debug Kubernetes Client",
      "type": "go",
      "request": "launch",
      "mode": "debug",
      "program": "${fileDirname}",
      "env": {
        "KUBECONFIG": "/Users/yourname/.kube/config",
        "KUBERNETES_POD_IP": "127.0.0.1",
        "HOSTNAME": "your-app-pod-7cd8b768f8-6fxmz",
        "NAMESPACE": "your-app-namespace"
      }
    }
  ]
}

This tells VSCode’s debugger to load your kubeconfig so your Go program can talk to the Kubernetes API server just like kubectl does. You don’t need to export these environment variables in your terminal since VSCode will set them for the debug session.

Verifying the Debug Session

Now run the debugger in VSCode. If your local cluster is up and kubectl works, you should see output similar to:

Found 3 pods

if your program has the logic to list pods via Kubernetes API.
You can now set breakpoints anywhere in your code and inspect how your client interacts with the Kubernetes API.

Summary

Debugging a Go program that uses the Kubernetes client SDK locally is straightforward once you understand the mechanism:

• Docker uses docker.sock for API access
• Kubernetes uses your kubeconfig for API access

As long as you have a local cluster running and a valid kubeconfig, you can fully debug your program in VSCode just like any normal Go application.