Remote debugging with Delve

Remote debugging

Previously we discussed local debugging with Goland IDE. Currently we'll discuss how to remotely debug the program, which is working inside Docker container, with Visual Studio Code and Goland IDE.

In a local debugging IDE manages everything — compiles the program, starts it, connects to it.

But sometimes you may need to do a remote debugging. In this case our program should be started regardless of IDE. It will a developer/devops work to prepare program for remote debugging.

Why do we need a remote debugging? It seems to me that it is a rare case, but here are the situation one may need it:

program can't be tested locally, only in specific remote environment;
one have to analyze the bug, that is reproduced on remote environment only — during the integration testing, for example;

Before starting to implement remote debugging, one must understand, that bugs can fixed easily on early stages of development. Debugging locally one can run program locally with go run или delve debug or any IDE; but with remote debugging one have to wait for program to be deployed to remote environment.

Delve

Debug tool, used in Goland IDE or Visual Studio Code, is Delve.

Delve + IDE debugging always works like this:

Delve is started as a server application, listen for a network connections on some network port.
Delve runs our program (compiled binary or the GO source code).
Delve allows Goland IDE or Visual Studio Code to connect to it, receives breakpoints data.
On breakpoint Delve pauses the program. Delve sends variables data to IDE.

Delve — command line tool. All it's CLI parameters are defined here.
Besides having network API Delve also has command line debugging options to debug directly from command line (without IDE). But we are not going to use this option.

So we have to start Delve and provide remote connection to it from our IDE.

The sources

For better understanding of this article, i've created a Github repository with all related code.

Docker

Docker container is popular type of deployment. Recently we discussed mimnimal Docker image building and Docker swarm deployment. Docker container here can be used to demonstrate remote connection.

Let's set the Dockerfile up:

FROM rhaps1071/golang-1.14-alpine-git AS build
WORKDIR /
COPY . .
RUN CGO_ENABLED=0 go get -ldflags "-s -w -extldflags '-static'" github.com/go-delve/delve/cmd/dlv
RUN CGO_ENABLED=0 go build -gcflags "all=-N -l" -o ./app

FROM scratch
COPY --from=build /go/bin/dlv /dlv
COPY --from=build /app /app
ENTRYPOINT [ "/dlv" ]

Here i used two-stage build and static compilation to make minimal docker image FROM scratch.
There two files in resulting Docker image — /go/bin/dlv — Delve; /app — our program.

GO Compilation flags are supported by many GO tools - build, clean, get, install, list, run, test.
Due to this, Delve binary, received with go get, is also statically compiled. By default, Delve binary compiled in Alpine Linux environment depends on two libraries. One can check it with ldd tool:

ldd /go/bin/dlv 
/lib/ld-musl-x86_64.so.1 (0x7f761f8b7000)
libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f761f8b7000)

-gcflags "all=-N -l" flags, which are used to compile the main program, are required to make Delve debugging our program properly.

To build the container one may use docker build -f ./docker/debug/Dockerfile -t debug ., which is implemented as Makefile command docker-build-debug, so the same command will work as make docker-build-debug.

Container run

Let's start our image as a container by using of docker-compose.

The docker-compose.yml contents is below:

version: "3"

services:
	debug:
	build:
		dockerfile: docker/debug/Dockerfile
		context: ../../
	ports:
		- 2345:2345
	command: "--listen=:2345 --headless=true --log=true --log-output=debugger,debuglineerr,gdbwire,lldbout,rpc --accept-multiclient --api-version=2 exec ./app"

Previously we set up the ENTRYPOINT in our image as /dlv, so now in the command parameter we pass Delve arguments only. Parameters above are passed to make Delve work as network server and to enable rich logging.

Let's call docker-compose -f ./docker/debug/docker-compose.yml up and check the logs out.
This command is also available as make docker-run-debug.

Starting debug_debug_1 ... done
Attaching to debug_debug_1
debug_1  | API server listening at: [::]:2345
debug_1  | 2020-07-10T13:36:06Z debug layer=rpc API server pid = 1
debug_1  | 2020-07-10T13:36:06Z info layer=debugger launching process with args: [./app]

The logs mean that the debug server is started on 2345 port and it is ready to accept connections.

Let's connect to it from IDEs.

Visual Studio Code

We have to open our project in IDE. The source code represent the program in our container.

Let's create/modify .vscode/launch.json file to have following configuration:

{
	"version": "0.2.0",
	"configurations": [
		{
			"name": "Attach",
			"type": "go",
			"request": "attach",
			"mode": "remote",
			"remotePath": "",
			"port":2345,
			"host":"127.0.0.1",
			"showLog": true,
			"trace": "log",
			"logOutput": "rpc"
		}
	]
}

"request": "attach" — allows VSCode to attach to running Delve, instead of starting new debug session locally;
port,host — the network host and port of our Delve. By using docker-compose we started Docker container locally and published it's port 2345 to the host (localhost).
remotePath — of the critical parameters, which affects successful breakpoints setting. It is the path to the sources folder, which was used during the compile stage. We compiled our binary by Dockerfile, using WORKDIR /, so our directory is root directory — let's leave remotePath blank.

Before doing anything in IDE, let's check that our Docker container is working currently.

Next, run "Attach" debug task:

We should set breakpoints and we need them to stay (visually in IDE). Each breakpoint setting is reflected in Docker containter logs — there should not be errors.

Goland IDE

All settings in this IDE can be changed in graphical interface, without editing configuration files.
Important thing — enable modules integration:

Click Run — Edit configurations, add new debug configuration, select "Go Remote":

Same as in VSCode, we should set breakpoints and ensure they are in place. We should check Docker container logs. If there are any errors like could not find file somedir/main.go, you need to enable GO modules integration.

Remote debugging with Delve