a73x

high effort, low reward
← Posts

Debugging in Go

Intro to Delve

In the beginning there was the print statement... technically it was probably inspecting registers directly but I wasn't present for that era. And as man smashed lighting into rocks and convinced them to think, Delve was born to debug Go programs. Delve enables developers to inspect the internals their running Go programs, manipulate them at run time, and even rewind time (more on that later).

Anyone that has debugged a Go program from an IDE has interacted with Delve. In the same spirit as the standardisation of the LSP protocol, Microsoft proposed a "Debug Adapter Protocol". This provides a generic way for an IDE to interact with a debugger tool. So Delve implements this protocol and an IDE gets to communicate with it with relatively little Delve specific nuance. I've had mixed experience debugging from an IDE. I find VS Code occasionally fails to set breakpoints and find Goland to be much more reliable As time goes on however, I found myself using Delve directly more and more frequently due to how much functionality is embedded within the CLI tool.

How does Delve work?

Delve is built for Go, so it understands the semantics of a running Go binary. On Linux, it uses ptrace to attach and control running processes, and DWARF data to make the executing assembly human readable. DWARF data is included by default with all Go binaries, and provides a bidirectional mapping between the executing assembly and the original source code.

By default optimisations such as inlining functions also occur, which can make debugging tricky so to disable these optimisations during development you can use:

go build -gcflags=all="-N -l"

Basic usage

Assuming a working Go environment, you can install Delve directly with

go install github.com/go-delve/delve/cmd/dlv@latest

For demonstration purposes I have a basic main.go.

package main

import (
	"fmt"
	"time"
)

func main() {
	run := true
	i := 0
	for run {
		fmt.Println(i)
		i += 1
		time.Sleep(1 * time.Second)
	}
}

There are several ways we can launch this for debugging. Keep in mind that the go build has optimisations enabled.

Launching

Directly

dlv debug ./main.go

Attach

go build ./main.go
./main # in a different terminal
dlv attach $(pgrep main) # assuming you only have 1 process called main

Exec

go build ./main.go
dlv exec ./main

The CLI

Once attached, you'll be greeted with a blank canvas

dlv debug ./main.go
Type 'help' for list of commands.
(dlv)

From here help or h will provide you with a list of all available commands. The most common ones I use are:

Currently we're sat somewhere in the runtime (press l to check)

Since we know our entry point into the main function in main.go we can write b main.main to set a breakpoint on that function.

(dlv) b main.main
Breakpoint 1 set at 0x49846a for main.main() ./main.go:8

Equivalently we could have written b main.go:8

Currently out program is not running, so lets continue to the breakpoint we've just set

(dlv) c
> [Breakpoint 1] main.main() ./main.go:8 (hits goroutine(1):1 total:1) (PC: 0x4b736e)
     3:	import (
     4:		"fmt"
     5:		"time"
     6:	)
     7:	
=>   8:	func main() {
     9:		run := true
    10:		i := 0
    11:		for run {
    12:			fmt.Println(i)
    13:			i += 1
(dlv)

From here we can repeatedly press s to step through the program, the source code will be printed each time.

(dlv) s
> main.main() ./main.go:9 (PC: 0x4b7372)
     4:		"fmt"
     5:		"time"
     6:	)
     7:	
     8:	func main() {
=>   9:		run := true
    10:		i := 0
    11:		for run {
    12:			fmt.Println(i)
    13:			i += 1
    14:			time.Sleep(1 * time.Second)
(dlv) s
> main.main() ./main.go:10 (PC: 0x4b7377)
     5:		"time"
     6:	)
     7:	
     8:	func main() {
     9:		run := true
=>  10:		i := 0
    11:		for run {
    12:			fmt.Println(i)
    13:			i += 1
    14:			time.Sleep(1 * time.Second)
    15:		}
(dlv) s
> main.main() ./main.go:11 (PC: 0x4b7382)
     6:	)
     7:	
     8:	func main() {
     9:		run := true
    10:		i := 0
=>  11:		for run {
    12:			fmt.Println(i)
    13:			i += 1
    14:			time.Sleep(1 * time.Second)
    15:		}
    16:	}
(dlv)

Nothing really interesting here, lets set a breakpoint on line 12 and continue

(dlv) b 12
Breakpoint 2 set at 0x4b738e for main.main() ./main.go:12
(dlv) c
> [Breakpoint 2] main.main() ./main.go:12 (hits goroutine(1):1 total:1) (PC: 0x4b738e)
     7:	
     8:	func main() {
     9:		run := true
    10:		i := 0
    11:		for run {
=>  12:			fmt.Println(i)
    13:			i += 1
    14:			time.Sleep(1 * time.Second)
    15:		}
    16:	}

Here we can step into the function with s and backout with so

(dlv) s
> fmt.Println() /usr/lib/go/src/fmt/print.go:313 (PC: 0x4b26ae)
   308:	}
   309:	
   310:	// Println formats using the default formats for its operands and writes to standard output.
   311:	// Spaces are always added between operands and a newline is appended.
   312:	// It returns the number of bytes written and any write error encountered.
=> 313:	func Println(a ...any) (n int, err error) {
   314:		return Fprintln(os.Stdout, a...)
   315:	}
   316:	
   317:	// Sprintln formats using the default formats for its operands and returns the resulting string.
   318:	// Spaces are always added between operands and a newline is appended.
(dlv) so
0
> main.main() ./main.go:13 (PC: 0x4b740e)
Values returned:
	n: 2
	err: error nil

     8:	func main() {
     9:		run := true
    10:		i := 0
    11:		for run {
    12:			fmt.Println(i)
=>  13:			i += 1
    14:			time.Sleep(1 * time.Second)
    15:		}
    16:	}

We can see Println returned an int and a nil error.

With our breakpoint set on 12 we can hit c a few times and... we're stuck in an infinite loop. A quick look at locals shows us that run=true and there is no sign of stopping.

(dlv) locals
run = true
i = 3

Rather interestingly we can manipulate this value directly and see what happens to the execution of our program.

(dlv) set run=false
(dlv) c
3
Process 40996 has exited with status 0

Turns out we just quit!

We can also use the on command to change the behavior of how a breakpoint behaves when hit.

(dlv) b main.go:14
Breakpoint 1 set at 0x4b7413 for main.main() ./main.go:14
(dlv) help on
Executes a command when a breakpoint is hit.

	on <breakpoint name or id> <command>
	on <breakpoint name or id> -edit
	

Supported commands: print, stack, goroutine, trace and cond. 
To convert a breakpoint into a tracepoint use:
	
	on <breakpoint name or id> trace

The command 'on <bp> cond <cond-arguments>' is equivalent to 'cond <bp> <cond-arguments>'.

The command 'on x -edit' can be used to edit the list of commands executed when the breakpoint is hit.
(dlv) on 1 stack
(dlv) c
0
> [Breakpoint 1] main.main() ./main.go:14 (hits goroutine(1):1 total:1) (PC: 0x4b7413)
	Stack:
		0  0x00000000004b7413 in main.main
		    at ./main.go:14
		1  0x00000000004460d2 in runtime.main
		    at /usr/lib/go/src/runtime/proc.go:285
		2  0x000000000047d281 in runtime.goexit
		    at /usr/lib/go/src/runtime/asm_amd64.s:1693
     9:		run := true
    10:		i := 0
    11:		for run {
    12:			fmt.Println(i)
    13:			i += 1
=>  14:			time.Sleep(1 * time.Second)
    15:		}
    16:	}
(dlv)

By default breakpoints stop program execution, which may be undesirable An alternative is to use trace

(dlv) trace main.go:14
Tracepoint 1 set at 0x4b7413 for main.main() ./main.go:14
(dlv) c
0
> goroutine(1): main.main()
1
> goroutine(1): main.main()
2
> goroutine(1): main.main()
3
> goroutine(1): main.main()
4
> goroutine(1): main.main()
5
> goroutine(1): main.main()

Which isn't very useful by itself, but we can combine it with the on command (you can use ctrl-c to stop execution)

(dlv) on 1 locals
(dlv) c
6
> goroutine(1): main.main()
	run: true
	i: 7
7
> goroutine(1): main.main()
	run: true
	i: 8
8
> goroutine(1): main.main()
	run: true
	i: 9
9
> goroutine(1): main.main()
	run: true
	i: 10

Starlark

Delve also supports Starlark scripting (a Python dialect) for automation and gives access to a rich set of functions for interacting with the debugger.

When you source a Starlark script, any main function runs automatically and functions prefixed with a command_ because available as custom commands. So command_foo is available as foo with the Delve prompt.

For instance given the following code snippet:

package main

import (
	"fmt"
	"math/rand"
)

type Foo struct {
}

func (f *Foo) A(i int) {
	fmt.Println(i)
}

func (f *Foo) B(i int) {
	i += 2
	fmt.Println(i)
}

func (f *Foo) C(i int) {
	i += 10
	fmt.Println(i)
}

func main() {
	f := Foo{}

	f.A(rand.Intn(10))
	f.B(rand.Intn(10))
	f.C(rand.Intn(10))
}

We can list all Foo methods with this script:

def main():
	for f in functions("main.\\(\\*foo\\)").funcs:
		print(f)
		bp = create_breakpoint(
			{
				"functionname": f,
				"line": -1,
				"tracepoint": true,
				"loadargs": {
					"followpointers": true,
				},
			}, none, none, false)
		dlv_command("on " + str(bp.breakpoint.id) + " args")

Attaching trace points instantly and printing args.

(dlv) source debug.star
main.(*Foo).A
main.(*Foo).B
main.(*Foo).C
(dlv) c
> goroutine(1): main.(*Foo).A((*main.Foo)(0xc000118f60), 6)
6
> goroutine(1): main.(*Foo).B((*main.Foo)(0xc000118f60), 8)
10
> goroutine(1): main.(*Foo).C((*main.Foo)(0xc000118f60), 4)
14
Process 52692 has exited with status 0

Rewinding Time

For those with rr installed, Delve has native support for recording and replaying executions deterministically.

Take a look at this production ready code:

package main

import (
	"fmt"
	"log"
	"math/rand"
)

func main() {
	for {
		i := rand.Intn(10)
		fmt.Println(i)
		if i == 5 {
			log.Fatal("uhoh")
		}
	}
}

Whilst the cause of the program exiting is quite obvious, the same principle can be applied to other non deterministic problems.

% dlv debug --backend=rr ./main.go                                                          
rr: Saving execution to trace directory `/home/a73x/.local/share/rr/__debug_bin4196056735-0'.
7
6
4
4
4
4
4
2
3
0
0
4
4
4
8
9
8
5
2025/09/28 15:05:26 uhoh
Type 'help' for list of commands.
(dlv)

From here we can set a breakpoint on the offending line and run the program.

(dlv) b main.go:14
Breakpoint 1 set at 0x4c7075 for main.main() ./main.go:14
(dlv) c
7
6
4
4
4
4
4
2
3
0
0
4
4
4
8
9
8
5
> [Breakpoint 1] main.main() ./main.go:14 (hits goroutine(1):1 total:1) (PC: 0x4c7075)
Completed event: 457
     9:	func main() {
    10:		for {
    11:			i := rand.Intn(10)
    12:			fmt.Println(i)
    13:			if i == 5 {
=>  14:				log.Fatal("uhoh")
    15:			}
    16:		}
    17:	}
(dlv) locals
i = 5

We can clearly see the cause was i equaling 5. Lets continue.

(dlv) c
2025/09/28 15:08:50 uhoh
Process 0 has exited with status 0

What if we want to examine that moment again? With traditional debugging, we would have to restart the process and hope to hit the same case again. But with rr recording:

(dlv) rw
> [Breakpoint 1] main.main() ./main.go:14 (hits goroutine(1):2 total:2) (PC: 0x4c7075)
Completed event: 457
     9:	func main() {
    10:		for {
    11:			i := rand.Intn(10)
    12:			fmt.Println(i)
    13:			if i == 5 {
=>  14:				log.Fatal("uhoh")
    15:			}
    16:		}
    17:	}

Since the execution is saved, we are able to rewind and step back in time as many times as we need.

That flaky test that only fails once a blue moon? Record it with rr in a loop until you capture the failure, then step through the exact execution that caused the bug. No more guessing or chasing down red herrings.