At a Glance

Prerequisites

Module 1: Setup

Debugging

More important than knowing how to make something work is knowing what to do when it doesn't. Debugging is probably one of the most important skills you can learn, since things rarely work perfectly the first time.

Printf Debugging

A common method of debugging is inserting print statements throughout the program to log program flow and variable states. Although this has its place in debugging and is still useful, it has many drawbacks, especially in an embedded context.

Requires recompilation every time we want to look at something different - For embedded, this also means reflashing.
Requires cleanup after the problem is found.
Requires dump functions for anything more than an integer.
Printf is huge - For embedded, adding printf support can eat up a lot of RAM and flash.
Printf is slow - It takes a long time to actually format the string, then transmitting it through your debug output has an additional delay.
- This can actually block the main program for a noticeable amount of time.
- This will wreck havoc with any timing-sensitive operations such as race conditions.
It requires a debug output - In Module 2: Hello World, we needed a UART-to-USB adapter to see the serial output.
It can't handle crashes and relies on the debug output to be working.

Of course, like any tool, this has its place. If you're okay with the drawbacks, printf is great for convenient, human-readable continuous logging. We have a set of LOG macros for human-readable print statements in our HAL, such as the one you used in Module 2: Hello World.

GPIO Debugging

A higher-speed variant of this technique on embedded platforms is toggling GPIOs at certain points and using an oscilloscope or logic analyzer to trace execution. I find this most useful for timing-sensitive debugging.

As an example, I tend to use this technique to determine bottlenecks in functions by toggling the IO before and after chunks of code and looking at the timing conditions. This is useful because it doesn't require a programmer to be attached.

GDB

In software, knowing how to use a debugger like GDB is invaluable. Debuggers allow us to do things like step through running programs, inspect variables, and set breakpoints without recompilation. Since it can access the target's memory, it doesn't take any effort to dump entire structures or walk through arrays.

GDB is usually the first tool I reach for when something's wrong. Since it doesn't require recompilation, I can attach it to running programs or microcontrollers without any hassle. My workflow usually looks like this:

Attach to the target using GDB
Set up breakpoints on the functions or lines I suspect
Run the program until my breakpoints are hit
Look at the function's parameters, local variables, and the stack trace for any anomalies
Continue execution and repeat until I've found the problem

Useful Commands

Fun fact: GDB can handle short forms of commands. For example, bt instead of backtrace. GDB also supports tab completion for pretty much everything.

Command	Example/Short Form	Purpose
help help cmd	h h b	Prints GDB command help Prints help for the specific command
run	r	Begins the program and runs until next breakpoint or program end
continue	c	Continues the program until next breakpoint or program end
step	s	Single-step code, descending into functions
next	n	Single-step code without descending into functions
finish	fin	Run until the function/loop/etc. is complete
backtrace	bt	Print the stacktrace
up		Go up one context level in the stack
down		Go down one context level in the stack
print variable	p/x foo	Prints the specified variable - this command is very powerful and can be used with arbitrary memory addresses
break function break file:line break location if expr	b prv_func b module.c:50 b module.c:50 if x > 20	Set a breakpoint at the specified function Set a breakpoint at the specified line in the specified file Set a breakpoint that will only stop if the expression is true
info break	i b	Lists breakpoints
delete delete #breakpoint	d d 1	Delete all breakpoints Delete the specified breakpoint

Example

Let's create a simple program to test GDB with. To start, you should be in our development environment.

# Make sure you're sshed into the vagrant box
vagrant ssh
# Should report vagrant
whoami
cd ~/shared/firmware
# Check the state of your local repo
git status
# Move back into the getting started branch
git checkout wip_getting_started

# Initialize new project
make new PROJECT=debug_test

Now, just like before, let's create a new main.c in the project's src folder.

#include "log.h"
#include "objpool.h"
#include <stdint.h>
#include <string.h>

#define MAX_NODES 10

typedef struct LinkedNode {
  struct LinkedNode *next;
  struct LinkedNode *prev;
  uint32_t data;
} LinkedNode;

typedef struct LinkedList {
  LinkedNode *head;
  LinkedNode *tail;
  ObjectPool pool;
  LinkedNode nodes[MAX_NODES];
} LinkedList;

static void prv_init_list(LinkedList *list) {
  memset(list, 0, sizeof(*list));
  objpool_init(&list->pool, list->nodes, NULL, NULL);
}

static bool prv_add_node(LinkedList *list, uint32_t data) {
  LinkedNode *node = objpool_get_node(&list->pool);
  if (node == NULL) {
    return false;
  }

  node->data = data;
  if (list->head == NULL) {
    list->head = node;
    list->tail = node;
  } else {
    list->tail->next = node;
    node->prev = list->tail;
  }

  return true;
}

static void prv_remove_node(LinkedList *list, LinkedNode *node) {
  if (node == list->tail) {
    list->tail = node->prev;
  }

  if (node == list->head) {
    list->head = node->next;
  }

  if (node->prev != NULL) {
    node->prev->next = node->next;
  }

  if (node->next != NULL) {
    node->next->prev = node->prev;
  }

  // This returns a statuscode, but we'll ignore it for now. Just assume it worked.
  objpool_free_node(&list->pool, node);
}

int main(void) {
  LinkedList list = { 0 };
  prv_init_list(&list);

  for (uint32_t i = 0; i < MAX_NODES; i++) {
    printf("Adding %d to list\n", i);
    prv_add_node(&list, i);
  }

  for (int i = MAX_NODES; i > 0; i--) {
    printf("Removing node %d (data %d) from list\n", i - 1, list.nodes[i - 1].data);
    prv_remove_node(&list, &list.nodes[i - 1]);
  }

  printf("List head: 0x%p tail: 0x%p\n", list.head, list.tail);
  printf("All nodes should be empty:\n");

  for (int i = 0; i < MAX_NODES; i++) {
    printf("%d: %d\n", i, list.nodes[i].data);
  }

  return 0;
}

Just as side note, now would be a perfect time to commit your work! Initial revisions that might change drastically as you try to fix bugs are usually a good reference for reverting to when your fixes don't quite work out.

To begin debugging with GDB, we have a special gdb target in our build system.

make gdb PROJECT=debug_test PLATFORM=x86

You should see the following prompt appear:

GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from build/bin/x86/debug_test...done.
(gdb)

First, let's try to run it as is.

(gdb) r
Starting program: /home/titus/projects/firmware/build/bin/x86/debug_test
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Adding 0 to list
Adding 1 to list
Adding 2 to list
Adding 3 to list
Adding 4 to list
Adding 5 to list
Adding 6 to list
Adding 7 to list
Adding 8 to list
Adding 9 to list
Removing node 9 (data 9) from list

Program received signal SIGSEGV, Segmentation fault.
0x0000000000400876 in prv_remove_node (node=0x7fffffffe380, list=0x7fffffffe268)
    at projects/debug_test/src/main.c:54
54          node->next->prev = node->prev;
(gdb)

Woah, what happened? It seems like we hit a segfault, but GDB was able to catch it. Let's take a closer look at what caused that.

(gdb) p *list
$1 = {head = 0x7fffffffe2a8, tail = 0x7fffffffe2a8, pool = {nodes = 0x7fffffffe2a8, context = 0x0,
    init_node = 0x0, num_nodes = 10, node_size = 24, free_bitset = 0}, nodes = {{next = 0x0,
      prev = 0x0, data = 0}, {next = 0x0, prev = 0x7fffffffe2a8, data = 1}, {next = 0x0,
      prev = 0x7fffffffe2a8, data = 2}, {next = 0x0, prev = 0x7fffffffe2a8, data = 3}, {next = 0x0,
      prev = 0x7fffffffe2a8, data = 4}, {next = 0x0, prev = 0x7fffffffe2a8, data = 5}, {next = 0x0,
      prev = 0x7fffffffe2a8, data = 6}, {next = 0x0, prev = 0x7fffffffe2a8, data = 7}, {next = 0x0,
      prev = 0x7fffffffe2a8, data = 8}, {next = 0x0, prev = 0x7fffffffe2a8, data = 9}}}
(gdb) p *node
$2 = {next = 0x0, prev = 0x7fffffffe2a8, data = 9}
(gdb) p node->prev->data
$3 = 0
(gdb)

Hmm, it seems like the next and previous pointers aren't quite right. You can see that node->prev is the same value as list->head and list->tail. Let's set a breakpoint where next and prev are set and poke around a bit.

(gdb) b main.c:37
Breakpoint 1 at 0x4007f6: file projects/debug_test/src/main.c, line 37.
(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /home/titus/projects/firmware/build/bin/x86/debug_test
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Adding 0 to list
Adding 1 to list

Breakpoint 1, prv_add_node (data=1, list=0x7fffffffe268) at projects/debug_test/src/main.c:37
37          list->tail->next = node;
(gdb) p *list->tail
$4 = {next = 0x0, prev = 0x0, data = 0}
(gdb) n
38          node->prev = list->tail;
(gdb) p *list->tail
$5 = {next = 0x7fffffffe2c0, prev = 0x0, data = 0}
(gdb) n
main () at projects/debug_test/src/main.c:65
65        for (uint32_t i = 0; i < MAX_NODES; i++) {
(gdb) p list
$10 = {head = 0x7fffffffe2a8, tail = 0x7fffffffe2a8, pool = {nodes = 0x7fffffffe2a8, context = 0x0,
    init_node = 0x0, num_nodes = 10, node_size = 24, free_bitset = 1020}, nodes = {{
      next = 0x7fffffffe2c0, prev = 0x0, data = 0}, {next = 0x0, prev = 0x7fffffffe2a8, data = 1}, {
      next = 0x0, prev = 0x0, data = 0}, {next = 0x0, prev = 0x0, data = 0}, {next = 0x0,
      prev = 0x0, data = 0}, {next = 0x0, prev = 0x0, data = 0}, {next = 0x0, prev = 0x0,
      data = 0}, {next = 0x0, prev = 0x0, data = 0}, {next = 0x0, prev = 0x0, data = 0}, {
      next = 0x0, prev = 0x0, data = 0}}}
(gdb) q
A debugging session is active.

        Inferior 1 [process 9829] will be killed.

Quit anyway? (y or n) y

Wait a minute, the tail wasn't updated with the new node! Let's modify prv_add_node by moving list->tail = node; right above the return statement.

static bool prv_add_node(LinkedList *list, uint32_t data) {
  LinkedNode *node = objpool_get_node(&list->pool);
  if (node == NULL) {
    return false;
  }

  node->data = data;
  if (list->head == NULL) {
    list->head = node;
  } else {
    list->tail->next = node;
    node->prev = list->tail;
  }
  list->tail = node;

  return true;
}

Now, let's rerun make gdb PROJECT=debug_test PLATFORM=x86 and see what we get.

(gdb) r
Starting program: /home/titus/projects/firmware/build/bin/x86/debug_test
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Adding 0 to list
Adding 1 to list
Adding 2 to list
Adding 3 to list
Adding 4 to list
Adding 5 to list
Adding 6 to list
Adding 7 to list
Adding 8 to list
Adding 9 to list
Removing node 9 (data 9) from list
Removing node 8 (data 8) from list
Removing node 7 (data 7) from list
Removing node 6 (data 6) from list
Removing node 5 (data 5) from list
Removing node 4 (data 4) from list
Removing node 3 (data 3) from list
Removing node 2 (data 2) from list
Removing node 1 (data 1) from list
Removing node 0 (data 0) from list
List head: 0x(nil) tail: 0x(nil)
All nodes should be empty:
0: 0
1: 0
2: 0
3: 0
4: 0
5: 0
6: 0
7: 0
8: 0
9: 0
[Inferior 1 (process 10454) exited normally]
(gdb) q

Yay! That seems to have fixed this simple example. Hopefully, you were able to see how GDB allows you to observe the state and data of your program without recompiling or relying on debug output.

Here's another great place to commit your changes! It's a nice, cohesive change that can be summarized in a line or two if neccessary.

Module 3: Debugging