Page Comparison

...

Command	Example/Short Form	Purpose
help help cmd	h h b	Prints GDB command help Prints help for the specific command
run	r	Begins the program and runs until next breakpoint or program end
continue	c	Continues the program until next breakpoint or program end
step	s	Single-step code, descending into functions
next	n	Single-step code without descending into functions
finish	fin	Run until the function/loop/etc. is complete
backtrace	bt	Print the stacktrace
up		Go up one context level in the stack
down		Go down one context level in the stack
print variable	p/x foo	Prints the specified variable - this command is very powerful and can be used with arbitrary memory addresses
break function break file:line break location if expr	b prv_func b module.c:50 b module.c:50 if x > 20	Set a breakpoint at the specified function Set a breakpoint at the specified line in the specified file Set a breakpoint that will only stop if the expression is true
info break	i b	Lists breakpoints
delete delete #breakpoint	d d 1	Delete all breakpoints Delete the specified breakpoint

Example

Note
This example is currently incorrect - the program will function correctly.

Let's create a simple program to test GDB with. To start, you should be in our development environment.

Code Block

language	bash
title	Terminal

# Make sure you're sshed into the vagrant box
vagrant ssh
# Should report vagrant
whoami
cd ~/shared/firmware
# Check the state of your local repo
git status
# Move back into the getting started branch
git checkout wip_getting_started

# Initialize new project
make new PROJECT=debug_test

Now, just like before, let's create a new main.c in the project's src folder.

Code Block

language	cpp
title	main.c
linenumbers	true

#include <stdint.h>
#include "soft_timer.h"
#include "loggpio.h"
#include "objpoolinterrupt.h"
#include <stdint"log.h>h"
#include <string"wait.h>h"

#define MAX_NODES 10DEBUG_TEST_TIMEOUT_MS 50

typedef struct LinkedNodeDebugTestStorage {
  structGPIOAddress LinkedNode *nextleds;
  struct LinkedNode *prevsize_t num_leds;
  uint32size_t datacurrent_led;
} LinkedNodeDebugTestStorage;

typedefstatic struct LinkedList {
  LinkedNode *head;
  LinkedNode *tail;
  ObjectPool pool;
  LinkedNode nodes[MAX_NODES];
} LinkedList;

static void prv_init_list(LinkedList *list) {
  memset(list, 0, sizeof(*list));
  objpool_init(&list->pool, list->nodes, NULL, NULL);
}

static bool prv_add_node(LinkedList *list, uint32_t data) {
  LinkedNode *node = objpool_get_node(&list->pool);
  if (node == NULL) {
    return false;
  }

  node->data = data;
  if (list->head == NULL) {
    list->head = node;
    list->tail = node;
  } else {
    list->tail->next = node;
    node->prev = list->tail;
  }

  return true;
}

static void prv_remove_node(LinkedList *list, LinkedNode *node) {
  if (node == list->tail) {
    list->tail = node->prev;
  }

  if (node == list->head) {
    list->head = node->next;
  }

  if (node->prev != NULL) {
    node->prev->next = node->next;
  }

  if (node->next != NULL) {
    node->next->prev = node->prev;
  }

  // This returns a statuscode, but we'll ignore it for now. Just assume it worked.
  objpool_free_node(&list->pool, node);
}

int main(void) {
  LinkedList list = { 0 };
  prv_init_list(&list);

  for (uint32_t i = 0; i < MAX_NODES; i++) {
    printf("Adding %d to list\n", i);
    prv_add_node(&list, i);
  }

  for (int i = MAX_NODES; i > 0; i--) {
    printf("Removing node %d (data %d) from list\n", i - 1, list.nodes[i - 1].data);
    prv_remove_node(&list, &list.nodes[i - 1]);
  }

  printf("List head: 0x%p tail: 0x%p\n", list.head, list.tail);
  printf("All nodes should be empty:\n");

  for (int i = 0; i < MAX_NODES; i++) {
    printf("%d: %d\n", i, list.nodes[i].data);
  }

  return 0;
}

Info
Just as side note, now would be a perfect time to commit your work! Initial revisions that might change drastically as you try to fix bugs are usually a good reference for reverting to when your fixes don't quite work out.

To begin debugging with GDB, we have a special gdb target in our build system.

Code Block
make gdb PROJECT=debug_test PLATFORM=x86

You should see the following prompt appear:

Code Block

GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from build/bin/x86/debug_test...done.
(gdb)

First, let's try to run it as is.

Code Block

(gdb) r
Starting program: /home/titus/projects/firmware/build/bin/x86/debug_test
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Adding 0 to list
Adding 1 to list
Adding 2 to list
Adding 3 to list
Adding 4 to list
Adding 5 to list
Adding 6 to list
Adding 7 to list
Adding 8 to list
Adding 9 to list
Removing node 9 (data 9) from list

Program received signal SIGSEGV, Segmentation fault.
0x0000000000400876 in prv_remove_node (node=0x7fffffffe380, list=0x7fffffffe268)
    at projects/debug_test/src/main.c:54
54          node->next->prev = node->prev;
(gdb)

Woah, what happened? It seems like we hit a segfault, but GDB was able to catch it. Let's take a closer look at what caused that.

Code Block

(gdb) p *list
$1 = {head = 0x7fffffffe2a8, tail = 0x7fffffffe2a8, pool = {nodes = 0x7fffffffe2a8, context = 0x0,
    init_node = 0x0, num_nodes = 10, node_size = 24, free_bitset = 0}, nodes = {{next = 0x0,
      prev = 0x0, data = 0}, {next = 0x0, prev = 0x7fffffffe2a8, data = 1}, {next = 0x0,
      prev = 0x7fffffffe2a8, data = 2}, {next = 0x0, prev = 0x7fffffffe2a8, data = 3}, {next = 0x0,
      prev = 0x7fffffffe2a8, data = 4}, {next = 0x0, prev = 0x7fffffffe2a8, data = 5}, {next = 0x0,
      prev = 0x7fffffffe2a8, data = 6}, {next = 0x0, prev = 0x7fffffffe2a8, data = 7}, {next = 0x0,
      prev = 0x7fffffffe2a8, data = 8}, {next = 0x0, prev = 0x7fffffffe2a8, data = 9}}}
(gdb) p *node
$2 = {next = 0x0, prev = 0x7fffffffe2a8, data = 9}
(gdb) p node->prev->data
$3 = 0
(gdb)

Hmm, it seems like the next and previous pointers aren't quite right. You can see that node->prev is the same value as list->head and list->tail. Let's set a breakpoint where next and prev are set and poke around a bit.

Code Block

(gdb) b main.c:37
Breakpoint 1 at 0x4007f6: file projects/debug_test/src/main.c, line 37.
(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /home/titus/projects/firmware/build/bin/x86/debug_test
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Adding 0 to list
Adding 1 to list

Breakpoint 1, prv_add_node (data=1, list=0x7fffffffe268) at projects/debug_test/src/main.c:37
37          list->tail->next = node;
(gdb) p *list->tail
$4 = {next = 0x0, prev = 0x0, data = 0}
(gdb) n
38          node->prev = list->tail;
(gdb) p *list->tail
$5 = {next = 0x7fffffffe2c0, prev = 0x0, data = 0}
(gdb) n
main () at projects/debug_test/src/main.c:65
65        for (uint32_t i = 0; i < MAX_NODES; i++) {
(gdb) p list
$10 = {head = 0x7fffffffe2a8, tail = 0x7fffffffe2a8, pool = {nodes = 0x7fffffffe2a8, context = 0x0,
    init_node = 0x0, num_nodes = 10, node_size = 24, free_bitset = 1020}, nodes = {{
      next = 0x7fffffffe2c0, prev = 0x0, data = 0}, {next = 0x0, prev = 0x7fffffffe2a8, data = 1}, {
      next = 0x0, prev = 0x0, data = 0}, {next = 0x0, prev = 0x0, data = 0}, {next = 0x0,
      prev = 0x0, data = 0}, {next = 0x0, prev = 0x0, data = 0}, {next = 0x0, prev = 0x0,
      data = 0}, {next = 0x0, prev = 0x0, data = 0}, {next = 0x0, prev = 0x0, data = 0}, {
      next = 0x0, prev = 0x0, data = 0}}}
(gdb) q
A debugging session is active.

        Inferior 1 [process 9829] will be killed.

Quit anyway? (y or n) y

Wait a minute, the tail wasn't updated with the new node! Let's modify prv_add_node by moving list->tail = node; right above the return statement.

Code Block

static bool prv_add_node(LinkedList *list, uint32_t data) {
  LinkedNode *node = objpool_get_node(&list->pool);
  if (node == NULL) {
    return false;
  }

  node->data = data;
  if (list->head == NULL) {
    list->head = node;
  } else {
    list->tail->next = node;
    node->prev = list->tail;
  }
  list->tail = node;

  return true;
}

Now, let's rerun make gdb PROJECT=debug_test PLATFORM=x86 and see what we get.

Code Block

(gdb) r
Starting program: /home/titus/projects/firmware/build/bin/x86/debug_test
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Adding 0 to list
Adding 1 to list
Adding 2 to list
Adding 3 to list
Adding 4 to list
Adding 5 to list
Adding 6 to list
Adding 7 to list
Adding 8 to list
Adding 9 to list
Removing node 9 (data 9) from list
Removing node 8 (data 8) from list
Removing node 7 (data 7) from list
Removing node 6 (data 6) from list
Removing node 5 (data 5) from list
Removing node 4 (data 4) from list
Removing node 3 (data 3) from list
Removing node 2 (data 2) from list
Removing node 1 (data 1) from list
Removing node 0 (data 0) from list
List head: 0x(nil) tail: 0x(nil)
All nodes should be empty:
0: 0
1: 0
2: 0
3: 0
4: 0
5: 0
6: 0
7: 0
8: 0
9: 0
[Inferior 1 (process 10454) exited normally]
(gdb) q

...

DebugTestStorage s_storage;

static void prv_init_leds(DebugTestStorage *storage) {
  GPIOAddress leds[] = {
    { .port = GPIO_PORT_C, .pin = 8 },  //
    { .port = GPIO_PORT_C, .pin = 9 },  //
    { .port = GPIO_PORT_C, .pin = 6 },  //
    { .port = GPIO_PORT_C, .pin = 7 },  //
  };

  GPIOSettings led_settings = {
    .direction = GPIO_DIR_OUT,
    .state = GPIO_STATE_HIGH
  };

  storage->leds = leds;
  storage->num_leds = SIZEOF_ARRAY(leds);
  storage->current_led = 0;

  for (size_t i = 0; i < SIZEOF_ARRAY(leds); i++) {
    gpio_init_pin(&leds[i], &led_settings);
  }
}

static void prv_timeout_cb(SoftTimerID timer_id, void *context) {
  DebugTestStorage *storage = context;

  // Toggle the current LED
  GPIOAddress *led = &storage->leds[storage->current_led];
  LOG_DEBUG("Toggling LED P%d.%x\n", led->port, led->pin);
  gpio_toggle_state(led);

  // Next LED
  storage->current_led = (storage->current_led + 1) % storage->num_leds;

  soft_timer_start_millis(DEBUG_TEST_TIMEOUT_MS, prv_timeout_cb, storage, NULL);
}

int main(void) {
  interrupt_init();
  soft_timer_init();
  gpio_init();

  prv_init_leds(&s_storage);
  uint32_t buffer[20];
  LOG_DEBUG("%ld LEDs set up\n", s_storage.num_leds);
  soft_timer_start_millis(DEBUG_TEST_TIMEOUT_MS, prv_timeout_cb, &s_storage, NULL);

  size_t i = 0;
  while (true) {
    wait();
    buffer[i % SIZEOF_ARRAY(buffer)] += i;
    i++;
  }

  return 0;
}

Info
Just as side note, now would be a perfect time to commit your work! Initial revisions that might change drastically as you try to fix bugs are usually a good reference for reverting to when your fixes don't quite work out.

To begin debugging with GDB, we have a special gdb target in our build system.

Code Block

language	bash
title	Terminal

# This works with STM32 as well - just remove PLATFORM=x86
make gdb PROJECT=debug_test PLATFORM=x86

You should see the following prompt appear:

Code Block

language	bash
title	GDB

GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from build/bin/x86/debug_test...done.
(gdb)

First, let's try to run it as is.

Code Block

language	bash
title	GDB

(gdb) r
Starting program: /home/vagrant/shared/firmware/build/bin/x86/debug_test
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[0] projects/debug_test/src/main.c:58: 4 LEDs set up
[0] projects/debug_test/src/main.c:43: Toggling LED P2.8

Program received signal SIGSEGV, Segmentation fault.
0x0000000000400c01 in printf (__fmt=0x401623 "[%u] %s:%u: Toggling LED P%d.%x\n")
    at /usr/include/x86_64-linux-gnu/bits/stdio2.h:104
104       return __printf_chk (__USE_FORTIFY_LEVEL - 1, __fmt, __va_arg_pack ());
(gdb)

Woah, what happened? It seems like we hit a segfault, but GDB was able to catch it. Let's take a closer look at what caused that.

First, we use backtrace to print the state of the stack. We can see that the segfault is occuring in prv_timeout_cb, which is the timer callback we registered. We can use up and down to move between function scopes in the stack. In our case, we move up to prv_timeout_cb. Since it's a segfault, we're probably attempting to access invalid memory. Looking at led and storage, we see that they're completely invalid. Since led is derived from storage, we should look at where storage is defined.

Info

title	What's a callback?

A callback is just a function that we pass to be called at a later time! We also commonly provide user context variables since the function has likely gone out of scope.

For example, on line 61, soft_timer_start_millis takes in a callback function and context variable. Once the specified time has elapsed, the callback function is run (prv_timeout_cb) with the provided context (&s_storage).

Code Block

language	bash
title	GDB

(gdb) bt
#0  0x0000000000400c11 in printf (__fmt=0x401623 "[%u] %s:%u: Toggling LED P%d.%x\n")
    at /usr/include/x86_64-linux-gnu/bits/stdio2.h:104
#1  prv_timeout_cb (timer_id=<optimized out>, context=0x7fffffffd9c0)
    at projects/debug_test/src/main.c:45
#2  0x0000000000400e7e in prv_soft_timer_interrupt () at libraries/ms-common/src/x86/soft_timer.c:38
#3  prv_soft_timer_handler (interrupt_id=<optimized out>)
    at libraries/ms-common/src/x86/soft_timer.c:49
#4  <signal handler called>
#5  0x0000000000400ad0 in main () at projects/debug_test/src/main.c:64
(gdb) up
#1  prv_timeout_cb (timer_id=<optimized out>, context=0x7fffffffd9c0)
    at projects/debug_test/src/main.c:45
45        LOG_DEBUG("Toggling LED P%d.%x\n", led->port, led->pin);
(gdb) p led
$1 = (GPIOAddress *) 0x800000c029c0
(gdb) p *led
Cannot access memory at address 0x800000c029c0
(gdb) l
40      static void prv_timeout_cb(SoftTimerID timer_id, void *context) {
41        DebugTestStorage *storage = context;
42
43        // Toggle the current LED
44        GPIOAddress *led = &storage->leds[storage->current_led];
45        LOG_DEBUG("Toggling LED P%d.%x\n", led->port, led->pin);
46        gpio_toggle_state(led);
47
48        // Next LED
49        storage->current_led = (storage->current_led + 1) % storage->num_leds;
(gdb) p storage
$2 = (DebugTestStorage *) 0x7fffffffd9c0
(gdb) p *storage
$3 = {leds = 0x7fffffffd9c0, num_leds = 3257981071963740416, current_led = 6301696}
(gdb)

Now we know it's segfaulting because storage is invalid, but why is that? storage is provided through the context parameter of soft_timer_start_millis, which is called on lines 51 and 62. Looking at the code, the first call to soft_timer_start_millis is on line 62, so it should be equal to s_storage.

Let's set a breakpoint on prv_timeout_cb and compare storage with s_storage, since they should be identical. Since storage is set in the function, we'll step forward a few lines so that it's initialized. Due to optimizations, although it's the first line in the code, it won't necessarily be the first thing to be executed.

Tip
You could also just set a breakpoint on the line: `b main.c:41` and step forward one line (`s` or `n`) so it's executed.

Code Block

language	bash
title	GDB

(gdb) b prv_timeout_cb
Breakpoint 1 at 0x400bd6: file projects/debug_test/src/main.c, line 40.
(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /home/vagrant/shared/firmware/build/bin/x86/debug_test
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[0] projects/debug_test/src/main.c:60: 4 LEDs set up
Breakpoint 1, prv_timeout_cb (timer_id=0, context=0x602130 <s_storage>)
    at projects/debug_test/src/main.c:40
40      static void prv_timeout_cb(SoftTimerID timer_id, void *context) {
(gdb) n
45        LOG_DEBUG("Toggling LED P%d.%x\n", led->port, led->pin);
(gdb)
40      static void prv_timeout_cb(SoftTimerID timer_id, void *context) {
(gdb)
44        GPIOAddress *led = &storage->leds[storage->current_led];
(gdb)
41        DebugTestStorage *storage = context;
(gdb)
40      static void prv_timeout_cb(SoftTimerID timer_id, void *context) {
(gdb) p storage
$1 = (DebugTestStorage *) 0x602130 <s_storage>
(gdb) p &s_storage
$2 = (DebugTestStorage *) 0x602130 <s_storage>
(gdb)

So the first iteration seems to match up. Let's continue the program execution and the next time the breakpoint is hit.

Code Block

language	bash
title	GDB

(gdb) c
Continuing.
[0] projects/debug_test/src/main.c:45: Toggling LED P2.8

Breakpoint 1, prv_timeout_cb (timer_id=1, context=0x7fffffffd9c0)
    at projects/debug_test/src/main.c:40
40      static void prv_timeout_cb(SoftTimerID timer_id, void *context) {
(gdb) n
45        LOG_DEBUG("Toggling LED P%d.%x\n", led->port, led->pin);
(gdb)
40      static void prv_timeout_cb(SoftTimerID timer_id, void *context) {
(gdb)
44        GPIOAddress *led = &storage->leds[storage->current_led];
(gdb)
41        DebugTestStorage *storage = context;
(gdb)
40      static void prv_timeout_cb(SoftTimerID timer_id, void *context) {
(gdb) p storage
$3 = (DebugTestStorage *) 0x7fffffffd9c0
(gdb) p *storage
$4 = {leds = 0x7fffffffd9c0, num_leds = 5124990741377889536, current_led = 6301696}
(gdb)

storage has changed and it's obviously invalid now. If we continue running the program, we'll hit a segfault.

So like we said earlier, soft_timer_start_millis provides context, which storage is assigned as. Other than line 62, context is provided on line 51 as &storage.

That actually passes the address of the storage variable (which is on the stack) when we actually want to pass the value of storage that contains the address of s_storage. So the fix is to replace &storage with storage. With that change, it no longer segfaults!

Info
Here's another great place to commit your changes! It's a nice, cohesive change that can be summarized in a line or two if necessary.

Hopefully, you were able to see how GDB allows you to observe the state and data of your program without recompiling or relying on debug output.

Info

HereThere's another great place to commit your changes! It's a nice, cohesive change that can be summarized in a line or two if neccessary.actually another issue in this code. The expected behavior is for all LEDs to toggle in sequence, but the values are all wrong. Can you figure out why?

Hint: Be careful with addresses to stack-allocated variables!

Version	Old Version 3	New Version 4
Changes made by	Titus Chow	Titus Chow
Saved on	Jan 15, 2018	May 04, 2018

Versions Compared

Key

Example