GDB seems to have been clearly made with source-level debugging in mind, the kind where the source code of the program is available. Sometimes this is not the case, or you just don’t want to bother looking for the source code. Here I’ll put a bunch of useful commands.

• layout asm: enable TUI mode and show assembly instructions

• layout regs: enable TUI mode and show CPU registers

• set disassembly-flavor intel: use Intel-like disassembly, which is more familiar to those writing or debugging assembly code on Windows

• Ctrl+P, Ctrl+N: serve as alternatives to “up” and “down” when the active TUI window is not the command window

• Ctrl+X, O: switch the active TUI window

• Ctrl+X, (A or Shift+A or Ctrl+A): toggle TUI mode

• x/5i $pc: check out 5 instructions, starting at the program counter

• ni (nexti) and si (stepi): next/step, but on an instruction level

• b *0x<addr>: breakpoint on a specific address

• info registers: look at CPU registers

• info frame: look at current stack frame, potentially useful for finding function arguments and local vars

• x/8g $sp: look at (64-bit words) in the stack

x64 Linux C/C++ conventions

This is a set of additional notes written during the process of disassembling a library.

For function calls, the arguments are stored in the following order:

• RDI register
• RSI register
• RDX register
• RCX register
• R8 register
• R9 register
• rest are pushed onto the stack in reverse order

When a function is called, it will typically perform the following operations:

push rbp
mov  rbp, rsp
sub  rsp, (amount of space used for local variables)
...
add  rsp, (same amount)
leave ; or mov rsp,rbp and pop rbp
ret

This means that any reference to memory at [rbp-0x…] is using the function’s local variables.

If a function returns a value, it will be stored in the EAX register.

If it is a C++ function, its name will be “mangled”: its argument types will be included in the name. The utility c++filt can convert these mangled names to their standard forms. If the function is part of an object, it will require the first parameter to be the object’s address.

Seems like the same is true for constructors and destructors: their first parameter is an address that will point to the object in question.