Part 30 Buffer Overflow

Crash Triage

So far we have learned to find a general crash with Fuzzing

But here we have a very big problem

Fuzzer
may generate hundreds or even thousands of different crashes for just one bug

So after Fuzzing we need to figure out

Which crash is really important

Which crashes are actually related to a common bug

Where exactly in the program did the crash occur

And most importantly

What was the root cause of the crash

We call this stage Crash Triage

That is, sorting and examining the crashes to find out which ones are really worth investigating

The first thing we examine

Suppose the Fuzzer caused a crash with these inputs

AAAA
BBBBBBBB
test123
AAAAAAAAAAAAAA
hello_world

We should not quickly conclude that we have 5 bugs here

It is possible that all five inputs ultimately lead to a specific command in the program Get

For example

0x401234: mov byte ptr [rax], dl

And this command causes a Crash

So what is important to us is not just the input file

We need to see where the Crash occurred and under what conditions

That is

Input
↓
Program
↓
Fault
↓
Where did it happen
↓
Why did it happen

2 Faulting Instruction

One of the most important things to find when investigating a Crash is
"Faulting Instruction"

That is, the instruction that the CPU was executing when the Exception or Fault occurred

For example

Program received signal SIGSEGV

RIP = 0x401234

0x401234:
mov byte ptr [rax], dl

Here, just knowing that the program gave us "SIGSEGV" is not enough

We need to ask

What value did RAX have? Why was this value invalid? Where did this value come from?

For example "RAX" may point to an invalid address

So instead of just saying

Program crashed

we need to go a step deeper

Crash
↓
Faulting Instruction
↓
Invalid Register / Memory
↓
Why did this happen

Crash itself is not the cause

Crash is the result of a previous event

3 Stack Trace

Then we go to the "Stack Trace"

For example

#0 0x401234 in process_data()
#1 0x401567 in handle_input()
#2 0x401789 in main()

That means the path the program took to Crash was roughly

main()
↓
handle_input()
↓
process_data()
↓
CRASH

This information is very important for us

Because now we know which part of the program we should check first

For example, we can go to "process_data()" in IDA or Ghidra and see exactly what happened before the Crash

Crash Deduplication

Now suppose the Fuzzer gives us a thousand crashes

If we have information like this for each crash

Signal
Faulting Address
Faulting Instruction
Stack Trace

We can put similar crashes together

For example

Crash 1 → process_data → 0x401234
Crash 2 → process_data → 0x401234
Crash 3 → process_data → 0x401234
Crash 4 → parse_packet → 0x402010

The first to third crashes are probably in the same family

Crash 1
Crash 2
Crash 3
↓
Crash Family A

But the fourth crash has a different path

Crash 4
↓
Crash Family B

Of course, there is an important point here

The equality of the "Faulting Address" alone does not prove that two crashes are necessarily the same bug

There may be several different paths to the same instruction Get

So for Deduplication we usually check several indicators together

For example

Faulting Instruction

Call Stack

Registers

Memory State

Crash Type

Input Behavior

The goal is to separate duplicate crashes from truly different crashes

Sanitizers are very helpful here

For example, if the program was run with "AddressSanitizer" or "ASan"

Instead of just seeing a simple message like this

Segmentation fault

we may have much more information

For example

ERROR: AddressSanitizer:

heap-buffer-overflow

READ of size 4

#0 process_data

#1 handle_input

#2 main

ASan
can give more information about the type of error and the location of the unauthorized access and in some errors it also reports information about the allocation and access location

As a result, the inspection path can go from this

Crash

to

Crash
↓
Error Type
↓
Invalid Access
↓
Stack Trace
↓
Suspicious Function
↓
Root Cause

Instead of just knowing that the program crashed