ReverseEngineering

Working, tested code for every chapter of the book The Art of Exploit Development: Vulnerability Research and Exploitation on Today's Hardened Systems.

https://github.com/KazamaDono/taoxd

Mobile App Reverse Engineering: Get started with discovering, analyzing, and exploring the internals of Android and iOS apps

@reverseengine

Reverse Engineering with Terraform

Dissecting the dark web Reverse Engineering

@reverseengine

Golang loader that uses vulnerable drivers to terminate EDR and antivirus processes before dropping Formbook

https://youtu.be/Wz6ROkJ3AZg?si=cO8uan3vRo7NTZZs

Silverseal is a Linux framework containing a bootkit, rootkit loader and a rootkit

https://github.com/Idov31/Silverseal

@reverseengine

Flicker and fall: rooting the Philips Hue Bridge both remotely and wirelessly

https://blog.thalium.re/posts/rooting-the-philips-hue-bridge-remotely-and-wirelessly

@reverseengine

A Year of Hacking with LLMs

https://sites.google.com/site/zhiniangpeng/blogs/Hacking-with-LLMs-Eng

@reverseengine

Documentation for binutils

https://sourceware.org/binutils/docs

@reverseengine

LD

https://sourceware.org/binutils/docs/ld

@reverseengine

That's why after finding the Crash, we need to enter the analysis stage

Fuzzer
↓
Crash
↓
Debugger
↓
Root Cause Analysis
↓
Bug Classification

Don't confuse Root Cause and Crash

Always keep these two separate

Crash
↓
What happened

But

Root Cause
↓
Why did it happen

For example:

Crash
↓
SIGSEGV
↓
Invalid Memory Access

The Root Cause is still unknown

We may find out after investigation

Input
↓
Unsafe Copy
↓
Buffer Overflow
↓
Memory Corruption
↓
Crash

Here we have found the root cause

From the Reverse Engineer's perspective:

When you have a Crash, keep asking yourself these questions

Where did the CPU crash

Which instruction was problematic

Which register or address was involved

Where did this value come from

What data caused this situation

Where was the first place where the wrong behavior occurred

The last question is the most important

Because the Crash may have occurred at one point but the Bug may have occurred several functions earlier

When We found a crash
We don't say we found the cause of the bug
First we find the location of the crash
Then we examine the problematic instruction, then we look at the registers and memory, then we follow the data path back to where the problem actually occurred
That is

Crash
↓
Where
↓
How
↓
Why
↓
Root Cause

And this is exactly the difference between seeing a crash and understanding a bug

@reverseengine

Part 31 Buffer Overflow

Root Cause Analysis

Finding the Real Cause of a Crash

In the previous part, we learned how to classify crashes when a Fuzzer finds them. Now we want to go a step deeper. This time, we don't just want to know where the program crashed.

We want to understand.

Why it crashed.

Because saying.

Program crashed.

doesn't really help.

It's almost like a car broke down and the mechanic just said.

The car broke down.

Well, thanks for this great discovery.

Crash.

itself is not a bug.

Suppose the debugger shows this.

SIGSEGV.

This means that the program has reached an invalid memory access, but we still don't know why it happened.

It could be one of these.

Buffer Overflow.

Use After Free.

Out of Bounds Read.

Null Pointer Dereference.

So we have to go after the crash itself. Let's go

Crash
↓
What happened
↓
Why happened
↓
Root Cause

Find the problematic instruction

Suppose the Debugger shows this instruction

mov eax,[rcx]

and the program crashes right here

Here the first question is

What is the value of RCX

For example

RCX = 0x00000000000000000

Well here the program is actually reading something from address zero, so the path could be like this

mov eax,[rcx]
↓
RCX = 0
↓
Invalid Memory Access
↓
Crash

But we haven't found the root cause yet

We need to understand
Why RCX is zero

Checking the Registers
After the Crash, we check the status of the Registers

For example:
RAX = 0x000000001
RBX = 0x00000000
RCX = 0x00000000
RDX = 0x00000020
RIP = 0x401234

Here RIP tells us where the CPU was approximately at the time of the Crash
And RCX can also give us important information about the address that was supposed to be used
But the main question is still the same

RCX
↓
Where did it come from

Let's go back
Suppose we have these instructions before the Crash

mov rcx,[rbp-20h]

mov eax,[rcx]

So RCX got its value from here

mov rcx,[rbp-20h]

Now we need to go back one more step
And see

[rbp-20h]

How did it get its value
For example, we may have had something like this before

mov [rbp-20h], rdi

Now we ask
Where did RDI come from
And we go back like this

Crash
↓
RCX
↓
[rbp-20h]
↓
RDI
↓
Function Argument
↓
Input

Here we are getting closer to the source of the problem

General Analysis Path
This is the path we work with a lot during Root Cause Analysis

Crash
↓
Faulting Instruction
↓
Register / Memory
↓
Previous Instruction
↓
Data Origin
↓
Root Cause

That is, we do not just look at the last instruction
We look at how this situation arose

This is where Reverse Engineering really shows itself

A simple example:

Suppose the code is

C
void process(char *input)
{
char buffer[8];

strcpy(buffer, input);

printf("%s", buffer);

}

Now suppose the input is too large

input
↓
strcpy
↓
buffer
↓
Stack Corruption
↓
Crash

The program may crash later, for example, inside printf

If we just say

Crash in printf

Our analysis is incomplete

Because the main problem may have occurred much earlier

Where is the Root Cause

Here

buffer = 8 bytes

input = larger than 8 bytes

strcpy
↓
No bounds checking
↓
Stack Buffer Overflow

That is, strcpy wrote more data than the buffer capacity

and caused memory corruption

So

Crash
≠
Root Cause

Crash
is the final result

Root Cause

Where the problem actually occurred

Another example

Suppose the debugger shows

mov rax,[rbx]

and

RBX = 0

We may say

Null Pointer Dereference

But it is still better to stop here Let's sit

We need to check

RBX
↓
How did it get its value
↓
Why did it become NULL
↓
Should it have been checked before
↓
Root Cause

We may find that a function returned a NULL value but the program used it without checking it, so mov rax,[rbx] itself is only where the problem showed itself, not the problem itself

Why is this step important for Fuzzing

Suppose the Fuzzer found this

Crash found

input_0421.dat

So what now
We still don't know what we are dealing with
It may be:

Crash
↓
Heap Buffer Overflow

or

Crash
↓
Use After Free

or

Crash
↓
Out of Bounds Read

or

Crash
↓
Null Pointer Dereference

این دو تا رو همیشه جدا نگه دارید

Crash
↓
چه اتفاقی افتاد

ولی

Root Cause
↓
چرا این اتفاق افتاد

مثلا:

Crash
↓
SIGSEGV
↓
Invalid Memory Access

هنوز Root Cause مشخص نیست
ممکنه بعد از بررسی بفهمیم

Input
↓
Unsafe Copy
↓
Buffer Overflow
↓
Memory Corruption
↓
Crash

اینجا دیگه علت اصلی رو پیدا کردیم

از دید Reverse Engineer:

وقتی یک Crash دارید مدام این سوال‌ها رو از خودتون بپرسید

CPU
کجا Crash کرد

کدوم Instruction مشکل‌دار بود

کدوم Register یا Address درگیر بود

این مقدار از کجا اومده

چه داده‌ای باعث رسیدن به این وضعیت شده

اولین جایی که رفتار اشتباه ایجاد شده کجاست

آخرین سؤال از همه مهم‌تره

چون ممکنه Crash در یک نقطه اتفاق بیفته ولی Bug چند تابع قبل‌تر ایجاد شده باشه


وقتی یک Crash پیدا کردیم
مستقیم نمیگیم علت باگ رو فهمیدیم
اول محل Crash رو پیدا میکنیم
بعد Instruction مشکل‌دار رو بررسی میکنیم بعد Register ها و Memory رو نگاه میکنیم بعد مسیر داده رو به عقب دنبال میکنیم تا برسیم به جایی که مشکل واقعا ایجاد شده
یعنی

Crash
↓
Where
↓
How
↓
Why
↓
Root Cause

و این دقیقا تفاوت بین دیدن Crash و فهمیدن باگ هست

@reverseengine

بخش سی و یکم بافر اور فلو

Root Cause Analysis

پیدا کردن علت واقعی Crash


توی قسمت قبل یاد گرفتیم وقتی Fuzzer کلی Crash پیدا می‌کنه چطور اون‌ها رو دسته‌بندی کنیم حالا میخوایم یک مرحله عمیق‌تر بشیم این بار فقط نمی‌خوایم بدونیم برنامه کجا Crash کرده

می‌خوایم بفهمیم
چرا Crash کرده
چون اینکه بگیم

Program crashed

خیلی کمک خاصی نمیکنه

تقریبا مثل اینه که ماشین خراب شده باشه و مکانیک فقط بگه
ماشین خراب شد
خب ممنون از این کشف بزرگ😂

Crash
خودش باگ نیست
فرض کنید Debugger اینو نشون بده

SIGSEGV

این یعنی برنامه به یک دسترسی نامعتبر به حافظه رسیده ولی هنوز نمیدونیم چرا این اتفاق افتاده
ممکنه علتش یکی از این‌ها باشه

Buffer Overflow

Use After Free

Out of Bounds Read

Null Pointer Dereference

پس باید از خود Crash عقب‌ تر بریم

Crash
↓
What happened
↓
Why happened
↓
Root Cause



پیدا کردن دستور مشکل دار

فرض کنید Debugger این دستور رو نشون بده

mov eax,[rcx]

و دقیقاً همینجا برنامه Crash کنه
اینجا اولین سوال اینه

RCX چه مقداری داشته
مثلا

RCX = 0x0000000000000000

خب اینجا برنامه عملا داره از آدرس صفر چیزی بخونه پس مسیر میتونه این شکلی باشه

mov eax,[rcx]
↓
RCX = 0
↓
Invalid Memory Access
↓
Crash

ولی هنوز علت اصلی رو پیدا نکردیم
باید بفهمیم
RCX چرا صفر شده



بررسی Register ها
بعد از Crash وضعیت Registerها رو بررسی می‌کنیم

مثلا:
RAX = 0x00000001
RBX = 0x00000000
RCX = 0x00000000
RDX = 0x00000020
RIP = 0x401234

اینجا RIP به ما میگه CPU موقع Crash تقریبا کجا بوده
و RCX هم میتونه اطلاعات مهمی درباره آدرسی که قرار بوده استفاده بشه بده
ولی سوال اصلی هنوز همونه

RCX
↓
از کجا اومده


برگردیم عقب
فرض کنید قبل از Crash این دستورات رو داریم

mov rcx,[rbp-20h]
mov eax,[rcx]

پس RCX از اینجا مقدار گرفته

mov rcx,[rbp-20h]

حالا باید یک مرحله دیگه عقب بریم
و ببینیم

[rbp-20h]

چطور مقدار گرفته
مثلا ممکنه قبل‌تر چیزی شبیه این داشته باشیم

mov [rbp-20h], rdi

حالا میپرسیم
RDI از کجا اومده
و همینطور عقب میریم

Crash
↓
RCX
↓
[rbp-20h]
↓
RDI
↓
Function Argument
↓
Input

اینجا کم‌کم داریم به منشا مشکل نزدیک میشیم

مسیر کلی تحلیل
این مسیریه که موقع Root Cause Analysis زیاد باهاش کار میکنیم

Crash
↓
Faulting Instruction
↓
Register / Memory
↓
Previous Instruction
↓
Data Origin
↓
Root Cause

یعنی فقط به آخرین دستور نگاه نمیکنیم
دنبال این میگردیم که این وضعیت چطور به وجود اومده

اینجاست که Reverse Engineering واقعا خودش رو نشون میده

یک مثال ساده:

فرض کنید کد اینه
C
void process(char *input)
{
char buffer[8];

strcpy(buffer, input);

printf("%s", buffer);
}

حالا فرض کنید ورودی خیلی بزرگ باشه

input
↓
strcpy
↓
buffer
↓
Stack Corruption
↓
Crash

ممکنه برنامه کمی بعدتر مثلاً داخل printf Crash کنه
اگر فقط بگیم

Crash در printf

تحلیل‌مون ناقصه
چون ممکنه مشکل اصلی خیلی قبل‌تر ایجاد شده باشه

Root Cause کجاست
اینجا

buffer = 8 bytes

input = larger than 8 bytes

strcpy
↓
No bounds checking
↓
Stack Buffer Overflow

یعنی strcpy بیشتر از ظرفیت buffer داده نوشته
و باعث خراب شدن حافظه شده
پس

Crash
≠
Root Cause

Crash
نتیجه نهایی اتفاقه
Root Cause
جاییه که مشکل واقعا ایجاد شده

یک مثال دیگه
فرض کنید Debugger اینو نشون بده

mov rax,[rbx]

و

RBX = 0

ممکنه بگیم

Null Pointer Dereference

ولی باز هم بهتره همینجا متوقف نشیم

باید بررسی کنیم

RBX
↓
چطور مقدار گرفته
↓
چرا NULL شده
↓
آیا باید قبلش بررسی می‌شده
↓
Root Cause

ممکنه متوجه بشیم یک تابع مقدار NULL برگردونده ولی برنامه بدون بررسی اون رو استفاده کرده پس خود mov rax,[rbx] فقط جاییه که مشکل خودش رو نشون داده نه لزواما جایی که مشکل ایجاد شده

چرا این مرحله برای Fuzzing مهمه

فرض کنید Fuzzer اینو پیدا کرده

Crash found

input_0421.dat

خب حالا چی
هنوز نمی‌دونیم با چه چیزی طرفیم
ممکنه:

Crash
↓
Heap Buffer Overflow

یا

Crash
↓
Use After Free

یا

Crash
↓
Out of Bounds Read

یا

Crash
↓
Null Pointer Dereference

برای همین بعد از پیدا شدن Crash باید وارد مرحله تحلیل بشیم

Fuzzer
↓
Crash
↓
Debugger
↓
Root Cause Analysis
↓
Bug Classification


Root Cause و Crash رو قاطی نکنید

C
int verify(int input)
{
int a = input ^ 0x31;

int junk = 900;
junk *= 5;

int b = a + 20;

int temp = 100;
temp ^= 55;

int result = b * 3;

return result == 300;
}

Assume only input is tainted.
Now, determine the taint path:

input
↓
?
↓
?
↓
?
↓
comparison

And identify which variables should not be tainted at all.

@reverseengine

Taint Analysis

Tracking data throughout the program
So far, we have learned how to trace data paths using Forward Slicing and Backward Slicing.

Now, let's take it a step further.

Taint Analysis
This involves marking a specific piece of data and then observing where it travels within the program and what it affects.

Suppose the program receives an input:

C
int input = get_input();

We consider input to be "Tainted"—
meaning, for the time being, we assume:

input = TAINTED

Now, if this value enters a calculation:

int x = input + 10;

Then:

input
↓
x

x is also considered Tainted.
If we subsequently have:

C
int y = x * 5;

The path becomes:

input
↓
x
↓
y

Thus, y is also influenced by the initial input.

Now, if we have:

C
int result = y ^ 0x55;

The path becomes:

input
↓
x
↓
y
↓
result

And if, finally, we have:

if (result == 100)

We realize that the initial input has influenced this condition.

This is one of the key applications of Taint Analysis.

A simple example:

C
int check(int input)
{
int x = input ^ 0x55;
int y = x + 10;

int temp = 500;
temp *= 20;

if (y == 100)
return 1;

return 0; }

We mark the input as "Tainted."
So, its path becomes:

input
↓
x
↓
y
↓
comparison

However, this part:

temp
↓
temp * 20

...is not connected to the final condition.
Therefore, the taint is not propagated to it;
meaning temp remains "Clean."

Taint in Assembly:
When we delve into reverse engineering, we usually don't have a variable explicitly named input.
We might see something like this:

mov eax, [rbp-20h]
xor eax, 55h
add eax, 10
cmp eax, 100

Let's assume:
[rbp-20h] = input

So, the data path becomes:

[rbp-20h]
↓
EAX
↓
XOR 55h
↓
ADD 10
↓
CMP 100

Here, we can see that the input enters the calculations directly and eventually reaches the comparison.
Taint analysis isn't limited to user input;
any data can serve as a taint starting point.
For example:

PowerShell
File Data
Network Data
Configuration
Function Argument
Registry Value
Environment Variable

For instance, if a function takes this:
C
void process(char *data)

We can designate data as the starting point
and then trace which functions this data enters as the program proceeds.
The path might look like this:

Input
↓
Parser
↓
Transformation
↓
Function A
↓
Function B
↓
Comparison

Or even:

Input
↓
Buffer
↓
Memory
↓
Function
↓
Output

An important point:
Taint analysis
does not necessarily imply that the exact initial value remains unchanged;
the value might be modified multiple times.

For example:

Input
↓
XOR
↓
ADD
↓
SHIFT
↓
SUB
↓
Comparison

What we track is whether the new value still depends on the initial data.
In other words, the core question is:

Is this value influenced by the tainted data? ...is situated...

It is not the case that...

Is the value exactly the same as the initial value?

We call the transfer of a "taint" from one value to another "Taint Propagation."
For example:

a = input;
b = a + 5;
c = b ^ 0x44;

The path looks like this:

input
↓
a
↓
b
↓
c

This means the taint moves along with the data.
However, if we have:

C++
int x = 100;
int y = x + 20;

Since x did not come from input:

x = CLEAN
y = CLEAN

So, the taint does not enter this path.
Now, for a more realistic example: suppose we have a complex function:

Function
│
├── Input
├── Calculation A
├── Junk Code
├── Function B
├── Calculation C
├── Comparison
└── Return

If we mark the input as "tainted," analysis might lead us to this conclusion:

Input
↓
Calculation A
↓
Function B
↓
Calculation C
↓
Comparison

...and the "Junk Code" section might not be part of the taint path at all.
This is precisely where we see why Taint Analysis is so powerful when combined with Slicing.

Slicing tells us:

What elements are related to this specific point?

Taint Analysis tells us:

Where this specific piece of data went and what it affected.

The two concepts are closely related but not exactly the same.

Exercise:

Examine this function:

Taint Analysis

دنبال کردن یک داده در کل برنامه
تا اینجا با Forward Slicing و Backward Slicing یاد گرفتیم چطور مسیر داده رو پیدا کنیم

حالا یه قدم جلوتر میریم

Taint Analysis
یعنی یه داده مشخص رو علامت گذاری کنیم و بعد ببینیم این داده توی ادامه برنامه کجاها میره و روی چه چیزهایی تاثیر میذاره

فرض کنید برنامه یه ورودی میگیره:

C

int input = get_input();

ما input رو Tainted در نظر میگیریم
یعنی فعلا فرض میکنیم

input = TAINTED

حالا اگه این مقدار وارد یه محاسبه بشه

int x = input + 10;

پس

input
↓
x

x هم Tainted حساب میشه
اگه بعدا داشته باشیم
C
int y = x * 5;

مسیر میشه

input
↓
x
↓
y

پس y هم تحت تاثیر ورودی اولیه قرار گرفته

حالا اگه داشته باشیم
C
int result = y ^ 0x55;

مسیر میشه

input
↓
x
↓
y
↓
result

و اگه آخرش داشته باشیم

if (result == 100)

میفهمیم ورودی اولیه روی این شرط تاثیر گذاشته

این یکی از کاربردهای مهم Taint Analysis هست

یک مثال ساده:

C
int check(int input)
{
int x = input ^ 0x55;
int y = x + 10;

int temp = 500;
temp *= 20;

if (y == 100)
return 1;

return 0;
}

ما input رو Tainted میکنیم
پس مسیرش میشه

input
↓
x
↓
y
↓
comparison

ولی این قسمت

temp
↓
temp * 20

به شرط نهایی وصل نیست
پس Taint هم بهش منتقل نمیشه
یعنی temp همچنان Clean هست

Taint
توی Assembly
وقتی وارد Reverse Engineering میشیم معمولا دیگه متغیری به اسم input نداریم
ممکنه چیزی شبیه این ببینیم

mov eax, [rbp-20h]
xor eax, 55h
add eax, 10
cmp eax, 100

فرض کنیم
json
[rbp-20h] = ورودی

پس مسیر داده میشه

[rbp-20h]
↓
EAX
↓
XOR 55h
↓
ADD 10
↓
CMP 100

اینجا میتونیم ببینیم ورودی مستقیم وارد محاسبات شده و در نهایت به مقایسه رسیده
Taint
فقط برای ورودی کاربر نیست
هر داده ای میتونه نقطه شروع Taint باشه
مثلا:

Powershell
File Data
Network Data
Configuration
Function Argument
Registry Value
Environment Variable

مثلا اگه یه تابع اینو بگیره
C
void process(char *data)

میتونیم data رو نقطه شروع قرار بدیم
بعد دنبال کنیم که این داده توی ادامه برنامه وارد چه توابعی میشه
ممکنه مسیرش این شکلی باشه

Input
↓
Parser
↓
Transformation
↓
Function A
↓
Function B
↓
Comparison

یا حتی

Input
↓
Buffer
↓
Memory
↓
Function
↓
Output

یه نکته مهم
Taint Analysis
یعنی لزوما مقدار دقیق اولیه همونطوری باقی مونده نه
ممکنه مقدار چند بار تغییر کنه

مثلا:


Input
↓
XOR
↓
ADD
↓
SHIFT
↓
SUB
↓
Comparison

چیزی که دنبال میکنیم اینه که ببینیم مقدار جدید هنوز به داده اولیه وابسته هست یا نه
یعنی سؤال اصلی اینه

آیا این مقدار تحت تاثیر داده Tainted قرار گرفته

نه اینکه

آیا مقدار دقیقا همون مقدار اولیه عسا


به انتقال Taint از یه مقدار به مقدار دیگه میگیم
Taint Propagation
مثلا

a = input;
b = a + 5;
c = b ^ 0x44;

مسیرش میشه

input
↓
a
↓
b
↓
c

یعنی Taint همراه داده حرکت میکنه
ولی اگه داشته باشیم
C++
int x = 100;
int y = x + 20;

چون x از Input نیومده

x = CLEAN
y = CLEAN

پس Taint وارد این مسیر نمیشه
حالا یه مثال واقعی تر
فرض کنید یه تابع پیچیده داریم

Function
│
├── Input
├── Calculation A
├── Junk Code
├── Function B
├── Calculation C
├── Comparison
└── Return

اگه ورودی رو Tainted کنیم ممکنه بعد از تحلیل به این نتیجه برسیم

Input
↓
Calculation A
↓
Function B
↓
Calculation C
↓
Comparison

و بخش Junk Code اصلاً توی مسیر Taint نباشه
اینجا دقیقا میبینیم چرا Taint Analysis در کنار Slicing خیلی قدرتمنده

Slicing میگه

چه چیزهایی به این نقطه مربوط میشن

Taint Analysis میگه

این داده مشخص کجاها رفته و روی چه چیزهایی تاثیر گذاشته

این دوتا خیلی به هم نزدیکن ولی دقیقا یکی نیستن

تمرین:

این تابع رو بررسی کنید
C
int verify(int input)
{
int a = input ^ 0x31;

int junk = 900;
junk *= 5;

int b = a + 20;

int temp = 100;
temp ^= 55;

int result = b * 3;

return result == 300;
}

فرض کنید فقط input رو Tainted کردیم
حالا مسیر Taint رو مشخص کنید

input
↓
?
↓
?
↓
?
↓
comparison

و مشخص کنید کدوم متغیرها اصلا نباید Tainted بشن

@reverseengine

Hardware
│
▼
Interrupt
│
▼
CPU
│
▼
Kernel

For example, an I O operation is completed
Hardware can notify the CPU with an Interrupt that the operation is completed
Then the Kernel can manage this event

The difference between these three concepts:

System Call

A program request to get a service from the operating system

Exception

An event that occurs as a result of executing a command or condition related to the CPU

Interrupt

Declaration of an event that usually comes from hardware

Trap

Is also an architectural and operating system term that is used to transfer control to a Handler. Depending on the source you read, the exact definition of this term may vary slightly, so it's better not to just memorize one definition. The important thing is to understand what causes the CPU to change the execution path and enter the corresponding Handler

Why is this important for Reverse Engineering

Here the topic becomes more interesting for us
When you are reversing a binary

You may see instructions like
syscall
or behaviors related to Exception and Page Fault
If you have only memorized the assembly you will see
syscall
But if you understand the OS you will understand what happened behind this instruction

User Code
↓
System Call
↓
CPU / Kernel Entry
↓
Kernel Handler
↓
Operation
↓
Return

That is, there is a big difference between just seeing an instruction and understanding what happens behind that instruction in the operating system

These are the basic concepts of many things that we will deal with later in OS and Reverse Engineering

@reverseengine

Trap

and Entering from User Mode to Kernel Mode

So far we have understood that programs run in User Mode and when they ask the operating system for a sensitive task, they use System Call

But we have an important question

How exactly does the CPU go from normal program execution in User Mode to Kernel Mode

Here the concept of Trap comes into play

Trap

means a controlled transition from normal program execution to Kernel, meaning the program is running in User Mode

User Mode
│
│ Service Request
▼
Trap
│
▼
Kernel Mode

After the Kernel has done its job, control returns to the program

What is the difference between System Call and Trap

These two are not the same

System Call

Actually, the program requests the operating system

For example, the program says I want to read from a file

or write something

or get memory

For example:

read()
write()
mmap()

But Trap is a mechanism that causes the CPU to transfer execution control to the Kernel

So very simply
Program
│
│ System Call
▼
Trap / System call entry
│
▼
Kernel

That is, the System Call tells what service I want

Trap

or the entry path causes the program to enter the Kernel

Example x86 64:

In Linux on the x86 64 architecture, a System Call is usually executed with the
syscall

command

Before executing this command, the program places the necessary information into specific registers

Concept:

Registers
│
├── System Call Number

├── Argument 1
├── Argument 2
└──
│
▼
syscall
│
▼
Kernel

Kernel

When it enters, it checks the System Call number to find out exactly what the program requested. For example, the program may request to read information from a file. So the Kernel needs to know which System Call this request is related to.

How does the Kernel know what System Call was requested? Here, one of the important things is the System Call Number. Each System Call in Linux has a specific number. Of course, these numbers depend on the architecture and ABI. There are
That is, we should not think that the number of a System Call is the same on all systems

Conceptually:

System Call Number
+
Arguments
│
▼
syscall
│
▼
Kernel


Kernel

Using this information, it finds the path to that request

Kernel

What does it do when it enters
When the CPU enters the Kernel, the Kernel must check the program request

Simply
User Code
│
▼
syscall
│
▼
Kernel Entry
│
▼
Check the request
│
▼
Execution of the operation
│
▼
Return Value
│
▼
User Mode

Kernel

It may check things like access permissions and the validity of some information

For example, a User Space program should not be able to manipulate any memory address it wants without control
The operating system plays the role of a watchdog here because if any program could do anything with system resources, the operating system would practically become useless

Trap

Is not just for System Calls
Here is an important point We have

When we say Trap, we should not think that it is only used for System Call
In the operating system and CPU architecture, control transfer to the Kernel can occur for various reasons

For example:

System Call

Exception

Some hardware events via Interrupt, of course, these concepts are not exactly the same and should be examined separately

What is Exception

Exception

It happens when the CPU encounters a special condition while executing an instruction

For example:

Program
│
▼
Execution of instruction
│
▼
Special conditions
│
▼
Exception
│
▼
Kernel Handler

One of its famous examples is

Page Fault

Suppose the program accesses a memory page, but that page does not have the necessary conditions for access at that moment

The CPU
creates a Page Fault

Then the Kernel enters the story and checks what should be done

An important point:

Page Fault

does not necessarily mean Crash

The operating system may be able to handle it completely normally

For example, it can prepare the desired page and the program can continue again Give

Interrupt

Usually generated by hardware or hardware controller, for example, a device wants to notify the CPU of an event

Simply:

Interrupt
اعلام یه رویداد که معمولا از طرف سخت افزار میاد
Trap
هم یه اصطلاح معماری و سیستم عاملیه که برای انتقال کنترل به یه Handler استفاده میشه بسته به منبعی که میخونید ممکنه تعریف دقیق این اصطلاح کمی فرق داشته باشه پس بهتره فقط یه تعریف حفظ نکنید مهم اینه بفهمی چه اتفاقی باعث میشه CPU مسیر اجرا رو عوض کنه و وارد Handler مربوطه بشه

چرا این موضوع برای Reverse Engineering مهمه

اینجا موضوع برای ما جالب تر میشه
وقتی دارید یه باینری رو Reverse میکنید

ممکنه به دستورهایی مثل
syscall
برسی
یا رفتارهایی مربوط به Exception و Page Fault ببینید
اگر فقط اسمبلی رو حفظ کرده باشید میبینید
syscall
ولی اگر OS رو فهمیده باشید میفهمید پشت این دستور چه اتفاقی افتاده
User Code
↓
System Call
↓
CPU / Kernel Entry
↓
Kernel Handler
↓
Operation
↓
Return

یعنی فرق زیادی هست بین اینکه فقط یه دستور رو ببینید و اینکه بفهمید پشت اون دستور چه اتفاقی توی سیستم عامل میوفته

این مفاهیم پایه خیلی از چیزهاییه که بعدا توی OS و Reverse Engineering باهاشون سروکار داریم

@reverseengine