System Call
How does a program talk to the Kernel?

So far we have worked with things like:
lua
fork()
exec()
wait()
exit()

But there is an important question

How can these functions request the Kernel to do something?

For example, how can a program running in User Space:

Open a file?

Read data?

Create a new process?

Get memory?

Process
terminate itself?

Original answer:
System Call

User Mode and Kernel Mode

The CPU
usually executes the program at a limited access level, which we call:
User Mode


The program in this mode cannot directly do anything it wants with the system

For example, it cannot directly manipulate:
Hardware
Kernel Memory
Page Tables
Device Control

Because if any program had such access, a malicious program could destroy the entire system😁
And people have already found enough other ways to do this

Kernel Mode

The Kernel is executed at a higher access level:

Kernel Mode

In this mode, the Kernel can access sensitive system resources
So the overall structure is approximately as follows:

User Space
│
│ System Call
↓
Kernel Space
│
↓
Hardware / Resources

The program does not enter the Kernel directly
But uses a controlled gateway, this gateway is the System Call

A simple example
Suppose the program wants to Read a file
At the program level we might have something like this:

read(fd, buffer, size);

The program requests:

I want to read a certain amount of data from this File Descriptor

At the end of the request, it should reach the Kernel

The Kernel checks:

Who is this Process?

Does it have access?

Is the fd
valid?

Is the buffer address valid?

How much data should be read?

Then it performs the operation and returns the result

General System Call Path

In simple terms:

Program
│
↓
Library / API Wrapper
│
↓
System Call Mechanism
│
↓
CPU → Kernel Mode
│
↓
Kernel
│
↓
Operation
│
↓
Return Value
│
↓
User Mode

That is, the program enters the Kernel from a specific path to perform sensitive operations

System Call is not the same as Function Call

Let's not confuse the two

A normal Function Call, for example:

result = add(a, b);

Usually it is done in the same execution space of the program

But System Call creates a crossing of the User and Kernel access boundaries

So:

Function Call
↓
Same execution space

System Call
↓
User Mode
↓
Kernel Mode
↓
Return to User Mode

This difference is very important in Reverse Engineering

Fork() and wait()
also use the same path

For example, in a C program we write:

fork();

But this does not mean a simple Function like:

C
int add(int a, int b)

In Unix-like systems this request must finally reach the Kernel so that the Kernel can create a new Process

The same idea exists for operations like:
shell

read
write
open
mmap
fork
exec
wait

Of course, one important point

What we see in the code is not necessarily the raw System Call itself, it may be a Library Wrapper that finally performs the appropriate System Call

How does System Call
reach the Kernel?

In modern architectures, the CPU has a special mechanism for this transfer. For example, in x86-64 Linux, the command:

syscall

is used to enter the System Call path. Conceptually:

User Code
↓
Arguments are prepared
↓
System Call Number
↓
syscall
↓
CPU
↓
Kernel Entry
↓
Kernel Handler

The Kernel
checks and processes the request after it has entered.

Why is it important to us?
Suppose you see this inside a Binary:

syscall

You are no longer dealing with just a regular assembly instruction

This instruction can indicate that the program passes through:

User Space
↓
Kernel Space

On the other hand, if you see APIs like:

CreateFile
ReadFile
VirtualAlloc
CreateProcess


in the program code, you should know that these APIs ultimately depend on Kernel mechanisms to perform a large number of system operations, so in RE you should be able to see this chain in your mind:

Application
↓
Library / API
↓
System Call
↓
Kernel
↓
Resource / Hardware

This perspective will come in handy later when analyzing Processes, Memory, File System, Threads, and even more complex behavior of Binary.