System calls and kernel mode

Why it exists

An operating system must protect itself and the other programs from any one program. A user program therefore may not touch hardware, other programs' memory or the kernel's data directly. But programs need services that only the kernel can perform: open a file, create a process, send data over a network. A system call is the controlled doorway: the program asks, the processor switches into a privileged state at a fixed entry point of the kernel, the kernel checks the request and performs it, and control returns to the program. GATE asks which instructions are privileged, how a system call differs from a function call and from an interrupt, what happens in what order, and which library calls actually use system calls.

Prerequisites

A system call enters the kernel through the same mechanism as an interrupt: the processor saves the return address, changes mode and jumps to an address from a vector.

Precise definitions

Kernel. The part of the operating system that runs with full privileges, always in memory: process and memory management, file systems, device drivers, interrupt handling.

Dual-mode operation. The hardware has (at least) two modes, shown by a mode bit: user mode and kernel mode (also called supervisor, system or monitor mode). User programs run in user mode, the kernel in kernel mode.

Privileged instructions. Instructions that are legal only in kernel mode; executing one in user mode causes a trap (an exception) to the kernel, which usually terminates the program. Typical examples:

• instructions that perform I/O or access device registers (in isolated I/O),
• halt the processor,
• change the mode bit (to kernel),
• enable or disable interrupts,
• set the timer, load the page table base register or other memory-management registers,
• load the base and limit registers,
• clear memory protection information.

Instructions such as add, load from the program's own memory and the trap instruction itself are not privileged.

System call. A request, made by a user program, for a service of the kernel, performed by executing a special instruction (trap, syscall, svc, int) that deliberately causes a software interrupt. The steps:

1. The program places the system call number and the arguments in agreed places (registers, or the stack, or a memory block whose address is in a register).
2. It executes the trap instruction. The hardware pushes the return address and status on the kernel stack, sets the mode bit to kernel, and loads the PC from a trap vector: the program cannot choose where the kernel starts.
3. The kernel's handler validates the request: a valid call number, and any pointer arguments pointing into the caller's own address space. It then uses the number to index the system call table, a table of kernel routine addresses, and runs the routine.
4. The result (or an error code) is placed where the program will find it.
5. A return-from-trap instruction restores the PC and the status and switches back to user mode.

Library wrapper. In C a program calls an ordinary function such as write(). It is a small routine in the C library that loads the number and arguments and executes the trap. Higher-level functions such as printf build the text in a user-space buffer and call write only when the buffer is flushed. Functions such as strlen or sqrt need no kernel service and make no system call at all.

Kinds of system calls. Process control (fork, exec, exit, wait), file management (open, read, write, close), device management, information maintenance (getpid, time), communication (pipe, sockets, shared memory) and protection.

Mode switch versus context switch. A system call changes the mode (user to kernel and back) within the same process; the kernel code runs on behalf of the calling process. A context switch replaces the running process by another one. A mode switch is much cheaper; a context switch may follow a system call only if the call blocks or the scheduler chooses another process (context switching).

Protection beyond the mode bit. Dual mode is the base of three protections: of the CPU (a timer interrupt returns control to the kernel so that no program can keep the CPU forever), of memory (base and limit registers or page tables confine a program to its own addresses) and of I/O (all I/O instructions are privileged, so all I/O goes through system calls).

How it works

The animation runs one system call, write with the number 4, on a machine with a trap instruction. Watch the MODE register: it is user until the trap, kernel while the handler runs, and user again only when the return-from-trap executes. The return address goes onto the kernel stack, the handler address comes from the trap vector, and the call number selects the routine from the table.

Worked examples

#define _POSIX_C_SOURCE 200809L
#include <errno.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
 
int main(void) {
    /* write() is a library wrapper around the write system call */
    long n = write(1, "hello\n", 6);
    printf("write returned %ld\n", n);
    errno = 0;
    long bad = write(-1, "x", 1);
    printf("write on an invalid descriptor returned %ld, errno says: %s\n", bad, strerror(errno));
    return 0;
}

hello
write returned 6
write on an invalid descriptor returned -1, errno says: Bad file descriptor

Cheat box

Traps and GATE patterns

Connections

The entry is an interrupt taken on purpose (a software interrupt). Saving the return address is the mechanism of a call, but on the kernel stack and with a mode change. The process services most often requested are those of the process model, and when a call blocks the kernel performs a context switch. Ordinary parameter passing between functions is treated in parameter passing.

PYQs

Which one or more of the following options
guarantees that a computer system will transition
from user mode to kernel mode?

Which of the following standard C library functions
will always invoke a system call when executed
from  a single-threaded process in a UNIX/Linux
operating system?

Let the time taken to switch between user and
kernel modes of execution be t1 while the time
taken to switch between two processes be t2. Which
of the following is TRUE?

Self-check

Revision card

References