A C program, from source to exit

Follow one small C program through the compiler, linker, loader, operating system, CPU and down to the logic gates of the ALU.

Stage 2 of 19Compiler

Lexical analysis

The character stream is cut into tokens by the longest-match rule.

Lexical analysis: characters to tokens

Real lexer output

Source characters

int main(void) {
    int x = 3;
    int y = x * 2 + 1;
    return y;
}

keywordidentifierconstantoperatorseparator

Token stream (0)

#	lexeme	class
No tokens yet

Produced by: token stream from pycparser's CLexer (real, independent C lexer). pycparser 3.11. Classes (keyword, identifier, constant, operator, separator) are the textbook categories; the raw token type is pycparser's.

Step 0

The lexer sees only a stream of characters. Its job is to cut it into the smallest meaningful units, called tokens.

0 / 25

Concepts at this stage

Lexical analysis