ABL electronic PIC Microcontrollers PIC18 User Manual

Token is the smallest element of a C program that is meaningful to the compiler.
The parser separates tokens from the input stream by creating the longest token
possible using the input characters in a left–to–right scan.

mikroC recognizes following kinds of tokens:

- keywords,
- identifiers,
- constants,
- operators,
- punctuators (also known as separators).

Here is an example of token extraction. Let’s have the following code sequence:

inter =  a+++b;

First, note that 

inter

would be parsed as a single identifier, rather than as the

keyword 

int

followed by the identifier 

er

.

The programmer who wrote the code might have intended to write

inter = a + (++b)

but it won’t work that way. The compiler would parse it as the following seven
tokens:

inter

// identifier

=

// assignment operator

a

// identifier

++

// postincrement operator

+

// addition operator

b

// identifier

;

// semicolon separator

Note that 

+++

parses as 

++

(the longest token possible) followed by 

+

.

mikroC - C Compiler for Microchip PIC microcontrollers

mikroC

making it simple...

38

MikroElektronika:  Development  tools  -  Books  -  Compilers

page