Tokens

Tokens are the smallest meaningful pieces of text that a compiler understands. Tokenization is the very first thing that the compiler does when starting to process a source code file. There are different types of tokens, including identifiers, numbers, strings, constants, and special symbols. An Object Pascal program is made up of tokens and separators. A separator is either a blank space or a comment. There must be at least one separator between tokens.

In Object Pascal source code, you can use any Unicode characters for identifiers, but it makes a lot of sense to stick to just ASCII characters and consistently use just the English language in your code. Different Object Pascal language elements are in English, and it is easier to work with other programmers who might use other languages as their mother tongue.

The Delphi Code Editor displays different fragments of Delphi source code using different colors. Some words are displayed in bold and some others in italics. The color scheme of the Code Editor can be customized in the IDE Options, but here we will stick to the defaults.

The Object Pascal language defines a set of reserved words that the compiler knows about, and they cannot be used as identifiers in our code. Keywords are displayed in bold, so they are easy to spot. Keywords have a special meaning to the compiler. For example, if you try to save your program as program, you would get an error message that program is a reserved word and cannot be used as an identifier. If, for some reason, you really want to use a keyword as an identifier in your code, you can escape it by prepending the identifier with the & character.

Identifiers are used for constants, types, variables, procedures, functions, units, programs, and fields in records. An identifier must begin with a letter or an underscore character and cannot contain spaces. Letters, digits, and underscore characters are allowed after the first character. Identifiers and reserved words are not case sensitive. Sometimes, you may want to qualify an identifier with its unit name to avoid ambiguity.