§297  TEX82  PART 21: INTRODUCTION TO THE SYNTACTIC ROUTINES 119

297. Introduction to the syntactic routines. Let’s pause a moment now and try to look at the Big Picture. The TEX program consists of three main parts: syntactic routines, semantic routines, and output routines. The chief purpose of the syntactic routines is to deliver the user’s input to the semantic routines, one token at a time. The semantic routines act as an interpreter responding to these tokens, which may be regarded as commands. And the output routines are periodically called on to convert box-and-glue lists into a compact set of instructions that will be sent to a typesetter. We have discussed the basic data structures and utility routines of TEX, so we are good and ready to plunge into the real activity by considering the syntactic routines.

Our current goal is to come to grips with the get next procedure, which is the keystone of TEX’s input mechanism. Each call of get next sets the value of three variables cur cmd , cur chr , and cur cs , representing the next input token.

cur cmd denotes a command code from the long list of codes given above; cur chr denotes a character code or other modifier of the command code; cur cs is the eqtb location of the current control sequence,

if the current token was a control sequence, otherwise it’s zero.

Underlying this external behavior of get next is all the machinery necessary to convert from character files to tokens. At a given time we may be only partially finished with the reading of several files (for which was specified), and partially finished with the expansion of some user-defined macros and/or some macro parameters, and partially finished with the generation of some text in a template for , and so on. When reading a character file, special characters must be classified as math delimiters, etc.; comments and extra blank spaces must be removed, paragraphs must be recognized, and control sequences must be found in the hash table. Furthermore there are occasions in which the scanning routines have looked ahead for a word like ‘ ’ but only part of that word was found, hence a few characters must be put back into the input and scanned again.

To handle these situations, which might all be present simultaneously, TEX uses various stacks that hold information about the incomplete activities, and there is a finite state control for each level of the input mechanism. These stacks record the current state of an implicitly recursive process, but the get next procedure is not recursive. Therefore it will not be difficult to translate these algorithms into low-level languages that do not support recursion. ⟨ Global variables 13 ⟩ +≡ cur cmd : eight bits ; { current command set by get next } cur chr : halfword ; { operand of current command } cur cs : pointer ; { control sequence found here, zero if none found } cur tok : halfword ; { packed representative of cur cmd and cur chr }

120  PART 21: INTRODUCTION TO THE SYNTACTIC ROUTINES  TEX82 §298

298. The print cmd chr routine prints a symbolic interpretation of a command code and its modifier. This is used in certain ‘ ´ ’ error messages, and in the implementation of diagnostic routines like .

The body of print cmd chr is a rather tedious listing of print commands, and most of it is essentially an inverse to the primitive routine that enters a TEX primitive into eqtb. Therefore much of this procedure appears elsewhere in the program, together with the corresponding primitive calls.

define chr cmd (#) ≡ begin print (#); print ASCII (chr code); end

⟨ Declare the procedure called print cmd chr 298 ⟩ ≡ procedure print cmd chr (cmd : quarterword ; chr code : halfword );

begin case cmd of

left brace: chr cmd ( ␣ ␣ ); right brace: chr cmd ( ␣ ␣ ); math shift : chr cmd ( ␣ ␣ ␣ ); mac param: chr cmd ( ␣ ␣ ␣ ); sup mark : chr cmd ( ␣ ␣ ); sub mark : chr cmd ( ␣ ␣ ); endv : print ( ␣ ␣ ␣ ); spacer : chr cmd ( ␣ ␣ ); letter : chr cmd ( ␣ ␣ ); other char : chr cmd ( ␣ ␣ ); ⟨ Cases of print cmd chr for symbolic printing of primitives 227 ⟩ othercases print ( ␣ ␣ ) endcases; end;

This code is used in section 252.

299. Here is a procedure that displays the current command. procedure show cur cmd chr ;

begin begin diagnostic; print nl ( ); if mode ≠ shown mode then

begin print mode(mode); print ( ␣ ); shown mode ← mode; end;

print cmd chr (cur cmd , cur chr ); print char ( ); end diagnostic(false); end;