134  PART 24: GETTING THE NEXT TOKEN  TEX82 §332

332. Getting the next token. The heart of TEX’s input mechanism is the get next procedure, which we shall develop in the next few sections of the program. Perhaps we shouldn’t actually call it the “heart,” however, because it really acts as TEX’s eyes and mouth, reading the source files and gobbling them up. And it also helps TEX to regurgitate stored token lists that are to be processed again.

The main duty of get next is to input one token and to set cur cmd and cur chr to that token’s command code and modifier. Furthermore, if the input token is a control sequence, the eqtb location of that control sequence is stored in cur cs ; otherwise cur cs is set to zero.

Underlying this simple description is a certain amount of complexity because of all the cases that need to be handled. However, the inner loop of get next is reasonably short and fast.

When get next is asked to get the next token of a line, it sets cur cmd = cur chr = cur cs = 0 in the case that no more tokens appear on that line. (There might not be any tokens at all, if the end line char has ignore as its catcode.)

333. The value of par loc is the eqtb address of ‘ ’. This quantity is needed because a blank line of input is supposed to be exactly equivalent to the appearance of ; we must set cur cs ← par loc when detecting a blank line. ⟨ Global variables 13 ⟩ +≡ par loc: pointer ; { location of ‘ ’ in eqtb } par token: halfword ; { token representing ‘ ’ }

334. ⟨ Put each of TEX’s primitives into the hash table 226 ⟩ +≡ primitive( , par end , 256); { cf. scan file name } par loc ← cur val ; par token ← cs token flag + par loc;

335. ⟨ Cases of print cmd chr for symbolic printing of primitives 227 ⟩ +≡ par end : print esc( );

336. Before getting into get next , let’s consider the subroutine that is called when an ‘ ’ control sequence has been scanned or when the end of a file has been reached. These two cases are distinguished by cur cs , which is zero at the end of a file. procedure check outer validity ;

var p: pointer ; { points to inserted token list }

q: pointer ; { auxiliary pointer } begin if scanner status ≠ normal then

begin deletions allowed ← false; ⟨ Back up an outer control sequence so that it can be reread 337 ⟩; if scanner status > skipping then ⟨ Tell the user what has run away and try to recover 338 ⟩ else begin print err ( ␣ ); print cmd chr (if test , cur if );

print ( ␣ ␣ ␣ ␣ ␣ ␣ ␣ ); print int (skip line);

      help3 ( ␣         ␣       ␣        ␣        ␣  ␣       ␣  )
      (  ␣    ␣  ␣     ␣       ␣    ␣   ␣   ␣`       ´ ␣   ␣  )
      (  ␣        ␣`    ´ ␣ ´ ␣        ␣ ␣`    ´ ␣    ␣     ␣  );

if cur cs ≠ 0 then cur cs ← 0

else help line[2] ← ␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ ; cur tok ← cs token flag + frozen fi ; ins error ; end;

deletions allowed ← true;

§337  TEX82  PART 24: GETTING THE NEXT TOKEN 135

end; end; 337. An outer control sequence that occurs in a will not be reread, since the error recovery for is not very powerful. ⟨ Back up an outer control sequence so that it can be reread 337 ⟩ ≡

if cur cs ≠ 0 then

begin if (state = token list ) ∨ (name < 1) ∨ (name > 17) then

begin p ← get avail ; info(p) ← cs token flag + cur cs ; back list (p);

{ prepare to read the control sequence again } end;

cur cmd ← spacer ; cur chr ← ␣ ; { replace it by a space }

end

This code is used in section 336.

338. ⟨ Tell the user what has run away and try to recover 338 ⟩ ≡ begin runaway ; { print a definition, argument, or preamble }

if cur cs = 0 then print err (  ␣  )
else begin cur cs ← 0; print err (  ␣       ␣        ␣  );

end; print ( ␣ ␣ ␣ ); ⟨ Print either ‘ ’ or ‘ ’ or ‘ ’ or ‘ ’, and insert

tokens that should lead to recovery 339 ⟩; print ( ␣ ␣ ); sprint cs (warning index );

help4 ( ␣       ␣   ␣    ␣         ␣ ␣`  ´ ␣       ␣  )
( ␣    ␣    ␣     ␣   ␣      ␣  ␣  ␣  )
( ´ ␣   ␣  ␣        ␣   ␣  ␣   ␣     ␣  ␣  )
(  ´ ␣      ␣    ␣`  ´ ␣  ␣`  ´ ␣   ␣   ␣   ␣    ␣  );

error ; end

This code is used in section 336.

339. The recovery procedure can’t be fully understood without knowing more about the TEX routines that should be aborted, but we can sketch the ideas here: For a runaway definition we will insert a right brace; for a runaway preamble, we will insert a special token and a right brace; and for a runaway argument, we will set long state to outer call and insert . ⟨ Print either ‘ ’ or ‘ ’ or ‘ ’ or ‘ ’, and insert tokens that should lead to

recovery 339 ⟩ ≡ p ← get avail ;

case scanner status of

defining : begin print ( ); info(p) ← right brace token + ;

end; matching : begin print ( ); info(p) ← par token; long state ← outer call ;

end; aligning : begin print ( ); info(p) ← right brace token + ; q ← p; p ← get avail ;

link (p) ← q; info(p) ← cs token flag + frozen cr ; align state ← −1000000; end;

absorbing : begin print ( ); info(p) ← right brace token + ;

end; end; { there are no other cases } ins list (p)

This code is used in section 338.

340. We need to mention a procedure here that may be called by get next . procedure firm up the line; forward ; 341. Now we’re ready to take the plunge into get next itself. Parts of this routine are executed more often than any other instructions of TEX.

define switch = 25 { a label in get next } define start cs = 26 { another }

procedure get next ; { sets cur cmd , cur chr , cur cs to next token }

label restart , { go here to get the next input token } switch, { go here to eat the next character from a file } reswitch, { go here to digest it again } start cs , { go here to start looking for a control sequence } found , { go here when a control sequence has been found } exit ; { go here when the next input token has been got }

var k: 0 . . buf size; { an index into buffer } t: halfword ; { a token } cat : 0 . . 15; { cat code(cur chr ), usually } c, cc: ASCII code; { constituents of a possible expanded code } d: 2 . . 3; { number of excess characters in an expanded code }

begin restart : cur cs ← 0; if state ≠ token list then ⟨ Input from external file, goto restart if no input found 343 ⟩ else ⟨ Input from token list, goto restart if end of list or if a parameter needs to be expanded 357 ⟩; ⟨ If an alignment entry has just ended, take appropriate action 342 ⟩;

exit : end;

342. An alignment entry ends when a tab or occurs, provided that the current level of braces is the same as the level that was present at the beginning of that alignment entry; i.e., provided that align state has returned to the value it had after the ⟨u_{j}⟩ template for that entry. ⟨ If an alignment entry has just ended, take appropriate action 342 ⟩ ≡

if cur cmd ≤ car ret then

if cur cmd ≥ tab mark then

if align state = 0 then ⟨ Insert the ⟨v_{j}⟩ template and goto restart 789 ⟩

This code is used in section 341.

343. ⟨ Input from external file, goto restart if no input found 343 ⟩ ≡

begin switch: if loc ≤ limit then { current line not yet finished }

begin cur chr ← buffer [loc]; incr (loc); reswitch: cur cmd ← cat code(cur chr ); ⟨ Change state if necessary, and goto switch if the current

character should be ignored, or goto reswitch if the current character changes to another 344 ⟩; end

else begin state ← new line;

⟨ Move to next line of file, or goto restart if there is no next line, or return if a line has

finished 360 ⟩; check interrupt ; goto switch; end;

end

This code is used in section 341.

344. The following 48-way switch accomplishes the scanning quickly, assuming that a decent Pascal compiler has translated the code. Note that the numeric values for mid line, skip blanks , and new line are spaced apart from each other by max char code + 1, so we can add a character’s command code to the state to get a single number that characterizes both.

define any state plus (#) ≡ mid line + #, skip blanks + #, new line + # ⟨ Change state if necessary, and goto switch if the current character should be ignored, or goto reswitch if

the current character changes to another 344 ⟩ ≡ case state + cur cmd of ⟨ Cases where character is ignored 345 ⟩: goto switch; any state plus (escape): ⟨ Scan a control sequence and set state ← skip blanks or mid line 354 ⟩; any state plus (active char ): ⟨ Process an active-character control sequence and set state ← mid line 353 ⟩; any state plus (sup mark ): ⟨ If this sup mark starts an expanded character like or , then goto

reswitch, otherwise set state ← mid line 352 ⟩; any state plus (invalid char ): ⟨ Decry the invalid character and goto restart 346 ⟩; ⟨ Handle situations involving spaces, braces, changes of state 347 ⟩

othercases do nothing

endcases

This code is used in section 343.

345. ⟨ Cases where character is ignored 345 ⟩ ≡ any state plus (ignore), skip blanks + spacer , new line + spacer

This code is used in section 344.

346. We go to restart instead of to switch, because state might equal token list after the error has been dealt with (cf. clear for error prompt ). ⟨ Decry the invalid character and goto restart 346 ⟩ ≡

begin print err (  ␣    ␣        ␣  ␣       ␣  );

  help2 ( ␣     ␣      ␣    ␣ ␣   ´ ␣    ␣   ␣    ␣    ␣  )
  (  ␣   ␣ ´ ␣      ␣    ␣  ␣    ␣  );

deletions allowed ← false; error ; deletions allowed ← true; goto restart ;

end

This code is used in section 344.

347. define add delims to(#) ≡ # + math shift , # + tab mark , # + mac param, # + sub mark , # + letter , # + other char

⟨ Handle situations involving spaces, braces, changes of state 347 ⟩ ≡ mid line + spacer : ⟨ Enter skip blanks state, emit a space 349 ⟩; mid line + car ret : ⟨ Finish line, emit a space 348 ⟩; skip blanks + car ret , any state plus (comment ): ⟨ Finish line, goto switch 350 ⟩; new line + car ret : ⟨ Finish line, emit a 351 ⟩; mid line + left brace: incr (align state); skip blanks + left brace, new line + left brace: begin state ← mid line; incr (align state);

end; mid line + right brace: decr (align state); skip blanks + right brace, new line + right brace: begin state ← mid line; decr (align state);

end; add delims to(skip blanks ), add delims to(new line): state ← mid line;

This code is used in section 344.

348. When a character of type spacer gets through, its character code is changed to ␣ = ´ 40 . This means that the ASCII codes for tab and space, and for the space inserted at the end of a line, will be treated alike when macro parameters are being matched. We do this since such characters are indistinguishable on most computer terminal displays. ⟨ Finish line, emit a space 348 ⟩ ≡

begin loc ← limit + 1; cur cmd ← spacer ; cur chr ← ␣ ; end

This code is used in section 347.

349. The following code is performed only when cur cmd = spacer . ⟨ Enter skip blanks state, emit a space 349 ⟩ ≡

begin state ← skip blanks ; cur chr ← ␣ ; end

This code is used in section 347.

350. ⟨ Finish line, goto switch 350 ⟩ ≡ begin loc ← limit + 1; goto switch; end

This code is used in section 347.

351. ⟨ Finish line, emit a 351 ⟩ ≡ begin loc ← limit + 1; cur cs ← par loc; cur cmd ← eq type(cur cs ); cur chr ← equiv (cur cs ); if cur cmd ≥ outer call then check outer validity ; end

This code is used in section 347.

352. Notice that a code like becomes if not followed by a hex digit.

define is hex (#) ≡ (((# ≥ ) ∧ (# ≤ )) ∨ ((# ≥ ) ∧ (# ≤ ))) define hex to cur chr ≡

if c ≤ then cur chr ← c − else cur chr ← c − + 10; if cc ≤ then cur chr ← 16 ∗ cur chr + cc − else cur chr ← 16 ∗ cur chr + cc − + 10

⟨ If this sup mark starts an expanded character like or , then goto reswitch, otherwise set

state ← mid line 352 ⟩ ≡ begin if cur chr = buffer [loc] then

if loc < limit then

begin c ← buffer [loc + 1]; if c < ´ 200 then { yes we have an expanded char } begin loc ← loc + 2; if is hex (c) then

if loc ≤ limit then

begin cc ← buffer [loc]; if is hex (cc) then begin incr (loc); hex to cur chr ; goto reswitch; end; end;

if c < ´ 100 then cur chr ← c + ´ 100 else cur chr ← c − ´ 100 ; goto reswitch; end; end;

state ← mid line; end

This code is used in section 344.

353. ⟨ Process an active-character control sequence and set state ← mid line 353 ⟩ ≡ begin cur cs ← cur chr + active base; cur cmd ← eq type(cur cs ); cur chr ← equiv (cur cs ); state ← mid line; if cur cmd ≥ outer call then check outer validity ; end

This code is used in section 344.

354. Control sequence names are scanned only when they appear in some line of a file; once they have been scanned the first time, their eqtb location serves as a unique identification, so TEX doesn’t need to refer to the original name any more except when it prints the equivalent in symbolic form.

The program that scans a control sequence has been written carefully in order to avoid the blowups that might otherwise occur if a malicious user tried something like ‘ ´ ’. The algorithm might look at buffer [limit + 1], but it never looks at buffer [limit + 2].

If expanded characters like ‘ ’ or ‘ ’ appear in or just following a control sequence name, they are converted to single characters in the buffer and the process is repeated, slowly but surely. ⟨ Scan a control sequence and set state ← skip blanks or mid line 354 ⟩ ≡

begin if loc > limit then cur cs ← null cs { state is irrelevant in this case } else begin start cs : k ← loc; cur chr ← buffer [k]; cat ← cat code(cur chr ); incr (k);

if cat = letter then state ← skip blanks

else if cat = spacer then state ← skip blanks

else state ← mid line; if (cat = letter ) ∧ (k ≤ limit ) then ⟨ Scan ahead in the buffer until finding a nonletter; if an expanded

code is encountered, reduce it and goto start cs ; otherwise if a multiletter control sequence is found, adjust cur cs and loc, and goto found 356 ⟩

else ⟨ If an expanded code is present, reduce it and goto start cs 355 ⟩;

cur cs ← single base + buffer [loc]; incr (loc);

end; found : cur cmd ← eq type(cur cs ); cur chr ← equiv (cur cs );

if cur cmd ≥ outer call then check outer validity ; end

This code is used in section 344.

355. Whenever we reach the following piece of code, we will have cur chr = buffer [k −1] and k ≤ limit +1 and cat = cat code(cur chr ). If an expanded code like or appears in buffer [(k − 1) . . (k + 1)] or buffer [(k − 1) . . (k + 2)], we will store the corresponding code in buffer [k − 1] and shift the rest of the buffer left two or three places. ⟨ If an expanded code is present, reduce it and goto start cs 355 ⟩ ≡

begin if buffer [k] = cur chr then if cat = sup mark then if k < limit then

begin c ← buffer [k + 1]; if c < ´ 200 then { yes, one is indeed present } begin d ← 2; if is hex (c) then if k + 2 ≤ limit then

begin cc ← buffer [k + 2]; if is hex (cc) then incr (d); end; if d > 2 then

begin hex to cur chr ; buffer [k − 1] ← cur chr ; end

else if c < ´ 100 then buffer [k − 1] ← c + ´ 100

else buffer [k − 1] ← c − ´ 100 ; limit ← limit − d; first ← first − d; while k ≤ limit do

begin buffer [k] ← buffer [k + d]; incr (k); end; goto start cs ; end; end;

end

This code is used in sections 354 and 356.

356. ⟨ Scan ahead in the buffer until finding a nonletter; if an expanded code is encountered, reduce it and goto start cs ; otherwise if a multiletter control sequence is found, adjust cur cs and loc, and

goto found 356 ⟩ ≡

begin repeat cur chr ← buffer [k]; cat ← cat code(cur chr ); incr (k); until (cat ≠ letter ) ∨ (k > limit ); ⟨ If an expanded code is present, reduce it and goto start cs 355 ⟩; if cat ≠ letter then decr (k); { now k points to first nonletter } if k > loc + 1 then { multiletter control sequence has been scanned }

begin cur cs ← id lookup(loc, k − loc); loc ← k; goto found ; end;

end

This code is used in section 354.

357. Let’s consider now what happens when get next is looking at a token list. ⟨ Input from token list, goto restart if end of list or if a parameter needs to be expanded 357 ⟩ ≡

if loc ≠ null then { list not exhausted } begin t ← info(loc); loc ← link (loc); { move to next } if t ≥ cs token flag then { a control sequence token }

begin cur cs ← t − cs token flag ; cur cmd ← eq type(cur cs ); cur chr ← equiv (cur cs ); if cur cmd ≥ outer call then

if cur cmd = dont expand then ⟨ Get the next token, suppressing expansion 358 ⟩ else check outer validity ;

end else begin cur cmd ← t div ´ 400 ; cur chr ← t mod ´ 400 ;

case cur cmd of

left brace: incr (align state); right brace: decr (align state); out param: ⟨ Insert macro parameter and goto restart 359 ⟩;

othercases do nothing

endcases; end;

end else begin { we are done with this token list }

end token list ; goto restart ; { resume previous level } end

This code is used in section 341.

358. The present point in the program is reached only when the expand routine has inserted a special marker into the input. In this special case, info(loc) is known to be a control sequence token, and link (loc) = null .

define no expand flag = 257 { this characterizes a special variant of relax } ⟨ Get the next token, suppressing expansion 358 ⟩ ≡

begin cur cs ← info(loc) − cs token flag ; loc ← null ; cur cmd ← eq type(cur cs ); cur chr ← equiv (cur cs ); if cur cmd > max command then

begin cur cmd ← relax ; cur chr ← no expand flag ; end;

end

This code is used in section 357.

359. ⟨ Insert macro parameter and goto restart 359 ⟩ ≡ begin begin token list (param stack [param start + cur chr − 1], parameter ); goto restart ; end

This code is used in section 357.

360. All of the easy branches of get next have now been taken care of. There is one more branch. define end line char inactive ≡ (end line char < 0) ∨ (end line char > 255)

⟨ Move to next line of file, or goto restart if there is no next line, or return if a line has

finished 360 ⟩ ≡ if name > 17 then ⟨ Read next line of file into buffer , or goto restart if the file has ended 362 ⟩ else begin if ¬terminal input then { line has ended }

begin cur cmd ← 0; cur chr ← 0; return; end;

if input ptr > 0 then { text was inserted during error recovery } begin end file reading ; goto restart ; { resume previous level } end;

if selector < log only then open log file; if interaction > nonstop mode then

begin if end line char inactive then incr (limit ); if limit = start then { previous line was empty }

print nl ( ␣ ␣ ␣ ␣ ␣ ␣` ´ ); print ln; first ← start ; prompt input ( ); { input on-line into buffer } limit ← last ; if end line char inactive then decr (limit ) else buffer [limit ] ← end line char ; first ← limit + 1; loc ← start ; end

else fatal error ( ␣ ␣ ␣ ␣ ␣ ␣ );

{ nonstop mode, which is intended for overnight batch processing, never waits for on-line input } end

This code is used in section 343.

361. The global variable force eof is normally false; it is set true by an command. ⟨ Global variables 13 ⟩ +≡ force eof : boolean; { should the next be aborted early? }

362. ⟨ Read next line of file into buffer , or goto restart if the file has ended 362 ⟩ ≡ begin incr (line); first ← start ; if ¬force eof then

begin if input ln(cur file, true) then { not end of file }

firm up the line { this sets limit } else force eof ← true; end;

if force eof then

begin print char ( ); decr (open parens ); update terminal ; { show user that file has been read } force eof ← false; end file reading ; { resume previous level } check outer validity ; goto restart ; end;

if end line char inactive then decr (limit ) else buffer [limit ] ← end line char ; first ← limit + 1; loc ← start ; { ready to read } end

This code is used in section 360.

363. If the user has set the pausing parameter to some positive value, and if nonstop mode has not been selected, each line of input is displayed on the terminal and the transcript file, followed by ‘ ’. TEX waits for a response. If the response is simply carriage return, the line is accepted as it stands, otherwise the line typed is used instead of the line in the file. procedure firm up the line;

var k: 0 . . buf size; { an index into buffer }

begin limit ← last ;

if pausing > 0 then

if interaction > nonstop mode then begin wake up terminal ; print ln; if start < limit then

for k ← start to limit − 1 do print (buffer [k]); first ← limit ; prompt input ( ); { wait for user response } if last > first then

begin for k ← first to last − 1 do { move line down in buffer }

buffer [k + start − first ] ← buffer [k]; limit ← start + last − first ; end; end;

end;

364. Since get next is used so frequently in TEX, it is convenient to define three related procedures that do a little more: get token not only sets cur cmd and cur chr , it also sets cur tok , a packed halfword version of the current

token. get x token, meaning “get an expanded token,” is like get token, but if the current token turns out to be

a user-defined control sequence (i.e., a macro call), or a conditional, or something like or or , it is eliminated from the input by beginning the expansion of the macro or the evaluation of the conditional.

x token is like get x token except that it assumes that get next has already been called. In fact, these three procedures account for all uses of get next , except for two places in the “inner loop” when cur tok need not be set, and except when the arguments to are being scanned.

365. No new control sequences will be defined except during a call of get token, or when compresses a token list, because no new control sequence is always true at other times. procedure get token; { sets cur cmd , cur chr , cur tok }

begin no new control sequence ← false; get next ; no new control sequence ← true; if cur cs = 0 then cur tok ← (cur cmd ∗ ´ 400 ) + cur chr else cur tok ← cs token flag + cur cs ; end;