174  PART 27: BUILDING TOKEN LISTS  TEX82 §464
464. Building token lists. The token lists for macros and for other things like  and  and

are produced by a procedure called scan toks .

Before we get into the details of scan toks , let’s consider a much simpler task, that of converting the current string into a token list. The str toks function does this; it classifies spaces as type spacer and everything else as type other char .

The token list created by str toks begins at link (temp head ) and ends at the value p that is returned. (If p = temp head , the list is empty.) function str toks (b : pool pointer ): pointer ; { changes the string str pool [b . . pool ptr ] to a token list }

var p: pointer ; { tail of the token list } q: pointer ; { new node being added to the token list via store new token } t: halfword ; { token being appended } k: pool pointer ; { index into str pool }

begin str room(1); p ← temp head ; link (p) ← null ; k ← b; while k < pool ptr do

begin t ← so(str pool [k]); if t = ␣ then t ← space token else t ← other token + t; fast store new token(t); incr (k); end;

pool ptr ← b; str toks ← p; end;

465. The main reason for wanting str toks is the next function, the toks , which has similar input/output characteristics.

This procedure is supposed to scan something like ‘ ’, i.e., whatever can follow ‘ ’, and it constructs a token list containing something like ‘ ’. function the toks : pointer ;

var old setting : 0 . . max selector ; { holds selector setting } p, q, r: pointer ; { used for copying a token list } b: pool pointer ; { base of temporary string }

begin get x token; scan something internal (tok val , false); if cur val level ≥ ident val then ⟨ Copy the token list 466 ⟩ else begin old setting ← selector ; selector ← new string ; b ← pool ptr ;

case cur val level of

int val : print int (cur val ); dimen val : begin print scaled (cur val ); print ( );

end; glue val : begin print spec(cur val , ); delete glue ref (cur val );

end; mu val : begin print spec(cur val , ); delete glue ref (cur val );

end; end; { there are no other cases } selector ← old setting ; the toks ← str toks (b); end;

§466  TEX82  PART 27: BUILDING TOKEN LISTS 175

end; 466. ⟨ Copy the token list 466 ⟩ ≡

begin p ← temp head ; link (p) ← null ; if cur val level = ident val then store new token(cs token flag + cur val ) else if cur val ≠ null then

begin r ← link (cur val ); { do not copy the reference count } while r ≠ null do

begin fast store new token(info(r)); r ← link (r); end; end;

the toks ← p; end

This code is used in section 465.

467. Here’s part of the expand subroutine that we are now ready to complete: procedure ins the toks ;

begin link (garbage) ← the toks ; ins list (link (temp head )); end;

468. The primitives  ,  ,  ,  ,  , and  are defined

as follows.

define number code = 0 { command code for } define roman numeral code = 1 { command code for } define string code = 2 { command code for } define meaning code = 3 { command code for } define font name code = 4 { command code for } define job name code = 5 { command code for }

⟨ Put each of TEX’s primitives into the hash table 226 ⟩ +≡ primitive( , convert , number code); primitive( , convert , roman numeral code); primitive( , convert , string code); primitive( , convert , meaning code); primitive( , convert , font name code); primitive( , convert , job name code);

469. ⟨ Cases of print cmd chr for symbolic printing of primitives 227 ⟩ +≡ convert : case chr code of

number code: print esc( ); roman numeral code: print esc( ); string code: print esc( ); meaning code: print esc( ); font name code: print esc( ); othercases print esc( ) endcases; 470. The procedure conv toks uses str toks to insert the token list for convert functions into the scanner; ‘ ’ control sequences are allowed to follow ‘ ’ and ‘ ’. procedure conv toks ;

var old setting : 0 . . max selector ; { holds selector setting } c: number code . . job name code; { desired type of conversion } save scanner status : small number ; { scanner status upon entry } b: pool pointer ; { base of temporary string }

begin c ← cur chr ; ⟨ Scan the argument for command c 471 ⟩; old setting ← selector ; selector ← new string ; b ← pool ptr ; ⟨ Print the result of command c 472 ⟩; selector ← old setting ; link (garbage) ← str toks (b); ins list (link (temp head )); end;

471. ⟨ Scan the argument for command c 471 ⟩ ≡

case c of

number code, roman numeral code: scan int ; string code, meaning code: begin save scanner status ← scanner status ; scanner status ← normal ;

get token; scanner status ← save scanner status ;

end; font name code: scan font ident ; job name code: if job name = 0 then open log file; end { there are no other cases }

This code is used in section 470.

472. ⟨ Print the result of command c 472 ⟩ ≡

case c of

number code: print int (cur val ); roman numeral code: print roman int (cur val ); string code: if cur cs ≠ 0 then sprint cs (cur cs )

else print char (cur chr ); meaning code: print meaning ; font name code: begin print (font name[cur val ]);

if font size[cur val ] ≠ font dsize[cur val ] then begin print ( ␣ ␣ ); print scaled (font size[cur val ]); print ( ); end;

end; job name code: print (job name); end { there are no other cases }

This code is used in section 470.

473. Now we can’t postpone the difficulties any longer; we must bravely tackle scan toks . This function returns a pointer to the tail of a new token list, and it also makes def ref point to the reference count at the head of that list.

There are two boolean parameters, macro def and xpand . If macro def is true, the goal is to create the token list for a macro definition; otherwise the goal is to create the token list for some other TEX primitive: , , , , , , , , or . In the latter cases a left brace must be scanned next; this left brace will not be part of the token list, nor will the matching right brace that comes at the end. If xpand is false, the token list will simply be copied from the input using get token. Otherwise all expandable tokens will be expanded until unexpandable tokens are left, except that the results of expanding ‘ ’ are not expanded further. If both macro def and xpand are true, the expansion applies only to the macro body (i.e., to the material following the first left brace character).

The value of cur cs when scan toks begins should be the eqtb address of the control sequence to display in “runaway” error messages. function scan toks (macro def , xpand : boolean): pointer ;

label found , done, done1 , done2 ; var t: halfword ; { token representing the highest parameter number }

s: halfword ; { saved token }

p: pointer ; { tail of the token list being built }

q: pointer ; { new node being added to the token list via store new token }

unbalance: halfword ; { number of unmatched left braces }

hash brace: halfword ; { possible ‘# ’ token } begin if macro def then scanner status ← defining else scanner status ← absorbing ; warning index ← cur cs ; def ref ← get avail ; token ref count (def ref ) ← null ; p ← def ref ; hash brace ← 0; t ← zero token; if macro def then ⟨ Scan and build the parameter part of the macro definition 474 ⟩ else scan left brace; { remove the compulsory left brace } ⟨ Scan and build the body of the token list; goto found when finished 477 ⟩;

found : scanner status ← normal ; if hash brace ≠ 0 then store new token(hash brace); scan toks ← p; end;

474. ⟨ Scan and build the parameter part of the macro definition 474 ⟩ ≡

begin loop

begin get token; { set cur cmd , cur chr , cur tok }

if cur tok < right brace limit then goto done1 ;

if cur cmd = mac param then ⟨ If the next character is a parameter number, make cur tok a match

token; but if it is a left brace, store ‘left brace, end match’, set hash brace, and goto done 476 ⟩; store new token(cur tok ); end;

done1 : store new token(end match token);

if cur cmd = right brace then ⟨ Express shock at the missing left brace; goto found 475 ⟩; done: end

This code is used in section 473.

475. ⟨ Express shock at the missing left brace; goto found 475 ⟩ ≡

begin print err (  ␣ ␣  ); incr (align state);

help2 (  ␣   ␣   ␣    ␣      ␣   ␣    ␣         ␣    ␣`  ´  )

(  ␣ ´ ␣     ␣  ␣         ␣  ␣`  ´  ); error ; goto found ;

end

This code is used in section 474.

476. ⟨ If the next character is a parameter number, make cur tok a match token; but if it is a left brace,

store ‘left brace, end match’, set hash brace, and goto done 476 ⟩ ≡ begin s ← match token + cur chr ; get token; if cur cmd = left brace then

begin hash brace ← cur tok ; store new token(cur tok ); store new token(end match token); goto done; end;

if t = zero token + 9 then begin print err ( ␣ ␣ ␣ ␣ ); help1 ( ´ ␣ ␣ ␣ ␣ ␣#␣ ␣ ␣ ␣ ); error ; end

else begin incr (t);

if cur tok ≠ t then

begin print err (  ␣    ␣  ␣        ␣  );

  help2 ( ´ ␣        ␣   ␣     ␣   ␣      ␣    ␣    ␣     ␣   ␣#  )

(  ␣`  ´ ␣  ␣      ␣    ␣   ␣   ␣  ); back error ;

end; cur tok ← s; end;

end

This code is used in section 474.

477. ⟨ Scan and build the body of the token list; goto found when finished 477 ⟩ ≡ unbalance ← 1; loop begin if xpand then ⟨ Expand the next part of the input 478 ⟩

else get token; if cur tok < right brace limit then

if cur cmd < right brace then incr (unbalance)

else begin decr (unbalance); if unbalance = 0 then goto found ; end

else if cur cmd = mac param then

if macro def then ⟨ Look for parameter number or ## 479 ⟩; store new token(cur tok ); end

This code is used in section 473.

478. Here we insert an entire token list created by the toks without expanding it further. ⟨ Expand the next part of the input 478 ⟩ ≡

begin loop

begin get next ; if cur cmd ≤ max command then goto done2 ; if cur cmd ≠ the then expand else begin q ← the toks ;

if link (temp head ) ≠ null then begin link (p) ← link (temp head ); p ← q; end;

end; end;

done2 : x token

end

This code is used in section 477.

479. ⟨ Look for parameter number or ## 479 ⟩ ≡ begin s ← cur tok ;

if xpand then get x token

else get token; if cur cmd ≠ mac param then

if (cur tok ≤ zero token) ∨ (cur tok > t) then begin print err ( ␣ ␣ ␣ ␣ ␣ ␣ ); sprint cs (warning index );

    help3 (  ␣     ␣  ␣    ␣##␣       ␣  ␣# ␣  )
    ( ␣     ␣ ␣ ␣   ␣         ␣         ␣        ␣   ␣  )

(  ␣   ␣       ␣   ␣ ´ ␣     ␣  ␣      ␣    ␣   ␣     ␣##  ); back error ; cur tok ← s;

end else cur tok ← out param token − + cur chr ;

end

This code is used in section 477.

480. Another way to create a token list is via the command. The sixteen files potentially usable for reading appear in the following global variables. The value of read open[n] will be closed if stream number n has not been opened or if it has been fully read; just open if an but not a has been done; and normal if it is open and ready to read the next line.

define closed = 2 { not open, or at end of file } define just open = 1 { newly opened, first line not yet read }

⟨ Global variables 13 ⟩ +≡ read file: array [0 . . 15] of alpha file; { used for } read open: array [0 . . 16] of normal . . closed ; { state of read file[n] }

481. ⟨ Set initial values of key variables 21 ⟩ +≡ for k ← 0 to 16 do read open[k] ← closed ;

482. The read toks procedure constructs a token list like that for any macro definition, and makes cur val point to it. Parameter r points to the control sequence that will receive this token list. procedure read toks (n : integer ; r : pointer );

label done; var p: pointer ; { tail of the token list }

q: pointer ; { new node being added to the token list via store new token } s: integer ; { saved value of align state } m: small number ; { stream number }

begin scanner status ← defining ; warning index ← r; def ref ← get avail ; token ref count (def ref ) ← null ; p ← def ref ; { the reference count } store new token(end match token); if (n < 0) ∨ (n > 15) then m ← 16 else m ← n; s ← align state; align state ← 1000000; { disable tab marks, etc. } repeat ⟨ Input and store tokens from the next line of the file 483 ⟩; until align state = 1000000; cur val ← def ref ; scanner status ← normal ; align state ← s; end; 483. ⟨ Input and store tokens from the next line of the file 483 ⟩ ≡

begin file reading ; name ← m + 1; if read open[m] = closed then ⟨ Input for from the terminal 484 ⟩ else if read open[m] = just open then ⟨ Input the first line of read file[m] 485 ⟩

else ⟨ Input the next line of read file[m] 486 ⟩; limit ← last ; if end line char inactive then decr (limit ) else buffer [limit ] ← end line char ; first ← limit + 1; loc ← start ; state ← new line; loop begin get token;

if cur tok = 0 then goto done; { cur cmd = cur chr = 0 will occur at the end of the line } if align state < 1000000 then { unmatched ‘ ’ aborts the line }

begin repeat get token;

until cur tok = 0;

align state ← 1000000; goto done;

end; store new token(cur tok ); end;

done: end file reading

This code is used in section 482.

484. Here we input on-line into the buffer array, prompting the user explicitly if n ≥ 0. The value of n is set negative so that additional prompts will not be given in the case of multi-line input. ⟨ Input for from the terminal 484 ⟩ ≡

if interaction > nonstop mode then if n < 0 then prompt input ( ) else begin wake up terminal ; print ln; sprint cs (r); prompt input ( ); n ← −1;

end

else fatal error (  ␣       ␣     ␣    ␣        ␣  ␣       ␣  )

This code is used in section 483.

485. The first line of a file must be treated specially, since input ln must be told not to start with get . ⟨ Input the first line of read file[m] 485 ⟩ ≡

if input ln(read file[m], false) then read open[m] ← normal else begin a close(read file[m]); read open[m] ← closed ;

end

This code is used in section 483.

486. An empty line is appended at the end of a read file. ⟨ Input the next line of read file[m] 486 ⟩ ≡

begin if ¬input ln(read file[m], true) then begin a close(read file[m]); read open[m] ← closed ; if align state ≠ 1000000 then

begin runaway ; print err (  ␣     ␣      ␣ ); print esc(  );
help1 (  ␣     ␣   ␣          ␣  ); align state ← 1000000; error ;

end; end;

end

This code is used in section 483.