144  PART 25: EXPANDING THE NEXT TOKEN  TEX82 §366

366. Expanding the next token. Only a dozen or so command codes > max command can possibly be returned by get next ; in increasing order, they are undefined cs , expand after , no expand , input , if test , fi or else, cs name, convert , the, top bot mark , call , long call , outer call , long outer call , and end template.

The expand subroutine is used when cur cmd > max command . It removes a “call” or a conditional or one of the other special operations just listed. It follows that expand might invoke itself recursively. In all cases, expand destroys the current token, but it sets things up so that the next get next will deliver the appropriate next token. The value of cur tok need not be known when expand is called.

Since several of the basic scanning routines communicate via global variables, their values are saved as local variables of expand so that recursive calls don’t invalidate them. ⟨ Declare the procedure called macro call 389 ⟩ ⟨ Declare the procedure called insert relax 379 ⟩ procedure pass text ; forward ; procedure start input ; forward ; procedure conditional ; forward ; procedure get x token; forward ; procedure conv toks ; forward ; procedure ins the toks ; forward ; procedure expand ;

var t: halfword ; { token that is being “expanded after” } p, q, r: pointer ; { for list manipulation } j: 0 . . buf size; { index into buffer } cv backup: integer ; { to save the global quantity cur val } cvl backup, radix backup, co backup: small number ; { to save cur val level , etc. } backup backup: pointer ; { to save link (backup head ) } save scanner status : small number ; { temporary storage of scanner status }

begin cv backup ← cur val ; cvl backup ← cur val level ; radix backup ← radix ; co backup ← cur order ; backup backup ← link (backup head ); if cur cmd < call then ⟨ Expand a nonmacro 367 ⟩ else if cur cmd < end template then macro call

else ⟨ Insert a token containing frozen endv 375 ⟩; cur val ← cv backup; cur val level ← cvl backup; radix ← radix backup; cur order ← co backup; link (backup head ) ← backup backup; end;

367. ⟨ Expand a nonmacro 367 ⟩ ≡ begin if tracing commands > 1 then show cur cmd chr ;

case cur cmd of

top bot mark : ⟨ Insert the appropriate mark text into the scanner 386 ⟩; expand after : ⟨ Expand the token after the next token 368 ⟩; no expand : ⟨ Suppress expansion of the next token 369 ⟩; cs name: ⟨ Manufacture a control sequence name 372 ⟩; convert : conv toks ; { this procedure is discussed in Part 27 below } the: ins the toks ; { this procedure is discussed in Part 27 below } if test : conditional ; { this procedure is discussed in Part 28 below } fi or else: ⟨ Terminate the current conditional and skip to 510 ⟩; input : ⟨ Initiate or terminate input from a file 378 ⟩; othercases ⟨ Complain about an undefined macro 370 ⟩ endcases; end

This code is used in section 366.

§368  TEX82  PART 25: EXPANDING THE NEXT TOKEN 145

368. It takes only a little shuffling to do what TEX calls . ⟨ Expand the token after the next token 368 ⟩ ≡

begin get token; t ← cur tok ; get token; if cur cmd > max command then expand else back input ; cur tok ← t; back input ; end

This code is used in section 367.

369. The implementation of is a bit trickier, because it is necessary to insert a special ‘dont expand ’ marker into TEX’s reading mechanism. This special marker is processed by get next , but it does not slow down the inner loop.

Since macros might arise here, we must also clear the scanner status temporarily. ⟨ Suppress expansion of the next token 369 ⟩ ≡

begin save scanner status ← scanner status ; scanner status ← normal ; get token; scanner status ← save scanner status ; t ← cur tok ; back input ;

{ now start and loc point to the backed-up token t } if t ≥ cs token flag then

begin p ← get avail ; info(p) ← cs token flag + frozen dont expand ; link (p) ← loc; start ← p; loc ← p; end;

end

This code is used in section 367.

370. ⟨ Complain about an undefined macro 370 ⟩ ≡

begin print err (  ␣       ␣  );

help5 (  ␣       ␣        ␣  ␣   ␣   ␣  ␣   ␣   ␣  )
( ␣    ␣     ␣       ␣   ␣     ␣    ´ ␣  ␣   ␣  )
(  ␣  ␣      ␣`      ´ ␣    ␣`  ´ ␣   ␣   ␣  )
(  ␣      ␣`       ´ ␣         ␣    ␣  )

(  ␣ ´ ␣      ␣     ␣        ␣   ␣  ); error ;

end

This code is used in section 367.

371. The expand procedure and some other routines that construct token lists find it convenient to use the following macros, which are valid only if the variables p and q are reserved for token-list building.

define store new token(#) ≡ begin q ← get avail ; link (p) ← q; info(q) ← #; p ← q; { link (p) is null } end

define fast store new token(#) ≡ begin fast get avail (q); link (p) ← q; info(q) ← #; p ← q; { link (p) is null } end 372. ⟨ Manufacture a control sequence name 372 ⟩ ≡

begin r ← get avail ; p ← r; { head of the list of characters } repeat get x token;

if cur cs = 0 then store new token(cur tok ); until cur cs ≠ 0; if cur cmd ≠ end cs name then ⟨ Complain about missing 373 ⟩; ⟨ Look up the characters of list r in the hash table, and set cur cs 374 ⟩; flush list (r); if eq type(cur cs ) = undefined cs then

begin eq define(cur cs , relax , 256); { N.B.: The save stack might change } end; { the control sequence will now match ‘ ’ }

cur tok ← cur cs + cs token flag ; back input ; end

This code is used in section 367.

373. ⟨ Complain about missing 373 ⟩ ≡

begin print err (  ␣ ); print esc(  ); print ( ␣  );

help2 (  ␣       ␣        ␣      ␣   ␣  ␣    ␣      ␣  )

(  ␣      ␣       ␣       ␣   ␣  ); back error ;

end

This code is used in section 372.

374. ⟨ Look up the characters of list r in the hash table, and set cur cs 374 ⟩ ≡ j ← first ; p ← link (r); while p ≠ null do

begin if j ≥ max buf stack then begin max buf stack ← j + 1; if max buf stack = buf size then overflow ( ␣ , buf size); end;

buffer [j] ← info(p) mod ´ 400 ; incr (j); p ← link (p);

end; if j > first + 1 then

begin no new control sequence ← false; cur cs ← id lookup(first , j − first ); no new control sequence ← true; end

else if j = first then cur cs ← null cs { the list is empty }

else cur cs ← single base + buffer [first ] { the list has length one }

This code is used in section 372.

375. An end template command is effectively changed to an endv command by the following code. (The reason for this is discussed below; the frozen end template at the end of the template has passed the check outer validity test, so its mission of error detection has been accomplished.) ⟨ Insert a token containing frozen endv 375 ⟩ ≡

begin cur tok ← cs token flag + frozen endv ; back input ; end

This code is used in section 366.

376. The processing of involves the start input subroutine, which will be declared later; the processing of is trivial. ⟨ Put each of TEX’s primitives into the hash table 226 ⟩ +≡

primitive( , input , 0); primitive( , input , 1);

377. ⟨ Cases of print cmd chr for symbolic printing of primitives 227 ⟩ +≡

input : if chr code = 0 then print esc(  ) else print esc(  );

378. ⟨ Initiate or terminate input from a file 378 ⟩ ≡ if cur chr > 0 then force eof ← true

else if name in progress then insert relax
else start input

This code is used in section 367.

379. Sometimes the expansion looks too far ahead, so we want to insert a harmless into the user’s input. ⟨ Declare the procedure called insert relax 379 ⟩ ≡ procedure insert relax ;

begin cur tok ← cs token flag + cur cs ; back input ; cur tok ← cs token flag + frozen relax ; back input ; token type ← inserted ; end;

This code is used in section 366.

380. Here is a recursive procedure that is TEX’s usual way to get the next token of input. It has been slightly optimized to take account of common cases. procedure get x token; { sets cur cmd , cur chr , cur tok , and expands macros }

label restart , done; begin restart : get next ; if cur cmd ≤ max command then goto done; if cur cmd ≥ call then

if cur cmd < end template then macro call

else begin cur cs ← frozen endv ; cur cmd ← endv ; goto done; { cur chr = null list }

end else expand ; goto restart ;

done: if cur cs = 0 then cur tok ← (cur cmd ∗ ´ 400 ) + cur chr else cur tok ← cs token flag + cur cs ; end;

381. The get x token procedure is equivalent to two consecutive procedure calls: get next ; x token. procedure x token; { get x token without the initial get next }

begin while cur cmd > max command do begin expand ; get next ; end;

if cur cs = 0 then cur tok ← (cur cmd ∗ ´ 400 ) + cur chr else cur tok ← cs token flag + cur cs ; end; 382. A control sequence that has been ’ed by the user is expanded by TEX’s macro call procedure.

Before we get into the details of macro call , however, let’s consider the treatment of primitives like , since they are essentially macros without parameters. The token lists for such marks are kept in a global array of five pointers; we refer to the individual entries of this array by symbolic names top mark , etc. The value of top mark is either null or a pointer to the reference count of a token list.

define top mark code = 0 { the mark in effect at the previous page break } define first mark code = 1 { the first mark between top mark and bot mark } define bot mark code = 2 { the mark in effect at the current page break } define split first mark code = 3 { the first mark found by } define split bot mark code = 4 { the last mark found by } define top mark ≡ cur mark [top mark code] define first mark ≡ cur mark [first mark code] define bot mark ≡ cur mark [bot mark code] define split first mark ≡ cur mark [split first mark code] define split bot mark ≡ cur mark [split bot mark code]

⟨ Global variables 13 ⟩ +≡ cur mark : array [top mark code . . split bot mark code] of pointer ; { token lists for marks }

383. ⟨ Set initial values of key variables 21 ⟩ +≡ top mark ← null ; first mark ← null ; bot mark ← null ; split first mark ← null ; split bot mark ← null ;

384. ⟨ Put each of TEX’s primitives into the hash table 226 ⟩ +≡ primitive( , top bot mark , top mark code); primitive( , top bot mark , first mark code); primitive( , top bot mark , bot mark code); primitive( , top bot mark , split first mark code); primitive( , top bot mark , split bot mark code);

385. ⟨ Cases of print cmd chr for symbolic printing of primitives 227 ⟩ +≡ top bot mark : case chr code of

first mark code: print esc( ); bot mark code: print esc( ); split first mark code: print esc( ); split bot mark code: print esc( ); othercases print esc( ) endcases;

386. The following code is activated when cur cmd = top bot mark and when cur chr is a code like top mark code. ⟨ Insert the appropriate mark text into the scanner 386 ⟩ ≡

begin if cur mark [cur chr ] ≠ null then begin token list (cur mark [cur chr ], mark text ); end

This code is used in section 367.

387. Now let’s consider macro call itself, which is invoked when TEX is scanning a control sequence whose cur cmd is either call , long call , outer call , or long outer call . The control sequence definition appears in the token list whose reference count is in location cur chr of mem.

The global variable long state will be set to call or to long call , depending on whether or not the control sequence disallows in its parameters. The get next routine will set long state to outer call and emit , if a file ends or if an control sequence occurs in the midst of an argument. ⟨ Global variables 13 ⟩ +≡ long state: call . . long outer call ; { governs the acceptance of }

388. The parameters, if any, must be scanned before the macro is expanded. Parameters are token lists without reference counts. They are placed on an auxiliary stack called pstack while they are being scanned, since the param stack may be losing entries during the matching process. (Note that param stack can’t be gaining entries, since macro call is the only routine that puts anything onto param stack , and it is not recursive.) ⟨ Global variables 13 ⟩ +≡ pstack : array [0 . . 8] of pointer ; { arguments supplied to a macro }

389. After parameter scanning is complete, the parameters are moved to the param stack . Then the macro body is fed to the scanner; in other words, macro call places the defined text of the control sequence at the top of TEX’s input stack, so that get next will proceed to read it next.

The global variable cur cs contains the eqtb address of the control sequence being expanded, when macro call begins. If this control sequence has not been declared , i.e., if its command code in the eq type field is not long call or long outer call , its parameters are not allowed to contain the control sequence . If an illegal appears, the macro call is aborted, and the will be rescanned. ⟨ Declare the procedure called macro call 389 ⟩ ≡ procedure macro call ; { invokes a user-defined control sequence }

label exit , continue, done, done1 , found ;

var r: pointer ; { current node in the macro’s token list } p: pointer ; { current node in parameter token list being built } q: pointer ; { new node being put into the token list } s: pointer ; { backup pointer for parameter matching } t: pointer ; { cycle pointer for backup recovery } u, v: pointer ; { auxiliary pointers for backup recovery } rbrace ptr : pointer ; { one step before the last right brace token } n: small number ; { the number of parameters scanned } unbalance: halfword ; { unmatched left braces in current parameter } m: halfword ; { the number of tokens or groups (usually) } ref count : pointer ; { start of the token list } save scanner status : small number ; { scanner status upon entry } save warning index : pointer ; { warning index upon entry } match chr : ASCII code; { character used in parameter }

begin save scanner status ← scanner status ; save warning index ← warning index ;

warning index ← cur cs ; ref count ← cur chr ; r ← link (ref count ); n ← 0;

if tracing macros > 0 then ⟨ Show the text of the macro being expanded 401 ⟩;

if info(r) ≠ end match token then ⟨ Scan the parameters and make link (r) point to the macro body;

but return if an illegal is detected 391 ⟩; ⟨ Feed the macro body and its parameters to the scanner 390 ⟩;

exit : scanner status ← save scanner status ; warning index ← save warning index ;

end;

This code is used in section 366.

390. Before we put a new token list on the input stack, it is wise to clean off all token lists that have recently been depleted. Then a user macro that ends with a call to itself will not require unbounded stack space. ⟨ Feed the macro body and its parameters to the scanner 390 ⟩ ≡

while (state = token list ) ∧ (loc = null ) do end token list ; { conserve stack space }

begin token list (ref count , macro); name ← warning index ; loc ← link (r);

if n > 0 then

begin if param ptr + n > max param stack then begin max param stack ← param ptr + n; if max param stack > param size then overflow ( ␣ ␣ , param size); end;

for m ← 0 to n − 1 do param stack [param ptr + m] ← pstack [m];

param ptr ← param ptr + n;

end

This code is used in section 389.

391. At this point, the reader will find it advisable to review the explanation of token list format that was presented earlier, since many aspects of that format are of importance chiefly in the macro call routine.

The token list might begin with a string of compulsory tokens before the first match or end match. In that case the macro name is supposed to be followed by those tokens; the following program will set s = null to represent this restriction. Otherwise s will be set to the first token of a string that will delimit the next parameter. ⟨ Scan the parameters and make link (r) point to the macro body; but return if an illegal is

detected 391 ⟩ ≡ begin scanner status ← matching ; unbalance ← 0; long state ← eq type(cur cs ); if long state ≥ outer call then long state ← long state − 2; repeat link (temp head ) ← null ;

if (info(r) > match token + 255) ∨ (info(r) < match token) then s ← null

else begin match chr ← info(r) − match token; s ← link (r); r ← s; p ← temp head ; m ← 0;

end; ⟨ Scan a parameter until its delimiter string has been found; or, if s = null , simply scan the delimiter

string 392 ⟩; { now info(r) is a token whose command code is either match or end match }

until info(r) = end match token;

end

This code is used in section 389.

392. If info(r) is a match or end match command, it cannot be equal to any token found by get token. Therefore an undelimited parameter—i.e., a match that is immediately followed by match or end match— will always fail the test ‘cur tok = info(r)’ in the following algorithm. ⟨ Scan a parameter until its delimiter string has been found; or, if s = null , simply scan the delimiter

string 392 ⟩ ≡ continue: get token; { set cur tok to the next token of input }

if cur tok = info(r) then ⟨ Advance r; goto found if the parameter delimiter has been fully matched,

otherwise goto continue 394 ⟩; ⟨ Contribute the recently matched tokens to the current parameter, and goto continue if a partial match

is still in effect; but abort if s = null 397 ⟩; if cur tok = par token then

if long state ≠ long call then ⟨ Report a runaway argument and abort 396 ⟩; if cur tok < right brace limit then

if cur tok < left brace limit then ⟨ Contribute an entire group to the current parameter 399 ⟩ else ⟨ Report an extra right brace and goto continue 395 ⟩

else ⟨ Store the current token, but goto continue if it is a blank space that would become an undelimited

parameter 393 ⟩; incr (m); if info(r) > end match token then goto continue; if info(r) < match token then goto continue;

found : if s ≠ null then ⟨ Tidy up the parameter just scanned, and tuck it away 400 ⟩

This code is used in section 391.

393. ⟨ Store the current token, but goto continue if it is a blank space that would become an undelimited

parameter 393 ⟩ ≡ begin if cur tok = space token then

if info(r) ≤ end match token then

if info(r) ≥ match token then goto continue; store new token(cur tok ); end

This code is used in section 392.

394. A slightly subtle point arises here: When the parameter delimiter ends with ‘# ’, the token list will have a left brace both before and after the end match. Only one of these should affect the align state, but both will be scanned, so we must make a correction. ⟨ Advance r; goto found if the parameter delimiter has been fully matched, otherwise goto continue 394 ⟩ ≡

begin r ← link (r); if (info(r) ≥ match token) ∧ (info(r) ≤ end match token) then

begin if cur tok < left brace limit then decr (align state);

goto found ;

end else goto continue; end

This code is used in section 392.

395. ⟨ Report an extra right brace and goto continue 395 ⟩ ≡ begin back input ; print err ( ␣ ␣ ); sprint cs (warning index ); print ( ␣ ␣ ␣ ␣ );

help6 ( ´ ␣   ␣      ␣ ␣`  ´ ␣    ␣     ´ ␣    ␣  ␣     ␣  )
(  ␣        ␣` #      ´ ␣   ␣`    ´ ␣     ␣  )
(  ␣      ␣  ␣   ␣      ␣       ␣    ␣   ␣`     ´ ␣  )
( ´ ␣    ␣        ␣    ␣     ␣  ␣  ␣      ␣ ␣  )
(  ␣    ␣     ␣  ␣   ␣    ␣  ␣   ␣        ␣   ␣  )

(  ␣`  ´ ␣   ␣         ␣    ␣    ␣`  ´ ␣   ␣  ␣    ␣  ␣  ); incr (align state);

long state ← call ; cur tok ← par token; ins error ; end { a white lie; the won’t always trigger a runaway }

This code is used in section 392.

396. If long state = outer call , a runaway argument has already been reported. ⟨ Report a runaway argument and abort 396 ⟩ ≡

begin if long state = call then begin runaway ; print err ( ␣ ␣ ␣ ); sprint cs (warning index ); print ( ␣ ␣ );

  help3 ( ␣       ␣   ´ ␣         ␣ ␣`  ´ ␣       ␣  ␣  ␣     ␣  )
  (  ␣        ␣  ␣   ␣    ␣     ␣   ␣   ␣  ␣  )

( ␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ ); back error ; end;

pstack [n] ← link (temp head ); align state ← align state − unbalance; for m ← 0 to n do flush list (pstack [m]); return; end

This code is used in sections 392 and 399.

397. When the following code becomes active, we have matched tokens from s to the predecessor of r, and we have found that cur tok ≠ info(r). An interesting situation now presents itself: If the parameter is to be delimited by a string such as ‘ ’, and if we have scanned ‘ ’, we want to contribute one ‘ ’ to the current parameter and resume looking for a ‘ ’. The program must account for such partial matches and for others that can be quite complex. But most of the time we have s = r and nothing needs to be done.

Incidentally, it is possible for  tokens to sneak in to certain parameters of non-  macros. For
example, consider a case like ‘  #  ’ where the first  is not followed by an exclamation

point. In such situations it does not seem appropriate to prohibit the , so TEX keeps quiet about this bending of the rules. ⟨ Contribute the recently matched tokens to the current parameter, and goto continue if a partial match is

still in effect; but abort if s = null 397 ⟩ ≡ if s ≠ r then

if s = null then ⟨ Report an improper use of the macro and abort 398 ⟩ else begin t ← s;

repeat store new token(info(t)); incr (m); u ← link (t); v ← s;

loop begin if u = r then if cur tok ≠ info(v) then goto done else begin r ← link (v); goto continue;

end; if info(u) ≠ info(v) then goto done; u ← link (u); v ← link (v); end; done: t ← link (t); until t = r; r ← s; { at this point, no tokens are recently matched } end

This code is used in section 392.

398. ⟨ Report an improper use of the macro and abort 398 ⟩ ≡

begin print err (  ␣  ␣ ); sprint cs (warning index ); print ( ␣     ´ ␣     ␣   ␣  );

  help4 ( ␣   ␣    ␣     ␣`            ´ ␣    ␣   ␣    ␣  )
  (  ␣`  ´ ␣     ␣`   ´ ␣     ␣       ␣        ␣     ␣  )
  (  ␣  ␣  ␣       ␣     ␣   ␣     ␣    ␣   ␣   ␣  )

(  ␣  ␣   ␣        ␣      ␣  ␣ ´ ␣        ␣  ); error ; return;

end

This code is used in section 397.

399. ⟨ Contribute an entire group to the current parameter 399 ⟩ ≡ begin unbalance ← 1; loop begin fast store new token(cur tok ); get token;

if cur tok = par token then

if long state ≠ long call then ⟨ Report a runaway argument and abort 396 ⟩; if cur tok < right brace limit then

if cur tok < left brace limit then incr (unbalance)

else begin decr (unbalance); if unbalance = 0 then goto done1 ; end;

end; done1 : rbrace ptr ← p; store new token(cur tok );

end

This code is used in section 392.

400. If the parameter consists of a single group enclosed in braces, we must strip off the enclosing braces. That’s why rbrace ptr was introduced. ⟨ Tidy up the parameter just scanned, and tuck it away 400 ⟩ ≡

begin if (m = 1) ∧ (info(p) < right brace limit ) ∧ (p ≠ temp head ) then begin link (rbrace ptr ) ← null ; free avail (p); p ← link (temp head ); pstack [n] ← link (p); free avail (p); end

else pstack [n] ← link (temp head ); incr (n); if tracing macros > 0 then

begin begin diagnostic; print nl (match chr ); print int (n); print ( );

show token list (pstack [n − 1], null , 1000); end diagnostic(false);

end; end

This code is used in section 392.

401. ⟨ Show the text of the macro being expanded 401 ⟩ ≡ begin begin diagnostic; print ln; print cs (warning index ); token show (ref count ); end diagnostic(false); end

This code is used in section 389.