§268  TEX82  PART 19: SAVING AND RESTORING EQUIVALENTS 109

268. Saving and restoring equivalents. The nested structure provided by ‘ … ’ groups in TEX means that eqtb entries valid in outer groups should be saved and restored later if they are overridden inside the braces. When a new eqtb value is being assigned, the program therefore checks to see if the previous entry belongs to an outer level. In such a case, the old value is placed on the save stack just before the new value enters eqtb. At the end of a grouping level, i.e., when the right brace is sensed, the save stack is used to restore the outer values, and the inner ones are destroyed.

Entries on the save stack are of type memory word . The top item on this stack is save stack [p], where p = save ptr − 1; it contains three fields called save type, save level , and save index , and it is interpreted in one of four ways:

1) If save type(p) = restore old value, then save index (p) is a location in eqtb whose current value should be destroyed at the end of the current group and replaced by save stack [p − 1]. Furthermore if save index (p) ≥ int base, then save level (p) should replace the corresponding entry in xeq level .

2) If save type(p) = restore zero, then save index (p) is a location in eqtb whose current value should be destroyed at the end of the current group, when it should be replaced by the current value of eqtb[undefined control sequence].

3) If save type(p) = insert token, then save index (p) is a token that should be inserted into TEX’s input when the current group ends.

4) If save type(p) = level boundary , then save level (p) is a code explaining what kind of group we were previously in, and save index (p) points to the level boundary word at the bottom of the entries for that group.

define save type(#) ≡ save stack [#].hh.b0 { classifies a save stack entry }

110  PART 19: SAVING AND RESTORING EQUIVALENTS  TEX82 §269

define save level (#) ≡ save stack [#].hh.b1 { saved level for regions 5 and 6, or group code } define save index (#) ≡ save stack [#].hh.rh { eqtb location or save stack location } define restore old value = 0 { save type when a value should be restored later } define restore zero = 1 { save type when an undefined entry should be restored } define insert token = 2 { save type when a token is being saved for later use } define level boundary = 3 { save type corresponding to beginning of group } 269. Here are the group codes that are used to discriminate between different kinds of groups. They allow TEX to decide what special actions, if any, should be performed when a group ends.

Some groups are not supposed to be ended by right braces. For example, the ‘$’ that begins a math formula causes a math shift group to be started, and this should be terminated by a matching ‘$’. Similarly, a group that starts with should end with , and one that starts with should end with .

define bottom level = 0 { group code for the outside world }

define simple group = 1 { group code for local structure only }

define hbox group = 2 { code for ‘ ’ }

define adjusted hbox group = 3 { code for ‘ ’ in vertical mode }

define vbox group = 4 { code for ‘ ’ }

define vtop group = 5 { code for ‘ ’ }

define align group = 6 { code for ‘ ’, ‘ ’ }

define no align group = 7 { code for ‘ ’ }

define output group = 8 { code for output routine }

define math group = 9 { code for, e.g, ‘ ’ }

define disc group = 10 { code for ‘ ’ }

define insert group = 11 { code for ‘ ’, ‘ ’ }

define vcenter group = 12 { code for ‘ ’ }

define math choice group = 13 { code for ‘ ’ }

define semi simple group = 14 { code for ‘ ’ }

define math shift group = 15 { code for ‘$ $’ }

define math left group = 16 { code for ‘ ’ }

define max group code = 16 ⟨ Types in the outer block 18 ⟩ +≡

group code = 0 . . max group code; { save level for a level boundary }

270. The global variable cur group keeps track of what sort of group we are currently in. Another global variable, cur boundary , points to the topmost level boundary word. And cur level is the current depth of nesting. The routines are designed to preserve the condition that no entry in the save stack or in eqtb ever has a level greater than cur level .

271. ⟨ Global variables 13 ⟩ +≡ save stack : array [0 . . save size] of memory word ; save ptr : 0 . . save size; { first unused entry on save stack } max save stack : 0 . . save size; { maximum usage of save stack } cur level : quarterword ; { current nesting level for groups } cur group: group code; { current group type } cur boundary : 0 . . save size; { where the current level begins }

272. At this time it might be a good idea for the reader to review the introduction to eqtb that was given above just before the long lists of parameter names. Recall that the “outer level” of the program is level one, since undefined control sequences are assumed to be “defined” at level zero. ⟨ Set initial values of key variables 21 ⟩ +≡

save ptr ← 0; cur level ← level one; cur group ← bottom level ; cur boundary ← 0; max save stack ← 0; 273. The following macro is used to test if there is room for up to six more entries on save stack . By making a conservative test like this, we can get by with testing for overflow in only a few places.

define check full save stack ≡

if save ptr > max save stack then begin max save stack ← save ptr ; if max save stack > save size − 6 then overflow ( ␣ , save size); end

274. Procedure new save level is called when a group begins. The argument is a group identification code like ‘hbox group’. After calling this routine, it is safe to put five more entries on save stack .

In some cases integer-valued items are placed onto the save stack just below a level boundary word, because this is a convenient place to keep information that is supposed to “pop up” just when the group has finished. For example, when ‘ ’ is being treated, the 100pt dimension is stored on save stack just before new save level is called.

We use the notation saved (k) to stand for an integer item that appears in location save ptr + k of the save stack.

define saved (#) ≡ save stack [save ptr + #].int procedure new save level (c : group code); { begin a new level of grouping }

begin check full save stack ; save type(save ptr ) ← level boundary ; save level (save ptr ) ← cur group; save index (save ptr ) ← cur boundary ; if cur level = max quarterword then

overflow ( ␣ , max quarterword − min quarterword );

{ quit if (cur level + 1) is too big to be stored in eqtb } cur boundary ← save ptr ; incr (cur level ); incr (save ptr ); cur group ← c; end;

275. Just before an entry of eqtb is changed, the following procedure should be called to update the other data structures properly. It is important to keep in mind that reference counts in mem include references from within save stack , so these counts must be handled carefully. procedure eq destroy (w : memory word ); { gets ready to forget w }

var q: pointer ; { equiv field of w } begin case eq type field (w) of call , long call , outer call , long outer call : delete token ref (equiv field (w)); glue ref : delete glue ref (equiv field (w)); shape ref : begin q ← equiv field (w); { we need to free a block }

if q ≠ null then free node(q, info(q) + info(q) + 1);

end; { such a block is 2n + 1 words long, where n = info(q) } box ref : flush node list (equiv field (w));

othercases do nothing

endcases; end;

276. To save a value of eqtb[p] that was established at level l, we can use the following subroutine. procedure eq save(p : pointer ; l : quarterword ); { saves eqtb[p] }

begin check full save stack ; if l = level zero then save type(save ptr ) ← restore zero else begin save stack [save ptr ] ← eqtb[p]; incr (save ptr ); save type(save ptr ) ← restore old value;

end; save level (save ptr ) ← l; save index (save ptr ) ← p; incr (save ptr ); end; 277. The procedure eq define defines an eqtb entry having specified eq type and equiv fields, and saves the former value if appropriate. This procedure is used only for entries in the first four regions of eqtb, i.e., only for entries that have eq type and equiv fields. After calling this routine, it is safe to put four more entries on save stack , provided that there was room for four more entries before the call, since eq save makes the necessary test. procedure eq define(p : pointer ; t : quarterword ; e : halfword ); { new data for eqtb }

begin if eq level (p) = cur level then eq destroy (eqtb[p])

else if cur level > level one then eq save(p, eq level (p));

eq level (p) ← cur level ; eq type(p) ← t; equiv (p) ← e;

end;

278. The counterpart of eq define for the remaining (fullword) positions in eqtb is called eq word define. Since xeq level [p] ≥ level one for all p, a ‘restore zero’ will never be used in this case. procedure eq word define(p : pointer ; w : integer );

begin if xeq level [p] ≠ cur level then begin eq save(p, xeq level [p]); xeq level [p] ← cur level ; end;

eqtb[p].int ← w;

end;

279. The eq define and eq word define routines take care of local definitions. Global definitions are done in almost the same way, but there is no need to save old values, and the new value is associated with level one. procedure geq define(p : pointer ; t : quarterword ; e : halfword ); { global eq define }

begin eq destroy (eqtb[p]); eq level (p) ← level one; eq type(p) ← t; equiv (p) ← e; end;

procedure geq word define(p : pointer ; w : integer ); { global eq word define } begin eqtb[p].int ← w; xeq level [p] ← level one; end;

280. Subroutine save for after puts a token on the stack for save-keeping. procedure save for after (t : halfword );

begin if cur level > level one then begin check full save stack ; save type(save ptr ) ← insert token; save level (save ptr ) ← level zero; save index (save ptr ) ← t; incr (save ptr ); end;

end;

281. The unsave routine goes the other way, taking items off of save stack . This routine takes care of restoration when a level ends; everything belonging to the topmost group is cleared off of the save stack. ⟨ Declare the procedure called restore trace 284 ⟩ procedure back input ; forward ; procedure unsave; { pops the top level off the save stack }

label done;

var p: pointer ; { position to be restored } l: quarterword ; { saved level, if in fullword regions of eqtb } t: halfword ; { saved value of cur tok }

begin if cur level > level one then begin decr (cur level ); ⟨ Clear off top level from save stack 282 ⟩; end

else confusion( ); { unsave is not used when cur group = bottom level } end; 282. ⟨ Clear off top level from save stack 282 ⟩ ≡

loop begin decr (save ptr ); if save type(save ptr ) = level boundary then goto done; p ← save index (save ptr ); if save type(save ptr ) = insert token then ⟨ Insert token p into TEX’s input 326 ⟩ else begin if save type(save ptr ) = restore old value then

begin l ← save level (save ptr ); decr (save ptr ); end

else save stack [save ptr ] ← eqtb[undefined control sequence]; ⟨ Store save stack [save ptr ] in eqtb[p], unless eqtb[p] holds a global value 283 ⟩; end;

end; done: cur group ← save level (save ptr ); cur boundary ← save index (save ptr )

This code is used in section 281.

283. A global definition, which sets the level to level one, will not be undone by unsave. If at least one global definition of eqtb[p] has been carried out within the group that just ended, the last such definition will therefore survive. ⟨ Store save stack [save ptr ] in eqtb[p], unless eqtb[p] holds a global value 283 ⟩ ≡

if p < int base then

if eq level (p) = level one then begin eq destroy (save stack [save ptr ]); { destroy the saved value } stat if tracing restores > 0 then restore trace(p, ); tats end

else begin eq destroy (eqtb[p]); { destroy the current value } eqtb[p] ← save stack [save ptr ]; { restore the saved value } stat if tracing restores > 0 then restore trace(p, ); tats end

else if xeq level [p] ≠ level one then begin eqtb[p] ← save stack [save ptr ]; xeq level [p] ← l; stat if tracing restores > 0 then restore trace(p, ); tats end

else begin stat if tracing restores > 0 then restore trace(p, ); tats end

This code is used in section 282.

284. ⟨ Declare the procedure called restore trace 284 ⟩ ≡ stat procedure restore trace(p : pointer ; s : str number ); { eqtb[p] has just been restored or retained } begin begin diagnostic; print char ( ); print (s); print char ( ␣ ); show eqtb(p); print char ( ); end diagnostic(false); end; tats

This code is used in section 281.

285. When looking for possible pointers to a memory location, it is helpful to look for references from eqtb that might be waiting on the save stack. Of course, we might find spurious pointers too; but this routine is merely an aid when debugging, and at such times we are grateful for any scraps of information, even if they prove to be irrelevant. ⟨ Search save stack for equivalents that point to p 285 ⟩ ≡

if save ptr > 0 then

for q ← 0 to save ptr − 1 do

begin if equiv field (save stack [q]) = p then begin print nl ( ); print int (q); print char ( ); end; end

This code is used in section 172.

286. Most of the parameters kept in eqtb can be changed freely, but there’s an exception: The magnification should not be used with two different values during any TEX job, since a single magnification is applied to an entire run. The global variable mag set is set to the current magnification whenever it becomes necessary to “freeze” it at a particular value. ⟨ Global variables 13 ⟩ +≡ mag set : integer ; { if nonzero, this magnification should be used henceforth }

287. ⟨ Set initial values of key variables 21 ⟩ +≡ mag set ← 0;

288. The prepare mag subroutine is called whenever TEX wants to use mag for magnification. procedure prepare mag ;

begin if (mag set > 0) ∧ (mag ≠ mag set ) then begin print err ( ␣ ␣ ); print int (mag ); print ( ); print nl ( ␣ ␣ ␣ ␣ ␣ ␣ );

    help2 ( ␣   ␣      ␣    ␣   ␣            ␣     ␣   ␣    ␣  ␣ ´  )
    (  ␣  ␣   ␣            ␣   ␣    ␣       ␣  ␣    ␣  );

int error (mag set ); geq word define(int base + mag code, mag set ); { mag ← mag set } end;

if (mag ≤ 0) ∨ (mag > 32768) then

begin print err (  ␣            ␣   ␣    ␣       ␣  ␣  );
help1 (  ␣            ␣     ␣    ␣  ␣       ␣ ␣   ␣  ); int error (mag );

geq word define(int base + mag code, 1000);

end; mag set ← mag ; end;

§289  TEX82  PART 20: TOKEN LISTS 115

289. Token lists. A TEX token is either a character or a control sequence, and it is represented internally in one of two ways: (1) A character whose ASCII code number is c and whose command code is m is represented as the number 2^{8}m + c; the command code is in the range 1 ≤ m ≤ 14. (2) A control sequence whose eqtb address is p is represented as the number cs token flag + p. Here cs token flag = 2^{12} − 1 is larger than 2^{8}m + c, yet it is small enough that cs token flag + p < max halfword ; thus, a token fits comfortably in a halfword.

A token t represents a left brace command if and only if t < left brace limit ; it represents a right brace command if and only if we have left brace limit ≤ t < right brace limit ; and it represents a match or end match command if and only if match token ≤ t ≤ end match token. The following definitions take care of these token-oriented constants and a few others.

define cs token flag ≡ ´ 7777 { amount added to the eqtb location in a token that stands for a control

sequence; is a multiple of 256, less 1 }

define left brace token = ´ 0400 { 2^{8} · left brace } define left brace limit = ´ 1000 { 2^{8} · (left brace + 1) } define right brace token = ´ 1000 { 2^{8} · right brace } define right brace limit = ´ 1400 { 2^{8} · (right brace + 1) } define math shift token = ´ 1400 { 2^{8} · math shift } define tab token = ´ 2000 { 2^{8} · tab mark } define out param token = ´ 2400 { 2^{8} · out param } define space token = ´ 5040 { 2^{8} · spacer + ␣ } define letter token = ´ 5400 { 2^{8} · letter } define other token = ´ 6000 { 2^{8} · other char } define match token = ´ 6400 { 2^{8} · match } define end match token = ´ 7000 { 2^{8} · end match }

290. ⟨ Check the “constant” values for consistency 14 ⟩ +≡ if cs token flag + undefined control sequence > max halfword then bad ← 21;

116  PART 20: TOKEN LISTS  TEX82 §291

291. A token list is a singly linked list of one-word nodes in mem, where each word contains a token and a link. Macro definitions, output-routine definitions, marks, texts, and a few other things are remembered by TEX in the form of token lists, usually preceded by a node with a reference count in its token ref count field. The token stored in location p is called info(p).

Three special commands appear in the token lists of macro definitions. When m = match, it means that TEX should scan a parameter for the current macro; when m = end match, it means that parameter matching should end and TEX should start reading the macro text; and when m = out param, it means that TEX should insert parameter number c into the text at this point.

The enclosing and characters of a macro definition are omitted, but the final right brace of an output routine is included at the end of its token list.

Here is an example macro definition that illustrates these conventions. After TEX processes the text

                            # #      #     ## #  #

the definition of is represented as a token list containing

(reference count), letter , match #, match #, spacer ␣, , end match, out param 1, , letter , spacer ␣, mac param #, other char , out param 2, spacer ␣, out param 2.

The procedure scan toks builds such token lists, and macro call does the parameter matching.

Examples such as

                                   ␣

explain why reference counts would be needed even if TEX had no operation: When the token list for is being read, the redefinition of changes the eqtb entry before the token list has been fully consumed, so we dare not simply destroy a token list when its control sequence is being redefined.

If the parameter-matching part of a definition ends with ‘# ’, the corresponding token list will have ‘ ’ just before the ‘end match’ and also at the very end. The first ‘ ’ is used to delimit the parameter; the second one keeps the first from disappearing.