REF POPCOMPILE                                      John Gibson Apr 1996

         COPYRIGHT University of Sussex 1996. All Rights Reserved.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
<<<<<<<<<<<<<<<<<<<<<                             >>>>>>>>>>>>>>>>>>>>>>
<<<<<<<<<<<<<<<<<<<<<       POP-11 COMPILER       >>>>>>>>>>>>>>>>>>>>>>
<<<<<<<<<<<<<<<<<<<<<         PROCEDURES          >>>>>>>>>>>>>>>>>>>>>>
<<<<<<<<<<<<<<<<<<<<<                             >>>>>>>>>>>>>>>>>>>>>>
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

This file describes the operation of the Poplog Pop-11 compiler and  the
procedures  it  makes  available   for  reading  and  compiling   Pop-11
expressions, defining new syntactic constructs,  etc. It should be  read
in conjunction  with REF * VMCODE,  which  explains the  Poplog  Virtual
Machine interface used by this and other Poplog compilers.


         CONTENTS - (Use <ENTER> g to access required sections)

  1   Overview of the Pop-11 Compiler

  2   Compilation Procedures
      2.1   Stream
      2.2   Statement Sequence
      2.3   Expression Sequence
      2.4   Expression

  3   Utility Procedures
      3.1   Testing/Checking for Input Items
      3.2   Procedures Associated With define and procedure
      3.3   Procedures Associated With Other Syntax Constructs

  4   More On Expression Compilation

  5   Example Definitions of Syntax/Syntax Operator Words

  6   Compilation Contexts
      6.1   Statements Compiled at "Top Level"
      6.2   Compiling in "Executing" and "Non-Executing" Contexts
      6.3   Testing Compilation Context

  7   See Also



----------------------------------
1  Overview of the Pop-11 Compiler
----------------------------------

The Pop-11 compiler consists of a set of procedures corresponding to the
various syntactic constructs in the language, as specified by the syntax
diagrams given  in  REF * POPSYNTAX.  These  divide  into  two  classes:
meta-constructs like  expression  and  specific  syntax/syntax  operator
words like define. While there is a basic set of the latter class  which
constitute the 'normal' syntax of  Pop-11, the compiler itself makes  no
direct reference to these, since they are all dealt with via the general
mechanism of recognising words whose identprops are "syntax" or  "syntax
N", and by calling procedures from the identifier values of such words.

    This file therefore confines itself to procedures concerned with the
meta-constructs  (compilation-stream,   statement-sequence,   statement,
expression-sequence and expression), and deals only with those  specific
constructs that have generally-useful utility procedures associated with
them.

    All compiler procedures take source  code input items from the  list
proglist (using the  procedure itemread, see  REF * PROGLIST), and  emit
Poplog Virtual  Machine  code  directly (currently,  no  parse  tree  is
built). The interface procedure pop11_compile sets up proglist initially
to compile code either from a character stream or from an existing list,
and then commences compilation by calling  pop11_comp_stream to  compile
the input stream.


pop11_compile(source)                                        [procedure]
        Compiles Pop-11  program text  from an  item list  or  character
        source source. For example,

            pop11_compile([2 + 2]) =>
            ** 4

            pop11_compile('myprogram.p');

        This procedure is defined as

            define pop11_compile(source);
                dlocal  pop_default_type = '.p',
                        popprompt        = ': ',
                        proglist_state   = proglist_new_state(source);
                pop11_comp_stream()
            enddefine;

        That is, it sets up a new local proglist_state from the argument
        source (with  pop_default_type equal  to '.p'),  and then  calls
        pop11_comp_stream.

        See  REF * proglist_new_state  for  the  permissible  values  of
        source.


compile_mode                                                    [syntax]
        This facility, defined in LIB * COMPILE_MODE enables setting and
        clearing of the compiler flags pop_pop11_flags described  below.
        See HELP * COMPILE_MODE for more details.


trycompile(filename) -> bool                                 [procedure]
        If the file specified by filename exists, then calls

                subsystem_compile(filename, "pop11")

        and returns  true (assuming  successful compilation).  false  is
        returned if the file doesn't exist.


pop_pop11_flags -> int                                        [variable]
int -> pop_pop11_flags
        Flag bits in  this integer variable  control certain aspects  of
        the Pop-11 compiler's operation. These are defined and described
        in the file

            $usepop/pop/lib/include/pop11_flags.ph

        Note that  the value  of pop_pop11_flags  is localised  to  each
        compilation stream,  procedure  or lexical  block;  thus  when a
        given scope is terminated, the value of pop_pop11_flags is reset
        to its value on entry to that scope.

        See HELP * COMPILE_MODE  for the  syntax construct  compile_mode
        which enables  setting  and  clearing  of  these  flags  (either
        individually or in groups) via keyword arguments.


pop11_exec_compile(compile_p, has_result)                    [procedure]
        Calls an arbitrary compiler  procedure compile_p to compile  and
        plant VM code, and then executes the planted code immediately.

        The boolean argument has_result says whether compile_p  produces
        a result or not;  if true, then one  result is expected, and  is
        saved  before  executing  the  code  and  returned  after  (thus
        preventing it  from  being obscured  on  the stack  by  anything
        produced by the code execution).

        This procedure is basically just defined as

                sysEXEC_COMPILE(compile_p, has_result)

        except that  it locally  resets  certain global  Pop-11 compiler
        variables. Note that it also sets pop_autoload true locally.




-------------------------
2  Compilation Procedures
-------------------------


2.1  Stream
-----------

pop11_comp_stream()                                          [procedure]
        Sets up a new execute level compilation of Pop-11 code, and then
        compiles proglist until it becomes null. Aside from initialising
        certain global variables  locally, this  procedure is  basically
        just

                define pop11_comp_stream();
                    sysCOMPILE(termin, pop11_exec_stmnt_seq_to) ->
                enddefine;

        that is,  call  pop11_exec_stmnt_seq_to(termin) to  compile  and
        execute statements inside a new invocation of the VM compiler.


popclosebracket -> word_or_list                               [variable]
word_or_list -> popclosebracket
        Contains the closing 'bracket' (usually  a syntax word) for  the
        current syntax construct being compiled.  It can also be a  list
        of such words when the construct has a choice of closers.

        (A single word, or a word in a list, can also be given  inside a
        1-element vector. This means accept the word as a closer only if
        declared as a syntax word.)

        At top-level (i.e. directly inside pop11_comp_stream), the value
        of popclosebracket is <termin>.

        Syntax procedures  normally  set  this variable  locally  to  an
        appropriate value before commencing compilation of a  construct,
        and then call pop11_need_nextitem with  that value to check  for
        the closer at the end.


popclosebracket_exec -> word                                  [variable]
word -> popclosebracket_exec
        Contains the value  of popclosebracket established  by the  most
        recent  call   of  pop11_exec_stmnt_seq_to.   See  below   under
        pop11_exec_stmnt_seq_to.



2.2  Statement Sequence
-----------------------
A statement is  an expression  sequence, i.e. zero  or more  expressions
separated by commas; a statement-sequence then consists of zero or  more
such sequences, separated by semicolons  (note that the standard  syntax
operators "=>" and "==>" also  act as statement separators by  replacing
themselves on proglist with semicolons).

    Expression and statement  sequences are  largely interchangeable  in
Pop-11, except in a few places (such as procedure call arguments)  where
the latter are not allowed. At execute level, the end of each  statement
indicates where execution is to take place.


pop11_comp_stmnt_seq()                                       [procedure]
        Compiles and  plants  VM  code  for  a  sequence  of  statements
        separated by semicolons.


pop11_comp_stmnt_seq_to(closer) -> found                     [procedure]
        Same as pop11_comp_stmnt_seq except  that during compilation  of
        the sequence, popclosebracket is set locally to be the  argument
        closer, and after compilation pop11_need_nextitem is called with
        this argument to check the next item on proglist. The result  of
        pop11_need_nextitem is  returned (see  Utility Procedures  below
        for a description of pop11_need_nextitem).


pop11_exec_stmnt_seq_to(closer) -> found                     [procedure]
        As for pop11_comp_stmnt_seq_to, but executes the VM code planted
        at the end of each statement (by calling * sysEXECUTE).

        In addition to setting popclosebracket, this procedure also sets
        popclosebracket_exec locally to the same value. The main purpose
        of this is so that the test

                popclosebracket == popclosebracket_exec

        can  be  used  to  determine  whether  the  current  context  is
        'executing' or not, i.e. whether there are any syntax constructs
        currently compiling below the level of pop11_exec_stmnt_seq_to.

        [For example, a syntax procedure  like section wants to be  able
        to execute the statements in  its body immediately if  possible,
        but it cannot do so if  it occurs inside any construct which  is
        itself not executing code directly, e.g. define or if. While the
        context  of  define  can  be  recognised  by  the  VM   variable
        * pop_vm_compiling_list being non-empty, this  is not so for  an
        if  statement  outside  of  a  procedure;  however,  in   such a
        statement popclosebracket will be "endif", and so the above test
        will be false. section therefore uses the code

                if popclosebracket == popclosebracket_exec then
                    pop11_exec_stmnt_seq_to
                else
                    pop11_comp_stmnt_seq_to
                endif ("endsection");

        to compile its body.]

        To ensure  that  popclosebracket /==  popclosebracket_exec  when
        inside  any  other  procedure  that  sets  popclosebracket,  the
        actually value assigned to the two variables is not closer,  but
        the following:

                if ispair(closer) then
                    copy(closer)
                elseif closer == termin then
                    termin
                else
                    closer :: []
                endif ->> popclosebracket -> popclosebracket_exec;



2.3  Expression Sequence
------------------------

pop11_comp_expr_seq()                                        [procedure]
        Compiles and  plants  VM  code for  a  sequence  of  expressions
        separated by commas.

        A 'MISSING  SEPARATOR' mishap  will occur  if the  next item  on
        proglist after the sequence does not match the current value  of
        popclosebracket and  is  an  expression  opener  (an  expression
        opener is  anything except  <termin> or  a non-procedure  syntax
        word, i.e. a word whose identprops are "syntax" and whose  value
        is not a procedure).


pop11_comp_expr_seq_to(closer) -> found                      [procedure]
        Same as pop11_comp_stmnt_seq except  that during compilation  of
        the sequence, popclosebracket is set locally to be the  argument
        closer, and after compilation pop11_need_nextitem is called with
        this argument to check the next item on proglist. The result  of
        pop11_need_nextitem is  returned (see  Utility Procedures  below
        for a description of pop11_need_nextitem).



2.4  Expression
---------------
The procedure pop11_comp_prec_expr is the heart of the Pop-11  compiler:
it compiles a  single expression containing  operators upto a  specified
maximum precedence. Since an understanding of this procedure is  helpful
in writing new syntax  or syntax operator  constructs, its operation  is
described  in  greater  detail  in   the  section  More  On   Expression
Compilation below.


pop11_comp_prec_expr(prec, update) -> nextitem               [procedure]
        Compiles and plants VM code for a single expression, the  extent
        of which is determined as follows:

          # If the  first item  is  not an  expression opener,  or  is a
            syntax word equal to popclosebracket, then the expression is
            empty.

          # Otherwise, the expression has the generic form

               item  operator-expr  operator-expr ...

            where each operator expression  is either a syntax  operator
            followed  by  an  appropriate  code  body,  or  an  ordinary
            operator followed  by a  sub-expression. The  expression  is
            terminated by meeting either a non-operator, or an  operator
            whose  internally-represented  precedence  (see  below)   is
            greater than or equal to the argument prec.

        The prec argument is a  value specifying the limit for  operator
        precedences in this  expression; for efficiency  reasons, it  is
        supplied not in the normal identprops format, but in the form in
        which precedences are actually represented internally. If idprec
        is a normal identprops  precedence, then the corresponding  prec
        value is the positive integer given by

                prec = intof(abs(idprec) * 20)
                            + (if idprec > 0 then 1 else 0 endif)

        (E.g. an idprec of 4  will give a prec  of 81, whereas -4  would
        give 80.) Since an identprops precedence can range between -12.7
        and 12.7,  the normal  range for  prec  is 2  - 255;  any  value
        greater than 255 is  guaranteed to include  all operators in  an
        expression.

        The update  argument  is a  boolean  which if  true  causes  the
        expression to  be compiled  in update  mode rather  than  normal
        evaluate mode.

        The result of the procedure is the item following the expression
        (note that this remains as the next item on proglist).


pop_expr_prec -> int                                          [variable]
int -> pop_expr_prec
pop_expr_update -> bool                                       [variable]
bool -> pop_expr_update
        These two variables  are dynamic  locals of  pop_comp_prec_expr,
        and hold the  values of the  prec and update  arguments for  the
        current call of that procedure.


pop_expr_inst(item)                                 [procedure variable]
pop_expr_item -> item                                         [variable]
item -> pop_expr_item
        These   two    variables   are    also   dynamic    locals    of
        pop_comp_prec_expr,  and  together   constitute  a  buffer   for
        planting VM code. Their use  is described in More On  Expression
        Compilation below.


pop11_comp_expr()                                            [procedure]
        Compiles and  plants  VM  code  for  a  single  expression  with
        unlimited operator precedence. Same as

                pop11_comp_prec_expr(256, false)


pop11_comp_expr_to(closer) -> found                          [procedure]
        Same as pop11_comp_expr  except that during  compilation of  the
        expression, popclosebracket is  set locally to  be the  argument
        closer, and after compilation pop11_need_nextitem is called with
        this argument to check the next item on proglist. The result  of
        pop11_need_nextitem is returned.




---------------------
3  Utility Procedures
---------------------


3.1  Testing/Checking for Input Items
-------------------------------------

pop11_need_nextitem(item) -> found                           [procedure]
        Checks the next item from proglist to be (a) a member of item if
        item is  a list,  or  (b) equal  to  item otherwise.

        (item may also be, or contain, a word inside a 1-element vector.
        This means accept the word as equal only if it is declared  as a
        syntax word.)

        If neither condition holds, the mishap

            msw: MISPLACED SYNTAX WORD

        or

            mei: MISPLACED EXPRESSION item

        results (depending on  whether the  next item  is an  expression
        opener or not), where the INVOLVING list of the mishap is

            FOUND nextitem READING TO item

        Otherwise, the next item is removed from proglist and  returned.
        This procedure uses nextitem to examine the next item, so macros
        are expanded, autoloading can occur, etc.


pop11_need_nextreaditem(item) -> found                       [procedure]
        Same as pop11_need_nextitem, except that nextreaditem is used to
        get the next item, so that autoloading or macro expansion  can't
        occur.


pop11_try_nextitem(item) -> found                            [procedure]
        As for  pop11_need_nextitem, except  that false  is returned  if
        item (or a member of item) is not the next item. If it is,  then
        the next item is removed  from proglist and returned as  result.
        This procedure will expand macros and may cause autoloading.


pop11_try_nextreaditem(item) -> found                        [procedure]
        Same as pop11_try_nextitem, except that nextreaditem is used  to
        get the next item, so that autoloading or macro expansion  can't
        occur.



3.2  Procedures Associated With define and procedure
----------------------------------------------------

pop11_define_declare(id_name, globl_p, decl_p,      [procedure variable]
                                        idprops)
        The procedure in  this variable (which  is dynamically local  to
        pop11_compile) is called by the  define statement to declare  as
        an identifier the name of the procedure being defined, when such
        a declaration  is  needed.  This  is in  all  cases  EXCEPT  the
        following:

          # dlocal is specified,  i.e. define dlocal  .... In this  case
            the identifier is assumed to be declared already.

          # updaterof is specified without any declaration keywords, the
            name  is  already  declared   as  a  lexical  or   permanent
            identifier, and,  for a  nested  define, the  identifier  is
            'local' to the enclosing procedure (i.e. an lvars of it,  or
            a dlocal of it).

        The arguments are:

            Argument   Meaning
            --------   -------
            id_name    The name of the identifier to be declared.

            globl_p    The procedure sysGLOBAL  if "global" was  present
                       in the header and false otherwise.

            decl_p     The sys_ declaration procedure for a  declaration
                       keyword  if  present  (e.g.  sysVARS  for   vars,
                       sysCONSTANT, sysLVARS, etc), or false if none.

            idprops    A value for the identprops of the name, i.e.  the
                       identprops keyword specified, or false if none.

        The default  procedure  in  this variable  is  shown  below.  If
        POP11_DEFINE_CONSTANT is  set in  pop_pop11_flags  (compile_mode
        syntax :pop11 +defcon), then top-level defines are defaulted  to
        constant; if  POP11_DEFINE_PROCEDURE  is set  (:pop11  +defpdr),
        then identifiers are defaulted  to procedure-type.

        Also, in the case where a  nested define would default to  vars,
        it calls  pop11_vars_default_check if  POP11_VARS_CHECK  (:pop11
        +varsch) is set:

        define vars pop11_define_declare(id_name, globl_p, decl_p,
                                                           idprops);
            lvars globl_p, decl_p, idprops, id_name;

            unless decl_p then
                ;;; no explicit declaration given
                sysVARS -> decl_p;  ;;; initially, default to vars
                if pop_vm_compiling_list == [] then
                    ;;; top level define
                    if pop_pop11_flags &&/=_0 POP11_DEFINE_CONSTANT then
                        sysCONSTANT -> decl_p
                    endif
                else
                    ;;; nested define
                    if pop_pop11_flags &&/=_0 POP11_OLD_VARS then
                        ;;; old-style
                        if pop_pop11_flags &&/=_0 POP11_VARS_CHECK then
                            pop11_vars_default_check(p_idname) -> decl_p
                        endif
                    else
                        ;;; new style -- just defaults to lconstant
                        sysLCONSTANT -> decl_p
                    endif
                endif;
            endunless;

            unless idprops then
                ;;; no explicit identprops given
                if pop_pop11_flags &&/=_0 POP11_DEFINE_PROCEDURE then
                    "procedure"
                else
                    0
                endif -> idprops
            endunless;

            if globl_p
            and decl_p /== sysVARS and decl_p /== sysCONSTANT
            then
                mishap(0, 'ids: INCORRECT DEFINE SYNTAX (not
                                vars or constant after global')
            endif;

            ;;; do the declaration
            decl_p(id_name, idprops);
            if globl_p then globl_p(id_name) endif;
        enddefine;


pop11_define_props(id_name, p_name, updater)        [procedure variable]
                                         -> props
        The procedure in this variable is called by the define statement
        to return  the default  pdprops value  for the  procedure  being
        defined. (Note  that this  is  the DEFAULT  value, and  will  be
        overridden by an explicit "with_props" declaration.)

        The arguments are:

            Argument   Meaning
            --------   -------
            id_name    The name of the identifier the procedure  resides
                       in.

            p_name     The name of the procedure, i.e. the id_name  with
                       any section pathname removed.

            updater    true if updaterof was present, false if not.

        The standard version of this procedure appears below; by default
        it returns  p_name  as the  pdprops  value, but  for  a  lexical
        identifier  when  the  flag   POP11_NO_LEX_PDPROPS  is  set   in
        pop_pop11_flags (and pop_debugging is false), it returns false:

            define vars pop11_define_props(id_name, p_name, upd)
                                                            -> props;
                lvars id_name, p_name, upd, props;
                if isident(sys_current_ident(id_name)) /== "perm"
                and pop_pop11_flags &&/=_0 POP11_NO_LEX_PDPROPS
                and not(pop_debugging)
                then
                    ;;; false for lexical procedure props
                    false
                else
                    ;;; else default is procedure name as props
                    p_name
                endif -> props
            enddefine;

        (The  flag  POP11_NO_LEX_PDPROPS  corresponds  to   compile_mode
        syntax :pop11 -lprops.


pop11_comp_procedure(closer, id_name,  props) ->  p          [procedure]
        This procedure,  called  by  define  and  procedure,  compiles a
        procedure from just  before the  argument list up  to closer  (a
        word  or   list   containing  words   e.g.   "end"   "enddefine"
        "endprocedure"). id_name is the name of the procedure identifier
        (e.g. as read in by "define", or false if there is no name, e.g.
        in the  "procedure" syntactic  form.  props is  the item  to  be
        inserted in the pdprops of the procedure, or false. props can be
        over- ridden by with_props in  the header. The result  returned,
        p, is  of  the  same  type  as  the  result  of  sysENDPROCEDURE
        (described in REF * VMCODE),  i.e. either  the procedure  record
        created or a 'procedure compilation' structure.

        This  procedure  is  used  by  the  syntax  words  "define"  and
        "procedure". E.g. the latter could be defined as:

            define syntax procedure;
                pop11_comp_procedure([endprocedure end], false, false)
                        -> pop_expr_item;
                sysPUSHQ -> pop_expr_inst
            enddefine;

        Note  that  (providing  the  flag  POP11_OLD_VARS  is  clear  in
        pop_pop11_flags -- as it is by default), any formal argument  or
        result identifier that is not  locally declared (i.e. that  does
        not appear  in an  identifier  declaration or  dlocal  statement
        immediately following  the  procedure header)  is  defaulted  to
        lvars.

        (If POP11_OLD_VARS is set, e.g. with compile_mode syntax  :pop11
        +oldvar, then undeclared  formals default to  vars. However,  in
        this case,  if  POP11_VARS_CHECK  is also  set  (+varsch),  then
        pop11_comp_procedure    calls     the     variable     procedure
        pop11_vars_default_check on any such formal.)

        Note also that when POP11_VARS_CHECK is  set, it is an error  to
        have an explicit vars statement inside a procedure (dlocal  must
        be used  instead).  If  POP11_VARS_CHECK  is  clear,  a  warning
        message is issued instead (unless POP11_OLD_VARS is set).


pop11_vars_default_check(id_name) -> decl_p         [procedure variable]
        This variable is redundant except when POP11_OLD_VARS is set  in
        pop_pop11_flags (compile_mode syntax :pop11 +oldvar).

        In this case only, pop11_comp_procedure and pop11_define_declare
        call  the  procedure  in   this  variable,  whenever  the   flag
        POP11_VARS_CHECK is set  in pop_pop11_flags and  a default  vars
        declaration would be required (for  a formal argument or  result
        identifier in  the  former case,  or  a procedure  name  in  the
        latter).

        id_name is is the name of the identifier concerned; after taking
        any other  relevant  action,  the procedure  should  return  the
        declaration  procedure  decl_p  to   be  used  to  declare   the
        identifier. The standard value  of this procedure just  prints a
        warning message and returns sysVARS, i.e.

         define vars pop11_vars_default_check(id_name);
           lvars id_name;
           printf(';;; WARNING: %p DEFAULTED TO VARS IN PROCEDURE %p\n',
                       [%id_name, pdprops(hd(pop_vm_compiling_list))%]);
           sysVARS
         enddefine;

        (However,  in  Popc  compilation,  pop11_vars_default_check   is
        redefined to treat the condition as an error).



3.3  Procedures Associated With Other Syntax Constructs
-------------------------------------------------------

pop11_comp_constructor(closer) -> count_or_false             [procedure]
        Compiles and  plants code  to push  the elements  in a  list  or
        vector constructor,  i.e.  [...]  or {...},  after  the  opening
        bracket has been read. The closing bracket closer terminates the
        constructor expression (e.g. "]" or "}").

        If the  constructor  expression  did  not  contain  any  of  the
        'evaluate' keywords

                        %  ^  ^^

        and the flag POP11_CONSTRUCTOR_CONSTS is set in  pop_pop11_flags
        (compile_mode :pop11 +constr), the result count_or_false is  the
        number  of  elements  in  the  structure  (i.e.  the  number  of
        sysPUSHQs executed). Otherwise, false is returned.

        The count_or_false  result  allows  a syntax  procedure  to  use
        pop11_comp_constructor  inside   the   compile_p   argument   to
        * sysCONSTRUCT:

            define syntax {;
                sysCONSTRUCT(
                    procedure();
                        pop11_comp_constructor("}"), "sysvecons"
                    endprocedure, 2)
            enddefine;

        If count_or_false is a count, sysCONSTRUCT will then try to make
        the  structure   at  compile-time   (as  opposed   to   actually
        incorporating the  constructor  code into  the  procedure  being
        compiled, so that a  new structure will  be produced every  time
        the code is run). See REF * VMCODE.

        Note  that  if   POP11_CONSTRUCTOR_CONSTS  is   not  set,   then
        pop11_comp_constructor will always  return false, regardless  of
        whether the constructor  expression compiled contained  evaluate
        keywords or not. In Pop-11 terms this means that list and vector
        constructors will always produce  new structures every time  the
        code is run; the  alternative, with POP_CONSTRUCTOR_CONSTS  set,
        implies that constructor expressions  with no evaluate  keywords
        will produce compile-time constants (so that in a procedure, for
        example,  any  updating  of  the  structure's  components   will
        permanently change it in all invocations of that procedure).


pop11_comp_declaration(decl_p)                               [procedure]
pop11_comp_declaration(decl_p, globword)
        This procedure is used by the constant, vars, lvars, dlvars  and
        lconstant   constructs   to   read    declaration/initialisation
        statements for identifiers (see  REF * POPSYNTAX for the  syntax
        of these). The argument decl_p is a declaration procedure of the
        form

                decl_p(word, idprops)

        (e.g. sysVARS), which for each word appearing in the declaration
        is called  on the  word and  its specified  IDPROPS (the  latter
        defaulting to 0 when not specified).

        globword is  an optional  word argument  that can  be used  when
        declaring permanent identifiers, as follows:

            Argument        Meaning
            --------        -------
            "global"        Declare each identifier global, ie
                            sysGLOBAL(word).

            "nonglobal"     Declare each identifier nonglobal, ie
                            sysGLOBAL(word, false).

            "undef"         Declare each identifier global if the
                            bit POP11_PERM_GLOBAL is set in
                            pop_pop11_flags.

        If an identifier  declaration is followed  by an  initialisation
        expression then this is compiled, and, except in the case  where
        decl_p is sysLCONSTANT, the code

                sysPOP(word)

        is planted (and executed immediately  if at execute level).  For
        sysLCONSTANT, the expression  is evaluated  immediately and  the
        result item assigned with sysPASSIGN(item, word).


pop11_loop_start(lab)                                        [procedure]
        Adds the  label  lab  to  the  list  of  loop  iteration  labels
        referenced by nextloop, nextif and nextunless (lab will normally
        be a label produced by sysNEW_LABEL).


pop11_loop_end(lab)                                          [procedure]
        Adds the label lab to the list of loop end labels referenced  by
        quitloop, quitif and quitunless.




---------------------------------
4  More On Expression Compilation
---------------------------------

This  section  describes  the  procedure  pop11_comp_prec_expr  in  more
detail, and should  be read in  conjunction with Expression  Compilation
above.

    Essentially, the  procedure  has  two  phases:  phase  1  reads  and
examines the first item  of the expression, while  phase 2 loops  around
absorbing  operator  expressions  until  an  operator  is  found   whose
precedence is too high  to form part of  the current expression (or  the
next item is not  an operator). Phase 1  is responsible for  recognising
syntax words and calling their associated procedures, while phase 2 does
the same for syntax operators.

    Although the procedure emits Poplog  VM code directly (i.e.  without
constructing  an  intervening  parse  tree),  this  is  effected   via a
1-instruction  'buffer'   to  allow   possible  re-interpretation   of a
preceding instruction. The  buffer is represented  by the two  variables
pop_expr_inst  and   pop_expr_item,   which  contain   respectively   an
instruction-planting procedure (such as  sysPUSH) and an argument  value
to be given to it. Two 'dummy' code-planting procedures, pop11_EMPTY and
pop11_FLUSHED, are used in this context  to indicate that the buffer  is
empty, and also to distinguish whether  this is because no code has  yet
been compiled  (pop11_EMPTY),  or  because code  has  been  planted  but
flushed (pop11_FLUSHED); in  these cases the  value of pop_expr_item  is
irrelevant.

(Note that both pop_expr_inst and pop_expr_item are dynamically local to
pop11_comp_prec_expr, so  that nested  expressions have  no  interaction
with each other in respect of buffered instructions.)

    Phase 1 thus commences by  setting pop_expr_inst to pop11_EMPTY.  If
item is the next item on proglist and is one of the following, then item
is removed from proglist and the corresponding actions performed:

     #  Non-operator syntax word with procedure value P, not equal to
        popclosebracket:

             P();                            ;;; Call syntax procedure
             if pop_expr_inst == pop11_EMPTY then
                 pop11_FLUSHED -> pop_expr_inst  ;;; see below
             endif;

     #  Any other non-syntax, non-operator word:

             item -> pop_expr_item;
             sysPUSH -> pop_expr_inst;

     #  Any non-word except <termin>:

             item -> pop_expr_item;
             sysPUSHQ -> pop_expr_inst;

Phase 2 (to which all other cases go directly without changing the  next
item) then loops  while the  next item item  is an  operator word  whose
internally-represented precedence  op_prec  is  less  than  the  current
expression precedence  given by  pop_expr_prec  (see the  discussion  of
internal precedence  values in  Expression Compilation).  For each  such
operator, the word is  removed from proglist  and the following  actions
taken:

     #  syntax operator word with procedure value P:

             P();                        ;;; Call syntax op procedure
             if pop_expr_inst == pop11_EMPTY then
                 pop11_FLUSHED -> pop_expr_inst  ;;; see below
             endif;

     #  ordinary operator word:

             pop_expr_inst(pop_expr_item);       ;;; flush buffer
             item -> pop_expr_item;
             sysCALL -> pop_expr_inst;
             pop11_comp_prec_expr(op_prec || 1, false);

Note that for an  ordinary operator word, the  precedence passed to  the
recursive call of pop11_comp_prec_expr for the following  sub-expression
is the operator's precedence with bit  0 set. This value corresponds  to
the absolute value of its  signed identprops precedence, and causes  the
sub-expression to  include  operators  of negative  (but  not  positive)
identprops precedence of the same  absolute value (thus making  negative
operators associate to the right rather than to the left).

    Note also that after calling either syntax procedures in phase 1  or
syntax operator  procedures  in phase  2,  pop_expr_inst is  changed  to
pop11_FLUSHED if it remains pop11_EMPTY. This ensures that a  subsequent
syntax operator can use a test  for pop11_EMPTY to determine whether  it
occurred at the beginning of an expression, without forcing every syntax
procedure to make this change.

    pop11_comp_prec_expr finishes by flushing the instruction buffer  in
a mode  specified  by  the  value  of  the  update  argument  stored  in
pop_expr_update. For normal evaluate mode (false) this is just

            pop_expr_inst(pop_expr_item);

Update  mode  (true)   is  interpreted   as  meaning   that  the   final
instruction-planting procedure of the expression should have an updater,
which will plant  appropriate update-mode  code; the absence  of one  is
taken to mean that the compiled  code was illegal in an update  context,
and results in the mishap 'IMPERMISSIBLE UPDATE EXPRESSION'.  Otherwise,
the buffer is flushed by applying the updater:

            updater(pop_expr_inst)(pop_expr_item);

So that they work correctly with this scheme, all update-mode Poplog  VM
instructions  are  the   updaters  of   the  corresponding   normal-mode
procedures, e.g. sysPOP is the updater of sysPUSH, sysUCALL of  sysCALL,
etc.


pop11_EMPTY(dummy)                                           [procedure]
-> pop11_EMPTY(dummy)
pop11_FLUSHED(dummy)                                         [procedure]
-> pop11_FLUSHED(dummy)
        Both these procedures do nothing but erase their dummy argument.
        This  means  that  whenever  one   of  them  is  the  value   of
        pop_expr_inst,     it     is      always     safe     to      do
        pop_expr_inst(pop_expr_item) to flush the instruction buffer.

        Their updaters  are different,  however: on  the basis  that  an
        arbitrary piece  of  already-flushed  code  has  no  update-mode
        interpretation, the updater of pop11_FLUSHED produces the mishap
        'IMPERMISSIBLE UPDATE EXPRESSION'.

        On the other hand, the updater of pop11_EMPTY does allow that an
        empty expression  can  be meaningful  in  this context:  if  the
        (necessarily  true)  value  of  pop_expr_update  is  in   fact a
        procedure, then  this is  run  without arguments,  otherwise  no
        action is taken. (This facility is used, for example, by the  ->
        and ->>  syntax  operators  to  interpret  an  empty  assignment
        destination as a stack erase, by calling pop11_comp_prec_expr to
        compile the  following expression  with  an update  argument  of
        sysERASE(%0%).)




------------------------------------------------------
5  Example Definitions of Syntax/Syntax Operator Words
------------------------------------------------------

This section  gives  two  simple  examples  of  defining  new  syntactic
constructs. The first is the syntax word ", for pushing a quoted word:

    define syntax ";
        readitem() -> pop_expr_item;
        if isstring(pop_expr_item) then
            consword(pop_expr_item) -> pop_expr_item
        elseunless isword(pop_expr_item) then
            mishap(pop_expr_item, 1, 'iqw: INCORRECT QUOTED WORD');
        endif;
        pop11_need_nextitem(""") -> ;
        sysPUSHQ -> pop_expr_inst
    enddefine;

The  second  is  a  simplified  version  of  the  "("  syntax  operator,
illustrating  how  push  instructions,  etc  generated  in  phase  1  of
pop11_comp_prec_expr are re-interpreted in  phase 2 as procedure  calls.
It also demonstrates how a syntax operator, by testing for  pop11_EMPTY,
can behave differently depending  on whether it  opens an expression  or
not:

    define syntax -1 (;
         dlocal popclosebracket = ")";

        if pop_expr_inst == pop11_EMPTY then
            ;;; starts expression -- just compile parenthesised
            pop11_comp_stmnt_seq();              ;;; stmnt seq
            pop11_FLUSHED -> pop_expr_inst
        elseif pop_expr_inst == sysPUSH then
            ;;; re-interpret push as call of identifier
            pop11_comp_expr_seq();     ;;; compile arguments to call
            sysCALL -> pop_expr_inst
        elseif pop_expr_inst == sysPUSHQ then
            ;;; re-interpret as call of quoted structure
            pop11_comp_expr_seq();     ;;; compile arguments to call
            sysCALLQ -> pop_expr_inst
        else
            ;;; call of whatever's on the stack
            pop_expr_inst(pop_expr_item);              ;;; flush it
            sysPOP(sysNEW_LVAR() ->> pop_expr_item);   ;;; save in
                                                       ;;; temp lvar
            pop11_comp_expr_seq();     ;;; compile arguments to call
            sysCALL -> pop_expr_inst
        endif;
        ;;; check for closing bracket
        pop11_need_nextitem(")") ->
    enddefine;




-----------------------
6  Compilation Contexts
-----------------------

The context in which an expression  or statement is compiled can  change
in subtle ways.  One context  concerns the current  section, which  maps
words (see  REF * WORDS) onto  identifiers (see  REF * IDENT).  Changing
from one section  to another can  imply that some  words have  different
identifiers associated  with them  in the  new section,  with  different
syntactic properties or different values, or both. For full details  see
REF * SECTIONS.

Other  types  of  compile-time  context  are  concerned  with  whether a
procedure  is  currently  being  compiled  or  not,  and  with   whether
instructions can be  executed as soon  as they are  compiled or  whether
execution must be delayed until a larger structure has been compiled.

These two distinctions are  labelled by the phrases  "at top level"  and
"in executing context", which will now be explained.


6.1  Statements Compiled at "Top Level"
---------------------------------------
A statement (as defined in  REF * POPSYNTAX) is compiled at "top  level"
if it  is not  syntactically enclosed  within the  body of  an  explicit
procedure definition or procedure expression in the current  compilation
stream. In  that case  the  Pop-11 identifier  popexecute is  set  true.
Otherwise it  is false.  So,  in a  non-nested procedure  definition  or
procedure expression using one of the forms

        define <header>;
            <body>
        enddefine;

        procedure <header>;
            <body>
        endprocedure;

popexecute remains true  while the  <header> is being  compiled, but  is
made false as soon as compilation  of the <body> starts (i.e. after  the
the semi-colon has been read in, which is when the call of  sysPROCEDURE
(see REF * VMCODE)) occurs.  popexecute will  also be  set false  during
compilation of user-defined  syntactic forms  that invoke  sysPROCEDURE,
and it remains false until sysENDPROCEDURE is invoked, e.g. by enddefine
or endprocedure,  after which  popexecute is  restored to  its  previous
value. It  is  also  made  false by  lexical  blocks  in  "non-executing
contexts" explained below.

Most Pop-11  statement forms,  including loops,  conditionals,  variable
declarations, etc. can be used at top level, though a goto or go_on (see
REF * VMCODE) cannot. All statement forms  encountered at top level  are
executed immediately,  unless in  a "non-executing"  context,  explained
below.

If a statement other than a procedure definition or an expression  other
than a procedure expression, e.g. a  loop or conditional, occurs at  top
level then  statements  in its  body  will also  be  at top  level.  For
example, the  declaration  of  xxx  below  and  the  <instructions>  are
compiled at top level if the following conditional is at top level:

        if flag then
            vars xxx = 99;
            <instructions>
        endif;

During the compilation  of <instructions> the  value of popexecute  will
remain true  unless a  nested  procedure body  or nested  lexical  block
(using lblock ....  endlblock) is encountered.  If the same  conditional
expression occurred in  a procedure definition  it would not  be at  top
level and neither would the declaration and other embedded instructions.
In that context popexecute would be false throughout the compilation  of
the conditional.

During compilation of a procedure  body popexecute is automatically  set
false, but if  a macro  or syntax  word, or  the autoloading  mechanism,
invokes the  procedure  pop11_compile  or  anything  else  that  invokes
sysCOMPILE (see REF * VMCODE), then a  new compilation stream is set  up
temporarily and popexecute  (which is  local to  sysCOMPILE) starts  off
true  for  that  stream.  When  the  nested  compilation  is   finished,
compilation of the original procedure continues and popexecute takes  on
its previous value. (The  #_INCLUDE macro is  designed to merge  another
file into  the current  compilation  stream instead  of starting  a  new
compilation. So  it  will  not  change  the  value  of  popexecute.  See
REF * #_INCLUDE)

If a section (REF * SECTIONS)  or lexical block (HELP * LEXICAL)  occurs
at top level, in  an "executing" context (defined  below), then the  top
level statements within it are executed as soon as they are compiled. In
such a  case  the  instructions will  be  obeyed  even if  there  is  no
"endsection" or "endlblock", and a "misplaced syntax word" error will be
reported only  when  the compilation  stream  ends without  the  closing
bracket. Top level sections and lexical blocks may be nested within  one
another without affecting popexecute or immediate execution.


6.2  Compiling in "Executing" and "Non-Executing" Contexts
----------------------------------------------------------
However, not all top-level contexts (i.e. contexts not inside  procedure
bodies) are equivalent. This is because something like a conditional  or
loop statement at top level is implicitly a procedure and is compiled as
a single entity before it can be executed. Thus, if the body of a  loop,
or one of the branches  of a conditional includes declarations,  lexical
blocks or other statements, they will also  be at top level, but not  in
an "executing" context.

So the first lvars  declaration below is at  top level (with  popexecute
true) but in a non-executing context:

        if flag then
           lvars test1 = 99;
           lblock
                lvars test2 = 88;
                ......
           endlblock;
            .....
        endif;

Because the lexical block is in a non-executing context it is treated as
a nested procedure, i.e. as if surrounded by

        procedure .... endprocedure();

and therefore it sets popexecute false. This is because in this sort  of
context, the  nested lexical  block has  to be  treated as  an  embedded
procedure in order to  ensure that any Type-3  lexical variables in  its
scope  work  properly.  (See  REF * VMCODE,  Implementation  of  Lexical
Scoping.)

Although "test1" is declared inside an implicit procedure it remains  at
top level  and  can be  accessed  outside the  conditional,  though  the
assignment will not  be done  until the whole  conditional is  executed,
because the declaration  is in  a non-executing  context. Similarly  any
instructions in the lexical block nested in the conditional will not  be
done while the block  is being compiled.  Moreover, because the  lexical
block is  compiled  as  if  it were  a  nested  procedure,  all  lexical
variables declared  within  it,  such as  "test2",  are  not  accessible
outside  the  conditional.  During  compilation  of  the  lexical  block
popexecute is set false, for the reason explained above.

Top level lexical blocks  can be nested inside  other top level  lexical
blocks and for all of them the context will be executing if they are not
inside any explicit or implicit procedure (i.e. loops or conditionals).


6.3  Testing Compilation Context
--------------------------------
Macros and syntax  words that  have to behave  differently depending  on
whether they occur inside a procedure body or not, or whether they occur
in an executing context or not, cannot safely test the context by  using
popexecute as  this may  be  set false  inside a  non-executing  lexical
block, which  is  not  in  a  procedure body  but  inside  a  top  level
conditional or loop.

So, as explained  above and  in REF * VMCODE, the  test to  be used  for
whether the  current context  is  executing or  not (i.e.  whether  some
larger construct  has to  be  compiled before  the  current one  can  be
executed) is

            popclosebracket == popclosebracket_exec

And the test for whether the current context is in a procedure body is

            pop_vm_compiling_list /== []




-----------
7  See Also
-----------

For more information on the virtual machine instructions used above

    # REF * VMCODE

    # REF * POPSYNTAX

Other examples of syntactic extensions can be found in

    # LIB * FOREACH
         (documented in HELP * FOREACH)

    # LIB * FOREVERY
         (documented in HELP * FOREVERY)

    # HELP * DEFINE_FORM

    # HELP * FOR_FORM

For more complex examples see:

    # LIB * FACETS
         (documented in HELP * FACETS)

    # LIB * NEWOBJ
         (documented in HELP * NEWOBJ).

    $usepop/pop/lib/flavours/*.p
         (documented in TEACH * FLAVOURS)



--- C.all/ref/popcompile
--- Copyright University of Sussex 1996. All rights reserved.
