TEACH TEACHNEURAL                                Julian Clinton Jan 1990
                                            Modified A.Sloman 7 Mar 2000

Introduction To Poplog-Neural and on-line documentation.

This is a modified version of the Poplog-neural package.
See TEACH NEURAL

NOTE: some of these facilities are available with different procedure
names in the Popvision library produced by David Young and available
from the FreePoplog directory:

    http://www.cs.bham.ac.uk/research/poplog/freepoplog.html

See the MLP library and documentation contained therein.

         CONTENTS - (Use <ENTER> g to access required sections)

 -- How To Use This Documentation
 -- Introduction
 -- Software Description
 -- The LIB Neural Environment
 -- The Teach And Test Library
 -- The Generic Accessor Library
 -- Specific Network Types
 -- Rebuilding Poplog-Neural
 -- Using X Windows
 -- Linking C libraries
 -- Invoking Poplog-Neural
 -- Invoking Poplog-Neural With Graphical User Interface
 -- . X Windows
 -- How To Use LIB NETTRAINING
 -- Creating New Datatypes
 -- Using New Datatypes


-- How To Use This Documentation --------------------------------------

The aim  of this  file is  to teach  you how  to use  the  Poplog-Neural
package rather than what  to use neural networks  for. It describes  how
the software is organised and how to make use of the various parts.  The
best way  to  achieve  this  is  to read  about  the  various  parts  of
Poplog-Neural and  then move  on to  the examples  which aim  to  give a
suitable way of working.

NOTE added by A.Sloman, 29 Sep 2000.
The original version of this assumed that you would run popneural code
in the framework of a saved image with a special interface. The saved
image is no longer set up by default, and in any case users are more
likely to want to link the popneural libraries into their own programs.
This teach file has been modified to suit such users.

For most such users it will suffice to start with the section on
    How To Use LIB NETTRAINING


-- Introduction -------------------------------------------------------

LIB Neural is  a software  package providing  Poplog users  with  the
facilities to:

(a) teach themselves about neural networks

(b) create different networks and train them

(c) create new datatypes (e.g. sets, number ranges etc.) and example
    sets which can be applied to a neural network

(d) display information about the state of a network.

(e) allow networks to be embedded in programs and treated as standard
    procedures


-- Software Description -----------------------------------------------

Poplog-Neural can be thought of  as a collection of libraries  collected
into a single tool. The tool has the structure:

1. Poplog-Neural Environment Library (popneural)
    [Not relevant to this package]

2. Network Teach And Test Library (nettraining)
                                                   -!
3. Generic Accessor Library (netgenerics)           !
                                                    +- networkdefs
4. Specific Network Types (backprop, complearn)     !
                                                   -!

The aim  has  been  to produce  a  system  which can  have  new  network
architectures added to it without requiring a large amount of re-writing
of the software  (see HELP *NEWNETS  for a guide  to adding new  network
types).


-- The LIB Neural Environment --------------------------------------

The  environment  is  a  menu-driven  system  with  menus  for  creating
networks,  datatypes  and  examples,  teaching  and  testing   networks,
altering  environment  options  and  getting  help.  The   Poplog-Neural
environment is documented in REF *NEURALINDEX and *POPNEURAL.
It is no longer setup by default.

In this version of the package it is assumed that the popneural library
will be invoked from the editor and portions compiled as needed. I.e.
the "environment" described here is not necessarily avaiable, though
some of the files remain available as they contain useful documentation.


-- The Teach And Test Library -----------------------------------------

This library  provides  a set  of  generic functions  for  teaching  and
testing networks.  These  functions  are  documented  in  the  file
    REF *NETTRAINING.


-- The Generic Accessor Library ---------------------------------------

This library provides  a set of  network-independent accessor  functions
and a  facility  to support  the  addition  of new  network  types.  The
functions are described in REF *NETGENERICS.


-- Specific Network Types ---------------------------------------------

Network types are  stored as libraries  so they can  be loaded and  used
without the overheads of the other libraries being required.

The libraries provided by the current release of Poplog-Neural are:

(1) backprop      - back-propagation networks (REF *BACKPROP)
(2) complearn     - competitive learning networks (REF *COMPLEARN)



-- Rebuilding Poplog-Neural -------------------------------------------
[This section is no longer relevant. Skip to end of section.]

When you receive Poplog-Neural, you should rebuild the saved images  you
are supplied with.  You may  also need to  rebuild the  system when  you
upgrade to a different version of the operating system.

The base command for rebuilding Poplog-Neural is:

    neural -m           (UNIX)
    MKNEURAL            (VMS)

Certain additional arguments can be provided (described below).


-- Using X Windows ----------------------------------------------------
[This section is no longer relevant. Skip to end of section.]

You can include an X user interface by using the "-x" option:

    neural -m -x        (UNIX)
    MKNEURAL -X         (VMS)

The look and feel of the Neural  you build will depend on the Poplog  on
which it is being built. By default, Poplog is built using the OPEN LOOK
widget set on Sun workstations (Sun 3 and SPARCstations) and using Motif
on all other  platforms. See  the Poplog Administrators  Guide for  more
information on rebuilding Poplog with different X widget sets.

Note that to build an X version of Neural may require you to setup  some
environment variable used by the operating system to locate the required
libraries. For  example, on  Sun workstations  the environment  variable
LD_LIBRARY_PATH may  have to  be defined  or extended  to point  to  the
appropriate library directory e.g.:

    setenv LD_LIBRARY_PATH /usr/openwin/lib

Normally, Poplog should have the required information defined within it.
You can check  which libraries  Poplog will try  to use  by starting  up
Pop-11 and checking the value of XLINK_LIBS:

    % pop11

    Sussex POPLOG etc.

    Setpop
    : XLINK_LIBS =>
    ** [-L/usr/lib/X11R4 -L/usr/lib/Motif1.1 -lXm -lXt -lX11]



-- Linking C libraries ------------------------------------------------

Instructions for compiling the C libraries

    SEE $poplocal/local/neural/src/c/*

Select one of the compall* scripts that is suitable for your
environment. E.g. on most unix systems compall.gcc or compall should
work. You will have to set the environment variable $popneural

Run the script with an argument that corresponds to the current system,
for which there should be a matching sub-directory in $popneural/bin

E.g. if you are on a PC running linux you could do

    setenv popneural (...whatever...)

then
    compall.gcc pclinux

This should put some compiled files in $popneural/bin/pclinux

For a new machine and operating system you may have to create a new
directory, and run compall.gcc with a different argument.

Then edit $popneural/lib/cload.p to make sure that it copes with you
machine and operating system.


-- Invoking Poplog-Neural ---------------------------------------------

[This section is no longer relevant]

Poplog-Neural can be invoked from within any of the Poplog languages  or
direct from the UNIX or DCL prompt.

To start the system from the operating system prompt, type:

    neural              (UNIX)
or
    NEURAL              (VMS)

This will start Poplog-Neural in standard terminal mode.


-- Invoking Poplog-Neural With Graphical User Interface ---------------
[These sections are no longer relevant]

-- . X Windows --------------------------------------------------------

[This section is no longer relevant]

Ensure that you are running an X11R4 server on your workstation  console
or X terminal and in a shell or XTerm window, type:

    neural -x

This will invoke Poplog-Neural using X windows.

The "look and feel" of the interface will depend on how the Poplog which
Neural uses was linked. By default,  Sun 3 and SPARCstation Poplogs  are
linked to use OPEN LOOK. On all other platforms, Motif is the default.


-- How To Use LIB NETTRAINING -----------------------------------------

Load the neural net training library:

    lib nettraining;

To do  this you  will  need to  have  defined the  environment  variable
$popneural  and   have  added   '$popneural/lib'  to -popuseslist-   and
-popautolist-.

NOTE: all this is done automatically by the command
    uses neural;

Before going any further, we need to look at what our data consists  of.
In this example, which is teaching a network the exclusive-OR rule,  the
data is simply going  to be a  set of boolean values  of the inputs  and
target outputs. (The exclusive-OR problem was shown to be impossible for
a single layer network to learn in the late '60s and contributed to  the
temporary loss of interest in  neural-like networks among many  research
groups).

The exclusive-OR rule table looks like:

                    Input 1 Input 2     Output
                    --------------------------
                    false   false       false
                    true    false       true
                    false   true        true
                    true    true        false

and the example data can be specified as:

    vars xor_list = [[false false false]
                     [true false true]
                     [false true true]
                     [true true false]];

We also need to specify  what each example in  the set looks like.  This
template is also  a list of  lists, usually  one for each  field in  the
example. Here there are  two input fields followed  by one output  field
(it is also allowable to ignore a field by entering "none" or having the
data presented at both  input and output units  by typing "both").  Each
entry is of type "boolean". This can be specified as:

    vars xor_template = [[in boolean input1]
                         [in boolean input2]
                         [out boolean output]];

The third item  in each list  is an optional  field name. It  is a  good
habit to use this since it makes a print out of an example set easier to
understand. Create the example set with:

    nn_make_egs("xor_set", xor_template, xor_list);

and then generate the raw data:

    nn_generate_egs("xor_set");

-nn_generate_egs- creates the data and converts the high level datatypes
(e.g. boolean) into numbers  suitable to be applied  to the network  (in
this case the numbers  0.0 for false  and 1.0 for  true). To prove  that
this has been done, we  can display the contents  in one of the  example
set record slots:

    vars eg_rec = nn_example_sets("xor_set");
    eg_targ_data(eg_rec) =>
    ** <array [1 1 1 4]>

This shows that the output data to be presented to the network  consists
of four groups of data, each group being 1 element long. The input  data
consists of a similar array but each group contains two items of data.

    eg_in_data(eg_rec) =>
    ** <array [1 2 1 4]>

It  can  also  be  useful  to  show  the  contents  of  an  array  using
*ARRAYVECTOR. For example:

    arrayvector(eg_targ_data(eg_rec)) =>
    ** <pointer_to_float 0.0 1.0 1.0 0.0>

    arrayvector(eg_in_data(eg_rec)) =>
    ** <pointer_to_float 0.0 0.0 1.0 0.0 0.0 1.0 1.0 1.0>


Having created the  example set, we  can create a  network to learn  the
rule. From looking  at the examples,  we can see  that the network  will
need 2 input units and and 1 output unit. We will also give the  network
a hidden layer of 2 units (a hidden layer is a layer of units which  are
neither input or output units) since this is the topology which has been
shown to be capable of learning the exclusive-OR rule (shown below).

                               Output

                                 O
                                / \
                               /   \
                              /     \
                             O       O
                             | \   / |
                             |  \ /  |
                             |   X   |
                             |  / \  |
                             | /   \ |
                             O       O

                          Input 1  Input 2


The type of network used is  very important. In this case, our  examples
have both input and output values which implies using a back-propagation
network. A  back-propagation  network  learns by  comparing  the  actual
response for a  given input with  the target response  and altering  the
strengths of the connections  between the units  to reduce the  response
error.

(This compares with  a competitive  learning network  which attempts  to
'organise' the input  data on its  own by encouraging  units to  respond
strongly to  patterns  with  some  common  characteristics  and  not  to
others.)

The is  a  library  which implements  back-propagation  networks  called
BACKPROP (see REF *BACKPROP). This will have been loaded along  with
the NETTRAINING library. Create the network with:

    make_bpnet(2, {2 1}, 2.0, 0.2, 0.9)
                                -> nn_neural_nets("xor_net");

This creates a network with 2 input  units, 2 units in the hidden  layer
and 1 in the  output layer and  stores it in.  The weights (strength  of
connection between the units)  are each set to  a random number  between
-1.0 and +1.0 (a range  of 2.0). The last  two figures are the  learning
rate,  eta,  and   the  momentum,  alpha   (see  "Parallel   Distributed
Processing", McLelland & Rummelhart, for a detailed description of these
parameters).

We  can  print   the  weights   of  a   back-propagation  network   with
PR_BPWEIGHTS:

    pr_bpweights(nn_neural_nets("xor_net"));

    WEIGHTS
               bias     1     2
    Level 2
    unit   1:   0.57 -0.74 -0.56

    Level 1
    unit   1:   0.05  0.90  0.60
    unit   2:   0.48  0.88  0.73


This shows that the strength of the connection between the first unit in
hidden layer 1 and the second input unit is 0.60.

    vars results = nn_test_egs("xor_set", "xor_net", false);
    results =>
    ** [[false] [false] [false] [false]]

(You may get different results from this.)

What we actually want is

    ** [[false] [true] [true] [false]]

We want the network to learn to respond correctly to the input which  is
done with:

    nn_learn_egs("xor_set", "xor_net", 1000, true);

We can check how well the network has learnt by applying the example set
again:

    nn_test_egs("xor_set", "xor_net", false) =>
    ** [[false] [true] [true] [true]]

(Again you may find  that this does not  correspond with your  results.)
Although in this case the response was not correct, we can see that  the
network is beginning to learn the correct responses.

An important point to note is that the network you have created may  not
learn the correct  responses all the  time. This is  due to the  network
becoming stuck in a local minimum where weight changes made to make  one
response correct  are cancelled  out  by other  weight changes.  In  the
majority of  cases however,  if we  continue the  learning process,  the
network will end up being correctly trained.

    nn_learn_egs("xor_set", "xor_net", 5000, true);

    nn_test_egs("xor_set", "xor_net", false) =>
    ** [[false] [true] [true] [false]]      ;;; hooray !

We can see how the weights have changed from before:

    pr_bpweights(nn_neural_nets("xor_net"));


    WEIGHTS before training             WEIGHTS (after training)

               bias     1     2                    bias     1     2
    Level 2                             Level 2
    unit   1:   0.57 -0.74 -0.56        unit   1:  -4.11 -9.66  8.96

    Level 1                             Level 1
    unit   1:   0.05  0.90  0.60        unit   1:  -6.69  4.36  4.36
    unit   2:   0.48  0.88  0.73        unit   2:  -2.76  6.28  6.28


-- Creating New Datatypes ---------------------------------------------

There will be many occasions  when the datatypes defined by  NETGENERICS
are not sufficient. For example, suppose  we want to train a network  to
accept percentages of red,  green and blue in  a given light source  and
return the  colour  that  is  represented  by  that  light  source.  The
NETGENERICS library provides functions to  allow the user to define  new
datatypes.

The "percent" datatype is going to be an integer somewhere between 0 and
100. This can be created using the function NN_DECLARE_RANGE:

    nn_declare_range("percent", 0, 100);

(If we had wanted to  have the result returned as  a real instead of  an
integer, the third  argument would  have been replaced  with 100.0.  The
function which takes results from the output of the network and converts
it to a number  within the range  also coerces the result  to be of  the
type  of   the  number   specified   as  the   upper  bound   (see   REF
*NUMBERS/number_coerce).)

The set  of  colours  can  be  defined  using -nn_declare_set- (see  REF
*NETGENERICS). This takes  a set  name and a  list of  set members.  For
example:

    nn_declare_set("colour",
                   [red orange yellow green blue indigo violet]);


-- Using New Datatypes ------------------------------------------------

Having defined the required datatypes, we can create a couple of example
sets, one for teaching and one for testing. Both will take three  inputs
(the red, green and blue values) and a single output (the colour).

    nn_make_egs("colour_set",
                [[in percent red] [in percent green] [in percent blue]
                 [out colour colour]],
                [[100 0 0 red] [40 40 20 yellow] [0 100 0 green]
                 [0 0 100 blue] [40 0 60 violet] [30 0 70 indigo]]);

    nn_generate_egs("colour_set");

Finally we can  create the  network which  will attempt  to learn  these
examples. Before we do this, we will need to find out how many input and
output units are needed to represent the different datatypes being used.
This can be done in two ways:

1. By  using  the  function  -nn_units_needed-. This  takes  a  list  of
datatypes and returns the number of units required to map the  datatypes
onto a network. The input has  three sets of "percent" datatype and  the
output is a single "colour", so:

    nn_units_needed([percent percent percent]) =>   ;;; for input
    ** 3

    nn_units_needed([colour]) =>                    ;;; for output
    ** 7


2. By accessing the  example set record itself.  There are two slots  in
the record accessed by  the functions -eg_in_units- and  -eg_out_units-.
Once we  have  retrieved the  example  sets from  the  -nn_example_sets-
table, we can find out how many  input and output units are required  by
the example data:

    vars eg_rec = nn_example_sets("colour_set");

    eg_in_units(eg_rec) =>
    ** 3

    eg_out_units(eg_rec) =>
    ** 7


We will make our network with 3 input and 7 output units. There will  be
one hidden layer with 4 units:

    make_bpnet(3, {4 7}, 2.0, 0.2, 0.9) ->
                                        nn_neural_nets("col_net");

    pr_bpweights(nn_neural_nets("col_net"));

    WEIGHTS
               bias     1     2     3     4     5     6     7
    Level 2
    unit   1:   0.25  0.30  0.84  0.95 -0.42
    unit   2:  -0.13 -0.56  0.36  0.19  0.45
    unit   3:   0.44  1.00 -0.98  0.55 -0.53
    unit   4:  -0.01  0.73  0.04  0.78  0.67
    unit   5:  -0.67 -0.22 -0.60 -0.09  0.60
    unit   6:  -1.00  0.22 -0.99 -0.60 -0.10
    unit   7:  -0.93 -0.95 -0.43  0.02 -0.58

    Level 1
    unit   1:  -0.48  0.30 -0.46 -0.41
    unit   2:   0.57 -0.41 -0.98 -0.41
    unit   3:   0.45  0.04  0.44 -0.64
    unit   4:   0.48 -0.73  0.80  0.04


Now the example set "colour_set" can be learnt by the network:

    nn_learn_egs("colour_set", "col_net", 5000, false);

    nn_test_egs("colour_set", "col_net", false)=>
    ** [[red] [yellow] [green] [blue] [violet] [indigo]]

Again, you  may find  that  these results  don't agree  completely  with
yours. For example, the last result, "indigo", may actually be  "violet"
because the difference in values  between "indigo" and "violet" are  not
as great as they are with the other colours. This means the network will
have to be trained further in  order to be able to discriminate  between
them.

--- Copyright Integral Solutions Ltd. 1990. All rights reserved. ---
--- Copyright Integral Solutions Ltd. and the University of Sussex, 2000
        All rights reserved. ---

--- $poplocal/local/neural/teach/teachneural
--- Copyright University of Birmingham 2000. All rights reserved. ------
