Pup File Transfer Protocol Package




This package is a collection  of modules implementing the Pup  File and
Mail Transfer Protocols.  The package is used by the FTP  subsystem and
the Interim File System.



1. Overview


This  document  is  organized   as  a  general  overview   followed  by
descriptions  of each  of the  modules in  the package.   A  history of
revisions to the package is included at the end.

Before beginning the main  documentation, some general comments  are in
order.

     a.  The  File Transfer  Protocol is  (alas) complex;  this package
     requires the Pup package  and all of its supporting  packages plus
     some other  packages not specific  to Pup.  This  documentation is
     less tutorial than normal Alto package descriptions so  you should
     be prepared to consult its author.

     b.  This document describes the external program interfaces  for a
     particular implementation of the File Transfer Protocol,  and does
     not  deal with  the internal  implementation nor  the  reasons for
     design  choices in  the  protocol or  the  implementation.  Before
     considering  the  details   of  this  package,  you   should  read
     [Maxc]FtpSpec.press  to  get  the  flavor  of  how  the  File
     Transfer  Protocol  works.   The    directory  also  contains
     descriptions of the lower  level protocols on which FTP  is based.
     Detailed knowledge of these protocols is not necessary to use this
     package, but you  must be familiar with  the operation of  the Pup
     package.

     c.  This package and the protocol are under active  development so
     users should expect modifications and extensions.

     d.   This  package  is designed  to  run  under  several operating
     systems  and with  several file  systems.  Functions  are carefuly
     split  into  protocol-specific  and  environment-specific modules.
     This package  provides the  protocol modules;  you must  write the
     matching environment-specific modules.


1.1. Organization

The FTP  package comes  in four modules:  Server, User,  Utilities, and
Property lists.  The  utility and property  list modules are  shared by
the User and Server.

The User and  Server modules implement  their respective halves  of the
protocol exchanges.

                             ------------
                   Copyright Xerox Corporation 1982


Pup FTP Package            December 12, 1981                          2




The  Property  List   module  generates  and  parses   property  lists,
filesystem-independent descriptions of files.  When passed between User
and Server FTPs through the network byte stream, their form  is defined
by  protocol  as  a  parenthesized  list.   When  passed  between these
protocol modules and the user-supplied modules in a program,  they take
the form of a data structure defined by this package.

The Utility module  contains protocol routines  shared by the  User and
Server  modules  and  some  efficient  routines  for  transferring data
between a network stream and a disk stream.


1.2. File Conventions

The FTP package is distributed as file FTPPackage.dm, and  contains the
following files:

User
    FtpUserProt.br           User protocol common to file and mail
    FtpUserProtFile.br       User file commands
    FtpUserProtMail.br       User mail commands

Server
    FtpServProt.br           Server protocol common to file and mail
    FtpServProtFile.br       Server file commands
    FtpServProtMail.br       Server mail commands

Property lists
    FtpPListProt.br          Property list protocol
    FtpPListImpl.br          Implements a 'standard' property list
    FtpPListInit.br          Initialization

Utility
    FtpUtilB.br              Common protocol
    FtpUtilXfer.br           Unformatted data transfer
    FtpUtilDmpLd.br          Dump/Load data transfer
    FtpUtilCompB.br          Binary compare data transfer
    FtpUtilCompA.br          Binary compare data transfer
    FtpUtilA.br              Assembly-language utility code
    FtpUtilInit.br           Initialization

Definitions
    FtpProt.decl             Protocol parameters and structures

Command files
    CompileFtpPackage.cm     Compiles all files
    DumpFtpPackage.cm        A list of all binary files
    FtpPackage.cm            A list of all source files

All of these  modules are swappable, and  are broken up into  pieces no
larger  than  1024  words.   Modules  whose  names  end  in  "init" are
initialization code which should be executed once and thrown away.

The source files are kept with the subsystem sources in FTP.dm  and are
formatted for printing in a small fixed-pitch font such as  Gacha8 (use
the command 'Empress @FtpPackage.cm@').


Pup FTP Package            December 12, 1981                          3




1.3. Other Packages

FTP is a level 3 Pup  protocol and this package uses a number  of other
Alto software packages.  As always, files whose names end in "init" may
be discarded after initialization (except ContextInit.br).

Pup Package
    PupBSPOpenClose     PupBSPStreams.br    PupBSPProt.br
                        PupBSPa.br          PupBSPBlock.br
    PupRTP.br           PupRTPOpenClose     PupDummyGate.brPupRoute.br
    Pup1b.br            Pup1OpenClose       PupAl1a.br     Pup1Init.br
    PupAlEthb.br        PupAlEtha.br        PupAlEthInit.br
Context Package
    Context.br          ContextInit.br
Interrupt Package
    Interrupt.br        InterruptInit.br
Queue, Timer, and ByteBLT Packages
    AltoQueue.br        AltoTimer.br        AltoByteBLT.br
Time Package
    TimeConvB.br        TimeConvA.br        TimeIO.br
CmdScan Package
    Keyword.br          KeywordInit.br
Strings and Template Packages
    StringUtil.br       Template.br


1.4. Principal Data Structures

The following data  structures are of  interest to users,  and together
with the procedures described later, constitute the package interface.

PL       Property    List,    is    this    implementation's   internal
         representation of the protocol-specified property list.

FTPI     File  Transfer  Package Interface,  contains  pointers  to the
         network byte  stream, user disk  stream, log stream,  the file
         buffer, and various flags.

FTPSI    FTP Server Interface, is a vector of  user-supplied procedures
         constituting   the   interface   between   the   protocol  and
         environment-specific Server modules.

FtpCtx   FTP Context, is the process-private storage for an instance of
         a User  or Server  FTP.  It consists  of an  FTPI, and  if the
         process is a Server, an FTPSI.  This is a convenient place for
         the user-supplied modules  to keep process-private  data.  You
         can do this by adding items to the FtpCtx definition  and then
         recompiling everything.

The entire  FtpCtx need not  be filled  in all of  the time.   For each
group  of procedures,  the  items they  require will  be  specified.  A
general description of the contents of the FTPI part of an FtpCtx is in
order here.

bspSoc             a  pointer to  a  BSP socket  open to  a  remote FTP
                   process.

bspStream          a pointer  to the  stream in  the above  BSP socket.
                   Pup  package  experts will  recognize  that  this is


Pup FTP Package            December 12, 1981                          4




                   redundant, but it is often convenient and  makes the
                   code clearer.

dspStream          a pointer  to a  stream to  which this  package will
                   output   generally  useful   information,  including
                   copious   amounts   of   debugging   information  if
                   debugFlag is true.  The only operation that  need be
                   defined is 'Puts'.

debugFlag          a boolean.  If true, the protocol exchanges for this
                   context are output to dspStream as text,  along with
                   some other useful  information.  Use this!   It will
                   save you much head-scratching.

connFlag           a boolean.  This should  be true if bspSoc  is open.
                   The package will cooperate in maintaining this flag,
                   which is valuble when finishing.

serverFlag         a boolean.  This flag is tested by procedures in the
                   shared modules to determine whether the caller  is a
                   User or Server.

getCmdString       a pointer to the last string read by  the GetCommand
                   procedure  in  the  Utility  module.   Commands with
                   string    arguments    are    No,    Yes,   Version,
                   MailBoxException, and Comment.

The  following items  are used  by the  data transfer  routines  in the
Utility module.  The routines  are Alto-specific and in some  cases Ftp
subsystem-specific, so these items need not be filled in if you  do not
use the routines.

diskStream         a pointer  to a  disk stream.   It should  always be
                   opened in byte mode.

buffer             a pointer to a block of memory which can be used for
                   block transfer I/O  operations.  The bigger  this is
                   the faster things will go.

bufferLength       the length in words of the above buffer.


1.5. Programming Conventions

This  package  can  be  used  with  the  Bcpl  Overlay  package.   File
FtpOEPInit.br contains  a procedure  which will help  do this,  but you
should consult with the author.

This package does a lot of string manipulation, and uses  the following
conventions:

     a.  All strings are allocated from 'sysZone'.

     b.  Strings are represented  in data structures (such  as property
     lists) as addresses.  Zero means no string is present.

All of the procedures in this package expect to execute in contexts (in
the sense of  the Context package),  and expect CtxRunning  (defined by
the Context package) to point to an appropriately filled in FtpCtx.


Pup FTP Package            December 12, 1981                          5




1.6. Property Lists

In most contexts,  there are two property  lists: one generated  by the
client of  the package, and  one generated by  the package.   A client-
generated property list  is referred to as  a 'localPL', and it  is the
client's  responsibility  to  free  it when  it  is  no  longer needed.
Property lists created by  this package are referred to  as 'remotePLs'
since they are copies of property lists generated remotely; they should
never be freed by the client.



2. Server


The FTP Server module consists of three files: FtpServProt.br, routines
common to the file and mail servers, FtpServProtFile.br, file commands,
and  FtpServProtMail.br,  mail  commands.  The  server  module  has one
public procedure:

FtpServProt(timeout)
     which  carrys out  protocol  commands received  over  bspStream by
     calling  the  user-supplied  procedures in  FTPSI.   When  the BSP
     connection is  closed by the  remote User process,  this procedure
     returns.   FtpServProt  passes  'timeout'  to  GetCommand  (in the
     utility  module) when  waiting for  top-level  commands (retrieve,
     store,  delete,   etc.).   This  permits   the  server   to  break
     connections that don't seem to be doing anything.

This module uses the following fields in FtpCtx:  dspStream, bspStream,
bspSoc,  and  FTPSI.    The  manifest  constant  MTP   in  FtpProt.decl
conditionally  compiles  calls on  the  MTP commands.   The  package is
released with this switch false, since I expect only IFS will  need it.
All of the FTP  commands (Version, Store, Retrieve, etc.)  must contain
procedures (except the MTP ones if the MTP switch is false).  If you do
not wish to implement a command, it is sufficient to point  the command
at:

     and NYI(nil, nil) = valof
        [
        FTPM(markNo, 1, "Unimplemented Command")
        resultis false
        ]


in  which case  any  subsidiary procedures  for that  command  (such as
StoreFile and StoreCleanup  for the Store  command) need not  be filled
in.  FTPM is described in more detail below.


2.1. Version Command

By  convention, Version  is the  first command  exchanged over  a newly
opened FTP connection.  The User sends its protocol version  number and
a string such as "Maxc Pup Ftp User 1.04 19-Mar-77".   When FtpServProt
receives this command, it  replys with its protocol version  number and
then calls

     (CtxRunning>>FtpCtx.Version)(bspStream, nil)


Pup FTP Package            December 12, 1981                          6




which should generate some herald text:

     Wss(CtxRunning>>FtpCtx.bspStream,  "Alto Pup  FTP Server  1.13 14-
May-77")


2.2. Retrieve Command

When the  remote FTP User  process sends the  command 'Retrieve'  and a
property list  describing the files  it wants to  retrieve, FtpServProt
parses the property list and calls

     (CtxRunning>>FtpCtx.Retrieve)(remotePL, localPL)

which should decide whether to accept the command.  Retrieve's decision
may involve  checking passwords,  looking up  files, and  other actions
using  the  information  in  remotePL  plus  other environment-specific
information,   such  as   whether   the  requester   has   the  correct
capabilities, etc.  To refuse the request, Retrieve should call

     FTPM(markNo, code, string)

and  return false.   To  accept the  command,  it should  return  a new
property  list,  localPL,  describing a  file  matching  remotePL which
Retrieve is  willing to send.   FtpServProt returns this  PL to  you as
'localPL' in the next  call to Retrieve, so  that you can free  it.  On
the first call, localPL will be zero.  Some FTP implementations require
a minimum set of properties  here, but the whole subject of  who should
specify what properties is rather involved and beyond the scope of this
description.   For  more information,  consult  the  FTP specification.
This  package provides  a fast  procedure (in  the Utility  module) for
deciding  the 'type'  of a  file (text  or binary)  which you  may find
useful.

Property  lists in  retrieve requests  may specify  multiple  files, so
FtpServProt will continue to  call Retrieve until it returns  false (no
more files).  On each call, remotePL will be the same original  PL sent
from  the remote  User, and  localPL will  be the  last PL  returned by
Retrieve.  If  Retrieve supports  multiple file  requests then  it must
save some information  so that the next  time FtpServProt calls  it, it
can generate the next file.  If Retrieve does not support multiple file
requests then it should do its thing during the first call and remember
that it is finished.  The next time it is called it should return false
having only deallocated localPL (it should not call FTPM).

If Retrieve returns a PL, FtpServProt sends it back to the User to more
fully describe the file.   At this point the  User may back out  of the
transfer,  in  which  case  the next  procedure  will  be  skipped, and
RetrieveCleanup will  be called immediately.   If the User  indicates a
willingness to proceed, FtpServProt then calls

     (CtxRunning>>FtpCtx.RetrieveFile)(localPL, remotePL)

to transfer the file data.   This package provides a procedure  (in the
Utility  module) for  transferring data  from a  disk Stream  to  a BSP
Stream,  but  you  are  free write  your  own.   When  RetrieveFile has
finished the transfer, it should return true if everything went OK.  If
something bad happened, it should call

     FTPM(markNo, code, string)


Pup FTP Package            December 12, 1981                          7




and return false.  In any case, FtpServProt calls

     (CtxRunning>>FtpCtx.RetrieveCleanup)(localPL, ok, remotePL)

where 'ok' is false if  RetrieveFile returned false or the  User backed
out   of  the   command.   Note   that  if   Retrieve   returned  true,
RetrieveCleanup will  always be called,  but RetrieveFile may  not.  If
Retrieve allocates any resources  (such as opening a file)  they should
be deallocated here.

Finally,  FtpServProt calls  Retrieve  again, and  the  process repeats
until Retrieve returns false.


2.3. Store Command

When the remote FTP User process sends the command  'newStore' followed
by a property list describing the file, FtpServProt parses the property
list and calls

     (CtxRunning>>FtpCtx.Store)(remotePL)

which should decide  whether to accept  the command.  To  accept, Store
should return a property list (referred to as localPL below) specifying
the destination file (localPL will be passed to StoreCleanup so you can
free it).  To refuse  the command Store should call  FTPM(markNo, code,
string) and return false, in which case the next  procedure (StoreFile)
is not called.

If Store returns  true, FtpServProt sends the  PL to the User  and then
calls

     (CtxRunning>>FtpCtx.StoreFile)(remotePL, localPL)

to transfer the file data.   This package provides a procedure  (in the
Utility  module) for  transferring data  from a  BSP Stream  to  a disk
Stream, but you  may write your own.   When StoreFile has  finished the
transfer, it should  return true if  everything went OK.   If something
bad happened, it should call

     FTPM(markNo, code, string)

and return false.  Finally, FtpServProt calls

     (CtxRunning>>FtpCtx.StoreCleanup)(remotePL, ok, localPL)

where 'ok' is  true if StoreFile returned  true and the  User indicated
that everything went ok.  If 'ok' is false, StoreCleanup  should delete
the file,  since it is  almost certainly damaged.   Note that  if Store
returned true,  StoreCleanup will always  be called, but  StoreFile may
not.  If Store  allocates any resources (such  as opening a  file) they
should be deallocated here.


2.4. Delete Command

When the remote FTP User process sends the command 'Delete' followed by
a  property  list  describing  the  files  which  it  wants  to delete,
FtpServProt parses the property list and calls


Pup FTP Package            December 12, 1981                          8




     (CtxRunning>>FtpCtx.Delete)(remotePL, localPL)

which  should  decide  whether to  accept  the  command.   Don't delete
anything  yet!  The  User may  still back  out.  To  refuse  the delete
request,  Delete  should  call FTPM(markNo,  code,  string)  and return
false.  To  accept the command,  it should return  a new PL  with every
property it can find, so that  the User can be sure of the  identity of
file to be  deleted.  FtpServProt will return  this PL as  'localPL' in
the next call to Delete, so that it can be deallocted.

Property  lists  in  delete requests  may  specify  multiple  files, so
FtpServProt will continue  to call Delete  until it returns  false.  On
each call, remotePL will be  the same original PL sent from  the remote
User, and localPL  will be the last  PL returned by Delete.   If Delete
supports multiple file requests  then it must save some  information so
that the next time FtpServProt calls it, it can generate the PL for the
next file.  If Delete does  not support multiple file requests  then it
should  do its  thing during  the first  call and  remember that  it is
finished.  The  next time it  is called it  should return  false having
only deallocated localPL (it should not call FTPM).

If Delete returns a PL, FtpServProt sends it back to the User and waits
for  confirmation.   If  the  User  still  wants  to  delete  the file,
FtpServProt calls

     (CtxRunning>>FtpCtx.DeleteFile)(localPL, remotePL)

which should delete the file and return true.  If something goes wrong,
it should call

     FTPM(markNo, code, string)

and return false.  Finally, FtpServProtFile calls Delete again, and the
process repeats until Delete returns false.


2.5. Directory Command

When the remote FTP User process sends the command 'Directory' followed
by a property list naming  the files about which it  wants information,
FtpServProt parses the property lists and calls

     (CtxRunning>>FtpCtx.Directory)(remotePL, localPL)

which  should decide  whether  to accept  the command.   To  refuse the
request (because for  example the requestor  does not have  the correct
access capabilities) Directory  should call FTPM(markNo,  code, string)
and  return  false.   To  accept the  command  it  should  return  a PL
describing a file.

Property lists  in directory  requests may  specify multiple  files, so
FtpServProt will continue to call Directory until it returns false.  If
Directory  supports  multiple  file requests  then  it  must  save some
information so that the next time FtpServProt calls it, it can generate
the PL for the next file.  If Directory does not support  multiple file
requests then it should do its thing during the first call and remember
that it is finished.  The next time it is called it should return false
having only deallocated localPL (it should not call FTPM).


Pup FTP Package            December 12, 1981                          9




2.6. Rename Command

When the remote FTP User process sends the command 'Rename' followed by
two property lists describing the old and new files, FtpServProt parses
the property lists and calls

     (CtxRunning>>FtpCtx.Rename)(oldPL, newPL)

which should decide  whether to accept  the command.  The  FTP protocol
does not require that user  access information be present in  newPL, so
access checking  should be done  on oldPl only.   To refuse  the rename
request,  Rename  should  call FTPM(markNo,  code,  string)  and return
false.   Otherwise  it  should  rename  the  file  returning   true  if
successful.  If the rename operation fails, Rename should call

FTPM(markNo, code, string)

and return false.


2.7. Mail Protocol

File FtpServProtMail.br implements the server part of the Mail Transfer
Protocol.  This description ignores various critical sections and other
vital  considerations  which  must  be  handled  by  the  user-supplied
routines  in  order  to  provide  a  reliable  mail  service.   For the
semantics of the protocol see [Maxc]MailTransfer.press.


2.8. StoreMail Command

When  the  remote  FTP  User  process  sends  the  command 'StoreMail',
FtpServProt parses  the property  lists which follow  and for  each one
calls

     (CtxRunning>>FtpCtx.StoreMail)(remotePL)

which should return  true or false.  Returning  true has nothing  to do
with whether the mailbox is  valid, it just indicates that  the command
exchange may  continue.  If  the mailbox  is invalid,  StoreMail should
call   FTPM(markMailboxException,  code,   string)  and   return  true.
Returning false terminates  the exchange: StoreMailFile is  skipped and
StoreMailCleanup is  called.  StoreMail  is called with  a zero  PL the
last time  so that  it may  reply No and  return false  if none  of the
mailboxes are valid.

If StoreMail always returns true, FtpServProt tells the User process to
go ahead and send the mail, and then calls

     (CtxRunning>>FtpCtx.StoreMailMessage)()

to  transfer  the file  data.   When StoreMaiMessage  has  finished the
transfer, it should  return true if  everything went OK.   If something
went wrong, it should call

FTPM(markNo, code, string)

and return false.  Finally, FtpServProt calls

     (CtxRunning>>FtpCtx.StoreMailCleanup)(ok)


Pup FTP Package            December 12, 1981                         10




where 'ok'  is true  if StoreMailMessage returned  true and  the remote
User  indicated   that  everything   went  ok.    If  'ok'   is  false,
StoreMailCleanup should not deliver  the mail.  Note that  if StoreMail
is ever called, StoreMailCleanup is always called, but StoreMailMessage
may not be.   If StoreMail allocates any  resources (such as  opening a
file) they should be deallocated here.


2.9. RetrieveMail Command

When  the  remote  FTP  User  process  sends  the  command RetrieveMail
followed by a property list describing the mailbox,  FtpServProt parses
the property list into 'remotePList' and then enters a loop:

         First FtpServProt calls

              (CtxRunning>>FtpCtx.RetrieveMail)(remotePL, localPL)

         which should return  a PL describing  the next message  in the
         mailbox.  If there are no more unread messages in the mailbox,
         RetrieveMail should  return zero.  On  each call,  remotePL is
         the same original PL sent from the remote User, and localPL is
         the last PL returned by RetrieveMail, which should be freed by
         the  client.   On  the   first  call  localPL  is   zero.   If
         RetrieveMail returns a PL, FtpServProt calls.

              (CtxRunning>>FtpCtx.RetrieveMailMessage)(remotePL,
         localPL)

         which should transfer the file and return true.   If something
         goes  wrong,  it  should call  FTPM(markNo,  code  string) and
         return false.

Finally, FtpServProt calls

     (CtxRunning>>FtpCtx.RetrieveMailCleanup)(remotePL, ok).

If 'ok' is true, then RetrieveMailCleanup should flush the mailbox.  If
this  operation  fails,  RetrieveMailCleanup  should  call FTPM(markNo,
code, string) and  return false, otherwise  it should return  true.  If
any  resouces  were  allocated  during  the  command,  they  should  be
deallocated here.



3. User


The  FTP  User module  (files  FtpUserProt.br,  FtpUserProtFile.br, and
FtpUserProtMail.br) implements the User protocol exchanges.

Many of  the procedures in  this module report  results by  returning a
word containing an FTP mark code in the right byte and a subcode in the
left byte (referred to  below as 'subcode,,mark').  Marks  and subcodes
are the first two arguments to the FTPM procedure which is described in
more detail in the Utility section.  If the mark type is  'markNo', the
subcode describes the reason  why the Server refused; your  modules may
be able  to fix the  problem and retry  the command.  The  package will
output to dspStream text accompanying No, Version, and Comment marks.


Pup FTP Package            December 12, 1981                         11




3.1. Common User Protocol

File  FtpUserProt.bcpl contains  routines shared  by FtpUserProtFile.br
and FtpUserProtMail.br.  It  uses the bspStream, bspSoc,  and dspStream
fields in its FtpCtx and contains the following external procedures:

UserOpen(Version) = true|false
    UserOpen should  be called  after the BSP  Connection is  open.  It
    sends a version command  and aborts the connection  returning false
    if the Server's protocol is incompatible.  Otherwise it calls

         Version(stream, nil)

    which should generate some herald text:

         Wss(stream, "Alto Pup FTP User 1.13, 4 June 78").

    The herald string received from the Server is output to dspStream.

UserClose(abortIt)
    UserClose closes  the FTP connection,  aborting it if  'abortIt' is
    true.

UserFlushEOC() = true|false
    flushes bspStream up to the next command, and returns true if it is
    EndOfCommand.  If the stream closes or times out, it returns false.
    It calls UserProtocolError if it encounters anything except an EOC.

UserGetYesNo(flushEOC) = subcode,,mark
    flushes bspStream up  to the next command,  which must be  'Yes' or
    'No'.  If flushEOC is true, it then calls UserFlushEOC  and returns
    the Yes or No mark and accompanying subCode.  If the  stream closes
    or   times   out,    it   returns   false.     UserGetYesNo   calls
    UserProtocolError  if  it  encounters  anything  except  Yes  or No
    followed by EOC.

UserProtocolError()
    Writes an error  message to dspStream  and then calls  UserClose to
    abort the connection.


3.2. User File Operations

File FtpUserProtFile.br implements the User protocol for  standard file
operations.  It uses the bspStream, bspSoc, and dspStream fields in its
FtpCtx and contains the following external procedures:

UserStore(localL, StoreFile) = subcode,,mark
    Attempts  to send  the  file described  by 'localL'  to  the remote
    Server, calling the user-supplied procedure 'StoreFile' to transfer
    the data.  It returns zero if something catastrophic  happens (such
    as the  Server aborts  the connection), in  which case  retrying is
    probably futile.

    UserStore  sends PL  to the  Server for  approval.  The  Server can
    refuse the command at  this point, in which case  UserStore returns
    subcode,,markNo.  If the Server  accepts the command, it  returns a
    PL (remotePL) specifying the destination file, and UserStore calls

         StoreFile(localL, remotePL)


Pup FTP Package            December 12, 1981                         12




    which  should  transfer  the  file  data.   This  package  provides
    procedures for transferring  data from a  disk stream to  a network
    stream,  but you  are  free to  write your  own.   StoreFile should
    return   true  if   the  transfer   went  successfully.    If  some
    environment-specific  thing goes  wrong (such  as  an unrecoverable
    disk error), StoreFile should call FTPM(markNo, code, string, true)
    and return false.  UserStore  then asks the Server if  the transfer
    went successfully and returns subcode,,mark.  If mark is 'markYes',
    the file arrived at the Server safely.

UserRetrieve(localPL, Retrieve) = subcode,,mark
    Attempts to retrieve the file described by localPL from  the remote
    Server,  calling  the  user-supplied  procedure  'RetrieveFile'  to
    transfer the data.  UserRetrieve returns zero if  some catastrophic
    error  occurs,  markNo  if  the  Server  refuses  the  command, and
    markEndOfCommand if the everything goes OK.

    UserRetrieve sends  localPL to the  Server and waits  for approval.
    The Server  can refuse  the command  at this  point, in  which case
    UserRetieve  returns  subcode,,markNo.  If  the  Server  can handle
    property  lists that  specify  multiple files,  then  the following
    steps are taken for each file:

         If the Server has no more files matching localPL, UserRetrieve
         returns  subcode,,markEndOfCommand  (subcode  is  undefined in
         this  case).   Otherwise the  Server  sends  a fully-specified
         property list describing a  file which it is willing  to send.
         UserRetrieve parses this into remotePL and calls

              Retrieve(remotePL, localPL)

         which should decide whether  to accept the file.  To  skip the
         file, Retrieve should  return false.  UserRetrieve  so informs
         the  Server  and then  loops.   To accept  the  file, Retrieve
         should  return  a  procedure which  UserRetrieve  can  call to
         transfer  the  data.  Don't  open  the file  yet,  because the
         Server can  still back out,  in which case  UserRetrieve skips
         the  next  step and  just  loops.  If  Retrieve  returns true,
         UserRetrieve tells the Server to send the file and then calls

              RetrieveFile(remotePL, localPL)

         which should open the  file, transfer the data, and  close the
         file.  This package contains procedures for  transferring data
         from a network  stream to a disk  stream, but you are  free to
         write your own.  When  RetrieveFile is done, it  should return
         true   if  everything   went  OK,   or  false   after  calling
         FTPM(markNo,   code,   string)   if   something   went  wrong.
         UserRetrieve then loops.

UserDelete(localPL, Delete) = subcode,,mark
    Requests  the  remote  Server  to  delete  the  files  described by
    localPL,  calling  the  user-supplied  procedure  DeleteFile before
    allowing  the  server  to  actually  delete  anything.   UserDelete
    returns  zero  if some  catastrophic  error occurs,  markNo  if the
    Server refuses the command, and markEndOfCommand if  the everything
    goes OK.

    UserDelete sends localPL to the Server and waits for approval.  The


Pup FTP Package            December 12, 1981                         13




    Server  can  refuse  the  command  at  this  point,  in  which case
    UserDelete  returns  subcode,,markNo.   If  the  Server  can handle
    property  lists that  specify  multiple files,  then  the following
    steps are taken for each file:

         If the Server has  no more files matching the  original pList,
         UserDelete  returns subcode,,markEndOfCommand.   Otherwise the
         Server sends a fully-specified property list describing a file
         which it  is willing to  delete.  UserDelete parses  this into
         remotePL and calls

              Delete(remotePL, localPL)

         which  should  return  true  to  confirm  deleting   the  file
         described by  remotePL.  UserDelete passes  this answer  on to
         the Server and then loops.

UserDirectory(localPL, Directory) = subcode,,mark
    Requests the remote Server to describe in as much detail as  it can
    files matching localPL, and then calls the  user-supplied procedure
    Directory when the answers come back.

    UserDirectory sends localPL to the Server and waits for  an answer.
    The Server  can refuse  the command  at this  point, in  which case
    UserDirectory returns  subcode,,markNo.  If  the Server  can handle
    property  lists that  specify  multiple files,  then  the following
    steps are taken for each file:

         If   the  Server   has   no  more   files   matching  localPL,
         UserDirectory  returns  subcode,,markEndOfCommand.   Otherwise
         the Server  sends a property  list which  UserDirectory parses
         into remotePL and calls

              Directory(remotePL, localPL)

         and then loops.


3.3. User Mail Operations

File FtpUserProtMail.br implements the  user part of the  Mail Transfer
Protocol.  This description ignores various critical sections and other
vital  considerations  which  must  be  handled  by  the  user-supplied
routines  in  order  to  provide  a  reliable  mail  service.   For the
semantics of the protocol see MailTransfer.ears.

UserStoreMail(pListGen, ExcpHandler, Xfer)
    Attempts  to send  mail to  the mailboxes  described by  the pLists
    generated by pListGen.   It returns zero if  something catastrophic
    happens (such as the  Server aborts the connection), in  which case
    retrying is probably futile.

    UserStoreMail  repeatedly   calls  the   client-supplied  procedure
    pListGen which should supply a pList describing a recipient  of the
    message.   When the  last  recipient has  been  generated, pListGen
    should  return zero.   The Server  can refuse  the command  at this
    point, in which case UserStoreMail returns subcode,,markNo.  If the
    Server accepts  the command,  it may  still object  to some  of the
    mailboxes, in  which case  UserStoreMail calls  the client-supplied
    procedure


Pup FTP Package            December 12, 1981                         14




         ExcpHandler(subcode, index)

    which  should record  the  recipient as  rejected.   Recipients are
    numbered from  one, in the  order in which  they were  generated by
    pListGen.  Index is the number of the rejected recipient.  A string
    describing    why    the    recipient    was    rejected    is   in
    FtpCtx.getCmdString.

    If after rejecting any recipients, there are still some valid ones,
    UserStoreMail calls the client-supplied procedure

         Xfer()

    which should transfer the message text.  Xfer should return true if
    the transfer went successfully.  If some environment-specific thing
    goes wrong (such as an unrecoverable disk error), Xfer  should call
    FTPM(markNo,   code,   string,   true)   before   returning  false.
    UserStoreMail   then  asks   the  Server   if  the   transfer  went
    successfully.  The server can  reject some more recipients  at this
    point,  in  which  case  UserStoreMail  calls  the  client-supplied
    procedure   ExcpHandler  again.    Finally   UserStoreMail  returns
    subcode,,mark.   If  mark is  'markYes',  the mail  arrived  at the
    Server safely.

UserRetrieveMail(pList, RetrieveMail) = subCode,,mark
    Attempts  to  retrieve the  contents  of the  mailbox  described by
    'pList' from the remote Server, calling the user-supplied procedure
    'RetrieveMail' to transfer the data.  UserRetrieveMail returns zero
    if some catastrophic error occurs, markNo if the Server refuses the
    command, and markEndOfCommand if the everything goes OK.

    UserRetrieveMail sends pList to the Server and waits  for approval.
    The Server  can refuse  the command  at this  point, in  which case
    UserRetieveMail      returns       subcode,,markNo.       Otherwise
    UserRetrieveMail calls

         RetrieveMail(pList)

    which should transfer the file data.  When RetrieveMail is done, it
    should return true if everything went OK.



4. Utility Routines


The  utility  module  (files  FtpUtilB.br,   FtpUtilA.br,  FtpUtilXfer,
FtpUtilDmpLd, and FtpUtilInit.br) contains protocol routines  shared by
the  User  and  Server  modules,  and  some  routines  for  efficiently
manipulating disk streams.

InitFtpUtil()
    builds  some  internal  tables  and  streams,  getting  space  from
    sysZone.  You must call this procedure before starting a  Server or
    issuing any User commands.

FTPM(mark, subCode [0], string [], eoc [false], par0, par1, par2, par3,
    par4)
    sends the FTP command  'mark' to the remote FTP  process, including


Pup FTP Package            December 12, 1981                         15




    'subCode'  if the  command  requires one,  and 'string'  if  one is
    present.  Then, if 'eoc' is true, an EOC command is sent.  'String'
    is written to bspStream using the Template package, and may contain
    imbedded format information.   'Par0' through 'par4' are  passed as
    arguments  to  the  PutTemplate  call.   The  subcode   and  string
    arguments further explain certain commands.  For markNo, subCode is
    a machine-readable  explanation of why  a request was  refused, and
    'String'  is human-readable  text  such as  "UserName  and Password
    required".     Codes   are    tabulated   in    an    appendix   to
    FtpSpec.ears.  New codes may be registered on request.

GetCommand(timeout [-1]) = subCode,,mark
    flushes bspStream up to the  next command and returns the  mark and
    subcode (if any).  Returns false  if the stream closes or  it hangs
    for 'timeout'  miliseconds while waiting  for a byte  (the default,
    -1,  waits  forever).  Comment  commands  are  ignored.  GetCommand
    writes the strings  accompanying Version, No, and  Comment commands
    to dspStream and stores a pointer to them in FtpCtx.getCmdString.

The utility  module makes three  'process-relative streams' for  use by
the rest of the package.  The only operation defined is 'Puts'.

    lst       writes to dspStream
    dls       writes to dspStream if debugFlag is true
    dbls      writes to bspStream and if debugFlag to dspStream


For example,  Wss(dls,string) writes 'string'  to the  running process'
dspStream if the process' debugFlag is set.


4.1. Unformatted Data Transfer

File  FtpUtilXfer.br  contains  procedures  for   performing  efficient
operations on Alto OS disk  Streams.  They use the following  fields in
FtpCtx:   bspSoc,   bspStream,  dspStream,   diskStream,   buffer,  and
bufferLength.

DiskToNet(remotePL, localPL) = true|false
    Transfers bytes from diskStream to bspStream up to end-of-file, and
    returns true if everything went OK.

NetToDisk(remotePL, localPL) = true|false
    Transfers bytes  from bspStream to  diskStream until  it encounters
    another FTP command returning true if everything went OK.

FileType() = Text|Binary
    Resets diskStream,  scans it  looking for high  order bits  on, and
    then Resets  it again.  As  soon as it  encounters a byte  with the
    high order bit on, it returns 'Binary', otherwise (having  read the
    entire file) it returns 'Text'.

PrintBegin(localPL)
    Outputs the server filename in remotePL and the type and  byte size
    from localPL to dspStream.


Pup FTP Package            December 12, 1981                         16




4.2. Dump Format Data Transfer

File FtpUtilDmpLd.br contains procedures for transferring  data between
a disk and an FTP connection  in dump format.  They may be used  as the
inner  loops  of  user-supplied  data  transfer  procedures  passed  to
UserStore  and UserRetrieve  and will  create and  unbundle dump-format
files on the fly.  If you  don't want to handle dump format,  you don't
need this file.   Dump-file format is described  in an appendix  to the
Alto Executive documentation.

These procedures  use the same  fields in FtpCtx  and the same  Alto OS
routines as the unformatted transfer routines.  Buffer must be at least
130 words long.  Making it longer does not speed up the transfer.

DumpToNet(remotePL, localPL) = true|false
    Moves a  file from  diskStream to bspStream  converting it  to dump
    format, returning true if things go OK.  The filename is taken from
    the  name-body field  of localPL,  and the  creation date  from the
    creation date field.  To terminate a dump file, call DumpToNet with
    remotePL = 0.

LoadFromNet(remotePL, localPL) = true|false
    Moves a file from  bspStream to diskStream converting it  from dump
    format, returning false when it encounters an 'end block'.  When it
    encounters a file, it  returns true with the filename  and creation
    date in  remotePL.  If the  client wants the  file, he  should call
    LoadFromNet again with FtpCtx.diskStream  non zero; to skip  a file
    set diskStream to zero.


4.3. Binary Compare Data Transfer

Files FtpUtilCompB.br and FtpUtilCompA.br implement a binary compare of
a network stream and a disk stream.  If you don't want to do  this (not
many  people  will,  I  suspect),  then  you  don't  need  these files.
FtpUtilCompA contains two Block comparison procedures: one uses  a fast
machine code loop and the  other uses special microcode which  you must
load into the Alto's ram.

CompareNetWithDisk(remotePL, localPL) = true|false
    Compares  diskStream with  bspStream byte-by-byte  and  reports the
    results to dspStream.   If the two  streams are identical  (and the
    same length), then  a string of the  form "xxx identical  bytes" is
    output, otherwise a  string of the  form "First difference  at byte
    pos xxx" is output.  Returns  true if everything went OK,  false if
    something catastrophic happened to the network connection  (note in
    particular that a result of true implies nothing about  whether the
    two streams were identical).

    The  following fields  in  the FtpCtx  must be  set  up: bspStream,
    dspStream, diskStream, buffer, and bufferLength.



5. Property Lists


The  property  list module  (files  FtpPListProt.br,  FtpPList1.br, and
FtpPListInit.br)   translates    between   this    package's   internal


Pup FTP Package            December 12, 1981                         17




representation of  a property list  and the  protocol-specified network
representation.

The FTP protocol specifies the syntax of a property list and the syntax
of a  set of  properties sufficient for  standard file  operations, but
states that property lists are extensible.  Therefore the property list
module comes  in two parts:  a part that  knows the syntax  of property
lists, and a part which knows the syntax of individual  properties.  To
add new properties you need only modify the latter.

The  principal  data structure  in  this module  is  the  Property List
Keyword Table, or pListKT.  This table, built by InitFtpPlist, contains
(propertyName, propertyObject)  pairs.  PropertyNames are  strings such
as "Byte-size".   PropertyObjects know how  to Scan  (parse) properties
into  pLists, Generate  properties from  pLists,  Initialize properties
from a  pList full  of default  values, and  Free properties  stored in
pLists.


5.1. Property List Protocol

File  FtpPlistProt.br  implements four  operations  on  property lists.
This is the module  that knows the syntax  of a property list,  but not
the syntax of individual  properties.  Procedures in this file  use the
bspStream, bspSoc, and dspStream  fields of the FtpCtx and  contain the
following external procedures:

InitPList(defaultPL []) = PL
    Creates  an  empty  pList,  and initializes  it  to  be  a  copy of
    'defaultPL' if one was supplied.

FreePList(PL)
    Destroys PL and returns  0 to facilite writing PL  = FreePList(PL).
    If PL is zero, FreePList returns zero without doing anything.

ScanPList() = PL|false
    Expects to  find a  property list  in bspStream.   ScanPList parses
    this property list  and returns a PL  if it had proper  syntax.  If
    the property list is malformed, ScanPList calls  FTPM(markNo, code,
    string) and returns false.   If ScanPList encounters a  mark before
    starting  a PL  or  the connection  closes  or Gets  times  out, it
    returns false.

GenPList(PL)
    Generates a property list in network format from PL and sends it to
    bspStream.


5.2. The 'Standard' Properties

Files   FtpPlist1.br   and  FtpPlistInit.br   implement   the  standard
properties.  These files know the syntax of individual properties; they
contain  the operation  procedures for  the standard  property objects.
These files are  used by the FTP  subsystem and IFS and  are sufficient
for  performing  'standard'  file  operations.   If  you  wish  to  add
properties, these are the  modules which you must change.   In addition
to the property operations which are rather specialized to  their task,
there are a few generally useful procedures which are made external:

InitFtpPList()


Pup FTP Package            December 12, 1981                         18




    which makes the standard property objects and builds fplKT, getting
    space from sysZone.  This  procedure must be called  before calling
    any of the procedures in FtpPlist.br (which typically  means before
    starting a server or calling any procedures in the User module).

Nin(string, lvDest) = true|false
    Interprets 'string' as  a decimal number  and leaves the  result in
    'lvDest', ignoring leading blanks and terminating on end of string.
    A null string  results in lvDest getting  0.  Returns false  if the
    string contains any characters other than 0-9 and .

ParseDate(string, lvRes) = true|false
    Parses the  string format date  into an Alto  format date  which it
    puts into the two word vector at 'lvRes'.  Returns true if it could
    parse the date.  ParseDate expects the format of the string to bear
    some similarity to "day-month-year hour:minute:second".

WriteDT(stream, dt)
    converts 'dt' from 32 bit Alto date format to a string of  the form
    "dd-mmm-yy hh:mm:ss" and writes it to 'stream'.



6. Example


The following example  program makes use of  most of the  facilities in
the User part of the Ftp Package.  I have run it and it works.  It is a
rock-bottom  minimal User  Ftp with  no redeeming  features whatsoever.
More extensive and  realistic examples can be  found by looking  at the
sources for the Ftp subsystem.

The  main  procedure  FtpUserExample  performs   initialization,  which
consists of augmenting SysZone, initializing the Ftp and  Pup packages,
and creating and starting a context running the procedure 'User'.

User opens a BSP connection to Maxc, sets up its FtpCtx, gets and fills
a blank pList, and calls UserRetrieve.  When UserRetrieve returns, User
closes the connection, releases its resources and commits suicide.
//FtpUserExample.bcpl - Example Ftp User

//last modified April 9, 1978  4:24 PM

// The load command file is:
// Bldr/l/v 600/W FtpUserExample ↑
// ↑
// FtpUserProt FtpUserProtFile ↑
// FtpPListProt FtpPList1 ↑
// FtpUtilb FtpUtila FtpUtilXfer ↑
// ↑
// PupBspOpenClose PupBspStreams PupBspProt PupBspBlock PupBspA ↑
// PupRtpOpenClose PupRtp PupNameLookup ↑
// Pup1OpenClose Pup1B PupAl1A PupRoute PupDummyGate ↑
// PupAlEthB PupAlEthA ↑
// ↑
// Context ContextInit Interrupt ↑
// AltoQueue AltoTimer AltoByteBlt ↑
// Template CTime StringUtil Keyword ↑
// ↑
// FtpPlistInit FtpUtilInit KeywordInit ↑


Pup FTP Package            December 12, 1981                         19




// Pup1Init PupAlEthInit InterruptInit

get "FtpProt.decl"
get "Pup.decl"

external
[
//incoming procedures
InitFtpUtil; InitFtpPList; InitPupLevel1
GetFixed; CallSwat; AddToZone; Allocate; Free
InitializeContext; CallContextList; Enqueue
GetPartner; OpenLevel1Socket; OpenRTPSocket; CreateBSPStream
InitPList; FreePList; NetToDisk
UserRetrieve; UserOpen; UserClose; NetToDisk
ExtractSubstring; OpenFile; Closes; Wss

//incoming statics
sysZone; dsp; CtxRunning; UserName; UserPassword
]

let FtpUserExample() be
[
let v = GetFixed(10000)
if v eq 0 then CallSwat("GetFixed failed")
AddToZone(sysZone, v, 10000)
let ctxQ = vec 1; ctxQ!0 = 0
InitFtpUtil()
InitFtpPList()
InitPupLevel1(sysZone, ctxQ, 10)
Enqueue(ctxQ, InitializeContext(Allocate(sysZone, 1000), 1000,
 User, lenExtraCtx))
CallContextList(ctxQ!0) repeat
]

and User(ctx) be  //a context
[
let soc = Allocate(sysZone, lenBSPSoc)
let maxcPort = vec lenPort
unless GetPartner("Maxc", dsp, maxcPort, 0, socketFTP) do
   CallSwat("GetPartner failed")
OpenLevel1Socket(soc, 0, maxcPort)
unless OpenRTPSocket(soc) do
   CallSwat("OpenRTPSocket failed")

CtxRunning>>FtpCtx.bspStream = CreateBSPStream(soc)
CtxRunning>>FtpCtx.bspSoc = soc
CtxRunning>>FtpCtx.dspStream = dsp
CtxRunning>>FtpCtx.buffer = Allocate(sysZone, 256)
CtxRunning>>FtpCtx.bufferLength = 256
CtxRunning>>FtpCtx.debugFlag = true
unless UserOpen(Version) do
   CallSwat("UserOpen failed")

let localPL = InitPList()
localPL>>PL.UNAM = ExtractSubstring(UserName)
localPL>>PL.UPSW = ExtractSubstring(UserPassword)
localPL>>PL.SFIL = ExtractSubstring("Pup-Network.txt")

let mark = UserRetrieve(localPL, Retrieve)
if mark ne markEndOfCommand then


Pup FTP Package            December 12, 1981                         20




   CallSwat("UserRetrieve failed")
FreePList(localPL)
UserClose()
Free(sysZone, soc)
Free(sysZone, CtxRunning>>FtpCtx.buffer)
finish
]

and Version(stream, nil) be Wss(stream, "Example FTP User")

and Retrieve(remotePL, localPL) = RetrieveFile

and RetrieveFile(remotePL, localPL) = valof
[
let s = OpenFile(remotePL>>PL.NAMB, ksTypeWriteOnly, charItem)
CtxRunning>>FtpCtx.diskStream = s
unless NetToDisk(remotePL, localPL) do CallSwat("NetToDisk failed")
Closes(s)
resultis true
]



7. Revision History


March 30, 1977

First release.

May 15, 1977

Added Directory and Rename commands.  Server now handles property lists
which specify multiple files.  Added User and Server mail operations.

June 8, 1977

Overlay machinery was changed and some bugs were fixed.  Some structure
definitions changed, so recompilation of user programs is necessary.

July 17, 1977

DiskToNet  and  NetToDisk  moved  out  of  FtpUtilb  into  a  new  file
FtpUtilXfer.   Property  lists  reorganized,  causing  changes  to  the
calling interface in FTPSI.  Plist module now uses the Keyword routines
in the CmdScan package.   Recompilation of user programs  is necessary.
FtpUserDmpLd renamed FtpUtilDmpLd.  Timeouts cleaned up.

October 24, 1977

Example program added.

February 14, 1978

Files  FtpUtilCompB  and  FtpUtilCompA,  implementing   a  byte-by-byte
compare of the net stream with a disk stream added.

April 9, 1978


Pup FTP Package            December 12, 1981                         21




Implemented  the  new form  of  Store  in which  the  Server  returns a
property list specifying the  destination file.  The old form  is still
supported, but no longer documented.

June 1, 1978

FtpServProt.bcpl      split      out      of      FtpServProtFile.bcpl.
FtpServProtMail.bcpl updated to the current MTP.  Many  data structures
changed so recompiliation of user programs is necessary.

September 20, 1980

Parameters passed to client routines changed.  Both property  lists are
passed now.  Recompilation is necessary.

December 16, 1980

Timeouts  reworked.  Statics  'getCmdTimeout' and  'getPutTimeout', and
their    default     values,    manifests     'defGetCmdTimeout'    and
'defGetPutTimeout'  were  removed,  since  Pup  byte   stream  activity
timeouts, added  about a year  ago, do the  same job.   FtpServProt now
takes a timeout which it uses while waiting for top level commands.

December 12, 1981

Internal data structure rearranged.  Recompilation is necessary.