Loading

$Revision: 5.0.2.8 $

The document introduction.htm provides an overview of the Allegro CL documentation with links to all major documents. The document index.htm is an index with pointers to every documented object (operators, variables, etc.) The revision number of this document is below the title. These documents may be revised from time to time between releases.

1.0 Using the load function
     1.1 EOF encountered error during a load
     1.2 Special utilities for loading applications
2.0 Search lists
     2.1 Search List Structure
     2.2 Search lists variables
     2.3 Search list example
3.0 The Allegro Presto algorithm
     3.1 Comparison to autoloading
     3.2 Improved locality of reference
     3.3 Allegro Presto: basic usage
     3.4 Allegro Presto: advanced usage
     3.5 Libfasl loading
     3.6 What does the libfasl feature apply to?
     3.7 What does the libfasl feature do?
     3.8 When is a stub function fully loaded?
     3.9 When is libfasl loading invoked?
     3.10 What are the costs and advantages of using the libfasl feature?
     3.11 The in-package restriction
     3.12 Affect on users 1: keeping track of files
     3.13 What happens if Lisp cannot find a necessary fasl file?
     3.14 What happens if the file has changed?
     3.15 Affect on users 2: eq-ness of function objects
     3.16 Libfasl and excl:dumplisp

1.0 Using the load function

In Allegro CL, cl:load has additional keyword arguments, libfasl, searchlist, and system-library, described in the next three bullets.

• The libfasl argument allows you to take advantage of the libfasl option in Allegro CL. If the value of this argument is t, all defined functions in the file being loaded will be only partially loaded and will include a pointer to the file (under certain restrictions described in 3.5 Libfasl loading). The functions will be fully loaded (automatically and invisibly to the user) only when they are called. Using this option will save space in the image (since only necessary things are loaded in) at the cost of (1) a slight (probably unnoticeable) delay the first time a function is called and (2) the requirement that filename be available as long as the Lisp image (or any dumped progeny) is running. See 3.0 The Allegro Presto algorithm,  and 3.5 Libfasl loading for more information. The value of this keyword argument defaults to the value of the variable excl:*libfasl*.
• The search-list, if specified, should be a search list which will be used instead of sys:*load-search-list*. (See 2.0 Search lists.)
• The system-library argument, if true (default is nil) tells the system that the file being loaded is a system library file. The file should be a foreign library file, that is a shared object (.so) file on most Unix platforms, a shared-library file (.sl) on HP's, or a dynamic link library (.dll) file on Windows. Specifying this argument true has no practical effect on the running Lisp image but it tells generate-application not to copy the file to the application directory (or any subdirectory) when building an application. (It is a bad idea to copy system libraries to an application directory because (1) it may be illegal to distribute them; (2) having an application use a personal copy of a system library which may be different from the version installed on the system can lead to subtle errors. For example, when CLIM is loaded on Windows, it loads the system library user32.dll with system-library t, and so user32.dll does not get copied to CLIM application directories created with generate-application.) See delivery.htm for more information.

Note that certain global variables are bound when load is loading a file. Therefore, setting those variables in an initialization file (such as .clinit.cl) will not have the desired effect. (See also the next section which discusses setting global values in an initialization file). The following table shows what variables are bound by load:

The top-level command :ld calls the load function. :ld will accept multiple filenames but it does not accept keyword arguments. :ld reads its arguments as strings and does not accept pathnames or streams as filename arguments. Therefore, to load the file test.cl in the current directory, the following :ld command will work:

:ld test.cl

There are other keyword arguments for load in Allegro CL other than those specified by the ANSI standard, such as unreferenced-lib-names, used when load loads foreign (C or Fortran) object files. See foreign_functions.htm for details of these additional keyword arguments.

1.1 EOF encountered error during a load

If you try to load a file that contains an incomplete form (because of, e.g., a missing closing parenthesis), load signals an error with condition end-of-file. Consider the following file missing-paren.cl:

(defun foo nil nil)
(defun bar (a b) (+ a b)

The closing parenthesis is missing from the definition of bar. When Lisp tries to load this file, it signals an error:

USER(1): :ld missing-paren.cl
; Loading /net/rubix/usr/tech/dm/missing-paren.cl.
Error: eof encountered on stream #<EXCL::CHARACTER-INPUT-FILE-STREAM
#p"/net/rubix/usr/tech/dm/missing-paren.cl" pos 45 @ #xa91f02>
starting at position 20.
  [condition type: END-OF-FILE]

Restart actions (select using :continue):
  0: retry the load of missing-paren.cl
[1] USER(2):

Note the line:

starting at position 20.

That indicates that the incomplete form starts at position 20 in the file. Opening the file in Emacs and entering the command C-u 20 C-f (assuming standard keybindings) should bring you to the beginning of the incomplete form.

1.2 Special utilities for loading applications

Allegro CL does provide certain utilities for loading collections of files, in particular collections of files which comprise an application. The primary utility for this purpose is the defsystem utility, described in defsystem.htm. A couple of simpler utilities are also available: the macros excl:load-application and excl:tenuring.

excl:tenuring allows evaluation of forms with all live objects tenured at the end of the evaluation. You might want to call load within that macro. See the discussion in gc.htm for more information.

excl:load-application allows you to specify the environment in which the loading takes place.

2.0 Search lists

Now that logical pathnames are available in Lisp, search lists may be obsolete. However, they are preserved for backward compatibility.

This section describes how load and require find the files they are called upon to load. Novice users, who keep the files to load in the same directory as they run Lisp need not concern themselves with the subtleties of this section. However, the facilities defined in this section are very powerful, allowing complicated mechanisms to be defined for finding files not in the current directory. If such capability will be useful to you, you should read on.

The functions cl:require and cl:load (and the top-level command :ld) read files. The Common Lisp standard specifies how the filename argument to these functions is processed to yield the true name of the file to be read: the argument is merged (using the Common Lisp function merge-pathnames) with the value of *default-pathname-defaults*. This file-finding mechanism is inadequate for all but the simplest of applications although logical pathnames may be adequate.

Addressing the need for a more flexible file-finding mechanism, Allegro CL incorporates search lists, data structures embodying information about how the functions load and require will find a file to load into Lisp. Both functions have their own search list.

Autoloading (where an action like calling a function triggers the loading of the module containing the associated functionality) also uses a search list. We do not discuss it much in this document since autoloading is usually invisible to the user.

Search lists permit the user to specify a sequence of pathnames to be merged with the filename argument to load or require. We henceforth refer to this filename argument as the supplied name. From among those pathnames resulting from the sequence of merging operations, the first pathname that satisfies the given criteria will be loaded. Further, it is possible to specify side effects, e.g. to compile a file if the source file has been modified more recently than the corresponding compiled (fasl) file. This very general mechanism provides considerable control over file-loading operations.

The libfasl utility allows functionality to be kept on disk until actually needed. It permits users to run smaller images at the cost of slightly increased time to call functionality stored on disk but that cost is paid during the first call only; section 3.5 describes the libfasl utility.

You can build a standard image. Which is controlled by the presto argument to build-lisp-image (see building_images.htm). Or you can load individual files in libfasl mode by specifying the libfasl argument to load true.

A libfasl image has most of the default Lisp functionality stored in a file called files.bu, which is called the bundle file. When you load a module (for example, inspect, foreign-functions, trace, etc.) the system must know to look in the bundle file instead of looking for the fasl file in the Lisp library. The :lib-bundle search list keyword argument provides this functionality.

2.1 Search List Structure

A search list is a recursive data structure. It may be a symbol, string, pathname, or a list of search lists. (The car of a list is interpreted specially.) The elements of a search list are processed sequentially. If the search list is recursive, it is processed in depth-first fashion. The goal of processing a search list is to locate a file to be loaded -- in effect a search list returns the pathname of a file. It is often convenient to refer to search lists as returning a pathname, especially when discussing recursive search lists. It is an error if a file cannot be located within the given constraints. The different kinds of search lists are handled as follows:

• Symbols. If the symbol is bound, then its value is interpreted as a search list. If the symbol is unbound, it is ignored. If the symbol is nil, the denoted file is the filename supplied as an argument to load or require.
• Strings and pathnames. The file is determined by merging the string or pathname with the supplied name. If search-element is a string or pathname search list, the denoted file will be

(merge-pathnames supplied-name search-element)

• Lists. The car of the list determines how the elements of the list are processed. The car may be a keyword or a search list. If it is a keyword, it must be one of :lib-bundle, :first, :newest, :newest-ask-compile , :newest-do-compile, or :call. If it is not a keyword, the list is treated as if it began with the keyword :first.

We now define the effect of the keywords that can be the car of a list search list. In the definitions below, we borrow the Common Lisp notation &rest search-list to represent to the rest of the list.

For an example using :call, a user supplied function could look like:

2.2 Search lists variables

The search list for the load function is bound to the symbol system:*load-search-list*. The default value on many platforms for this symbol is

(#p"" #p(:type "fasl") #p(:type "cl") #p(:type "lisp"))

Thus load function looks for its file first by its supplied name in the current directory. If it cannot find the file there, it looks in the same place for a file with a "fasl" file type, then with a "cl" file type, and then with a "lisp" file type. If no file is found, then the search list returns nil indicating no satisfactory file corresponding to the supplied name was found.

The search list for the require function is bound to the symbol system:*require-search-list*. The initial value for this symbol on some platforms is:

require follows the same procedure as load for finding a file, except require does look in bundle file and (on the platform where the value was taken) in code/ , sys/, winapi/ and ole/ subdirectories of the Allegro directory, and load does not. If a file exists in both the bundle and in the code/ subdirectory, then the first file encountered will be in code/ and will be used. (This ensures that a fixed fasl file which was originally part of the bundle, supplied as a patch and placed in code/, will be used instead of the version in the bundle file.)

There is also an autoload search list, the value of sys:*autoload-search-list*, which is used by the system when an autoload is triggered. Autoloads are triggered by actions such as calling a function whose functionality is not presently in the system. (Not all such function trigger autoloads. Typically only the important functions of a facility.) Autoloading is usually nearly invisible to users. Most modules are autoloaded from the bundle. The structure of the search list, however, ensures that updated fasl files which replace fasl files in the bundle are found and used correctly.

2.3 Search list example

Suppose that you maintain a personal Lisp library in a subdirectory of your home-directory, and that you wish to have the load function look for files in this subdirectory. Let us assume this subdirectory is named lisplib. It would be reasonable to search your library directory after searching your current working directory. Further, say you would like to compile automatically all files that are in your working or library directories. The following definition of system:*load-search-list* in an initialization file (such as .clinit.cl) will accomplish this. Your home directory pathname is denoted with a tilde (`~').

If you start up Lisp in the directory ~/work, the expression

(load "foo")

will result in the following pathnames being generated.

If ~/work/foo.cl exists, it will be compiled, if necessary, and ~/work/foo.fasl will be loaded. If ~/work/foo.fasl exists but ~/work/foo.cl does not, ~/work/foo.fasl will be loaded. If neither of these files exists, the next two files are examined. If ~/lisplib/foo.cl exists, it will be compiled, if necessary, and ~/lisplib/foo.fasl will be loaded. If ~/lisplib/foo.fasl exists but ~/lisplib/foo.cl does not, ~/lisplib/foo.fasl will be loaded. If neither of these files exist, an error is signaled.

Note: if you change the value of *require-search-list*, you must include

(:newest (:lib-bundle #p"sys:;code;.fasl")
         #p"sys:;code;.fasl"
         #p"sys:;code;.cl"

so that Lisp will be able to find library files such as foreign.fasl.

3.0 The Allegro Presto algorithm

Under Allegro Presto, when a fasl file (a compiled Lisp file) is loaded into Lisp, only part of the information about a compiled function is loaded. The rest is kept in the fasl file and is loaded only when it is needed (typically, when the function is actually called). The partially-loaded function is called a stub function. The act of partially loading function definitions is called libfasl loading (distinguished from ordinary loading).

When the stub function is called, the system notices that only the stub definition is resident in the image so it automatically (and transparently to the user) retrieves the call-time function info from the disk file and, conceptually, replaces the stub definition with the complete definition. (We say conceptually because the exact mechanics and timing of the replacement are more complicated.)

The headings for the remainder of this section are:

3.1 Comparison to autoloading
3.2 Improved locality of reference
3.3 Allegro Presto: basic usage
3.4 Allegro Presto: advanced usage
3.5 Libfasl loading
3.6 What does the libfasl feature apply to?
3.7 What does the libfasl feature do?
3.8 When is a stub function fully loaded?
3.9 When is libfasl loading invoked?
3.10 What are the costs and advantages of using the libfasl feature?
3.11 The in-package restriction
3.12 Affect on users 1: keeping track of files
3.13 What happens if Lisp cannot find a necessary fasl file?
3.14 What happens if the file has changed?
3.15 Affect on users 2: eq-ness of function objects
3.16 Libfasl and excl:dumplisp

3.1 Comparison to autoloading

Autoloading is an image-size-reducing mechanism available in Allegro CL. Certain fasl files are not loaded into the standard image but are instead loaded only when certain functions associated with the fasl file are called. When a function that triggers autoloading is called (e.g. cl:trace, which autoloads trace.fasl), the entire fasl file is fully loaded with the complete definitions of all defined functions. Thus a call to a single function can cause many functions to be loaded. With libfasl loading, some portion of the fasl file is brought into Lisp, including stub definitions for all defined functions, but full definitions are only loaded for those functions when they are actually called, saving the space required for the call time info of functions not called.

This benefit over autoloading is particularly important when Allegro CL is run with the Emacs-Lisp interface and/or Allegro Common Windows and Allegro Composer. Those utilities use much Lisp functionality and so cause many of the library files (including, e.g., foreign.fasl, process.fasl, and mdproc.fasl) to be loaded. Without Allegro Presto, these files are fully loaded and most of the benefits of autoloading disappear.

3.2 Improved locality of reference

The space-saving benefit of libfasl loading has just been described. A second benefit is also provided: improved locality of reference. Locality of reference refers to the likelihood that related things are placed together (the better the locality of reference, the higher the likelihood). Improved locality of reference arises because code vectors of stub functions called in conjunction with one another are loaded in close proximity. A feature of loading call-time function info is that it is immediately stored in oldspace rather than starting in newspace and being tenured to oldspace after a period of time. Therefore, the code vectors for functions called in close proximity will (in general) be stored in close proximity. This is a benefit because if they are on the same page in memory, they will always be paged in and out together. Optimal locality of reference will occur when the code vectors are on the same page in memory. That cannot, of course, be guaranteed but is evidently much more likely to occur with Allegro Presto and libfasl loading.

3.3 Allegro Presto: basic usage

The simplest way to use Allegro Presto, and the way recommended for most beginning users, is simply to specify :presto t to excl:build-lisp-image when building a Lisp image. If you do that, the system will libfasl load whatever is required during the building of the image. Further, the value of excl:*libfasl* is set to t so all subsequent loads will also be libfasl loads (unless you override the value of excl:*libfasl* in some fashion).

If you specify :presto nil, Allegro Presto will be disabled in the resulting image.

If you specify :presto :enable, Allegro Presto will be enabled but *libfasl* will initially be nil.

The function of no arguments sys:prestop returns t or nil as Allegro Presto is or is not disabled in an image.

All of this is virtually transparent to the user, who will see no reduction in functionality.

3.4 Allegro Presto: advanced usage

We have provided the information and facilities for using Allegro Presto with the standard Allegro CL features (such as Allegro Composer and the foreign function interface). But we also provide tools for improving the performance of your application by gathering information about usage and providing that information to Lisp.

Suppose you are running a Lisp application and you are aware of the Allegro Presto utility: conceptually, what would you like to have happen? We would suggest that you will want the following:

1. Have functionality that is less likely to be used stored on disk until needed.
2. Have functionality that will certainly be used included in the image (to avoid paying the cost of fully loading functions you know will be called).

The simple use of Allegro Presto described above provides # 1: the fasl files that make up your application will be loaded in libfasl mode, which means that all functions (frequently or rarely used) will start as stubs and be fully loaded only when necessary. The time of loading will usually improve locality of reference.

So, how to get #2? What is necessary is to tell Lisp what functions will certainly be called and so should start fully loaded (thus saving the overhead required to resolve stub functions).

For example, consider Allegro Composer as an application. It is certain that anyone who uses Allegro Composer will call composer:start-composer (since that function actually starts the utility). Therefore, there is no point in partially loading that function since it will quickly be fully loaded anyway.

3.5 Libfasl loading

We have already defined libfasl loading above, but here again is a brief definition: when a compiled file is loaded in libfasl mode, only part of the definition of each function is loaded. The partially loaded function is called a stub function and it contains a pointer to the file where the remainder of the definition resides. When the function is actually called, the remainder of the function definition is loaded (automatically and transparently to the user). Thus, libfasl loading allows a user to keep some part of Lisp functionality on disk until it is actually needed.

In this and the following sections, we discuss details and consequences of libfasl loading, covering a number of topics identified in large headings in bold type.

3.6 What does the libfasl feature apply to?

The libfasl feature provides an alternate way of loading fasl (compiled Lisp) files. Therefore, it only applies to compiled functions defined in a fasl file (this includes library files). Functions defined and compiled within Lisp are not affected by the libfasl feature (since they are not loaded) and functions defined in Lisp source files which are loaded into Lisp are not affected by the libfasl feature (since they are not compiled). In order to avoid introducing package errors, only functions defined after an in-package form within a fasl file are libfasled.

3.7 What does the libfasl feature do?

When a function in a compiled file is loaded in libfasl mode, only part of the compiled function is loaded. A compiled function is composed of various parts, including, for example, the code vector, where the code executed when the function is called is stored. In libfasl mode, the smallest amount of information possible is loaded. Included in that information is the name of the file which contains the remainder of the complete function. The documentation string (if there is one) is also always available without accessing the file containing the rest of the function.

A partially loaded function is also called a stub function. When a partially loaded function is actually called, all remaining parts are loaded. A function that has all its parts loaded is said to be fully loaded. The process of loading the remainder of a stub function is called resolving the call-time information or fully loading.

3.8 When is a stub function fully loaded?

The following actions will cause the system to fully load a function definition:

1. Calling the function.
2. Calling excl:resident-function with the function object as the argument.
3. Getting the argument list of the function with excl:arglist. (Note that excl:arglist is called by other functions, e.g. describe, and by various Emacs commands.)
4. Compiling a call to the function. (This occurs because the argument list is needed by the compiler.)
5. Disassembling the function.
6. Writing the function as a constant out to a fasl file.

Applications (including CLIM and Allegro Composer) can fully load stub functions at any time of course.

3.9 When is libfasl loading invoked?

When the value of the :libfasl argument to load is true and load is loading a compiled file, functions will be loaded in libfasl mode (except see the paragraph headed 3.11 The in-package restriction below). If no value is specified for the :libfasl argument, it defaults to the value of  excl:*libfasl*. If :presto is specified true to excl:build-lisp-image, the initial value of that variable is t.

Several other functions cause load to be called including require, excl:load-system, and the top-level commands :ld and :cload. None of these functions or commands have the mechanism to specify whether loading should be in libfasl or normal mode, so the mode is determined by the value of excl:*libfasl*.

3.10 What are the costs and advantages of using the libfasl feature?

The costs are (1) a slightly increased time to execute functions the first time (but only the first time) they are called and (2) the necessity to ensure that the files where the unloaded functionality is stored are available to Lisp (and any descendent images made with excl:dumplisp).

The advantage of libfasl loading is that information which may not be necessary will not be loaded until needed. This allows for a smaller image size than otherwise without reducing functionality (reductions in image size of between 25% and 40% are normal, 50% has been seen).

This feature is particularly useful when loading a large file since in fact only some of the functions defined in the file may ever be called and space is saved by not loading parts of the functions which are never called. A further advantage is an improvement in locality of reference since code vectors used near one another will be loaded closely together and are thus likely to be located close to one another.

3.11 The in-package restriction

In a given source file, only functions defined after an in-package form will be loaded in libfasl mode. Functions whose definitions are located in the source file before an in-package form will always be fully loaded. (Therefore, all functions in files that do not contain an in-package form will be fully loaded.) This restriction guards against package errors and confusion.

3.12 Affect on users 1: keeping track of files

In general, operation of libfasl loading should be transparent to the user. The additional time required to fully load a function the first time it is called is not significant and may not even be noticeable. However, there are two issues that should concern users: location of fasl files (discussed here) and eq-ness of function objects (discussed below).

The location of the file containing the call-time function information for a function is stored in the loaded portion of the function definition. When the function is called, the system looks for that file to load the remainder of the function. At that time, the file must be exactly where the system expects it (the location of the fasl file which was loaded in libfasl mode), or an error will be signaled. See also the information under the heading 3.14 What happens if the file has changed?

Note that the file location is stored as an absolute pathname (regardless of how the file was specified to Lisp). Therefore, the current directory can be changed (with excl:chdir) without affecting the ability to find the files needed to resolve stub functions. However, this fact also affects excl:dumplisp. See dumplisp.htm for a discussion of that point. See sys:presto-build-lib which bundles all information in one file. That may be useful when dumping an image intended for a different machine.

3.13 What happens if Lisp cannot find a necessary fasl file?

If Lisp cannot find the file pointed to by a stub function, it clearly cannot continue computation since it has no code vector to execute. Therefore, Lisp signals an error. The error is similar to the following:

Note that if the file which cannot be found is files.bu (which holds the code vectors of the base Lisp system), the failure may be fatal since Lisp may not be able to find the call-time function info necessary to process the error, resulting in a recursive error. However, we believe that files.bu (and other Lisp library files) will be available while a Lisp image is running except under extraordinary circumstances. It is definitely a fatal error to remove such files while Lisp is running.

3.14 What happens if the file has changed?

If you change a compiled file which has already been libfasl loaded, (by, e.g., changing and compiling the source file) you must reload the changed compiled file in order to update the pointers within Lisp to the function definitions. The top-level command :cload, which does both a compile and a load, is very useful for ensuring that you have always loaded the most recently compiled file.

When a file is libfasl loaded, a pointer stored in the stub function points to the location in the file where a special code identifies the location as an appropriate location to begin loading the remainder of the function. If the pointer is found to point to a location that does not contain that special code, an error is signaled. The error is the same as shown above.

If the file exists, it is most likely to have been changed (corruption is possible but, presumably, rare). If the function is still defined in the file, it is most likely that the file was not reloaded and reloading will allow the function call to succeed. If the function is no longer defined in the file, reloading will (of course) make no difference and in that case, either calling the function in the first place was an error or the file where the function is now defined should be loaded.

Note that it is possible that after a file has been changed but not reloaded, the pointer in the stub function points to a location with the special code even though it no longer points to the correct definition (that is, it points to the definition of another function). We believe that this will be rare. When it occurs, however, Lisp will try to execute the wrong function code, likely leading to other errors (e.g. the wrong number of arguments passed) or strange results. This may lead to confusion and users should be aware of the possibility. This is an unavoidable problem. Providing sufficient checks to avoid this unlikely situation would so increase the cost of the libfasl feature that the benefits would be lost.

3.15 Affect on users 2: eq-ness of function objects

A stub function is not the same object as a fully loaded function. Suppose that a function foo is only partially loaded and another function, bar, calls foo. The function bar will contain a pointer to the stub definition of foo. Now suppose that foo is called independently of bar. foo will be fully loaded (probably into a different location) and the stub definition will be destructively modified to point to the full definition. However, bar will still point to the stub definition. Only at the next garbage collection will all pointers be updated and the stub function definition disposed of.

This means that things you might expect to be eq will not be eq, at least for a while. Thus, if the function is fully loaded and there has subsequently been a gc when you call excl:resident-function, the returned value will be eq to the argument. If the function is not fully loaded or there has not been a gc, the returned value will not be eq to the argument and other references to the function object may not be eq to the returned value until after a garbage collection.

3.16 Libfasl and excl:dumplisp

Because the files needed to resolve stub functions must be available when the stub function is called, there are a number of ways in which libfasl loading affects the excl:dumplisp utility. These are discussed in 15.0 Dumping a prestoized image in dumplisp.htm. Please look at that document before dumping an image containing stub functions.

Copyright (C) 1998-1999, Franz Inc., Berkeley, CA. All Rights Reserved.