Building Images

$Revision: 5.0.2.17 $

The document introduction.htm provides an overview of the Allegro CL documentation with links to all major documents. The document index.htm is an index with pointers to every documented object (operators, variables, etc.) The revision number of this document is below the title. These documents may be revised from time to time between releases.

1.0 Comparison with earlier releases
2.0 Comparison with excl:dumplisp
3.0 Comparison with excl:generate-application
4.0 The template of a call to build-lisp-image
5.0 Arguments to build-lisp-image 1: defaults inherited from the running image
6.0 Arguments to build-lisp-image 2: defaults not inherited from running image
7.0 Use of custom.cl
8.0 Building an image to include patches
9.0 Comparison of build-lisp-image with version 4.3's build/install_lisp
10.0 Minimal top levels
    10.1 Using the default minimal top-level
    10.2 Requiring the normal top-level in a minimal top-level lisp
    10.3. Top-level variables

There are many links to excl:build-lisp-image in this document. In fact, essentially all relevant information about excl:build-lisp-image is contained in this document.

The function excl:build-lisp-image can be used to create a new image (dxl) file. This will be a fresh image, fresh in the sense that it inherits little from the running image (only values of certain global variables used as argument defaults). Typical reasons for building images with excl:build-lisp-image include:

• Building a new image with that includes patches.
• Building a custom image which includes local functionality or with locally-desired initializations.

excl:build-lisp-image is a lisp function, so it is called from a running Lisp. We distinguish between the image of the running image, the running image, from the image being created, the new image.

excl:build-lisp-image spawns a separate process. On Windows, a "Create image" window displays information about the build process (you may have to expose this window). On Unix, information is printed to the listener. You will see many prompts: do not attempt to type to them.

1.0 Comparison with earlier releases

UNIX: in earlier versions of Allegro CL on Unix, images were created using a shell script build/install_lisp. This created a new executable image file from constituent parts (.o files at the start of the process, .fasl files once a proto-lisp had been built). The overriding defect of this method of creating custom images was that a suitable C compiler was often needed for the new Lisp image to be built. That was not a problem when C compilers were bundled with hardware, but such bundling is no longer standard. Starting with 5.0, building a new image is done entirely within Lisp.

WINDOWS: building a custom image from constituent parts was not supported in earlier releases of Allegro CL on Windows (releases up to 3.0.2). Customizations and patches were loaded at startup time. It was possible to dump (using save-image) a customized image but this generally inherited indiscriminately from the running image.

Release 4.3.2, an interim release on Windows, was something of a hybrid (using a separate executable named config.exe). The 4.3.2 behavior was unique to that single release.

2.0 Comparison with excl:dumplisp

Images can also be created with excl:dumplisp, as described in the documentation for that function and in the document dumplisp.htm. An image created by excl:dumplisp is essentially a copy of the currently running image, with all loaded functionality included and all values of variables preserved. In contrast, excl:build-lisp-image spawns a separate process which builds a new image out of constituent parts. The currently running image supplies only default values for certain arguments. The new image does not capture loaded or newly-defined functionality of the running image (thus, if the function user::foo is defined in the currently running image, it will not be defined in the new image created by excl:build-lisp-image).

This is why excl:build-lisp-image can be used to create a new image with new patches. Patches in the currently running image are not captured by the newly created image. Instead (as described below), patch files are built into the newly created image but the patches in the current image are not involved.

Note that excl:build-lisp-image calls excl:dumplisp in the spawned process. All arguments accepted by excl:dumplisp are also accepted by excl:build-lisp-image except the name argument (which is accepted but ignored since the required argument to excl:build-lisp-image specifies the new image name).

3.0 Comparison with excl:generate-application

excl:generate-application itself calls excl:build-lisp-image. generate-application is designed to produce a directory of files suitable for shipping to another machine or site (note that you must be licensed to distribute software built in Allegro CL – contact your Franz Inc. sales representative if you are unsure of licensing terms). In contrast, excl:build-lisp-image builds a single image file only. See the description of excl:generate-application and the document delivery.htm.

4.0 The template of a call to build-lisp-image

The "BUILD" module must be loaded for excl:build-lisp-image to work (that is, "BUILD" should be on the *modules* list). It will be loaded automatically when excl:build-lisp-image is called. You can force loading of the module by evaluating

(require :build)

excl:build-lisp-image takes one required and many keyword arguments. The required argument must be a string naming a file, with or without path information. If no path information is specified, the image file will be placed in the current directory (as returned by excl:current-directory). If relative path information is supplied, it will be resolved relative to the current directory. The image file name must have an extension (type). If it does not, it will not be found when Lisp is started. The standard extension (type) of an image file is .dxl but any extension will do. Note that on Windows machines, .dxl is registered and associated with Allegro CL, so double-clicking on a dxl file (in the Windows Explorer, say) executes lisp.exe with `-I <that file>.dxl' as arguments. When lisp.exe is called without an image specified with –I, it looks for an image (dxl) file with the same name (i.e. lisp) and in the same directory. You can copy lisp.exe to <anything else>.exe if you want it to find a different image file named <anything else>.dxl automatically.

Here is the template for a call to excl:build-lisp-image:

(excl:build-lisp-image image-file
                       &key ...keywords...
                             ...dumplisp-keywords...)

5.0 Arguments to build-lisp-image 1: defaults inherited from the running image

Certain defaults for keyword arguments to excl:build-lisp-image are inherited from the currently running image. It is important to understand that these inheritances are the only effect that values in the currently running image have on the new image being built.

6.0 Arguments to build-lisp-image 2: defaults not inherited from running image

Most of the remaining keyword arguments are listed in the next table. The keyword arguments to excl:dumplisp are also acceptable to excl:build-lisp-image. None of these arguments inherit values from the currently running image.

Table note: Finding available memory addresses

To successfully allocate the heap, you will need to move the starting address of the Lisp heap to a location large enough to support a contiguous address range specified by the heap size you chose. If you evaluate the following forms in Allegro CL, a memory map of the current state of virtual memory on your machine will be printed (except on the IBM RS/6000 where the output file will say only that the information is not available):

(ff:defforeign 'memory-status-dump :entry-point "memory_status_dump")
(memory-status-dump "foo.xxx")

The output differs for different platforms, but in all (except the RS/6000) cases, a set of address ranges is provided showing what ranges are allocated.

On Windows, if the filename argument is 0, then the Console is used as output.

Permissions on Windows are shown in the last four characters in the output:

• r - read access
• w - write access
• x - execute access
• c - mapped copy on write (and has not been copied yet).

You will need to locate a large chunk of free memory and specify to excl:build-lisp-image as a starting address that will support your heap size.

7.0 Use of custom.cl

The file sys:custom.cl in the Allegro directory is loaded into the new image at the end of the building process but just before the files specified by :lisp-files are loaded. If you are running excl:generate-application, sys:custom.cl is loaded before the files specified by the (required) input-files argument are loaded.

8.0 Building an image to include patches

During the build, all patch files in <Allegro Directory>/update relevant to the products included in the image (as coded by the filenames) are loaded into the image during the build.

9.0 Comparison of build-lisp-image with version 4.3's build/install_lisp

For users moving from version 4.3.x of Allegro CL on Unix, here are the arguments to install_lisp and their excl:build-lisp-image equivalents if any.

10.0 Minimal top levels

When building a lisp image, specifying nil for include-tpl will cause a greatly reduced top-level functionality to be built into the lisp. The sole purpose of this minimal top-level is to reduce the space used by the full top-level. Applications which do not require a top-level or which provide their own will often specify include-tpl nil. The minimal top level described here will be available in such images.

The entire text of the minimal top level functionality is given below. Note that this code is loaded only if include-tpl is nil. The top-level code when include-tpl is non-nil is quite different.

(defpackage :top-level
  (:nicknames :tpl)
  (:use :common-lisp :excl)
  (:import-from :excl excl::read-eval-print-loop)

  ;; These are the function handlers for the top-level commands.
  ;; They are user visible.
  (:export #:*read-eval-print-loop*	; user-defined read-eval-print-loop
	   #:*read*			; the top-level reader
	   #:*eval*			; the top-level evaler
	   #:*print*			; the top-level printer
	   ))

(provide :tpl-user)

(in-package :top-level)

;; simple default tpl handlers:
(defvar *read-eval-print-loop* 'default-read-eval-print-loop)
(setq *read* 'read)
(setq *eval* 'eval)
(setq *print* 'print)
(defvar *start-hooks* nil)

(declaim (special *break-level*))
(setq *break-level* 0)

(defun start-interactive-top-level (*terminal-io*
				    function args
				    &key initial-bindings
				    &aux vars vals)

  (declare (:discard-source-file-info))
  (setf (getf (excl::stream-property-list *terminal-io*) 'initial-listener)
    sys::*current-process*)

  ;; Compute the list of special variables and bindings for progv.
  (dolist (b initial-bindings)
    (unless (member (car b) vars :test #'eq)
      (push (car b) vars)
      (push (eval (cdr b)) vals)))
  (progv vars vals
    (setq vars nil vals nil)		;free up space
    (apply function args)))

(defun top-level-read-eval-print-loop ()
  (declare (:discard-source-file-info))
  (loop
    (setq *evalhook* nil *applyhook* nil)
    (catch ':top-level-reset (read-eval-print-loop :level 0))
    ))

(defun read-eval-print-loop (&key &allow-other-keys)
  (declare (:discard-source-file-info))
  (let (pop-type cval1 cval2)
    (loop
      (multiple-value-setq (pop-type cval1 cval2)
	(catch 'top-level-break-loop
	  (funcall *read-eval-print-loop*)))
      ;; If we get here and pop-type is not null, then a throw
      ;; to 'top-level-break-loop was done (by a different toplevel)
      (case pop-type
	((:pop :debug-pop)
	 (when (plusp cval1)
	   (excl::funcall-in-package :debug-pop :debugger
				     nil (1- cval1) (1- cval2))))
	(error "user toplevel can't handle this pop type: ~s" pop-type)))))

(defun default-read-eval-print-loop ()
  (loop
    ;; print the prompt
    (fresh-line *terminal-io*)
    (princ "// " *terminal-io*)
    (let* ((exp (funcall *read*))
	   (res (funcall *eval* exp)))
      (funcall *print* res))))

10.1 Using the default minimal top-level

The minimal top-level is set up by default to issue a "//" prompt.  It only accepts lisp evaluable expressions as "commands", and does not interpret any other top-level
commands.

Example on a sparc:

% lisp -I umsloadxcomp.dxl -qq
Loading /release/duane/acl50/src/libacl50pf23.so.
Mapping umsloadxcomp.dxl...done.
Mapping umclxcomp.pll.
Allegro CL 5.0.pre-final.23 [SPARC] (8/10/98 16:40)
Copyright (C) 1985-1998, Franz Inc., Berkeley, CA, USA.  All Rights Reserved.
// (room)
area  address(bytes)        cons        symbols        other bytes
                        8 bytes each  24 bytes each
                        (free:used)   (free:used)      (free:used)
Top #x81dc000
New #x8180000(376832)     918:3158      254:0         246536:37944
New #x8124000(376832)      -----         -----            -----
Old #x8000c40(1192896)    781:15523     135:5628      518896:397344
Root pages: 34
Lisp heap limit: 67108864

NIL 
// (exit)
; Exiting Lisp
%

10.2 Requiring the normal top-level in a minimal top-level lisp

If the power of the normal top-level is needed after a (non-runtime) minimal top-level lisp is built, :toplevel can be required. However, simply requiring :toplevel is not enough to start the regular top-level listener; instead, the listener must be invoked recursively, either by an error or by any command (such as cl:inspect) that starts a new listener level. At that time, top-level commands (such as :zoom, etc) can be invoked.

Note however that when :toplevel is required, the read-eval-print-loop is re-set, and so a :reset command will make it appear as if the Lisp had always had a normal top-level.

Example1:

;; In this example, an error can be debugged after the fact by requiring
;; the normal top-level.

% lisp -I umsloadxcomp.dxl -qq
Loading /release/duane/acl50/src/libacl50pf23.so.
Mapping umsloadxcomp.dxl...done.
Mapping umclxcomp.pll.
Allegro CL 5.0.pre-final.23 [SPARC] (8/10/98 16:40)
Copyright (C) 1985-1998, Franz Inc., Berkeley, CA, USA.  All Rights Reserved.
// (require :toplevel)
; Fast loading /release/duane/acl50/src/code/toplevel.fasl
;   Fast loading /release/duane/acl50/src/code/frame.fasl
;     Fast loading /release/duane/acl50/src/code/r/rframe.fasl

T 
// a
Error: Attempt to take the value of the unbound variable `A'.
  [condition type: UNBOUND-VARIABLE]

Restart actions (select using :continue):
 0: Try evaluating A again.
 1: Use :A instead.
 2: Set the symbol-value of A and use its value.
 3: Use a value without setting A.
[1] USER(1): :zo
; Autoloading for TOP-LEVEL::ZOOM-COMMAND:
; Fast loading /release/duane/acl50/src/code/tpl-debug.fasl
; Autoloading for package "DEBUGGER":
;   Fast loading /release/duane/acl50/src/code/debug.fasl
Evaluation stack:

 ->(EXCL::INTERNAL-INVOKE-DEBUGGER "Error" #<UNBOUND-VARIABLE @ #x81b9f8a> ...)
   (ERROR #<UNBOUND-VARIABLE @ #x81b9f8a>)
   (SYS::..CONTEXT-SAVING-RUNTIME-OPERATION)
   (EVAL A)
   (TPL:TOP-LEVEL-READ-EVAL-PRINT-LOOP)
   (TPL:START-INTERACTIVE-TOP-LEVEL #<BIDIRECTIONAL-TERMINAL-STREAM [initial terminal io] fd 0/1 @ #x80439ba>
                                    #<Function TOP-LEVEL-READ-EVAL-PRINT-LOOP @ #x804d22a> ...)

(to see any ghost frames, the disassembler must be loaded)
[1] USER(2): :res
USER(1): 

Example 2:

;; Note in this example a variable is inspected after setting it,
;; to indicate the state of the lisp before the new top-level is
;; pulled into the lisp.  Note also that we wrap a progn which will
;; return a final nil value, so as not to see a huge printout due to
;; the lack of *print-level*/*print-length* controls.

% lisp -I umsloadxcomp.dxl -qq
Loading /release/duane/acl50/src/libacl50pf23.so.
Mapping umsloadxcomp.dxl...done.
Mapping umclxcomp.pll.
Allegro CL 5.0.pre-final.23 [SPARC] (8/10/98 16:40)
Copyright (C) 1985-1998, Franz Inc., Berkeley, CA, USA.  All Rights Reserved.
// (progn (setq x (excl::get-objects 7)) nil)

NIL 
// (require :toplevel)
; Fast loading /release/duane/acl50/src/code/toplevel.fasl
;   Fast loading /release/duane/acl50/src/code/frame.fasl
;     Fast loading /release/duane/acl50/src/code/r/rframe.fasl

T 
// (inspect x)
; Autoloading for INSPECT:
; Fast loading /release/duane/acl50/src/code/inspect.fasl
A simple T vector (5649) @ #x81a506a
   0-> fixnum 5629 [#x000057f4]
   1-> The symbol CL:NIL
   2-> The symbol T
   3-> The symbol EXCL::ER-WNAERR
   4-> The symbol EXCL::ER-GENERAL-ERROR-HANDLER-ZERO
   5-> The symbol EXCL::ER-GENERAL-ERROR-HANDLER-ONE
   6-> The symbol EVAL
   7-> The symbol EXCL::INTERPRETED-FUNCALL
   8-> The symbol EXCL::+_2OP
   9-> The symbol EXCL::GC-AFTER
  10-> The symbol EXCL::*WITHOUT-INTERRUPTS*
  11-> The symbol EQUAL
  12-> The symbol *PACKAGE*
  13-> The symbol *LISP-PACKAGE*
  14-> The symbol *KEYWORD-PACKAGE*
  15-> The symbol EXCL::INTERN*
  16-> The symbol EXCL::FASL-FIND-PACKAGE
  17-> The symbol EXCL::CONVERT-TO-INTERNAL-FSPEC
  18-> The symbol *COMPILER-PACKAGE*
  19-> The symbol *SYSTEM-PACKAGE*
  20-> The symbol EXCL::CONVERT-TO-EXTERNAL-FSPEC
  21-> The symbol SYS::LISP-BREAKPOINT
  22-> The symbol EXCL::HANDLE-PENDING-SIGNAL
  23-> The symbol EXCL::SET-FUNCTION
  24-> The symbol EXCL::.INV-MACRO-FUNCTION
   ...
 5648-> The symbol NIL
[1i] USER(1): 

10.3. Top-level variables

The following variables are maintained or used by the minimal top-level:

If tpl:*read-eval-print-loop* is set to a value other than tpl::default-read-eval-print-loop, then the three read/eval/print variables are not used. Normally, this variable should not be set unless it is desired to remove all possible user interaction with lisp. If a replacement top-level is supplied, it is recommended that all possible errors be handled explicitly with handlers.

The top-level variables in the table with links are also discussed in 9.0 Top-level variables in top_level.htm.

Copyright (C) 1998-1999, Franz Inc., Berkeley, CA. All Rights Reserved.