Building Images

The index for the Allegro CL Documentation is in index.htm. The documentation is described in introduction.htm.

This document contains the following sections:

There are many links to build-lisp-image in this document. In fact, essentially all relevant information about build-lisp-image is contained in this document.

The function build-lisp-image can be used to create a new image (dxl) file. This will be a fresh image, fresh in the sense that it inherits little from the running image (only values of certain global variables used as argument defaults). Typical reasons for building images with build-lisp-image include:

• Building a new image that includes patches.
• Building a custom image which includes local functionality or with locally-desired initializations.

build-lisp-image is a lisp function, so it is called from a running Lisp. We distinguish between the image which calls build-lisp-image, the running image, from the image being created, the new image.

build-lisp-image spawns a separate process. On Windows, a "Create image" window displays information about the build process (you may have to expose this window). On Unix, information is printed to the listener. You will see many prompts: do not attempt to type to them.

1.0 Comparison with excl:dumplisp

Images can also be created with dumplisp, as described in the documentation for that function and in the document dumplisp.htm. An image created by excl:dumplisp is essentially a copy of the currently running image, with all loaded functionality included and all values of variables preserved. In contrast, build-lisp-image spawns a separate process which builds a new image out of constituent parts. The currently running image supplies only default values for certain arguments. The new image does not capture loaded or newly-defined functionality of the running image (thus, if the function user::foo is defined in the currently running image, it will not be defined in the new image created by build-lisp-image).

This is why build-lisp-image can be used to create a new image with new patches. Patches in the currently running image are not captured by the newly created image. Instead (as described below), patch files are built into the newly created image but the patches in the current image are not involved.

Note that build-lisp-image calls dumplisp in the spawned process. All arguments accepted by dumplisp are also accepted by build-lisp-image except the name argument (which is accepted but ignored since the required argument to build-lisp-image specifies the new image name).

2.0 Comparison with excl:generate-application

generate-application itself calls build-lisp-image. generate-application is designed to produce a directory of files suitable for shipping to another machine or site (note that you must be licensed to distribute software built in Allegro CL – contact your Franz Inc. account manager if you are unsure of licensing terms). In contrast, build-lisp-image builds a single image file only. See the description of generate-application and the document delivery.htm.

3.0 The template of a call to build-lisp-image

The "BUILD" module must be loaded for build-lisp-image to work (that is, "BUILD" should be on the *modules* list). It will be loaded automatically when build-lisp-image is called. You can force loading of the module by evaluating

(require :build)

build-lisp-image takes one required and many keyword arguments. The required argument must be a string naming a file, with or without path information. If no path information is specified, the image file will be placed in the current directory (as returned by current-directory). If relative path information is supplied, it will be resolved relative to the current directory. The image file name must have an extension (type). If it does not, it will not be found when Lisp is started. The standard extension (type) of an image file is .dxl but any extension will do. Note that on Windows machines, .dxl is registered and associated with Allegro CL, so double-clicking on a dxl file (in the Windows Explorer, say) executes lisp.exe with `-I [that file].dxl' as arguments. When lisp.exe is called without an image specified with –I, it looks for an image (dxl) file with the same name (i.e. lisp) and in the same directory. You can copy lisp.exe to [anything else].exe if you want it to find a different image file named [anything else].dxl automatically.

Here is the template for a call to build-lisp-image:

(excl:build-lisp-image image-file
                      
&key ...keywords... 
                      
      ...dumplisp-keywords...)

3.1 The character size in the resulting image

Allegro CL 6.0 supports two character sizes: 16-bit and 8-bit. Thus images and executables either use 16-bit characters or 8-bit characters. No image or executable supports characters of both sizes. See iacl.htm for information on character sizes in Allegro CL.

build-lisp-image creates a 16-bit charcater or an 8-bit character image as the new image is created using a 16-bit executable (mlisp and alisp on UNIX, mlisp.exe and alisp.exe on Windows) or an 8-bit executable (mlisp8 and alisp8 on UNIX, mlisp8.exe and alisp8.exe on Windows). The build-executable keyword argument to build-lisp-image specifies the executable that will be used to create the new image. The value of that argument defaults to the executable used to start the running image.

Therefore, if you want the character size of the enw image to be the same as the character size in the running image, you need not specify a value for the build-executable keyword argument. If you want a different character size, specify an appropriate executable as the value of that argument.

For example, if you are running a 16-bit character size image and you want to build an 8-bit character size image, call build-lisp-image like this:

(build-lisp-image "my-image.dxl"
   :build-executable "mlisp8" ;; On Windows "mlisp8.exe"
   ;; < other keyword arguments and values >
  )

(That call also works if you are running an 8-bit executable already, of course.)

For example, if you are running an 8-bit character size image and you want to build a 16-bit character size image, call build-lisp-image like this:

(build-lisp-image "my-image.dxl"
   :build-executable "mlisp"  ;; On Windows "mlisp.exe"
   ;; < other keyword arguments and values >
  )

(That call also works if you are running an 16-bit executable already, of course.)

Note: Unless you have created your own executable (which is uncommon, but see main.htm, all 16-bit character size executable are identical and all 8-bit character size executables are identical. Therefore, any executable of the correct character size may be specified.

4.0 Arguments to build-lisp-image 1: defaults inherited from the running image

Certain defaults for keyword arguments to build-lisp-image are inherited from the currently running image. It is important to understand that these inheritances are the only effect that values in the currently running image have on the new image being built.

5.0 Arguments to build-lisp-image 2: defaults not inherited from running image

Most of the remaining keyword arguments are listed in the next table. The keyword arguments to dumplisp are also acceptable to build-lisp-image. None of these arguments inherit values from the currently running image.

Table note: Finding available memory addresses

To successfully allocate the heap, you will need to move the starting address of the Lisp heap to a location large enough to support a contiguous address range specified by the heap size you chose. If you evaluate the following forms in Allegro CL, a memory map of the current state of virtual memory on your machine will be printed to the file filename.ext (except on the IBM RS/6000 where the output file will say only that the information is not available). Note that you may use any filename and extension.

(ff:def-foreign-call (memory-status-dump "memory_status_dump")
     ())

(memory-status-dump "filename.ext")

The output differs for different platforms, but in all (except the RS/6000) cases, a set of address ranges is provided showing what ranges are allocated.

On Windows, if the filename argument is 0, then the Console is used as output.

Permissions on Windows are shown in the last four characters in the output:

• r - read access
• w - write access
• x - execute access
• c - mapped copy on write (and has not been copied yet).

You will need to locate a large chunk of free memory and specify to build-lisp-image a starting address that will support your heap size.

5.1 Debugging on UNIX

build-lisp-image spawns another Lisp to build the desired image (that is one reason why the new image does not inherit from the calling image). This new Lisp is started in a way that it will exit after an error without any user interaction. That is usually what is desired but sometimes being able to interact with the Lisp building the new image is desirable. You can do that by starting that image yourself. Here is how you do that:

1. Call build-lisp-image (or generate-application with arguments (in addition to your own arguments)

:internal-debug "idout" :verbose t 

"idout" is a filename -- any filename with any valid path will do. Here is what the build-lisp-image form looks like:

(excl:build-lisp-image <image-file> 
                       :internal-debug "idout" :verbose t 
                       <your arguments>)

This causes the actual command starting the Lisp that builds the new dxl (along with other stuff) to be printed to the listener where the build-lisp-image command was issued.

Generated (by the :internal-debug argument) is the file "idout" which contains the Lisp forms passed to the spawned Lisp. It looks like:

(COMMON-LISP:SETQ EXCL::*BATCH-MODE* COMMON-LISP:NIL) 
(COMMON-LISP:SETQ EXCL::*BREAK-HOOK* #'EXCL::EXIT-ON-ERROR-HOOK) 
(EXCL:SET-CASE-MODE :CASE-SENSITIVE-LOWER) (COMMON-LISP:FORCE-OUTPUT) 
(COMMON-LISP:PROGN (COMMON-LISP:SETQ EXCL::*STORE-DOCUMENTATION* COMMON-LISP:NIL) 
(COMMON-LISP:SETQ EXCL:*RECORD-SOURCE-FILE-INFO* COMMON-LISP:NIL) 
(COMMON-LISP:SETQ EXCL:*LOAD-LOCAL-NAMES-INFO* COMMON-LISP:NIL) 
[...]

The first two lines tell the Lisp to exit on errors. The remaining lines depend on the arguments passed and may be different when you run build-lisp-image (or generate-application).

You are now ready for the next step. Presumably, the failure you wished to debug has occurred but ignore that for now.

2. In a shell (or better, an emacs shell buffer) start the build image by executing the command line captured from the attempt above.

Allegro CL will start and you should get a prompt. Start evaluating the forms in idout skipping the first two. That is, do not evaluate these:

(COMMON-LISP:SETQ EXCL::*BATCH-MODE* COMMON-LISP:NIL)
(COMMON-LISP:SETQ EXCL::*BREAK-HOOK* #'EXCL::EXIT-ON-ERROR-HOOK) 

but evaluate the forms after those two, like:

USER(1): (EXCL:SET-CASE-MODE :CASE-SENSITIVE-LOWER) 
USER(2): (COMMON-LISP:FORCE-OUTPUT) 

When you get to the form that causes the problem, you should get a error prompt and can do a :zoom etc.

6.0 Use of custom.cl

The file sys:custom.cl in the Allegro directory is loaded into the new image at the end of the building process but just before the files specified by :lisp-files are loaded. If you are running generate-application, sys:custom.cl is loaded before the files specified by the (required) input-files argument are loaded.

7.0 Building an image to include patches

During the build, all patch files in [Allegro Directory]/update relevant to the products included in the image (as coded by the filenames) are loaded into the image during the build.

8.0 Minimal top levels

When building a lisp image, specifying nil for include-tpl will cause a greatly reduced top-level functionality to be built into the lisp. The sole purpose of this minimal top-level is to reduce the space used by the full top-level. Applications which do not require a top-level or which provide their own will often specify include-tpl nil. The minimal top level described here will be available in such images.

The entire text of the minimal top level functionality is given below. Note that this code is loaded only if include-tpl is nil. The top-level code when include-tpl is true is quite different.

(defpackage :top-level
  (:nicknames :tpl)
  (:use :common-lisp :excl)
  (:import-from :excl excl::read-eval-print-loop)

  ;; These are the function handlers for the top-level commands.
  ;; They are user visible.
  (:export #:*read-eval-print-loop*	; user-defined read-eval-print-loop
	   #:*read*			; the top-level reader
	   #:*eval*			; the top-level evaler
	   #:*print*			; the top-level printer
	   ))

(provide :tpl-user)

(in-package :top-level)

;; simple default tpl handlers:
(defvar *read-eval-print-loop* 'default-read-eval-print-loop)
(setq *read* 'read)
(setq *eval* 'eval)
(setq *print* 'print)
(defvar *start-hooks* nil)

(declaim (special *break-level*))
(setq *break-level* 0)

(defun start-interactive-top-level (*terminal-io*
				    function args
				    &key initial-bindings
				    &aux vars vals)

  (declare (:discard-source-file-info))
  (setf (getf (excl::stream-property-list *terminal-io*) 'initial-listener)
    sys::*current-process*)

  ;; Compute the list of special variables and bindings for progv.
  (dolist (b initial-bindings)
    (unless (member (car b) vars :test #'eq)
      (push (car b) vars)
      (push (eval (cdr b)) vals)))
  (progv vars vals
    (setq vars nil vals nil)		;free up space
    (apply function args)))

(defun top-level-read-eval-print-loop ()
  (declare (:discard-source-file-info))
  (loop
    (setq *evalhook* nil *applyhook* nil)
    (catch ':top-level-reset (read-eval-print-loop :level 0))
    ))

(defun read-eval-print-loop (&key &allow-other-keys)
  (declare (:discard-source-file-info))
  (let (pop-type cval1 cval2)
    (loop
      (multiple-value-setq (pop-type cval1 cval2)
	(catch 'top-level-break-loop
	  (funcall *read-eval-print-loop*)))
      ;; If we get here and pop-type is not null, then a throw
      ;; to 'top-level-break-loop was done (by a different toplevel)
      (case pop-type
	((:pop :debug-pop)
	 (when (plusp cval1)
	   (excl::funcall-in-package :debug-pop :debugger
				     nil (1- cval1) (1- cval2))))
	(error "user toplevel can't handle this pop type: ~s" pop-type)))))


(defun default-read-eval-print-loop ()
  (loop
    ;; print the prompt
    (fresh-line *terminal-io*)
    (princ "// " *terminal-io*)
    (let* ((exp (funcall *read*))
	   (res (funcall *eval* exp)))
      (funcall *print* res))))

8.1 Using the default minimal top-level

The minimal top-level is set up by default to issue a "//" prompt. It only accepts lisp evaluable expressions as "commands", and does not interpret any other top-level commands.

Example on a sparc:

% lisp -I umsloadxcomp.dxl -qq
Loading /release/duane/acl60/src/libacl60pf23.so.
Mapping umsloadxcomp.dxl...done.
Mapping umclxcomp.pll.
Allegro CL 6.0.1 
Copyright (C) 1985-2000, Franz Inc., Berkeley, CA, USA.  All Rights Reserved.
// (room)
area  address(bytes)        cons        symbols        other bytes
                        8 bytes each  24 bytes each
                        (free:used)   (free:used)      (free:used)
Top #x81dc000
New #x8180000(376832)     918:3158      254:0         246536:37944
New #x8124000(376832)      -----         -----            -----
Old #x8000c40(1192896)    781:15523     135:5628      518896:397344
Root pages: 34
Lisp heap limit: 67108864


NIL 
// (exit)
; Exiting Lisp
%

8.2 Requiring the normal top-level in a minimal top-level lisp

If the power of the normal top-level is needed after a (non-runtime) minimal top-level lisp is built, :toplevel can be required. However, simply requiring :toplevel is not enough to start the regular top-level listener; instead, the listener must be invoked recursively, either by an error or by any command (such as inspect) that starts a new listener level. At that time, top-level commands (such as :zoom, etc) can be invoked.

Note however that when :toplevel is required, the read-eval-print-loop is re-set, and so a :reset command will make it appear as if the Lisp had always had a normal top-level.

Example1:

;; In this example, an error can be debugged after the fact by requiring
;; the normal top-level.

% lisp -I umsloadxcomp.dxl -qq
Loading /release/duane/acl60/src/libacl60pf23.so.
Mapping umsloadxcomp.dxl...done.
Mapping umclxcomp.pll.
Allegro CL 6.0
Copyright (C) 1985-2000, Franz Inc., Berkeley, CA, USA.  All Rights Reserved.
// (require :toplevel)
; Fast loading /acl60/src/code/toplevel.fasl
;   Fast loading /acl60/src/code/frame.fasl
;     Fast loading /acl60/src/code/r/rframe.fasl

T 
// a
Error: Attempt to take the value of the unbound variable `A'.
  [condition type: UNBOUND-VARIABLE]

Restart actions (select using :continue):
 0: Try evaluating A again.
 1: Use :A instead.
 2: Set the symbol-value of A and use its value.
 3: Use a value without setting A.
[1] USER(1): :zo
; Autoloading for TOP-LEVEL::ZOOM-COMMAND:
; Fast loading /acl60/src/code/tpl-debug.fasl
; Autoloading for package "DEBUGGER":
;   Fast loading /acl60/src/code/debug.fasl
Evaluation stack:

 ->(EXCL::INTERNAL-INVOKE-DEBUGGER "Error" #<UNBOUND-VARIABLE @ #x81b9f8a> ...)
   (ERROR #<UNBOUND-VARIABLE @ #x81b9f8a>)
   (SYS::..CONTEXT-SAVING-RUNTIME-OPERATION)
   (EVAL A)
   (TPL:TOP-LEVEL-READ-EVAL-PRINT-LOOP)
   (TPL:START-INTERACTIVE-TOP-LEVEL #<BIDIRECTIONAL-TERMINAL-STREAM [initial terminal io] fd 0/1 @ #x80439ba>
                                    #<Function TOP-LEVEL-READ-EVAL-PRINT-LOOP @ #x804d22a> ...)

(to see any ghost frames, the disassembler must be loaded)
[1] USER(2): :res
USER(1): 

Example 2:

;; Note in this example a variable is inspected after setting it,
;; to indicate the state of the lisp before the new top-level is
;; pulled into the lisp.  Note also that we wrap a progn which will
;; return a final nil value, so as not to see a huge printout due to
;; the lack of *print-level*/*print-length* controls.

% lisp -I umsloadxcomp.dxl -qq
Loading /acl60/src/libacl60pf23.so.
Mapping umsloadxcomp.dxl...done.
Mapping umclxcomp.pll.
Allegro CL 6.0
Copyright (C) 1985-2000, Franz Inc., Berkeley, CA, USA.  All Rights Reserved.
// (progn (setq x (excl::get-objects 7)) nil)

NIL 
// (require :toplevel)
; Fast loading /acl60/src/code/toplevel.fasl
;   Fast loading /acl60/src/code/frame.fasl
;     Fast loading /acl60/src/code/r/rframe.fasl

T 
// (inspect x)
; Autoloading for INSPECT:
; Fast loading /acl60/src/code/inspect.fasl
A simple T vector (5649) @ #x81a506a
   0-> fixnum 5629 [#x000057f4]
   1-> The symbol CL:NIL
   2-> The symbol T
   3-> The symbol EXCL::ER-WNAERR
   4-> The symbol EXCL::ER-GENERAL-ERROR-HANDLER-ZERO
   5-> The symbol EXCL::ER-GENERAL-ERROR-HANDLER-ONE
   6-> The symbol EVAL
   7-> The symbol EXCL::INTERPRETED-FUNCALL
   8-> The symbol EXCL::+_2OP
   9-> The symbol EXCL::GC-AFTER
  10-> The symbol EXCL::*WITHOUT-INTERRUPTS*
  11-> The symbol EQUAL
  12-> The symbol *PACKAGE*
  13-> The symbol *LISP-PACKAGE*
  14-> The symbol *KEYWORD-PACKAGE*
  15-> The symbol EXCL::INTERN*
  16-> The symbol EXCL::FASL-FIND-PACKAGE
  17-> The symbol EXCL::CONVERT-TO-INTERNAL-FSPEC
  18-> The symbol *COMPILER-PACKAGE*
  19-> The symbol *SYSTEM-PACKAGE*
  20-> The symbol EXCL::CONVERT-TO-EXTERNAL-FSPEC
  21-> The symbol SYS::LISP-BREAKPOINT
  22-> The symbol EXCL::HANDLE-PENDING-SIGNAL
  23-> The symbol EXCL::SET-FUNCTION
  24-> The symbol EXCL::.INV-MACRO-FUNCTION
   ...
 5648-> The symbol NIL
[1i] USER(1): 

8.3 Top-level variables

The following variables are maintained or used by the minimal top-level:

If *read-eval-print-loop* is set to a value other than tpl::default-read-eval-print-loop, then the three read/eval/print variables are not used. Normally, this variable should not be set unless it is desired to remove all possible user interaction with lisp. If a replacement top-level is supplied, it is recommended that all possible errors be handled explicitly with handlers.

The top-level variables in the table with links are also discussed in Top-level variables in top-level.htm.

Copyright (c) 1998-2000, Franz Inc. Berkeley, CA., USA. All rights reserved. Created 2000.10.5.