Miscellaneous Functionality

This document contains the following sections:

1.0 Introduction

This document describes functionality that does not naturally fit in any of the other overview documents.

2.0 The fasl reader/writer

The functions fasl-write and fasl-read provide a mechanism for writing Lisp data and subsequently reading it back into a Lisp image. It can handle many of the common Lisp data types. It can optionally detect circularity and structure sharing in the data and recreate the same topology up to eql-ness of components. The data is written in a binary file format similar to that used for compiled Lisp files, not in ASCII.

Among the advantages of fasl-read and fasl-write over standard Common Lisp read and print functions is that data does not have to be converted to its printed representation prior to being written (that conversion takes a significant amount of time for complex objects). The main disadvantage is that the files (unlike ASCII files with printed representations of Lisp objects) are not at all portable between versions of Lisp (or even between versions of Allegro CL on different platforms or between major releases of Allegro CL).

Not all Lisp objects may be written with fasl-write (and thus are not available to be read by fasl-read). The objects that cannot be written are CLOS objects, streams, stacks, or c-allocated data (cstructs). You can use fasl-open to open a file suitable for numerous fasl-write's. (fasl-read reads all of whatever file is specified to it.)

fasl-read reads the entire contents of its argument file. However, fasl-write, if its argument is a stream rather than a filename, writes data to the stream but does not close it. Thus you can open a stream and do multiple fasl-write's to it. The stream must have :element-type (unsigned-byte 8). fasl-open opens an appropriate stream, as shown in the example.

Here is a simple-minded example to show the correct syntax and give an idea of the effect:

(excl:fasl-write '(a b #1=#:c (#1#) #*1011 #(1 2 3) 
                  
1 1.1 1.1d0 1/2
                  
#c(1 1) #c(1.5 1.5) "abc")
     "test.fw")
(excl:fasl-read "test.fw")

(setq f (fasl-open "test2.fw")) 
(excl:fasl-write '(a b c) f) 
(excl:fasl-write '#1=(a b . #1#) f t) 
(excl:fasl-write '#(1 2 3 4.5) f)
(close f)

(excl:fasl-read "test2.fw")

3.0 Miscellaneous extensions

The table describes those extensions to Common Lisp that do not naturally fit elsewhere in the documentation. We only provide brief information in the table. Please follow the link to the documentation page for a full description.

4.0 Creating and using pll files

A pll file can be used in association with a Lisp image. It contains constant code vectors and strings that can be shared among many Lisp objects. When an image uses a pll file and a function is compiled, the new codevector is compared to codevectors in the pll file. If a match is found, the match is used and no new codevector is allocated. Similarly, if a constant string is specified (with excl:pure-string) and a matching string appears in the pll file, no new string is allocated and the match is used. Strings in a pll file cannot be modified. Attempting to do so causes an error. (Neither can codevectors be modified but there is no user-visible way to modify codevectors as there is with strings.)

Strings and codevectors in a pll file are not also (after being garbage collected) in the Lisp heap. Thus if a string has been successfully purified, it will not be in the heap after a global gc. A total count of strings and codevectors is shown in the output of (room t).

pll files are created with the cvdcvt program described next.

cvdcvt

Arguments: [-o outfilename] [-u] [file1 file2 …]

A .pll file, outfilename, is created holding all the unique code vectors and strings. If outfilename is omitted it defaults to code.blob.

If -u is specified, then no duplications of strings are done, otherwise for every string that has no lowercase characters in it and at least one uppercase character, a lowercase copy is added to the output file. This is the default and is useful for set-case-mode. If no files are specified, stdin is used.

.str (string) files and .cvs (code vector) files are combined without redundancies; if two files of the same extension have identical objects, the object from the file specified first to cvdcvt is retained, and the latter object is removed from the output. This allows for files (.cvs files especially) to be arranged by code vectors in order of execution, to provide for locality of reference. Those .cvs files that were produced by training techniques should be placed first in order to have the desired effect.

As said in the description, pll files are built out of cvs files and str files. cvs files are created with sys:write-codevectors and can be created by sys:flush-codevectors. str files are created with record-strings. See also record-code-vectors.

The following functions can be used to associate a pll file with an image, to find out which pll file is used with an image, and to use strings in the pll file.

5.0 MD5 support

Allegro CL, starting in release 6.0, provides support for MD5 (Message Digest, version 5) within Lisp. (Message digest represents text in the form of a single string of digits.)

The functions available are:

• md5-init
• md5-update
• md5-final
• md5-file
• md5-string
• hex-string-to-integer
• integer-to-hex-string

6.0 Base64 support

Allegro CL, starting in release 6.0, provides support for Base64 encoding within Lisp. Base64 encoding is a 6-bit representation scheme that uses the ASCII characters A-Z, a-z, 0-9, + and /. Since padding could be needed in converting multiples of 8-bits into base64, = characters are used, when necessary, as padding at the end of a converted string. Base64 encoding is described in the RFC2045 dcument (www.ietf.org/rfc/rfc2045.txt).

Four functions provide the Base64 support in Allegro CL:

• integer-to-base64-string, which takes an integer and an optional column arguments and returns the associated string (with newlines at the column location, if any).
• base64-string-to-integer, which takes a string and returns the associated integer.
• usb8-array-to-base64-string, which takes an (unsigned-byte 8) array and an optional column arguments and returns the associated string (with newlines at the column location, if any).
• base64-string-to-usb8-array, which takes a string and returns the associated (unsigned-byte 8) array.

Here are some examples, first using integer-to-base64-string and base64-string-to-integer:

cl-user(2): (integer-to-base64-string #xfeedfacefeedface)
"/u36zv7t+s4="
cl-user(3): (base64-string-to-integer "/u36zv7t+s4=")
18369614222061337294
cl-user(4): (format t "~x" *)
feedfacefeedface
nil
cl-user(5): 

And now using usb8-array-to-base64-string and base64-string-to-usb8-array:

;;  Note that we have specified ithout specifyin that value, 
;;  because of different end of line encoding, on UNIX, the result 
;;  of the cl-user(7) evaluation should be
;;  #(10 40 100 101 102 117 110 32 100 101 ...) and on Windows,
;;  the result of the cl-user(7) evaluation would be
;;  #(13 10 40 100 101 102 117 110 32 100 ...) -- starting
;;  with 13 rather than 10. The encoding in cl-user(6) will
;;  thus be different as well on Windows.

cl-user(5): (setq a (string-to-octets
                     (setq s
                       "
(defun deep-thought ()
  (sleep (years2secs 7500000))
  42)
")
                     :external-format (crlf-base-ef :latin1)))
#(10 40 100 101 102 117 110 32 100 101 ...)
cl-user(6): (usb8-array-to-base64-string a)
"CihkZWZ1biBkZWVwLXRob3VnaHQgKCkKICAoc2xlZXAgKHllYXJz
MnNlY3MgNzUwMDAwMCkpCiAgNDIpCgA="
cl-user(7): (base64-string-to-usb8-array *)
#(10 40 100 101 102 117 110 32 100 101 ...)
cl-user(8): (setq a2 *)
#(10 40 100 101 102 117 110 32 100 101 ...)
cl-user(9): (equalp a a2)
t
cl-user(10): (octets-to-string a2 :external-format (crlf-base-ef :latin1))
"
(defun deep-thought ()
  (sleep (years2secs 7500000))
  42)
"
61
61
cl-user(11): 

7.0 Support for encryption

Allegro CL provides implementations of some publically available encryption algorithms: blowfish (see Section 7.1 Support for Blowfish encryption) and rsa (see Section 7.2 Support for rsa encryption). Please note that we make no claims about the actual security provided by these encryption schemes.

7.1 Support for Blowfish encryption

The Blowfish algorithm, described on this page (and links from it), is a high speed symmetric cryptographic algorithm (or cipher). The same key is used to encrypt and decrypt the data. Blowfish encrypts blocks of 64 bits (8 octets) at a time. The functions below can automatically pad out the data to encrypt to be a multiple of 8 octets. Blowfish was designed by Bruce Schneier, a leading authority on cryptography and author of the book Applied Cryptography. Schneier writes in his book published in 1996: "I know of no successful cryptanalysis against Blowfish."

• blowfish-encrypt: encrypts an (unsigned-byte 8) array or a Lisp string using a specifed key.
• blowfish-decrypt: decrypts a value encrypted with blowfish-encrypt.
• blowfish-init: returns an object that holds the key already processed for doing encryption or decryption, saving time when doing multiple calls to blowfish-encrypt or blowfish-decrypt using the same key. (The object rather than the key is passed to those functions, saving the time needed to calculate the object from the key multiple times.)

Here are some examples of Blowfish encryption and decryption:

;; Example 1. string encrypting

cl-user(12): (blowfish-encrypt "my secret message" 
                        :key "my key")
#(57 27 110 242 191 19 182 150 1 5 ...)
24
cl-user(13): (blowfish-decrypt * :key "my key" :string t)
"my secret message"
cl-user(14): 

;; Example 2. (unsigned-byte 8) encrypting:

;; Here we allocate an (unsigned-byte 8) array with a size 
;; that is a multiple of 8 and fill it with data.  
;; We can do in-place encryption and decryption.
;; We specify no padding (since otherwise 8 bytes of padding would
;; have to be added and there's no room in this array for that):

;; Create our array:

cl-user(12): (setq aa (make-array 8 :element-type '(unsigned-byte 8) 
                        :initial-contents '(2 4 6 8 10 12 14 16)))
#(2 4 6 8 10 12 14 16)

;; Encrypt it in place

cl-user(13): (blowfish-encrypt aa :key "my key" 
               :in-place t :pad nil)
#(129 144 108 210 20 227 10 58)
8

;; Verify that it has been modified:

cl-user(14): aa
#(129 144 108 210 20 227 10 58)

;; Now decrypt in place. Notice how the arguments to 
;; blowfish-decrypt match those to blowfish-encrypt:

cl-user(15): (blowfish-decrypt aa :key "my key" 
                 :in-place t :pad nil)
#(2 4 6 8 10 12 14 16)
cl-user(16): 

;; And verify that the array is now back to normal:

cl-user(16): aa
#(2 4 6 8 10 12 14 16)
cl-user(17): 

;; Example 3. use of contexts

;; Create context which holds the key processed by 
;; blowfish to prepare it for encryption/decryption:

cl-user(21): (setq cc (blowfish-init "my key"))
#(141 90 172 196 250 88 140 57 179 211 ...)

;; Encrypt something using the context:

cl-user(22): (blowfish-encrypt "my message" :context cc)
#(75 202 37 143 4 243 181 205 211 126 ...)
16

;; And now decrypt it using the same context
;; to show the original string

cl-user(23): (blowfish-decrypt * :context cc :string t)
"my message"
cl-user(24): 

It is a common practice to send Blowfish keys to intended recipients using their RSA public keys. See Section 7.2 Support for rsa encryption for information on RSA encryption.

7.2 Support for rsa encryption

RSA is a public key cipher named after its inventors: Rivest, Shamir and Adleman. A public key ciper differs from a symmetric cypher like Blowfish (see Section 7.1 Support for Blowfish encryption) in two important ways:

1. There exist two keys: the Public key and the Private key.
2. Different keys are used for encryption and decryption.
3. One of the keys (the Public key) can be made public without making it possible to compute the other key (the Private key).

With RSA you can encrypt with the Public key and decrypt with the Private key or encrypt with the Private key and decrypt with the Public key. Typically one encrypts with the Public key to send a message to the person with the Private key.

RSA has never been proven to be secure. However the obvious way to crack the encryption involves factoring a very large number. There is no published way of factoring a large number that's better than a brute force attempt of trying all possible factors. Thus by making the key big enough you can be sure that it won't be possible to compute the factors by brute force search in a very long time. There may be other ways to crack RSA encryption that are simply not published yet.

One major downside to RSA is that it roughly 1000 times slower to encrypt and decrypt than a symmetric cipher like Blowfish. As a result people usually use RSA as means of transmitting a key for a symmetric cipher. For example if Alice wants to send Bob a large document securely she'll first go to Bob's web site and copy down his Public RSA key. Then she'll use a random number generator to create a 64 bit blowfish key. She'll encrypt the blowfish key with Bob's Public key and send the result to Bob. Then she'll encrypt her document using Blowfish and the key she generated. Bob will decrypt the first message from Alice using his Private RSA key. That will give him the Blowfish key he'll need to decrypt the second message from Alice.

Because the public key is known to all you have to be careful to not encrypt small values with an RSA public key since that gives you very little security. For example, suppose you decide to encrypt a 4 digit security code using an RSA public key. A person willing to steal your code need only encrypt the values 0000 through 9999 and compare them to your encrypted value to determine what the value encrypted was. If you want to encrypt a 4 digit security code XXXX then it's best to encrypt instead YYYYYYYYXXXX where the Y's are digits chosen randomly.

An RSA key pair consists of three integers: a modulus, a private exponent and a public exponent. The only number that must be kept secret is the private exponent. The public exponent is usually one of a set of common small numbers. The Allegro RSA key generator always chooses 17 as the public exponent.

An RSA key is represented in Allegro as a vector of three values:

1. t if this is the public key, nil if this is the private key. This value is to help you distinguish one key from the other and is not used in the encryption/decryption code.
2. the modulus value (approximately 1024 bits long).
3. the exponent value.

RSA is a block cipher: a sequence of octets are encrypted at once. The block size isn't fixed but is usually determined by the size of the modulus. In order to encrypt data whose length is not a multiple of the block size padding is done at the end of the value and information about the padding is added to the value. The format of this padding information is not standard among rsa encryption functions, thus you can't expect any function except rsa-decrypt to be able to decrypt a value encrypted with rsa-encrypt.

The functions associated with RSA encryption and ecryption are:

• rsa-encrypt: given a value to encrypt and a key, returned the encrypted value.
• rsa-decrypt: given a value returned by rsa-encrypt and a key, return the decrypted value.
• generate-rsa-keys: generates candidate public and private keys. This function takes a long time to complete and is not guaranteed to produce valid keys. Please see the description for details.

Because RSA encryption is resource intensive compared to symetric encoders like Blowfish, it is a common practice to encode using Blowfish and send Blowfish keys to intended recipients using their RSA public keys. See Section 7.1 Support for Blowfish encryption for information on Blowfish encryption.

Here are some examples of Blowfish encryption and decryption:

;; A call to generate-rsa-keys, such as the following, can take 
;; on the order of 10 minutes to complete. The example call
;; could have been made with ':verbose t' to get progress
;; information as it runs. 
;;
;; The return value is a list of the public and
;; private keys, both of which are vectors.  This list
;; is made the value of the variable 'keys'.

cl-user(12): (setq keys (generate-rsa-keys :verbose nil))
(#(t
   46966163069921561627913599098179694383015908573206517049125099728659553054438846018512904176959283177314807123575693727951554341934405797089936585940317631395106826826688294464956210080903479818549199568459705562548422892115525746166751074289213609596618613446079618857135830766959762009927055865884710796501
   17)
 #(nil
   46966163069921561627913599098179694383015908573206517049125099728659553054438846018512904176959283177314807123575693727951554341934405797089936585940317631395106826826688294464956210080903479818549199568459705562548422892115525746166751074289213609596618613446079618857135830766959762009927055865884710796501
   38678016645817756634752375727912689491895454119111249334573611541249043691890814368187097557495880263671017631179983070077750634534216538779947776656732155742116600759081667066940623643773618649101682703783799537062586749082756929026484684965113373035085734095558864183533679809881862774879514342408924287321))



;; Here we encrypt with the public key and decrypt with the private key
;; we could have encrypted with the private key and decrypted with the
;; public key as well.

cl-user(13): (rsa-encrypt "my secret message" (car keys))
#(102 136 69 180 180 27 185 63 132 137 ...)


cl-user(14): (rsa-decrypt * (cadr keys) :string t)
"my secret message"
cl-user(15): 

Copyright (c) 1998-2002, Franz Inc. Oakland, CA., USA. All rights reserved.
Documentation for Allegro CL version 6.2. The object described on this page has been modified in the 6.2 release; see the Release Notes.
Created 2002.2.26.