Streams

$Revision: 5.0.2.3 $

1.0 Introduction to streams in Allegro CL
2.0 Documenting object-oriented protocols
3.0 Stream classes
4.0 Generic functions for character input
5.0 Generic functions for character output
6.0 Generic functions for binary streams
7.0 Functions for efficient input and output of sequences
8.0 Creating streams
9.0 Miscellaneous stream functions

1.0 Introduction to streams in Allegro CL

Streams in Common Lisp have always been first-class objects and the stream type a first-class type. The inability of user code to customize or extend stream behavior has been an unfortunate limitation especially because the capabilities of the Common Lisp reader and printer can only be obtained through the interface of a stream. Suppose, for example, one wants to do normal Lisp printing while performing some simple output character translations. The only portable way to do this -- that is, without knowing about each implementation's internal functions -- would be to rewrite the whole printer from scratch for the single purpose of interposing a translation function around every call to write-char and write-string. This is clearly unacceptable. It should be possible to customize a stream to add such simple behavior without reimplementing large portions of Common Lisp.

When Common Lisp was first defined the Common Lisp Object System (CLOS) was not yet conceived. Some portable stream construction utilities were originally provided by the several "indirect" stream types - synonym-stream, concatenated-stream, etc. - but these provide only very limited kinds of customization. The subsequent ANSI standard for Common Lisp includes CLOS, so that is now the obvious mechanism to support user-customizable stream types.

Fairly late in the ANSI standardization process there was some consideration of closifying streams. The most complete proposal was by David N. Gray, then of Texas Instruments. Gray's proposal was only a draft and it acknowledged a number of problems and omissions. It was intended only as a starting point towards a complete specification. However, Gray and the standardization committee soon decided it best not to act because the change to the language would be large (causing a lot of work for implementors) and also because there was little actual experience with implementations of CLOS streams. Despite the feeling that it was not yet the time to adopt closified streams into the language standard, most members of the committee feel that redefining streams in terms of CLOS would be intrinsically worthwhile, and experimentation with extensions should be encouraged.

The implementation of streams in Allegro CL since release 4.0 (on Unix) is based largely on the X3J13 issue writeup entitled "STREAM-DEFINITION-BY-USER, Version 1, 22-Mar-89 by David N. Gray." Some of the text in this chapter is taken from that proposal, but there are numerous additions, modifications, clarifications, and comments specific to the Allegro implementation.

An important feature of the Gray proposal is that it is upward compatible with both the current language standard and earlier implementations of Allegro CL. Programmers need not know anything about closified streams unless they actually use its features.

The symbols naming classes and generic functions for the CLOS stream interface are exported from the excl package. In earlier releases on Unix, they were exported from the stream package. However, the stream package in 5.0 has been merged into the excl package, and stream has been made a nickname for excl.

The existing Common Lisp I/O functions cannot be made generic because in nearly every case the stream argument is optional and therefore cannot be specialized. It is therefore necessary to define new generic functions which are called internally by the standard Common Lisp functions. In order to make the meaning as obvious as possible, the names of the generic functions have been formed by prefixing "stream-" to the corresponding non-generic function (e.g. cl:terpri and excl:stream-terpri). Note that for all of these generic functions, the stream argument must be a stream object, not t or nil.

Having the generic input functions consistently return :eof at end-of-file, with the higher-level functions handling the eof-error-p and eof-value arguments, simplifies the generic function interface and makes it more efficient by not needing to pass through those arguments. Note that the functions that use this convention can only return a character or integer as a stream element, so there is no possibility of ambiguity.

2.0 Documenting object-oriented protocols

There is much uncertainty in the industry about how to document an object-oriented protocol. There are many ways to do it wrong, and it is far from clear how to do it right. Of course, problems with documentation can often be ascribed to poor design of the underlying system itself or else design based on assumptions that are not made explicit.

For example, the default method for stream-write-string is defined to do repeated calls to stream-write-char. The character-translation problem mentioned at the beginning of section 1 above could be implemented minimally by defining an :around method for stream-write-char, and everything ought to work. But what if some stream specialization -- say, the file-stream classes provided by the implementation -- optimizes out the repeated calls to stream-write-char in the interest of efficiency? The Gray proposal is silent whether this would conform, but it is clear that independently-written mixin classes will need to know each other's assumptions about which publicly-specializable generic functions do or do not call each other's publicly-specializable generic function. There is also controversy whether there should be a default method for a particular generic function or whether a method definition should be required on specialized stream classes. The difference is important if mixin methods normally do a call-next-method, because the question is left open whether the default method should be, should not be, or may optionally be shadowed.

We don't answer all these questions definitely here in part because we are still unsure of the answers. A better handle on the issues may develop over time and this specification may evolve accordingly.

This said, there are several kinds of information this document needs to specify:

The public superclasses (base and mixin) that may be used to define customized stream classes.
The generic functions that are defined on these classes, when and how they are called, and which ones call others.
The methods that are defined on these generic functions by the implementation, and what functionality is required if the user chooses to override any builtin methods.
Which builtin methods and built-in nongeneric functions call other public generic functions.
Which methods are required of any specialization of a class that are not provided by default methods on base classes.
Which public superclasses (if any) are required to be mixed into a user-customized class.

It is a legitimate criticism that this chapter does not exhaustively cover all these details. The legitimate excuse is that cogent, defensible design decisions have not yet been made for all of them. In particular, there is something of a trade-off between (5) and (6) depending on whether one believes subclassing should be defined in terms of class inheritance or in terms of generic function behavior protocol. It could be said that the current Allegro CL implementation still sits on the fence.

3.0 Stream classes

Two kinds of classes are mentioned here. Some are "mixin" classes intended to be used as super classes of user-defined stream classes. They are not intended to be directly instantiated; they primarily provide places to hang default methods. Others are classes actually instantiated by Allegro CL, for example, to service a call to open.

Those classes that are sufficiently complete to be meaningfully instantiated are labeled as Instantiable Class in their description pages while mixin classes are labeled simply as Class. You can, of course, further subclass both kinds of classes. Although most stream classes have their own description page, the pages do not contain more information than is present in this document and so we have not provided links.

Table of mixin (non-instantiable) stream classes
Class	Notes
`fundamental-stream`	This class is a subclass of `stream` and of standard-object. streamp will return true for an instance of any class that includes this.
`fundamental-input-stream`	A subclass of `fundamental-stream`. Its inclusion causes input-stream-p to return true. Note: any user-defined stream class that will do input must include this class. Bidirectional streams may be formed by including subclasses of both `fundamental-output-stream` and `fundamental-input-stream`.
`fundamental-output-stream`	A subclass of `fundamental-stream`. Its inclusion causes output-stream-p to return true. Note: any user-defined stream class that will do output must include this class. Bidirectional streams may be formed by including subclasses of both `fundamental-output-stream` and `fundamental-input-stream`.
`fundamental-character-stream`	A subclass of `fundamental-stream`. It provides a method for stream-element-type which returns `character`.
`fundamental-binary-stream`	A subclass of `fundamental-stream`. The Allegro CL implementation requires the `:element-type` keyword be provided to make-instance for streams of this class.
`fundamental-character-input-stream`	Includes `fundamental-input-stream` and `fundamental-character-stream`. Any user-defined stream class that will be used as an argument to read and friends must include this class.
`fundamental-character-output-stream`	Includes `fundamental-input-stream` and `fundamental-character-stream`. Any user-defined stream class that will be used as an argument to print, format, etc. must include this class.
`fundamental-binary-input-stream`	Includes `fundamental-input-stream` and `fundamental-binary-stream`.
`fundamental-binary-output-stream`	Includes `fundamental-output-stream` and `fundamental-binary-stream`.
`cl:file-stream`	See just below.
`cl:string-stream`	See just below.

ANSI Common Lisp mandates these seven subtypes of type stream: file-stream, string-stream, echo-stream, concatenated-stream, two-way-stream, broadcast-stream, and synonym-stream. The first two are mixin and the rest and instantiable.

Since in Allegro CL's implementation stream is a CLOS class (more precisely, a subclass of clos:standard-object) then the subtypes of stream must also be CLOS classes.

The stream types other than file-stream and string-stream are sometimes called indirect streams. The creator functions for these stream types (make-synonym-stream, make-echo-stream, make-broadcast-stream, make-concatenated-stream, and make-two-way-stream) are unmodified from the standard Common Lisp definitions. These stream types represent an early (and awkward) attempt in Common Lisp to obtain part of an extensible stream facility. These stream types may not be defined in an image but the mdoule defining them, streama, will be loaded if it is require'd or if (find-class 'x) is evaluated or make-x is called (where x is one of the classes). streama is also loaded automatically if any of the Common Lisp slot-readers for one of these streams is called. (streama stands for stream-ansi. The other module is streamc, which stands for stream-clos. It is loaded into every Lisp.)

The generic function approach does not integrate very well with indirect streams because the indirect stream classes cannot anticipate the full set of generic functions that user code may want to define over all streams in order to indirect them to the contained streams. It is arguable that no purely automatic mechanism can handle this, not even by defining a general method for no-applicable-method. For example, consider the problem a two-way-stream has choosing whether to pass to its input stream, output stream, or both a call to a generic function about which it knows nothing. For this reason there has been no attempt to extend or make customizable the five indirect streams.

The five indirect stream classes are instantiable, but at present cannot successfully be instantiated other than with their standard constructor functions (e.g. make-synonym-stream).

The file-stream class is customizable since it embodies the interface to the file system. file-stream is a mixin not intended to be instantiated directly. Its subclasses are normally instantiated by the open function (see below) although it is also possible to create them directly with make-instance. The string-stream subclass does not presently support customization.

Table of instantiable stream classes
Class	Notes
`cl:echo-stream`	See discussion above. Created with cl:make-echo-stream.
`cl:concatenated-stream`	See discussion above. Created with cl:make-concatenated-stream.
`cl:two-way-stream`	See discussion above. Created with cl:make-two-way-stream.
`cl:broadcast-stream`	See discussion above. Created with cl:make-broadcast-stream.
`cl:synonym-stream`.	See discussion above. Created with cl:make-synonym-stream.
`input-terminal-stream`	All six classes implement streams intended to support connection to sockets and input-output devices that connect to a "stream" of data rather than a fixed file stored in a file system. A normal call to open without the `:class` argument extension always creates a file stream. The initial value of `terminal-io` at startup is a `bidirectional-terminal-stream` stream. One essential difference between these two groups is evident for the bidirectional versions. The input and output sides of a bidirectional file stream access the same data storage (e.g. on a disk) and so need to cooperate closely about buffering when input and output operations are interleaved. They also share a single file-position pointer. The input and output sides of a socket stream are completely separate channels, and socket streams don't support file position at all. These classes all require an `:fn-in` and/or `:fn-out` initialization argument which should be a small integer that is a Unix file descriptor. (On Windows there is an internal hidden translation between the small integers and actual Windows "handles".) An error will be signaled if the required initializer is omitted. The input and output file descriptor of a bidirectional stream may be and typically are the same.
`output-terminal-stream`
`bidirectional-terminal-stream`
`input-binary-socket-stream`
`output-binary-socket-stream`
`bidirectional-binary-socket-stream`

4.0 Generic functions for character input

A character input stream class can be defined by including fundamental-character-input-stream and defining methods for the generic functions below. The builtin instantiable classes in Allegro CL already have appropriate methods, of course.

Generic function	Arguments	Notes
`stream-read-char`	stream	This reads one character from stream.
`stream-unread-char`	stream character	Undoes the last call to stream-read-char, as in unread-char. Returns `nil`.
`stream-read-char-no-hang`	stream	This is used to implement read-char-no-hang. It returns either a character, or `nil` if no input is currently available, or `:eof` if end-of-file is reached.
`stream-peek-char`	stream	Used to implement peek-char; this corresponds to the case where the peek-type argument is `nil`. It returns either a character or `:eof`.
`stream-listen`	stream	Used by listen. Returns true or false as there is or is not a characters available to be read.
`stream-read-line`	stream	Used by read-line. A string is returned as the first value. The second value is true if the string was terminated by end-of-file instead of the end of a line.
`stream-clear-input`	stream	Implements clear-input for stream, returning `nil`.

5.0 Generic functions for character output

A character output stream can be created by defining a class that includes fundamental-character-output-stream and defining methods for the generic functions below.

Generic function	Arguments	Notes
`stream-write-char`	stream character	Writes character to stream and returns character.
`stream-line-column`	stream	This function returns the column number where the next character will be written, or `nil` if that is not meaningful for stream. The first column on a line is numbered 0.
`stream-start-line-p`	stream	This is a predicate which returns `t` if stream is positioned at the beginning of a line, else returns `nil`. It is permissible to always return nil. This is called by stream-fresh-line.
`stream-write-string`	stream string &optional start end	Implements write-string. It writes string to stream, optionally delimited by start and end, which default to 0 and `nil`. The string argument is returned.
`stream-terpri`	stream	Writes an end of line, as for terpri. Returns `nil`.
`stream-fresh-line`	stream	Implements fresh-line.
`stream-finish-output`	stream	Implements finish-output
`stream-force-output`	stream	Implements force-output.
`stream-advance-to-column`	stream column	Writes enough blank space so that the next character will be written at the specified column. Returns true if the operation is successful, or nil if it is not supported for stream.
`stream-clear-input`	stream	Implements clear-input.

6.0 Generic functions for binary streams

A binary stream class can be defined by including either or both of fundamental-binary-input-stream and fundamental-binary-output-stream, and methods for one or both of the following generic functions. If you create a binary stream other than by using open you must also include a keyword initialization argument for :element-type.

Generic function	Arguments	Notes
`stream-read-byte`	stream	Used by read-byte; returns either an integer, or the symbol :eof if stream is at end-of-file.
`stream-write-byte`	stream integer	Used by write-byte; writes the integer to stream and returns the integer as the result.

7.0 Functions for efficient input and output of sequences

In the public comments on the first Draft Proposed ANSI Standard several reviewers noted that except for the write-string function, the language provided no means to request efficient input-output on large blocks of data. The ANSI standard consequently added two new functions to the language.

[Function]
lisp:read-sequence

Arguments: sequence stream &key :start :end

[Function]
lisp:write-sequence

Arguments: sequence stream &key :start :end

read-sequence destructively modifies the sequence argument, storing elements read from the stream. write-sequence successively writes elements of the sequence to the stream. As with other sequence functions, start defaults to 0 and end defaults to the length of the sequence.

The return value of read-sequence is the index of the first element not modified; eof is indicated if the return value is the same as start. The return value of write-sequence is the original sequence.

The stream argument must be a real stream; t and nil do not have special meanings as they do for some other stream functions.

The effect of these two functions is the same as iterating over the sequence performing the input-output operations one element at a time, but with the possibility of more efficient execution. Such optimization is to be expected only when the sequence is a vector with the same upgraded element-type as the stream. In any case, actual type errors are possible on input or output if an attempt is made to transfer an element incompatible with, respectively, the sequence or the stream.

Allegro CL also defines generic function versions of these functions. The nongeneric versions are implemented by calling these generic functions. Methods are defined for these functions that handle all legal calls, but not all legal calls will have highly efficient execution. The existing methods will handle vector sequence arguments efficiently, and will also handle start and end arguments efficiently. The Allegro CL implementation will also accept and efficiently transfer data to and from higher-dimension arrays in the usual row-major order, although this is an extension to the language and not defined by ANSI Common Lisp.

Generic function	Arguments	Notes
`stream-read-sequence`	stream sequence &optional start end	Destructively modifies sequence, storing in it elements read from stream.
`stream-write-sequence`	stream sequence &optional start end	Writes elements of sequence to stream.

The definition of these functions does not permit the useful ability to operate on sequence elements different from the stream element type. Such capability would clearly be desirable. (It would, for example, permit floats to be transferred efficiently, and allow an application needing to store several different successive sequences of different types to write them to a single stream and later reread the data.) However, the behavior of any such operation would depend on bit representations of objects in an implementation- and machine-dependent way, and would be incompatible with the strict definition of read-sequence and write-sequence, which require automatic conversion of data to the correct element type of the stream or sequence. Any such capability would therefore need to be implemented by a different pair of functions.

8.0 Creating streams

Streams of arbitrary class may be constructed by an explicit call to make-instance, but the Gray proposal did not address how to customize the stream created by open. We define a simple interface here. The Gray proposal also omits mention of constructor functions such as make-string-input-stream, make-string-output-stream, and their associated macros such as with-output-to-string. However, there is nothing these various operators do that can't be performed explicitly by user code including a call to make-instance. Unfortunately, Allegro CL's current string-stream subclasses do not (reliably) support specialization or even independent instantiation by make-instance. This is a bug in that some required initialization is performed by the make-string-...-stream function. This may be fixed in a future release.

[Function]
open

Arguments: pathname &rest initargs &key ... class &allow-other-keys

The open function has been extended to take a class keyword argument. open passes this argument to make-instance when it creates the stream, and as with make-instance, the argument may be a stream class object or a symbol naming such a class. If the class argument is not supplied or is nil, open selects one of the following built-in classes according to the direction and element-type arguments:

excl::character-input-file-stream
excl::character-output-file-stream
excl::character-bidirectional-file-stream
excl::binary-input-file-stream
excl::binary-output-file-stream
excl::binary-bidirectional-file-stream

These classes all contain file-stream and are variously mixed with

fundamental-character-input-stream
fundamental-character-output-stream
fundamental-binary-input-stream
fundamental-binary-output-stream

Although the file-stream subclasses returned by open are all instantiable, at present they require hidden initialization (for element-type upgrading, buffer allocation, etc.) and therefore they should only be created using open. It is fine to further specialize them, but you are required to create instances of your specializations of these stream classes using the class keyword argument to open rather than by calling make-instance yourself.

On Unix platforms, open will select a terminal-stream subclass rather than a file-stream subclass if it detects the argument file is a character special device or a named pipe. See socket.htm for more information about creating sockets in Allegro CL.

open is also modified with &allow-other-keys and &rest to pass all keyword arguments as initialization arguments to make-instance. This has the unfortunate side effect of removing error checking for misspelled keyword arguments. A useful future enhancement might be to code some explicit checks which are executed when the class keyword argument is not given.

9.0 Miscellaneous stream functions

The following functions are also defined (or in the case of standard Common Lisp functions, extended).

Generic function	Arguments	Notes
`cl:close`	stream &key abort	The existing function close is redefined to be a generic function but otherwise behaves as required by standard Common Lisp.
`cl:open-stream-p`	stream	This function is made generic. A default method is provided by class fundamental-stream which returns true if close has not been called on stream.
`cl:streamp`	object	The original proposal allowed but did not require these three existing predicates to be implemented as generic functions. In Allegro CL, streamp is not a generic function in order not to impact speed of opencoded type dispatching, but the other two functions are made generic. Normally, the default methods provided by classes fundamental-input-stream and fundamental-output-stream are sufficient.
`cl:input-stream-p`	stream
`cl:output-stream-p`	stream
`cl:stream-element-type`	stream	This existing function is made generic, but otherwise behaves the same. Class fundamental-character-stream provides a default method which returns character.
`stream-yes-or-no-p`	stream &optional format-string &rest args	The generic function analogues of cl:yes-or-no-p and cl:y-or-n-p.
`stream-y-or-n-p`	stream &optional format-string &rest args
`stream-input-fn`	stream	These accessors return the input or output file descriptor for streams that have them, or `nil`. For some streams the input and output file descriptors may be the same.
`stream-output-fn`	stream

The following function is used by the pretty printer to determine the output width to be used while pretty printing. Actually, the pretty printer uses the first of the following three values that returns non-nil.

the value of cl:*print-right-margin*
the value returned by stream-output-width
the value of excl::*default-right-margin*, an internal variable set when Lisp starts up to the apparent width of cl:*terminal-io*, if it can be determined, and to 72 otherwise.

Generic function	Arguments	Notes
`stream-output-width`	stream	Returns an integer width of stream or `nil`. The default method for this function returns `nil` for all streams.