F RANZ I NC. An In-Depth Look at Simple Streams By Duane Rettig October, 2002.

FRANZ INC.

An In-Depth Look at Simple Streams

By Duane Rettig

October, 2002

FRANZ INC.

What is a Lisp Stream?

Lisp

stream

stream

externaldevice

internal device

FRANZ INC.

An In-Depth Look at Simple Streams

• Simple Streams Design Goals• What is a Simple Stream?• Simple-stream Concepts

– Opening, closing, blocking, reading, writing

• Character Strategies• Common Windows Rewrite• Further changes in next version (references

distributed throughout presentation)

FRANZ INC.

Design Goals

• CL compliance• Bivalence• Thin Strategy layer that is uniform• Extensibility via specializations/mixins and

encapsulations• Minimal decision points in critical strategy code• Minimal calls to generic-functions• External-format ready for 8 and 16 bit lisps

FRANZ INC.

Gray vs. Simple Stream

Object functionality

buffer

API call

Strategy

Gray Stream

Simple Stream

Device interface

FRANZ INC.

Dual-channel stream

• Two octet buffers• External-format character translation

Ext. format

Device methods

Out buffer

API

In buffer

External device

FRANZ INC.

Single-channel stream

• One octet buffer• External-format character translation

Ext. format

Device methods

buffer

API

direction

Internal or External device

FRANZ INC.

String stream

• Zero, one, or two string buffers• No external-format translations• Plugable character/buffer strategies

Device methods

Out buffer

API

In buffer

Internal device

FRANZ INC.

Opening a Stream

• Creates or reuses a stream object

• Makes or retains a connection to a device

• Initializes character strategies

• Marks the stream as open

FRANZ INC.

Open

open make-instance (gf) ... shared-initialize (gf) shared-initialize :after [Method] device-open [Method]

FRANZ INC.

(Open): Simple encapsulation

(setq bun (open "sesame"))

(setq big-mac (make-instance 'all-beef :input-handle bun ...))

bunbig-mac

all-beef sesame

FRANZ INC.

(Open): shared-initialize

(defmethod shared-initialize :after ((stream simple-stream) slots

&rest initargs) (declare (ignore slots)) (unless (device-open stream initargs) (device-close stream t)) stream)

FRANZ INC.

(Open): device-open

• At device-open time:– All instance slots have been initialized– connection has been made or will be made by

device-open

• When device-open returns:– If non-nil, the stream is ready for appropriate

actions – If nil, then the open has failed.

FRANZ INC.

(Open): device-open

• device-open must:– Ensure a connection before returning– Ensure buffers are in place– Initialize/reinitialize pointers– Return nil for failure, or non-nil after success

• device-open must not:– Assume that the stream was closed– close the stream

FRANZ INC.

(Open): subclass example

(defclass file-with-header (file-simple-stream) ((header :initform nil :accessor file-header-info)))

(defmethod device-open ((stream file-with-header) options) (declare (ignore options)) (let ((success (call-next-method))) (when success (setf (file-header-info stream) (read-file-header stream)) t)))

FRANZ INC.

(Open): device-open :before methods

• In future releases, for string-simple-streams:– A catch-all primary method returns true for an

open stream.– Strategy installation functions are directional and

do not override.• install-string-character-strategy deprecated, replaced

by install-string-{input,output}-character-strategy

– All :before methods fire according to CPL, and may shadow default actions.

FRANZ INC.

(Open): device-open: string-input (future)

(defmethod device-open :before ((stream string-input-simple-stream) options) (with-stream-class (string-input-simple-stream stream) (let ((string (getf options :string))) (when (and string (null (sm buffer stream))) (let ((start (getf options :start)) (end (or (getf options :end) (length string)))) (setf (sm buffer stream) string (sm buffpos stream) start (sm buffer-ptr stream) end))))) (install-string-input-character-strategy stream) (add-stream-instance-flags stream :string :input :simple))

FRANZ INC.

(Open): device-open: string-output (future)(defmethod device-open :before ((stream string-output-simple-stream) options) (with-stream-class (string-output-simple-stream stream) (unless (sm out-buffer stream) (let ((string (getf options :string))) (if string (setf (sm out-buffer stream) string (sm max-out-pos stream) (length string)) (let ((buflen (max (device-buffer-length stream) 16))) (setf (sm out-buffer stream) (make-string buflen) (sm max-out-pos stream) buflen))))) (unless (sm control-out stream) (setf (sm control-out stream) *std-control-out-table*))) (install-string-output-character-strategy stream) (add-stream-instance-flags stream :string :output :simple))

FRANZ INC.

Closing a Stream

• Flushes output if any.• Breaks connection to device.• Secures against accidental future operations.• Does not change-class.

(defmethod close ((stream simple-stream) &key abort) (device-close stream abort))

FRANZ INC.

Close(defmethod device-close :around ((stream simple-stream) abort) (let (res) (when (pseudo::open-stream-p stream) (unwind-protect (progn (when (output-stream-p stream) (ignore-errors (if abort (clear-output stream) (force-output stream)))) (setq res (call-next-method))) (without-interrupts (pseudo::unset-open-flags stream) (setf (stream-input-handle stream) nil (stream-output-handle stream) nil)) (setf (stream-external-format stream) (find-external-format :void)) res))))

FRANZ INC.

(Close): device-close

• device-close should:– flush all data (unless aborting)– disconnect handles of any encapsulated streams– call lower-levels to close as necessary

• device-close should not:– operate on a closed stream– close an encapsulated stream

FRANZ INC.

Blocking

• Issues with listen– read direction only– covers character availability and not blocking– assumes character as the basic data unit

• stream-listen (carried over from Gray streams)

• read-no-hang-p and write-no-hang-p

FRANZ INC.

Blocking styles

• Non-blocking

• Blocking

• B/NB (blocking, then non-blocking)

Character: One element

Octet: One element

FRANZ INC.

Reading and Writing

• Mostly symmetrical

• Device methods obey B/NB discipline

FRANZ INC.

Basic read strategy

(block read (when (>= buffpos buffer-ptr) (let ((res (device-read stream nil 0 buffpos blocking))) (when (< res 0) (pseudo::do-error-handling)) (when (= res 0) (if blocking (pseudo::do-eof-handling) (return-from read nil))) (setq buffer-ptr res buffpos 0))) (prog1 (aref buffer buffpos) (incf buffpos)))))

FRANZ INC.

Basic write strategy

(block write (when (>= out-pos max-out-pos) (when (> out-pos 0) (erroring-device-write-whole stream nil 0 out-pos t)) (setq out-pos 0)) (setf (aref buffer out-pos) value) (incf out-pos))

FRANZ INC.

Implementing force-output, finish-output

• Both are implemented using device-write

• blocking argument determines whether force (nil) or finish (t)

• device-write buffer argument is :flush (future)

FRANZ INC.

Reading and Writing Sequences

• read-sequence and write-sequence– are width sensitive and by-element– require blocking semantics– do not return a count

• read-vector and write-vector– are octet-based– employ B/NB semantics– return a count, 0, or error code

FRANZ INC.

Deprecated generic function: device-extend

• History and relationship to device-read and device-write– structural differences– purity of reference

• Problems– one generic-function for two directions– inconsistent interface

FRANZ INC.

Replacement of device-extend

• Actions :input and :input-check become blocking argument to device-read

• Actions :output and :output-check become blocking argument to device-write

• Extra actions come through via device-read and device-write buffer argument

FRANZ INC.

Record orientation

• Via device-finish-record gf– dual-channel input– string output

FRANZ INC.

Other generic-functions

• device-buffer-length

• device-clear-input

• device-clear-output

• device-file-length

• device-file-position

FRANZ INC.

Strategies

• What is a strategy’s purpose?– To satisfy the high-level requirements of a

specified behavior in a uniform manner.

• Octet strategies are non-programmable.– read-byte, write-byte are cast in concrete.

• Character strategies are programmable...

FRANZ INC.

Character strategies

• Programmable, replaceable.

• External-formats use template functions for dual-channel and single-channel streams.

• String-streams use simple strategy functions.

• (future): Subclassed strategy functions can shadow more general ones.

• Occupy “joint” slots in the stream.

FRANZ INC.

Example strategy set for *terminal-io*

cl-user(5): :iterminal-simple-stream @ #x711e057a = #<terminal-simple-stream [initial terminal io] fd 0/1 @ #x711e057a> 0 Class --------> #<standard-class terminal-simple-stream> 1 j-unread-char -> #<Function dual-channel-unread-char> 2 j-write-chars -> #<Function (:efft dc-write-chars :latin1-base)> 3 j-write-char -> #<Function (:efft dc-write-char :latin1-base)> 4 j-read-chars -> #<Function (:efft dc-read-chars :latin1-base)> 5 j-read-char --> #<Function (:efft dc-read-char :latin1-base)> 6 j-listen -----> #<Function (:efft dc-listen :latin1-base)> ... 34 src-position-table -> The symbol nilcl-user(6):

FRANZ INC.

Example strategy set for string-output

cl-user(9): :istring-output-simple-stream @ #x719cb5e2 = #<string-output-simple-stream "" pos 0 @ #x719cb5e2> 0 Class --------> #<standard-class string-output-simple-stream> 1 j-unread-char -> The symbol nil 2 j-write-chars -> #<Function string-output-write-chars> 3 j-write-char -> #<Function string-output-write-char> 4 j-read-chars -> The symbol nil 5 j-read-char --> The symbol nil 6 j-listen -----> The symbol nil ... 33 out-buffer ---> A simple-string (4096) that starts "xxxxxxxxxxxxxxxx”cl-user(10):

FRANZ INC.

Example strategy set for string-input

cl-user(14): :istring-input-simple-stream @ #x719e74aa = #<string-input-simple-stream "abc" pos 0 @ #x719e74aa> 0 Class --------> #<standard-class string-input-simple-stream> 1 j-unread-char -> #<Function string-input-unread-char> 2 j-write-chars -> The symbol nil 3 j-write-char -> The symbol nil 4 j-read-chars -> #<Function string-input-read-chars> 5 j-read-char --> #<Function string-input-read-char> 6 j-listen -----> #<Function string-listen> ... 31 src-position-table -> The symbol nilcl-user(15):

FRANZ INC.

j-read-char

• Args: stream eof-error-p eof-value blocking

• Implements read-char and read-char-no-hang functionality directly (after argument resolution)

• blocking argument is nil/true dichotomy

FRANZ INC.

j-write-char

• Args: character stream

• Implements write-char directly (after argument resolution)

• Writing nil as the character argument will flush an external-format’s output state.

FRANZ INC.

j-read-chars

• Args: stream string search start end blocking

• Implements read-sequence, read-line

• blocking argument is nil, :bnb, or true

FRANZ INC.

j-write-chars

• Args: stream string start end

• Implements write-sequence, write-string

• Always blocks (for now)

FRANZ INC.

j-listen

• Args: stream

• Implements stream-listen directly

• Must have a complete character or an error condition in order to return true.

FRANZ INC.

j-unread-char

• Args: stream relaxed

• May need to unread multiple characters as part of a composing or encapsulating character.

• relaxed allows non-error unreads beyond the first unread.

FRANZ INC.

Taking strategies beyond ANS

• Other strategies can be created for operations that don’t fit into CL standard functionalities.

• Example: one-buffer ring fifo queue

FRANZ INC.

Common Windows low-level rewrite

• Source code comparison between Gray and simple streams

• Demo