Common Lisp Package: FAST-IO

README:

# fast-io **Now with [static-vectors](https://github.com/sionescu/static-vectors) support!** ```lisp (deftype octet '(unsigned-byte 8)) (deftype octet-vector '(simple-array octet (*))) ``` Fast-io is about improving performance to octet-vectors and octet streams (though primarily the former, while wrapping the latter). Imagine we're creating messages for the network. If we try and fill an octet-vector with 50 bytes, 50000 times, here are the results (SBCL 1.0.57): <table> <tr> <th></th> <th align=right><tt>vector-push-extend</tt>:</th> <th align=right><tt>flexi-streams</tt>:</th> <th align=right><tt>fast-io</tt>:</th> </tr> <tr> <td align=right>Time:</td> <td align=right>0.767s</td> <td align=right>2.545s</td> <td align=right>0.090s</td> </tr> <tr> <td align=right>Bytes consed:</td> <td align=right>104,778,352</td> <td align=right>274,452,768</td> <td align=right>18,373,904</td> </tr> </table> (See `t/benchmarks.lisp` for the exact code used.) It *should* be surprising that it takes a nontrivial effort to achieve relatively decent performance to octet-vectors, but probably isn't. However, fast-io provides a relatively straightforward interface for reading and writing either a stream or a vector: ```lisp ;;; Write a byte or sequence, optionally to a stream: (with-fast-output (buffer [STREAM | :vector | :static]) (fast-write-byte BYTE buffer)) (with-fast-output (buffer [STREAM | :vector | :static]) (fast-write-sequence OCTET-VECTOR buffer [START [END]])) ;;; Read from a vector or stream: (with-fast-input (buffer VECTOR [STREAM]) (fast-read-byte buffer)) (with-fast-input (buffer VECTOR [STREAM]) (let ((vec (make-octet-vector N))) (fast-read-sequence vec buffer [START [END]]))) ``` ## Multi-byte and Endianness Fast-io provides a host of read and write functions for big- and little-endian reads, in the following forms: * `write[u]{8,16,32,64,128}{-be,-le}`: E.g., `(write32-be VALUE BUFFER)` will write the specified 32-bit value to the specified buffer with a *big-endian* layout. Likewise, `(writeu16-le VALUE BUFFER)` will write an *unsigned* 16-bit value in *little-endian* layout. * `read[u]{8,16,32,64,128}{-be,-le}`: Similarly, `(read64-le BUFFER)` will read a 64-bit value from the buffer with little-endian layout. ## Static Vectors You may now specify `:static` instead of a stream to `WITH-OUTPUT-BUFFER`. This returns an octet-vector created with [static-vectors](https://github.com/sionescu/static-vectors), which means that passing the buffered data directly to a foreign function is now that much more efficient: ```lisp (let ((data (with-fast-output (buffer :static) (buffer-some-data buffer)))) (foreign-send (static-vectors:static-vector-pointer data)) (static-vectors:free-static-vector data)) ``` Note that the restriction for manually freeing the result remains. This avoids multiple inefficient (i.e., byte-by-byte) copies to foreign memory. ## Streams Obviously, the above API isn't built around Lisp streams, or even gray-streams. However, fast-io provides a small wrapper using `trivial-gray-streams`, and supports `{WRITE,READ}-SEQUENCE`: ```lisp (let ((stream (make-instance 'fast-io:fast-output-stream))) (write-sequence (fast-io:octets-from '(1 2 3 4)) stream)) ``` Both `fast-input-stream` and `fast-output-stream` support backing a stream, much like using the plain fast-io buffers. However, using the gray-streams interface is a 3-4x as slow as using the buffers alone. Simple benchmarks show the gray-streams interface writing 1M 50-byte vectors in about 1.7s, whereas simply using buffers is about 0.8s. Consing remains similar between the two.

FUNCTION

Public

Undocumented

FAST-READ-BYTE (INPUT-BUFFER &OPTIONAL (EOF-ERROR-P T) EOF-VALUE)

FAST-READ-SEQUENCE (SEQUENCE INPUT-BUFFER &OPTIONAL (START 0) END)

FAST-WRITE-BYTE (BYTE OUTPUT-BUFFER)

FAST-WRITE-SEQUENCE (SEQUENCE OUTPUT-BUFFER &OPTIONAL (START 0) END)

FINISH-OUTPUT-STREAM (STREAM)

MAKE-OCTET-VECTOR (LEN)

OCTETS-FROM (SEQUENCE)

READ128-BE (BUFFER)

READ128-LE (BUFFER)

READ16-BE (BUFFER)

READ16-LE (BUFFER)

READ32-BE (BUFFER)

READ32-LE (BUFFER)

READ64-BE (BUFFER)

READ64-LE (BUFFER)

READ8 (BUFFER)

READ8-BE (BUFFER)

READ8-LE (BUFFER)

READU128-BE (BUFFER)

READU128-LE (BUFFER)

READU16-BE (BUFFER)

READU16-LE (BUFFER)

READU32-BE (BUFFER)

READU32-LE (BUFFER)

READU64-BE (BUFFER)

READU64-LE (BUFFER)

READU8 (BUFFER)

READU8-BE (BUFFER)

READU8-LE (BUFFER)

WRITE128-BE (VALUE BUFFER)

WRITE128-LE (VALUE BUFFER)

WRITE16-BE (VALUE BUFFER)

WRITE16-LE (VALUE BUFFER)

WRITE32-BE (VALUE BUFFER)

WRITE32-LE (VALUE BUFFER)

WRITE64-BE (VALUE BUFFER)

WRITE64-LE (VALUE BUFFER)

WRITE8 (VALUE BUFFER)

WRITE8-BE (VALUE BUFFER)

WRITE8-LE (VALUE BUFFER)

WRITEU128-BE (VALUE BUFFER)

WRITEU128-LE (VALUE BUFFER)

WRITEU16-BE (VALUE BUFFER)

WRITEU16-LE (VALUE BUFFER)

WRITEU32-BE (VALUE BUFFER)

WRITEU32-LE (VALUE BUFFER)

WRITEU64-BE (VALUE BUFFER)

WRITEU64-LE (VALUE BUFFER)

WRITEU8 (VALUE BUFFER)

WRITEU8-BE (VALUE BUFFER)

WRITEU8-LE (VALUE BUFFER)

Private

INPUT-BUFFER-POS (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

INPUT-BUFFER-STREAM (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

INPUT-BUFFER-VECTOR (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

OUTPUT-BUFFER-FILL (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

OUTPUT-BUFFER-LAST (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

OUTPUT-BUFFER-LEN (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

OUTPUT-BUFFER-OUTPUT (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

OUTPUT-BUFFER-QUEUE (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

OUTPUT-BUFFER-VECTOR (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

Undocumented

CONCAT-BUFFER (BUFFER)

COPY-INPUT-BUFFER (INSTANCE)

COPY-OUTPUT-BUFFER (INSTANCE)

EXTEND (BUFFER &OPTIONAL (MIN 1))

FINISH-OUTPUT-BUFFER (OUTPUT-BUFFER)

FLUSH (OUTPUT-BUFFER)

INPUT-BUFFER-P (OBJECT)

SETFINPUT-BUFFER-POS (NEW-VALUE INSTANCE)

SETFINPUT-BUFFER-STREAM (NEW-VALUE INSTANCE)

SETFINPUT-BUFFER-VECTOR (NEW-VALUE INSTANCE)

MAKE-INPUT-BUFFER (&KEY ((VECTOR DUM54) NIL) ((POS DUM55) 0) ((STREAM DUM56) NIL))

MAKE-OUTPUT-BUFFER (&KEY ((VECTOR DUM0) (MAKE-OCTET-VECTOR *DEFAULT-OUTPUT-BUFFER-SIZE*)) ((FILL DUM1) 0) ((LEN DUM2) 0) ((QUEUE DUM3) NIL) ((LAST DUM4) NIL) ((OUTPUT DUM5) NIL))

SETFOUTPUT-BUFFER-FILL (NEW-VALUE INSTANCE)

SETFOUTPUT-BUFFER-LAST (NEW-VALUE INSTANCE)

SETFOUTPUT-BUFFER-LEN (NEW-VALUE INSTANCE)

SETFOUTPUT-BUFFER-OUTPUT (NEW-VALUE INSTANCE)

OUTPUT-BUFFER-P (OBJECT)

SETFOUTPUT-BUFFER-QUEUE (NEW-VALUE INSTANCE)

SETFOUTPUT-BUFFER-VECTOR (NEW-VALUE INSTANCE)

SIGNED-TO-UNSIGNED (VALUE SIZE)

UNSIGNED-TO-SIGNED (VALUE SIZE)

MACRO

Public

WITH-FAST-OUTPUT ((BUFFER &OPTIONAL OUTPUT) &BODY BODY)

Create `BUFFER`, optionally outputting to ``.

Undocumented

WITH-FAST-INPUT ((BUFFER VECTOR &OPTIONAL STREAM) &BODY BODY)

Private

Undocumented

MAKE-READERS (&REST BITLENS)

MAKE-WRITERS (&REST BITLENS)

READ-UNSIGNED-BE (SIZE BUFFER)

READ-UNSIGNED-LE (SIZE BUFFER)

WRITE-UNSIGNED-BE (VALUE SIZE BUFFER)

WRITE-UNSIGNED-LE (VALUE SIZE BUFFER)

VARIABLE

Public

Undocumented

*DEFAULT-OUTPUT-BUFFER-SIZE*

CLASS

Public

Undocumented

FAST-INPUT-STREAM

FAST-OUTPUT-STREAM

Private

Undocumented

INPUT-BUFFER

OUTPUT-BUFFER