Common Lisp Package: UK.CO.DEOXYBYTE-GZIP

The deoxybyte-gzip system provides a Lisp interface to Zlib including a regular function interface to gzipped files, a Gray-streams interface to gzipped files and utility gzip/gunzip functions built on the former. Functions are provided for inflating and deflating to and from Lisp octet vectors and Lisp octet streams, which may be tuned using the Zlib tuning parameters described in the Zlib C function deflateInit2. In addition, a basic implementation of the data structure described in RFC1952 is included, allowing a hybrid approach to reading gzip data, using native Lisp streams and Zlib inflate/deflate.

README:

FUNCTION

Public

COMPRESS (SOURCE DEST &KEY (SOURCE-START 0) SOURCE-END (DEST-START 0) (COMPRESSION +Z-DEFAULT-COMPRESSION+))

Compresses bytes in array SOURCE to array DEST, returning DEST and the compressed size in bytes.

DEFLATE-STREAM (SOURCE DEST &KEY (BUFFER-SIZE +DEFAULT-ZLIB-BUFFER-SIZE+) (COMPRESSION +Z-DEFAULT-COMPRESSION+) SUPPRESS-HEADER (WINDOW-BITS 15) (MEM-LEVEL 8) (STRATEGY DEFAULT-STRATEGY))

Deflates stream SOURCE to stream DEST. Arguments: - source (stream): A binary input stream. - dest (stream): A binary output stream. Key: - buffer-size (fixnum): The size of the internal buffer used by Zlib. - compression (fixnum): The Zlib compression factor (0-9, inclusive). - suppress-header (boolean): Exposes a feature of Zlib whereby the compressed data may be produced without a Zlib header, trailer or checksum. - window-bits (fixnum): The Zlib window-bits argument (9-15, inclusive). - mem-level (fixnum): The Zlib mem-level argument (1-9, inclusive). - strategy (fixnum): The Zlib strategy argument (one of :default-strategy , :filtered or :huffman-only ). Returns: - Number of bytes read. - Number of bytes written.

DEFLATE-VECTOR (SOURCE DEST &KEY (COMPRESSION +Z-DEFAULT-COMPRESSION+) SUPPRESS-HEADER (WINDOW-BITS 15) (MEM-LEVEL 8) (STRATEGY DEFAULT-STRATEGY) (BACKOFF 0))

Deflates vector SOURCE to vector DEST. Arguments: - source (vector): An octet vector. - dest (vector): An octet vector. Key: - compression (fixnum): The Zlib compression factor (0-9, inclusive). - suppress-header (boolean): Exposes an undocumented feature of Zlib whereby the compressed data may be produced without a Zlib header or checksum. - window-bits (fixnum): The Zlib window-bits argument (9-15, inclusive). - mem-level (fixnum): The Zlib mem-level argument (1-9, inclusive). - strategy (fixnum): The Zlib strategy argument (one of :default-strategy , :filtered or :huffman-only ). Returns: - The DEST vector, containing compressed data. - Number of bytes read. - Number of bytes written (consequently the end position of the compressed data in DEST).

GUNZIP (IN-FILESPEC &OPTIONAL OUT-FILESPEC)

Decompresses IN-FILESPEC to OUT-FILESPEC using the default compression level. Returns two values, OUT-FILESPEC and the number of bytes decompressed.

GUNZIP-PATHNAME (PATHNAME)

Returns a copy of PATHNAME. Any GZ type component is removed and the name component parsed to supply the new type, if possible. For example: foo.tar -> foo.tar foo.tar.gz -> foo.tar

GZ-CLOSE (GZ)

Closes GZ, if open.

GZ-EOF-P (GZ)

Returns T if GZ has reached EOF, or NIL otherwise.

GZ-MEMBER-CDATA (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

GZ-MEMBER-CEND (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

GZ-MEMBER-CM (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

GZ-MEMBER-CRC32 (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

GZ-MEMBER-FLG (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

GZ-MEMBER-ID1 (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

GZ-MEMBER-ID2 (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

GZ-MEMBER-ISIZE (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

GZ-MEMBER-MTIME (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

GZ-MEMBER-OS (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

GZ-MEMBER-XFL (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

GZ-MEMBER-XLEN (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

GZ-OPEN (FILESPEC &KEY (DIRECTION INPUT) (COMPRESSION +Z-DEFAULT-COMPRESSION+))

Opens FILESPEC for compression or decompression. Arguments: - filespec (pathname designator): The file to open. Key: - direction (keyword): The direction, one of either :input or :output , defaulting to :input . - compression (integer): The zlib compression level, if compressing. An integer between 0 and 9, inclusive.

GZ-READ (GZ BUFFER N)

Reads up to N bytes from GZ into octet vector BUFFER. Returns the number of bytes read, which may be 0.

GZ-READ-BYTE (GZ)

Returns a byte read from GZ, or :eof .

GZ-READ-STRING (GZ STR N)

Reads up to N characters from GZ into string STR. Returns a string or :eof . Relies on the CFFI conversion terminating the returned string, so the result may be shorter than the STR argument.

GZ-WRITE (GZ BUFFER N)

Writes up to N bytes in octet vector BUFFER to GZ. Returns the number of bytes written. BUFFER must be a simple-array of unsigned-byte 8.

GZ-WRITE-BYTE (GZ BYTE)

Writes BYTE to GZ and returns BYTE.

GZ-WRITE-STRING (GZ BUFFER)

Writes up to N characters in octet vector BUFFER to GZ. Returns the number of characters written.

GZIP (IN-FILESPEC &OPTIONAL OUT-FILESPEC)

Compresses IN-FILESPEC to OUT-FILESPEC using the default compression level. Returns two values, OUT-FILESPEC and the number of bytes compressed.

GZIP-OPEN (FILESPEC &KEY (DIRECTION INPUT) (ELEMENT-TYPE 'OCTET) (COMPRESSION +Z-DEFAULT-COMPRESSION+))

Opens a gzip stream for FILESPEC. Key: - direction (symbol): One of :input (the default) or :output - element-type (symbol): One of 'octet (the default) or 'string for input, or 'octet only for output. - compression (integer): The zlib compression level, if compressing. An integer between 0 and 9, inclusive. Returns: - A {defclass gzip-stream}

GZIP-PATHNAME (PATHNAME)

Returns a copy of PATHNAME. A GZ type component is added, unless already present. Any existing type component becomes part of the name component. For example: foo.gz -> foo.gz foo.tar -> foo.tar.gz

INFLATE-STREAM (SOURCE DEST &KEY (BUFFER-SIZE +DEFAULT-ZLIB-BUFFER-SIZE+) SUPPRESS-HEADER (WINDOW-BITS 15))

Inflates stream SOURCE to stream DEST. Arguments: - source (stream): A binary input stream. - dest (stream): A binary output stream. Key: - buffer-size (fixnum): The size of the internal buffer used by Zlib. - suppress-header (boolean): Exposes a feature of Zlib whereby the compressed data may be produced without a Zlib header, trailer or checksum. This must be set T if the data were compressed with the header suppressed. - window-bits (fixnum): The Zlib window-bits argument (9-15, inclusive). Returns: - Number of bytes read. - Number of bytes written.

INFLATE-VECTOR (SOURCE DEST &KEY SUPPRESS-HEADER (WINDOW-BITS 15) (BACKOFF 0))

Inflates vector SOURCE to vector DEST. Arguments: - source (vector): An octet vector. - dest (vector): An octet vector. Key: - compression (fixnum): The Zlib compression factor (0-9, inclusive). - suppress-header (boolean): Exposes an undocumented feature of Zlib whereby the compressed data may be produced without a Zlib header or checksum. This must be set T if the data were compressed with the header suppressed. - window-bits (fixnum): The Zlib window-bits argument (9-15, inclusive). Returns: - The DEST vector, containing decompressed data. - Number of bytes read. - Number of bytes written (consequently the end position of the decompressed data in DEST).

UNCOMPRESS (SOURCE DEST &KEY (SOURCE-START 0) SOURCE-END (DEST-START 0))

Uncompresses bytes in array SOURCE to array DEST, returning DEST and the compressed size in bytes.

Undocumented

ADLER32 (BUFFER &KEY (START 0) END ADLER32)

CRC32 (BUFFER &KEY (START 0) END CRC32)

SETFGZ-MEMBER-CDATA (NEW-VALUE INSTANCE)

SETFGZ-MEMBER-CEND (NEW-VALUE INSTANCE)

SETFGZ-MEMBER-CRC32 (NEW-VALUE INSTANCE)

SETFGZ-MEMBER-FLG (NEW-VALUE INSTANCE)

SETFGZ-MEMBER-ISIZE (NEW-VALUE INSTANCE)

SETFGZ-MEMBER-MTIME (NEW-VALUE INSTANCE)

SETFGZ-MEMBER-OS (NEW-VALUE INSTANCE)

SETFGZ-MEMBER-XFL (NEW-VALUE INSTANCE)

SETFGZ-MEMBER-XLEN (NEW-VALUE INSTANCE)

MAKE-GZ-MEMBER (&KEY ((ID1 DUM0) +ID1+) ((ID2 DUM1) +ID2+) ((CM DUM2) +CM-DEFLATE+) ((FLG DUM3) +FLAG-EXTRA+) ((MTIME DUM4) 0) ((XFL DUM5) 0) ((OS DUM6) +OS-UNKNOWN+) ((XLEN DUM7) 0) ((ISIZE DUM8) 0) ((CRC32 DUM9) 0) ((CDATA DUM10) (MAKE-ARRAY 0 ELEMENT-TYPE 'OCTET)) ((CEND DUM11) 0))

Private

FILE-DESCRIPTOR (STREAM &OPTIONAL DIRECTION)

Returns the Unix file descriptor associated with STREAM.

GZ-ERROR (GZ &OPTIONAL ERRNO MESSAGE)

Raises a {define-condition gz-io-error} . An ERRNO integer and MESSAGE string may be supplied. If ERRNO or MESSAGE are T, they are retrieved using *C-ERROR-NUMBER* and {defun gz-error-message} respectively.

GZ-ERROR-MESSAGE (GZ)

Returns a zlib error message string relevant to GZ.

GZ-OPEN-P (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

GZ-PTR (INSTANCE)

@arg[extid]{A @class{extid}} @return[sytemid]{puri:uri or nil} Returns the System ID part of this External ID.

MAKE-Z-STREAM

Makes an returns a new Z-STREAM with the ZALLOC, ZFREE and OPAQUE slots filled with null-pointers.

MAYBE-STANDARD-STREAM (DESIGNATOR)

Returns a standard stream (*standard-input* *standard-output* or *error-output*) if DESIGNATOR is a string that is STRING-EQUAL to one of "stdin", "stdout" or "stderr", otherwise returns DESIGNATOR. (Also works for "/dev/stdin" etc.) This function is useful where one of these strings may be given on a command line to indicate a system stream, rather than a file-stream is to be used.

READ-OCTET-LINE (STREAM)

Reads chunks of bytes up to the next newline or end of stream, returning them in a list. The newline is not included. Returns two values - a list of chunks and either NIL or T to indicate whether a terminating newline was missing. When the stream underlying the buffer is exhausted the list of chunks will be empty.

Z-ERROR (ERRNO &OPTIONAL MESSAGE)

Raises a {define-condition zlib-error} . A MESSAGE string may be supplied, otherwise it will be determined from ERRNO.

Z-STREAM-CLOSE (Z-STREAM OPERATION)

Signals the end of OPERATION (:inflate or :deflate) on Z-STREAM and frees the Z-STREAM memory.

Z-STREAM-OPEN (OPERATION &KEY (COMPRESSION +Z-DEFAULT-COMPRESSION+) SUPPRESS-HEADER (WINDOW-BITS 15) (MEM-LEVEL 8) (STRATEGY DEFAULT-STRATEGY))

Returns a new Z-STREAM initialised for OPERATION (:inflate or :deflate).

Z-STREAM-OPERATION (OPERATION SOURCE DEST INPUT-FN OUTPUT-FN BUFFER-SIZE &REST Z-STREAM-ARGS)

Implements Zlib compression/decompression using inflate/deflate on Lisp streams, as described in the Zlib Usage Example. Arguments: - operation (symbol): The operation type, either :inflate or :deflate . - source (stream): A Lisp octet input stream. - dest (stream): A Lisp octet output stream. - input-fn (function): A Lisp function that accepts 3 arguments, an input stream, a buffer and an integer n. The function must read up to n bytes from the input stream into the buffer and return the number of bytes read. - output-fn (function): A Lisp function that accepts 3 arguments, an output stream, a buffer and an integer n. The function must write up to n bytes to the output stream into the buffer and return the number of bytes written. - buffer-size (fixnum): The size of the buffer used in the compression/decompression step(s). Rest: - Keyword arguments passed to {defun z-stream-open} . See {defun z-stream-open} Returns: - Number of bytes read. - Number of bytes written.

Undocumented

%LINE-STREAM-READ-SEQUENCE (STREAM SEQUENCE START END)

%LINE-STREAM-WRITE-SEQUENCE (STREAM SEQUENCE START END)

%STREAM-READ-SEQUENCE (STREAM SEQ &OPTIONAL (START 0) END)

%STREAM-WRITE-SEQUENCE (STREAM SEQ &OPTIONAL (START 0) END)

COPY-GZ (INSTANCE)

COPY-GZ-MEMBER (INSTANCE)

EMPTY-TO-STREAM (BUFFER STREAM N)

FILL-FROM-STREAM (BUFFER STREAM N)

GZ-FLUSH (GZ FLUSH-MODE)

GZ-MEMBER-P (OBJECT)

SETFGZ-OPEN-P (NEW-VALUE INSTANCE)

GZ-P (OBJECT)

SETFGZ-PTR (NEW-VALUE INSTANCE)

GZ-SEEK (GZ OFFSET)

GZ-TELL (GZ)

MAKE-GZ (&KEY ((PTR DUM0) NIL) ((OPEN-P DUM1) NIL))

Z-VECTOR-OPERATION (OPERATION SOURCE DEST BACKOFF &REST Z-STREAM-ARGS)

MACRO

Public

WITH-GZ-FILE ((VAR FILESPEC &KEY (DIRECTION INPUT) COMPRESSION) &BODY BODY)

Executes BODY with VAR bound to a GZ handle structure created by opening the file denoted by FILESPEC. Arguments: - var (symbol): The symbol to be bound. - filespec (pathname designator): The file to open. Key: - direction (keyword): The direction, one of either :input or :output , defaulting to :input . - compression (integer): The zlib compression level, if compressing. An integer between 0 and 9, inclusive.

Undocumented

WITH-OPEN-GZIP ((VAR FILESPEC &REST ARGS) &BODY BODY)

GENERIC-FUNCTION

Private

Undocumented

ERRNO-OF (CONDITION)

STREAM-DELETE-FILE (STREAM)

CLASS

Public

GZ

A gzip handle. - ptr: A foreign pointer to a zlib struct. - open-p: A flag that, while T, indicates that the foreign pointer may be freed.

GZ-MEMBER

A gzip member as defined by RFC1952. - id1: IDentification 1. - id2: IDentification 2. - cm: Compression Method. - flg: FLaGs. - mtime: Modification TIME. - xfl: eXtra FLags. - os: Operating System. - xlen: eXtra LENgth. - isize: Input SIZE. - crc32: CRC-32. - cdata: Compressed DATA. - cend: Compressed data END. The length of compressed data in cdata, starting at index 0, if cdata is only partially filled.

GZIP-INPUT-STREAM

A stream that reads bytes from a compressed stream.

GZIP-OUTPUT-STREAM

A stream that writes characters to a compressed stream.

GZIP-STREAM

A gzip stream capable of reading or writing compressed data.

Undocumented

GZIP-LINE-INPUT-STREAM

CONDITION

Public

GZ-IO-ERROR

A condition raised when a gzip error occurs.

ZLIB-ERROR

A condition raised when a zlib error occurs.

CONSTANT

Public

+CM-DEFLATE+

CM (Compression Method)

+FLAG-COMMENT+

FLG (FLaGs) FCOMMENT

+FLAG-EXTRA+

FLG (FLaGs) FEXTRA

+FLAG-FHCRC+

FLG (FLaGs) FHCRC

+FLAG-NAME+

FLG (FLaGs) FNAME

+FLAG-TEXT+

FLG (FLaGs) FTEXT

+ID1+

ID1 (IDentification 1)

+ID2+

ID2 (IDentification 2)

+XFL-FASTEST+

XFL (eXtra FLags) compressor used fastest algorithm

+XFL-SLOWEST+

XFL (eXtra FLags) compressor used maximum compression, slowest algorithm

Undocumented

+OS-ACORN-RISCOS+

+OS-AMIGA+

+OS-ATARI-TOS+

+OS-CP/M+

+OS-FAT-FILESYSTEM+

+OS-HPFS-FILESYSTEM+

+OS-MACINTOSH+

+OS-NTFS-FILESYSTEM+

+OS-QDOS+

+OS-TOPS-20+

+OS-UNIX+

+OS-UNKNOWN+

+OS-VM/CMS+

+OS-VMS+

+OS-Z-SYSTEM+

Private

Undocumented

+DEFAULT-ZLIB-BUFFER-SIZE+

+OCTET-BUFFER-SIZE+