Common Lisp Package: COM.INFORMATIMAGO.COMMON-LISP.LISP-READER.READER

This package implements a standard Common Lisp reader. We implement a Common Lisp Reader to be able to read lisp sources. This is a complete standard compliant lisp reader, with additionnal hooks (token parser). A READTABLE-PARSE-TOKEN function takes a TOKEN as argument, and must return two values: - A boolean indicating whether the it could parse the token, - a parsed lisp object it could, or an error message (string) if not. See also the TOKEN functions, CONSTITUENT-TRAIT, SYNTAX-TABLE and CHARACTER-DESCRIPTION... License: AGPL3 Copyright Pascal J. Bourguignon 2006 - 2012 This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see <http://www.gnu.org/licenses/>

README:

FUNCTION

Public

COPY-READTABLE (&OPTIONAL (FROM-READTABLE *READTABLE*) (TO-READTABLE NIL))

DO: Copy the readtable. URL: <http://www.lispworks.com/documentation/HyperSpec/Body/f_cp_rdt.htm>

GET-DISPATCH-MACRO-CHARACTER (DISP-CHAR SUB-CHAR &OPTIONAL (READTABLE *READTABLE*))

RETURN: The dispatch macro character function. URL: <http://www.lispworks.com/documentation/HyperSpec/Body/f_set__1.htm>

GET-MACRO-CHARACTER (CHAR &OPTIONAL (READTABLE *READTABLE*))

RETURN: The macro character function. URL: <http://www.lispworks.com/documentation/HyperSpec/Body/f_set_ma.htm>

LIST-ALL-MACRO-CHARACTERS (&OPTIONAL (*READTABLE* *READTABLE*))

RETURN: A list of all the macro and dispatch-macro characters in the readtable.

MAKE-DISPATCH-MACRO-CHARACTER (CHAR &OPTIONAL (NON-TERMINATING-P NIL) (READTABLE *READTABLE*))

DO: Make the character a dispatch macro character in the readtable. URL: <http://www.lispworks.com/documentation/HyperSpec/Body/f_mk_dis.htm>

POTENTIAL-NUMBER-P (TOKEN &OPTIONAL (*READ-BASE* *READ-BASE*) (RATIO-MARKERS /))

TOKEN: A string containing the token to be tested. *READ-BASE*: The current radix. RATIO-MARKER: / in the standard readtable, but it could be something else... RETURN: Whether the TOKEN is a potential number.

READ (&OPTIONAL INPUT-STREAM (EOF-ERROR-P T) (EOF-VALUE NIL) (RECURSIVE-P NIL))

RETURN: An object read. URL: <http://www.lispworks.com/documentation/HyperSpec/Body/f_rd_rd.htm>

READ-DELIMITED-LIST (CHAR &OPTIONAL (INPUT-STREAM *STANDARD-INPUT*) (RECURSIVE-P NIL))

RETURN: A list of objects read. URL: <http://www.lispworks.com/documentation/HyperSpec/Body/f_rd_del.htm>

READ-FROM-STRING (STRING &OPTIONAL (EOF-ERROR-P T) (EOF-VALUE NIL) &KEY (START 0) (END NIL) (PRESERVE-WHITESPACE NIL))

RETURN: An object read from the string. URL: <http://www.lispworks.com/documentation/HyperSpec/Body/f_rd_fro.htm>

READ-PRESERVING-WHITESPACE (&OPTIONAL INPUT-STREAM (EOF-ERROR-P T) (EOF-VALUE NIL) (RECURSIVE-P NIL))

RETURN: An object read. URL: <http://www.lispworks.com/documentation/HyperSpec/Body/f_rd_rd.htm>

READTABLE-CASE (READTABLE)

RETURN: The case of the readtable. URL: <http://www.lispworks.com/documentation/HyperSpec/Body/f_rdtabl.htm>

SETFREADTABLE-CASE (VALUE READTABLE)

DO: Set the case of the readtable. URL: <http://www.lispworks.com/documentation/HyperSpec/Body/f_rdtabl.htm>

READTABLEP (OBJECT)

RETURN: Whether the object is a readtable. URL: <http://www.lispworks.com/documentation/HyperSpec/Body/f_rdta_1.htm>

SET-DISPATCH-MACRO-CHARACTER (DISP-CHAR SUB-CHAR NEW-FUNCTION &OPTIONAL (READTABLE *READTABLE*))

DO: Set the dispatch macro character function. URL: <http://www.lispworks.com/documentation/HyperSpec/Body/f_set__1.htm>

SET-INDIRECT-DISPATCH-MACRO-CHARACTER (DISP-CHAR SUB-CHAR FUNCTION-NAME &OPTIONAL (READTABLE *READTABLE*))

Like set-dispatch-macro-character, but with an indirect function, to enable TRACE and redefinitions of the dispatch macro character function.

SET-INDIRECT-MACRO-CHARACTER (CHAR FUNCTION-NAME &OPTIONAL (READTABLE *READTABLE*))

Like set-macro-character, but with an indirect function, to enable TRACE and redefinitions of the macro character function.

SET-MACRO-CHARACTER (CHAR NEW-FUNCTION &OPTIONAL (NON-TERMINATING-P NIL) (READTABLE *READTABLE*))

DO: Set then macro character function. URL: <http://www.lispworks.com/documentation/HyperSpec/Body/f_set_ma.htm>

SET-SYNTAX-FROM-CHAR (TO-CHAR FROM-CHAR &OPTIONAL (TO-READTABLE *READTABLE*) (FROM-READTABLE *STANDARD-READTABLE*))

DO: Copy the syntax between characters in the readtable. URL: <http://www.lispworks.com/documentation/HyperSpec/Body/f_set_sy.htm>

Private

ALL-DOTS-P (TOKEN)

RETURN: Whether the token is all dots, (excluding escaped dots).

COPY-HASH-TABLE (TABLE)

TABLE: (OR NULL HASH-TABLE) RETURN: If TABLE is NIL, then NIL, else a new HASH-TABLE with the same TEST, SIZE, REHASH-THRESHOLD REHASH-SIZE and KEY->VALUE associations than TABLE. (Neither the keys nor the values are copied).

EVAL-FEATURE (EXPRESSION STREAM)

Evaluates a feature expression as a BOOLEAN.

PARSE-DECIMAL-INTEGER-TOKEN (TOKEN)

integer ::= [sign] decimal-digit+ decimal-point

PARSE-FLOAT-1-TOKEN (TOKEN)

float ::= [sign] {decimal-digit}* decimal-point {decimal-digit}+ [exponent] exponent ::= exponent-marker [sign] {digit}+

PARSE-FLOAT-2-TOKEN (TOKEN)

float ::= [sign] {decimal-digit}+ [decimal-point {decimal-digit}*] exponent exponent ::= exponent-marker [sign] {digit}+

PARSE-INTEGER-TOKEN (TOKEN)

integer ::= [sign] digit+

PARSE-RATIO-TOKEN (TOKEN)

ratio ::= [sign] {digit}+ slash {digit}+

PARSE-SYMBOL-TOKEN (TOKEN)

symbol ::= symbol-name symbol ::= package-marker symbol-name symbol ::= package-marker package-marker symbol-name symbol ::= package-name package-marker symbol-name symbol ::= package-name package-marker package-marker symbol-name symbol-name ::= {alphabetic}+ package-name ::= {alphabetic}+

PARSE-TOKEN (TOKEN)

RETURN: okp ; the parsed lisp object if okp, or an error message if (not okp)

PROPER-LIST-P (OBJECT)

RETURN: whether object is a proper list NOTE: terminates with any kind of list, dotted, circular, etc.

READ-0/1 (INPUT-STREAM EOF-ERROR-P EOF-VALUE RECURSIVE-P PRESERVE-WHITESPACE-P FIRST-CHAR ALLOWED-ALL-DOTS)

DO: Read zero or one token. Use the *READTABLE*. INPUT-STREAM: The stream that is read. EOF-ERROR-P: Whether we should signal an END-OF-FILE error upon EOF. EOF-VALUE: Unless EOF-ERROR-P, the value to be returned in case of EOF. RECURSIVE-P: Whether the read is recursive. The *reference* table is reset only when RECURSIVE-P is false. PRESERVE-WHITESPACE-P: Whether a terminating whitespace will be unread or not. FIRST-CHAR: NIL or a CHARACTER that is used first, before reading the stream. This should be faster than UNREAD-CHAR it, and foremost, it allows for two unread character, this FIRST-CHAR plus an actual UNREAD-CHAR one. ALLOWED-ALL-DOTS: May be T in which case tokens containing only dots are allowed, or a (possibly empty) list of strings containing only dots that are explicitely allowed (others rejected). Typically ("."). RETURN: tokenp == t ; an unparsed (alldots) token. Or tokenp == :EOF ; the eof-value. Or tokenp == NIL ; a list of values read.

READ-1 (INPUT-STREAM EOF-ERROR-P EOF-VALUE RECURSIVE-P PRESERVE-WHITESPACE-P FIRST-CHAR ALLOWED-ALL-DOTS)

DO: Read exactly one token. Use the *READTABLE*. INPUT-STREAM: The stream that is read. EOF-ERROR-P: Whether we should signal an END-OF-FILE error upon EOF. EOF-VALUE: Unless EOF-ERROR-P, the value to be returned in case of EOF. RECURSIVE-P: Whether the read is recursive. The *reference* table is reset only when RECURSIVE-P is false. PRESERVE-WHITESPACE-P: Whether a terminating whitespace will be unread or not. FIRST-CHAR: NIL or a CHARACTER that is used first, before reading the stream. This should be faster than UNREAD-CHAR it, and foremost, it allows for two unread character, this FIRST-CHAR plus an actual UNREAD-CHAR one. ALLOWED-ALL-DOTS: May be T in which case tokens containing only dots are allowed, or a (possibly empty) list of strings containing only dots that are explicitely allowed (others rejected). Typically ("."). RETURN: The token read, or when EOF-ERROR-P is false and EOF occurs, EOF-VALUE.

READ-FEATURE (STREAM AFFIRMATIVEP)

Reads a feature expression, and possibly eats one following sexp

READ-RATIONAL-IN-BASE (STREAM ARG SUB-CHAR *READ-BASE*)

DO: Read a rational number in the base specified. RETURN: The rational read.

READ-TOKEN (INPUT-STREAM EOF-ERROR-P EOF-VALUE RECURSIVE-P PRESERVE-WHITESPACE-P FIRST-CHAR READTABLE)

DO: Implements parts of READ and READ-PRESERVING-WHITESPACE. INPUT-STREAM: The stream that is read. EOF-ERROR-P: Whether we should signal an END-OF-FILE error upon EOF. EOF-VALUE: Unless EOF-ERROR-P, the value to be returned in case of EOF. RECURSIVE-P: Whether the read is recursive. The *reference* table is reset only when RECURSIVE-P is false. PRESERVE-WHITESPACE-P: Whether a terminating whitespace will be unread or not. FIRST-CHAR: NIL or a CHARACTER that is used first, before reading the stream. This should be faster than UNREAD-CHAR it, and foremost, it allows for two unread character, this FIRST-CHAR plus an actual UNREAD-CHAR one. READTABLE: The readtable to use. RETURN: tokenp == t ; a token. Or tokenp == :EOF ; the eof-value. Or tokenp == NIL ; a list of values read. BUG: The handling of readtable-case :invert is wrong.

READER-DISPATCH-MACRO-ARRAY (STREAM ARG SUB-CHAR)

Standard #A dispatch macro reader.

READER-DISPATCH-MACRO-BINARY (STREAM ARG SUB-CHAR)

Standard #B dispatch macro reader.

READER-DISPATCH-MACRO-BIT-VECTOR (STREAM ARG SUB-CHAR)

Standard #* dispatch macro reader. URL: <http://www.lispworks.com/documentation/HyperSpec/Body/02_dhd.htm>

READER-DISPATCH-MACRO-CHAR (STREAM ARG SUB-CHAR)

Standard #\ dispatch macro reader.

READER-DISPATCH-MACRO-COMMENT (STREAM ARG SUB-CHAR)

Standard #| dispatch macro reader.

READER-DISPATCH-MACRO-COMPLEX (STREAM ARG SUB-CHAR)

Standard #C dispatch macro reader.

READER-DISPATCH-MACRO-FEATURE (STREAM ARG SUB-CHAR)

Standard #+ dispatch macro reader.

READER-DISPATCH-MACRO-FUNCTION (STREAM ARG SUB-CHAR)

Standard #' dispatch macro reader.

READER-DISPATCH-MACRO-HEXADECIMAL (STREAM ARG SUB-CHAR)

Standard #X dispatch macro reader.

READER-DISPATCH-MACRO-LABEL-DEFINITION (STREAM ARG SUB-CHAR)

Standard #= dispatch macro reader.

READER-DISPATCH-MACRO-LABEL-REFERENCE (STREAM ARG SUB-CHAR)

Standard ## dispatch macro reader.

READER-DISPATCH-MACRO-NOT-FEATURE (STREAM ARG SUB-CHAR)

Standard #- dispatch macro reader.

READER-DISPATCH-MACRO-OCTAL (STREAM ARG SUB-CHAR)

Standard #O dispatch macro reader.

READER-DISPATCH-MACRO-PATHNAME (STREAM ARG SUB-CHAR)

Standard #P dispatch macro reader.

READER-DISPATCH-MACRO-RADIX (STREAM ARG SUB-CHAR)

Standard #R dispatch macro reader.

READER-DISPATCH-MACRO-READ-EVAL (STREAM ARG SUB-CHAR)

Standard #. dispatch macro reader.

READER-DISPATCH-MACRO-STRUCTURE (STREAM ARG SUB-CHAR)

Standard #S dispatch macro reader.

READER-DISPATCH-MACRO-UNINTERNED (STREAM ARG SUB-CHAR)

Standard #: dispatch macro reader.

READER-DISPATCH-MACRO-UNREADABLE (STREAM ARG SUB-CHAR)

Standard #< dispatch macro reader.

READER-DISPATCH-MACRO-VECTOR (STREAM ARG SUB-CHAR)

Standard #( dispatch macro reader.

READER-MACRO-BACKQUOTE (STREAM CH)

Standard ` macro reader.

READER-MACRO-COMMA (STREAM CH)

Standard , macro reader.

READER-MACRO-LEFT-PARENTHESIS (STREAM CH)

Standard ( macro reader.

READER-MACRO-LINE-COMMENT (STREAM CH)

Standard ; macro reader.

READER-MACRO-QUOTE (STREAM CH)

Standard ' macro reader.

READER-MACRO-STRING (STREAM DELIM)

Standard " macro reader.

TRAITP (TRAIT TRAITS)

Returns whether the TRAIT is in the TRAITS 'set'.

Undocumented

%PARSE-INTEGER (STRING &KEY (START 0) (END NIL) (RADIX 10) (JUNK-ALLOWED NIL) &AUX (END (OR END (LENGTH STRING))))

CHECK-SYMBOLS

MAKE-TOKEN

READER-DISPATCH-MACRO-ERROR-INVALID (STREAM SUB-CHAR ARG)

READER-DISPATCH-MACRO-ERROR-UNDEFINED (STREAM CH SUB-CHAR)

READER-MACRO-DISPATCH-FUNCTION (STREAM CH)

READER-MACRO-ERROR-START (STREAM CH)

SERROR (CONDITION STREAM CONTROL-STRING &REST ARGUMENTS)

TEST-CASES (TEST-NAME CASES)

TEST-PROPER-LIST-P

TEST/POTENTIAL-NUMBER-P

TEST/READER

TOKEN-CHAR (TOKEN INDEX)

TOKEN-CHAR-TRAITS (TOKEN INDEX)

TOKEN-COLLECT-CHARACTER (TOKEN CHARACTER TRAITS)

TOKEN-DELIMITER-P (CHARACTER)

TOKEN-LENGTH (TOKEN)

TOKEN-TEXT (TOKEN)

TOKEN-TRAITS (TOKEN)

MACRO

Public

Undocumented

WITH-STANDARD-IO-SYNTAX (&BODY BODY)

Private

DEFPARSER (NAME ARGUMENTS &BODY BODY)

Defines a token parser function, which parses its argument token and returns three values: a ok flag; a type of value; and a value parsed from the token. When the ok flag is false, the type indicates whether it's a strong error, and the value returned is an error message. A strong error is a lexical error that is not ambiguous. A weak error is when the token could still be of another lexical category. In the body of the parser, there are macrolet defined to REJECT or ACCEPT the token, and to describe the parsed syntax with ALT, ZERO-OR-MORE, ONE-OR-MORE and OPT-SIGN.

Undocumented

TESTS (&REST CASES)

GENERIC-FUNCTION

Private

Undocumented

CHARACTER-DESCRIPTION (SYNTAX-TABLE CHARACTER)

SETFCHARACTER-DESCRIPTION (VAL SYNTAX-TABLE CHARACTER)

COPY-SYNTAX-TABLE (SYNTAX-TABLE)

PROCESS-CASE-FUNCTION (MODE)

SLOT-ACCESSOR

Public

READTABLE-PARSE-TOKEN (READTABLE)

The RETURN: The function used to parse a token that has been read.

SETFREADTABLE-PARSE-TOKEN (NEW-FUNCTION READTABLE)

The DO: Set the function used to parse a token that has been read.

READTABLE-SYNTAX-TABLE (READTABLE)

The RETURN: The syntax-table of the readtable.

SETFREADTABLE-SYNTAX-TABLE (NEW-VALUE OBJECT)

Set the RETURN: The syntax-table of the readtable.

Private

CHARACTER-DISPATCH (OBJECT)

A HASH-TABLE character -> dmc function.

CHARACTER-MACRO (OBJECT)

A macro character function.

Undocumented

CHARACTER-CONSTITUENT-TRAITS (OBJECT)

CHARACTER-SYNTAX (OBJECT)

VARIABLE

Public

*READ-BASE*

Controls the interpretation of tokens by READ as being integers or ratios. The value of *READ-BASE*, called the current input base, is the radix in which integers and ratios are to be read by the Lisp reader. The parsing of other numeric types (e.g., floats) is not affected by this option. The effect of *READ-BASE* on the reading of any particular rational number can be locally overridden by explicit use of the #O, #X, #B, or #nR syntax or by a trailing decimal point. URL: <http://www.lispworks.com/documentation/HyperSpec/Body/v_rd_bas.htm>

*READ-DEFAULT-FLOAT-FORMAT*

Controls the floating-point format that is to be used when reading a floating-point number that has no exponent marker or that has e or E for an exponent marker. Other exponent markers explicitly prescribe the floating-point format to be used. The printer uses *read-default-float-format* to guide the choice of exponent markers when printing floating-point numbers. URL: <http://www.lispworks.com/documentation/HyperSpec/Body/v_rd_def.htm>

*READ-EVAL*

If it is true, the #. reader macro has its normal effect. Otherwise, that reader macro signals an error of type reader-error. URL: <http://www.lispworks.com/documentation/HyperSpec/Body/v_rd_eva.htm>

*READ-SUPPRESS*

This variable is intended primarily to support the operation of the read-time conditional notations #+ and #-. If it is false, the Lisp reader operates normally. If the value of *read-suppress* is true, read, read-preserving-whitespace, read-delimited-list, and read-from-string all return a primary value of nil when they complete successfully. URL: <http://www.lispworks.com/documentation/HyperSpec/Body/v_rd_sup.htm>

*READTABLE*

The value of *READTABLE* is called the current readtable. It controls the parsing behavior of the Lisp reader, and can also influence the Lisp printer (e.g., see the function READTABLE-CASE). URL: <http://www.lispworks.com/documentation/HyperSpec/Body/v_rdtabl.htm>

Private

*INPUT-STREAM*

Bound to the input stream, during token parsing. Consequences are undefined, if any destructive operations are attempted on this stream.

*REFERENCES*

Used to implement #= and ##.

*STANDARD-READTABLE*

Only used by SET-SYNTAX-FROM-CHAR

Undocumented

*CD-CONSTITUENT-ALPHABETIC*

*CD-CONSTITUENT-INVALID*

*CD-INVALID*

*CD-WHITESPACE*

CLASS

Public

READTABLE

A READTABLE maps characters into syntax types for the Lisp reader; see Section 2 (Syntax). A readtable also contains associations between macro characters and their reader macro functions, and records information about the case conversion rules to be used by the Lisp reader when parsing symbols. Each simple character must be representable in the readtable. It is implementation-defined whether non-simple characters can have syntax descriptions in the readtable. URL: <http://www.lispworks.com/documentation/HyperSpec/Body/t_rdtabl.htm>

Private

CHARACTER-DESCRIPTION (SYNTAX-TABLE CHARACTER)

Description of one character. In the syntax tables, a single character description instance can be shared by several characters, but with copy-on-write.

SYNTAX-TABLE

STANDARD-CHARACTERS is a vector of CHARACTER-DESCRIPTION instances for the standard character codes below +STANDARD-CHARACTERS-LIMIT+. EXTENDED-CHARACTERS is NIL, or a HASH-TABLE mapping characters to CHARACTER-DESCRIPTIONS instances for the extended characters with codes above +STANDARD-CHARACTERS-LIMIT+. Extended characters without an entry in EXTENDED-CHARACTERS either have CONSTITUENT or INVALID CHARACTER-DESCRIPTION, depending on whether they're GRAPHIC-CHAR-P or not.

CONDITION

Public

MISSING-PACKAGE-ERROR

The error condition signaled when trying use an inexistant package.

MISSING-SYMBOL-ERROR

The error condition signaled when trying to read a symbol not exported from a package.

SIMPLE-END-OF-FILE

A simple end-of-file condition.

SIMPLE-READER-ERROR

A simple reader error condition.

SYMBOL-IN-MISSING-PACKAGE-ERROR

The error condition signaled when trying to read a symbol in an inexistant package.

SYMBOL-MISSING-IN-PACKAGE-ERROR

The error condition signaled when trying to read a symbol not exported from a package.

CONSTANT

Private

Undocumented

+CS-CONSTITUENT+

+CS-INVALID+

+CS-MULTIPLE-ESCAPE+

+CS-NON-TERMINATING-MACRO-CHARACTER+

+CS-SINGLE-ESCAPE+

+CS-TERMINATING-MACRO-CHARACTER+

+CS-WHITESPACE+

+CT-ALPHABETIC+

+CT-ALPHADIGIT+

+CT-DECIMAL-POINT+

+CT-DIGIT+

+CT-DOT+

+CT-DOUBLE-FLOAT-EXPONENT-MARKER+

+CT-FLOAT-EXPONENT-MARKER+

+CT-INVALID+

+CT-LONG-FLOAT-EXPONENT-MARKER+

+CT-MAX+

+CT-MINUS-SIGN+

+CT-PACKAGE-MARKER+

+CT-PLUS-SIGN+

+CT-RATIO-MARKER+

+CT-SHORT-FLOAT-EXPONENT-MARKER+

+CT-SIGN+

+CT-SINGLE-FLOAT-EXPONENT-MARKER+

+STANDARD-CHARACTERS-LIMIT+