Common Lisp Package: CL-CSV

README:

cl-csv

This library aims to simplify working with csvs to the bare minimum of tedium

  • reads/writes csvs from/to strings, streams and files
  • support streaming reads (allowing processing very large csvs, through read-csv's row-fn paramter)
  • supports custom data formating
  • settable quote, separator and quote-escapes
  • supports multiline quoted data
  • A test suite

Rationale

I had many scattered, not well tested, not easily runable pieces of csv code. I was unhappy with this situation then decided to refactor all of this into a single location. I wrote tests for it and had a library so I thought I might release it. This project started as extensions and bugfixes on arnesi's CSV.

I then looked around and saw there are other csv libs out there that probably mostly accomplished what I had set out to do. However, I already had code that was tested and had an easier license (BSD) so, I figured why not just release it anyway.

Other Available CSV libs

  • http://members.optusnet.com.au/apicard/csv-parser.lisp
  • http://www.cliki.net/fare-csv

Library Integration

  • data-table - functions for building data-tables from csv's, must (asdf:load-system :cl-csv-data-table)
  • clsql must (asdf:load-system :cl-csv-clsql)
    • import-from-csv
    • serial-import-from-csv
  • iterate - provides an in-csv driver clause for iterating over a CSV

Examples

`lisp ;; read a file into a list of lists (cl-csv:read-csv #P"file.csv") => (("1" "2" "3") ("4" "5" "6"))

;; read a file that's tab delimited (cl-csv:read-csv #P"file.tab" :separator #\Tab)

;; read a file and return a list of objects created from each row (cl-csv:read-csv #P"file.csv" :map-fn #'(lambda (row) (make-instance 'object :foo (nth 0 row) :baz (nth 2 row)))) ;; read csv from a string (streams also supported) (cl-csv:read-csv "1,2,3 4,5,6") => (("1" "2" "3") ("4" "5" "6"))

;; loop over a CSV for effect (let ((sum 0)) (cl-csv:do-csv (row #P"file.csv") (incf sum (parse-integer (nth 0 row)))) sum)

;; loop over a CSV using iterate (iter (for (foo bar baz) in-csv #P"file.csv") (collect (make-instance 'object :foo foo :baz baz))) `

Authors

;; Copyright (c) 2011 Russ Tyndall , Acceleration.net http://www.acceleration.net ;; Copyright (c) 2002-2006, Edward Marco Baringer ;; All rights reserved. ;; ;; Redistribution and use in source and binary forms, with or without ;; modification, are permitted provided that the following conditions are ;; met: ;; ;; - Redistributions of source code must retain the above copyright ;; notice, this list of conditions and the following disclaimer. ;; ;; - Redistributions in binary form must reproduce the above copyright ;; notice, this list of conditions and the following disclaimer in the ;; documentation and/or other materials provided with the distribution. ;; ;; - Neither the name of Edward Marco Baringer, nor BESE, nor the names ;; of its contributors may be used to endorse or promote products ;; derived from this software without specific prior written permission. ;; ;; THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS ;; "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT ;; LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR ;; A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT ;; OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, ;; SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT ;; LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, ;; DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY ;; THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT ;; (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE ;; OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

FUNCTION

Public

READ-CSV (STREAM-OR-STRING &KEY ROW-FN MAP-FN SAMPLE SKIP-FIRST-P ((SEPARATOR *SEPARATOR*) *SEPARATOR*) ((QUOTE *QUOTE*) *QUOTE*) ((ESCAPE *QUOTE-ESCAPE*) *QUOTE-ESCAPE*))

Read in a CSV by data-row (which due to quoted newlines may be more than one line from the stream) row-fn: passing this parameter will cause this read to be streaming and results will be discarded after the row-fn is called with data map-fn: used for manipulating the data by row during collection if specified; (funcall map-fn data) is collected instead of data sample: when a positive integer, only take that many samples from the input file skip-first-p: when true, skips the first line in the csv Keywords: separator: character separating between data cells. Defaults to *separator* quote: quoting character for text strings. Defaults to *quote* escape: escape character. Defaults to *quote-escape*

READ-CSV-ROW (STREAM-OR-STRING &KEY ((SEPARATOR *SEPARATOR*) *SEPARATOR*) ((QUOTE *QUOTE*) *QUOTE*) ((ESCAPE *QUOTE-ESCAPE*) *QUOTE-ESCAPE*) &AUX (CURRENT (MAKE-ARRAY 20 ELEMENT-TYPE 'CHARACTER ADJUSTABLE T FILL-POINTER 0)) (STATE WAITING) LINE LLEN (C ) (ELEN (LENGTH *QUOTE-ESCAPE*)))

Read in a CSV by data-row (which due to quoted newlines may be more than one line from the stream)

WRITE-CSV (ROWS-OF-ITEMS &KEY STREAM ((SEPARATOR *SEPARATOR*) *SEPARATOR*) ((QUOTE *QUOTE*) *QUOTE*) ((ESCAPE *QUOTE-ESCAPE*) *QUOTE-ESCAPE*) ((NEWLINE *NEWLINE*) *NEWLINE*) ((ALWAYS-QUOTE *ALWAYS-QUOTE*) *ALWAYS-QUOTE*))

Writes a csv to the given stream. rows-of-items: iterable Keywords: stream: stream to write to. Default: nil. nil - writes the rows to a string and returns it an open stream a pathname (overwrites if the file exists) quote: quoting character. Defaults to *quote* escape: escaping character. Defaults to *quote-escape* newline: newline character. Defaults to *newline always-quote: Defaults to *always-quote*

WRITE-CSV-ROW (ITEMS &KEY STREAM ((SEPARATOR *SEPARATOR*) *SEPARATOR*) ((QUOTE *QUOTE*) *QUOTE*) ((ESCAPE *QUOTE-ESCAPE*) *QUOTE-ESCAPE*) ((NEWLINE *NEWLINE*) *NEWLINE*) ((ALWAYS-QUOTE *ALWAYS-QUOTE*) *ALWAYS-QUOTE*))

Writes a list items to stream rows-of-items: iterable Keywords: stream: stream to write to. Default: nil. quote: quoting character. Defaults to *quote* escape: escaping character. Defaults to *quote-escape* newline: newline character. Defaults to *newline always-quote: Defaults to *always-quote*

Undocumented

CSV-PARSE-ERROR (MSG &REST ARGS)

READ-CSV-SAMPLE (STREAM-OR-STRING SAMPLE-SIZE &KEY ROW-FN MAP-FN SKIP-FIRST-P ((SEPARATOR *SEPARATOR*) *SEPARATOR*) ((QUOTE *QUOTE*) *QUOTE*) ((ESCAPE *QUOTE-ESCAPE*) *QUOTE-ESCAPE*))

Private

%OUT-STREAM (STREAM-OR-STRING)

creates a stream from the given thing, trying to DWIM

CHARS-IN (CHARS-TO-CHECK VALUE-TO-LOOK-THROUGH)

returns true if any of the chars-to-check is found in the value-to-look-through

Undocumented

%CHAR-IN (C TO-CHECK)

%ESCAPE-SEQ? (S I ESCAPE LLEN ELEN)

%IN-STREAM (STREAM-OR-STRING)

WHITE-SPACE? (C)

MACRO

Public

DO-CSV ((ROW-VAR STREAM-OR-PATHNAME &REST READ-CSV-KEYS) &BODY BODY)

row-var: a variable that is passed into _body_ stream-or-pathname: a stream or a pathname to read the CSV data from read-csv-keys: keys and values passed to the _read-csv_ function body: body of the macro

Private

CLAUSE-FOR-IN-CSV-1 (&KEY ((FOR VAR) NIL) ((IN-CSV INPUT) NIL) ((SKIPPING-HEADER SKIP-FIRST-P) NIL) ((SEPARATOR SEPARATOR) NIL) ((QUOTE QUOTE) NIL) ((ESCAPED-QUOTE ESCAPED-QUOTE) NIL))

IN-CSV driver for iterate

CLAUSE-SAMPLING-2 (&KEY ((SAMPLING EXPR) NIL) ((INTO VAR) NIL) ((SIZE SIZE) NIL))

resevoir sample the input

Undocumented

WITH-CSV-INPUT-STREAM ((NAME INP) &BODY BODY)

WITH-CSV-OUTPUT-STREAM ((NAME INP) &BODY BODY)

GENERIC-FUNCTION

Public

FORMAT-CSV-VALUE (VAL)

Print values in ways that are most cross compatible with the csv format

WRITE-CSV-VALUE (VAL CSV-STREAM &KEY FORMATTER QUOTE SEPARATOR ESCAPE ALWAYS-QUOTE (FORMATTER #'FORMAT-CSV-VALUE) '*QUOTE* (SEPARATOR *SEPARATOR*) (ESCAPE *QUOTE-ESCAPE*) (ALWAYS-QUOTE *ALWAYS-QUOTE*))

Writes val to csv-stream in a formatted fashion. Keywords formatter: used to format val. Defaults to format-csv-value. quote: quoting character. Defaults to *quote* escape: escaping character. Defaults to *quote-escape* newline: newline character. Defaults to *newline always-quote: Defaults to *always-quote*

Private

Undocumented

FORMAT-ARGS (CONDITION)

SETFFORMAT-ARGS (NEW-VALUE CONDITION)

FORMAT-CONTROL (CONDITION)

SETFFORMAT-CONTROL (NEW-VALUE CONDITION)

VARIABLE

Public

*DEFAULT-EXTERNAL-FORMAT*

the external format used for opening files

*NEWLINE*

Default newline string

*QUOTE*

Default quote character

*QUOTE-ESCAPE*

Default setting for escaping quotes

*SEPARATOR*

Default separator character

Private

*ALWAYS-QUOTE*

Default setting for always quoting

CONDITION

Public

Undocumented

CSV-PARSE-ERROR (MSG &REST ARGS)