Common Lisp Package: SWANK-CREW

Evaluate expressions on remote Lisps using the Swank protocol.

README:

FUNCTION

Public

CONNECT-LOCAL-WORKERS (BASE-PORT WORKER-COUNT)

Makes Swank connections to WORKER-COUNT workers on "localhost" and returns a WORKER-POOL containing them. The Swank servers for the workers must be listening on WORKER-COUNT consecutively numbered ports, starting from BASE-PORT.

CONNECT-WORKERS (HOST/PORT-ALIST)

Makes Swank connections to all the workers in HOST/PORT-ALIST and returns a WORKER-POOL containing them. HOST/PORT-ALIST is a list of (host-name . port) pairs

DISCONNECT-WORKERS (WORKER-POOL)

Closes the Swank connections of all connected workers in WORKER-POOL.

EVAL-FORM-ALL-WORKERS (WORKER-POOL FORM &KEY RESULT-DONE (REPLAY-REQUIRED T))

Evaluates FORM on all workers in WORKER-POOL. When RESULT-DONE is non-NIL, it must be a function of two arguments. In this case RESULT-DONE is called as each worker returns a result with two arguments, a non-negative integer representing the order in which the work was dispatched and the worker's result. If REPLAY-REQUIRED is true, which is the default, FORM will be remembered and evaluated again for side effects on any new worker that joins POOL.

EVAL-FORM-REPEATEDLY (WORKER-POOL RESULT-COUNT FORM &KEY (WORKER-COUNT (WHEN WORKER-POOL (WORKER-COUNT WORKER-POOL))))

Evaluates FORM, which must return a function of no arguments, on WORKER-COUNT workers in WORKER-POOL, then arranges for the workers to repeatedly call the function to create RESULT-COUNT results, which are returned in a list. WORKER-COUNT defaults to the number of workers in WORKER-POOL.

EVAL-REPEATEDLY-ASYNC-STATE (WORKER-POOL FORM INITIAL-STATE UPDATE-STATE &KEY (WORKER-COUNT (WHEN WORKER-POOL (WORKER-COUNT WORKER-POOL))))

Evaluates FORM, which must return a function of one argument, on WORKER-COUNT workers in WORKER-POOL, then arranges for the workers to repeatedly call the work function and send its results back to the master. The work function is passed a state argument, initially set to INITIAL-STATE, that the master can update asynchronously as it receives new results. On the master, the work state is initialized to INITIAL-STATE and UPDATE-STATE is called repeatedly to process results received from the workers. UPDATE-STATE is called with two arguments, the current work state and a list containing all unprocessed work results. UPDATE-STATE should return three values: the new work state, a boolean indicating whether the computation should end, and a boolean indicating whether the latest work state should be distributed to workers. When UPDATE-STATE's second return value is true, EVAL-REPEATEDLY-ASYNC-STATE tells the workers to stop and returns the latest work state.

PARALLEL-MAPCAR (WORKER-POOL MAKE-WORK LIST &OPTIONAL RESULT-DONE)

Traverses LIST, calling MAKE-WORK on each element to create a form that is then passed to a remote worker in WORKER-POOL for evaluation. Results of evaluating each form are collected into a list, which is returned when every remote worker is done. If RESULT-DONE is provided, then it must be a function of two arguments. In this case RESULT-DONE is called on the master as each worker returns a result. The arguments to RESULT-DONE are the position of the work in LIST and the worker's result.

PARALLEL-REDUCE (WORKER-POOL MAKE-WORK LIST INITIAL-VALUE REDUCER)

Traverses LIST, calling MAKE-WORK on each element to create a form that is then passed to a remote worker in WORKER-POOL for evaluation. As results are returned, REDUCER, a binary function, is used to combine successive results in the manner of REDUCE. INITIAL-VALUE is used as the starting value for the reduction computation. The form (parallel-reduce worker-pool make-work list initial-value reducer) is equivalent to (reduce reducer (mapcar (lambda (x) (eval (funcall make-work x))) list) :initial-value initial-value)

WORKER-COUNT (WORKER-POOL)

Returns the total number of workers in WORKER-POOL.

Private

ADD-EVAL (EVAL)

Adds EVAL to the list of in-progress computations.

ADD-EVALUATED-FORM (WORKER-POOL FORM)

Adds FORM to the set of forms that need to be evaluated on a new worker when it joins WORKER-POOL.

ALLOCATE-WORKER (WORKER-POOL &KEY WORKER-TO-AVOID)

Allocates and returns an idle worker from WORKER-POOL that is connected. If WORKER-POOL is being shut down, then NIL is returned.

ALLOCATE-WORKER-AND-EVALUATE (WORKER-POOL FORM SET-WORKER WORKER-DONE)

Allocates a connected worker from WORKER-POOL and sends it FORM to evaluate. Returns the worker that was successfully allocated, or NIL if WORKER-POOL is shutting down. SET-WORKER is a function that is called to tell the WORKER-DONE continuation what worker is evaluating FORM.

ASYNC-EVALUATE (FORM INITIAL-STATE ID MASTER-HOST-NAME MASTER-REX-PORT)

Evaluates FORM, which must return a work function of one argument, then calls that function repeatedly to produce results, each time passing it the current computation state. At first this state is INITIAL-STATE, but the master may update the state asynchronously. Each work result is sent to the machine called MASTER-HOST-NAME by making a Swank connection to its MASTER-REX-PORT and remotely evaluating an expression that records the result. ID is used on the master machine to process results and on the worker to update the state.

ASYNC-RESULTS-LOOP (ASYNC-EVAL UPDATE-STATE)

Handles incoming results for ASYNC-EVAL by calling UPDATE-STATE whenever ASYNC-EVAL holds unprocessed results. UPDATE-STATE is called with two arguments, the work state and a list of the unprocessed results.

ASYNC-WORK-FORM (FORM INITIAL-STATE WORKER-POOL ID)

Returns a form for evaluation on a client of WORKER-POOL that ensures the client is caught up and then evaluates FORM for the async evaluation request identified by ID.

CATCH-UP-IF-NECESSARY (MASTER-HOST-NAME MASTER-REX-PORT WORKER-POOL-ID POOL-COUNT)

Ensures that the current client is up to date, that it has evaluated all POOL-COUNT replay forms associated with the pool identified by WORKER-POOL-ID. If it is necessary to evaluate forms in order to catch up, they are fetched by making a Swank connection to MASTER-REX-PORT on host MASTER-HOST-NAME.

CLEAR-DEBUGGING-INFO

Sets all debugging global variables to NIL.

DEBUGGING-FORM (FORM)

Returns an expression that when evaluated returns the result of evaluating FORM. In addition, the returned expression arranges to update the values of *LAST-FORM-EVALED* and *LAST-RANDOM-STATE* for ease of debugging.

DISPATCH-WORK (WORKER-POOL MAKE-WORK LIST RESULT-DONE RETAIN-WORKERS REPLAY-REQUIRED)

Traverses LIST, calling MAKE-WORK on each element to create a form that is then passed to a remote worker in WORKER-POOL for evaluation. When RETAIN-WORKERS is true, each form is evaluated on a unique worker. Otherwise, workers are reused and may process multiple forms. In either case, the results of evaluating each form are collected into a list, which is returned when every remote worker is done. If RESULT-DONE is not NIL, then it must be a function of two arguments. In this case RESULT-DONE is called on the master as each worker returns a result. The arguments to RESULT-DONE are an integer counter, indicating the work form's position in LIST, and the result of evaluating the form. REPLAY-REQUIRED indicates whether new workers will have to perform the work generated by MAKE-WORK before being considered to be caught up.

ENSURE-CAUGHT-UP-THEN-EVALUATE (FORM WORKER-POOL REPLAY-REQUIRED)

Returns a form that when evaluated on a worker in WORKER-POOL ensures that the worker is caught up then evaluates FORM and returns its result. REPLAY-REQUIRED indicates whether new workers must evaluate FORM before being considered to be caught up.

EVAL-ON-MASTER (MAKE-WORK LIST RESULT-DONE)

Iterates over the members of LIST calling MAKE-WORK on each one. If RESULT-DONE is not NIL, it must be a function of two arguments. In this case, after each application of MAKE-WORK to a LIST member, RESULT-DONE is called with the member's position in LIST and MAKE-WORK's result.

EVALUATE-FORM (FORM MASTER-HOST-NAME MASTER-REX-PORT WORKER-POOL-ID POOL-COUNT REPLAY-REQUIRED)

Evaluates FORM, but first ensures that this worker has evaluated all POOL-COUNT replay forms associated with the worker pool identified by WORKER-POOL-ID on the master. When catching up is required, fetches forms by making a Swank connection to MASTER-REX-PORT on host MASTER-HOST-NAME. REPLAY-REQUIRED indicates whether FORM may need to be replayed in order to bring a worker up to date.

FETCH-AND-EVALUATE (MASTER-HOST-NAME MASTER-REX-PORT WORKER-POOL-ID LOCAL-COUNT)

Fetches from the master and then evaluates all forms required to catch up with other workers in the pool identified by WORKER-POOL-ID on the master. All replay forms after the first LOCAL-COUNT forms are fetched by making a Swank connection to MASTER-REX-PORT on host MASTER-HOST-NAME.

FIND-EVAL (ID)

Returns the running EVAL instance identified by ID, or NIL if no computation with that ID is currently running.

FIND-WORKER-POOL (WORKER-POOL-ID)

Returns the worker pool identified by WORKER-POOL-ID.

FREE-WORKER (WORKER WORKER-POOL)

Changes the state of WORKER to idle, so that it's available for allocation.

HANDLE-CONNECTION-CLOSED (DISCONNECTED-WORKER WORKER-POOL)

Called when the connection to DISCONNECTED-WORKER, a member of WORKER-POOL, is closed because of a call to SLIME-CLOSE, the death of the worker, or because of a communications error. Moves all uncompleted work intended for DISCONNECTED-WORKER to another idle connected worker in WORKER-POOL.

NO-WORKERS-P (WORKER-POOL)

Returns T if WORKER-POOL is NIL or contains no workers.

RECONNECT-WORKERS (WORKER-POOL)

Reconnects disconnected workers in WORKER-POOL.

RECORD-ASYNC-RESULT (ID RESULT WORKER-STATE-COUNTER)

Stores RESULT as one of the values produced by the async evaluation identified by ID. Returns a boolean indicating whether the worker should continue to call ASYNC-RESULT with additional results.

RECORD-REPEATED-RESULT (ID RESULT)

Stores RESULT as one of the values produced by the repeated evaluation identified by ID. Returns a boolean indicating whether the worker should continue to call RECORD-REPEATED-RESULT with additional results.

REMOVE-EVAL (EVAL)

Removes EVAL from the list of in-progress computations.

REPEATED-WORK-FORM (FORM WORKER-POOL ID)

Returns a form for evaluation on a client of WORKER-POOL that ensures the client is caught up and then evaluates FORM for the repeated evaluation request identified by ID.

REPEATEDLY-EVALUATE (FORM ID MASTER-HOST-NAME MASTER-REX-PORT)

Evaluates FORM, which must return a function of no arguments, then calls that function repeatedly to produce results. Each result is sent to the machine called MASTER-HOST-NAME by making a Swank connection to its MASTER-REX-PORT and remotely evaluating an expression that records the result. ID is used on the master machine to correctly record the result.

SEND-MANY-RESULTS (SEND-RESULT MASTER-HOST-NAME MASTER-REX-PORT)

Creates a Slime connection to MASTER-HOST-NAME on port MASTER-REX-PORT, then repeatedly calls SEND-RESULT with the new connection as argument. Returns when SEND-RESULT returns NIL.

UNEVALUATED-REPLAY-FORMS (WORKER-POOL-ID EVALUATED-COUNT)

For a worker that has executed EVALUATED-COUNT of the replay forms associated with the worker-pool identified by WORKER-POOL-ID, returns a list of the forms the worker needs to evaluate in order to be completely up to date.

WORKER-COUNTS (WORKER-POOL)

Returns the number of idle, busy, and disconnected workers in WORKER-POOL. This function executes without locking WORKER-POOL, so it may return inconsistent information.

Undocumented

MAKE-WORKER-POOL (CONNECT-INFOS CONNECT-WORKER)

GENERIC-FUNCTION

Private

CONNECT-WORKER (CONNECT-INFO CLOSE-HANDLER)

Creates a connection to a worker's Swank server using information in CONNECT-INFO. Passes the thunk CLOSE-HANDLER to SWANK-CLIENT:SLIME-CONNECT, so that it is invoked when the connection closes.

RECONNECT-WORKER (CONNECT-INFO CLOSE-HANDLER)

Reconnects to a Swank server using information in CONNECT-INFO. Passes the thunk CLOSE-HANDLER to SWANK-CLIENT:SLIME-CONNECT, so that it is invoked when the connection closes.

SLOT-ACCESSOR

Private

CONNECT-INFO (OBJECT)

Information used when connecting to this worker.

CONNECTION (OBJECT)

When non-NIL, an open Swank connection to the worker.

SETFCONNECTION (NEW-VALUE OBJECT)

When non-NIL, an open Swank connection to the worker.

DISCONNECTING (OBJECT)

Set non-NIL when the worker pool is being torn down to tell the reconnector thread it should exit.

SETFDISCONNECTING (NEW-VALUE OBJECT)

Set non-NIL when the worker pool is being torn down to tell the reconnector thread it should exit.

DONE (OBJECT)

Is the computation done?

SETFDONE (NEW-VALUE OBJECT)

Is the computation done?

DONE-CONDITION (OBJECT)

Condition variable notified when computation is done.

HOST-NAME (OBJECT)

Host the worker is running on.

SETFHOST-NAME (NEW-VALUE OBJECT)

Host the worker is running on.

ID (OBJECT)

Unique ID for this running evaluation request.

IDLE-WORKERS (OBJECT)

List of currently idle workers.

SETFIDLE-WORKERS (NEW-VALUE OBJECT)

List of currently idle workers.

LOCK (OBJECT)

Lock protecting the access to this instance.

MASTER-HOST-NAME (OBJECT)

Host name of the master.

PORT (OBJECT)

Port on which the worker's Swank server is listening for connections.

SETFPORT (NEW-VALUE OBJECT)

Port on which the worker's Swank server is listening for connections.

REPLAY-FORMS (OBJECT)

List containing all forms passed to EVAL-FORM-ALL-WORKERS that need to be replayed on new workers when they join the pool.

SETFREPLAY-FORMS (NEW-VALUE OBJECT)

List containing all forms passed to EVAL-FORM-ALL-WORKERS that need to be replayed on new workers when they join the pool.

REPLAY-FORMS-LOCK (OBJECT)

Lock that protects REPLAY-FORMS.

RESULTS (OBJECT)

List holding returned, but unprocessed, results.

SETFRESULTS (NEW-VALUE OBJECT)

List holding returned, but unprocessed, results.

RESULTS-AVAILABLE (OBJECT)

Condition notified when new results are available.

RESULTS-POSITION (OBJECT)

Position where the next result will be recorded.

SETFRESULTS-POSITION (NEW-VALUE OBJECT)

Position where the next result will be recorded.

REX-PORT (OBJECT)

Port number for Swank remote execution requests to the master.

SET-WORKER (OBJECT)

Function called to record which worker is evaluating a form.

SETFSET-WORKER (NEW-VALUE OBJECT)

Function called to record which worker is evaluating a form.

STATE (OBJECT)

State of the asynchronous computation, updated from results.

SETFSTATE (NEW-VALUE OBJECT)

State of the asynchronous computation, updated from results.

STATE-COUNTER (OBJECT)

Counter incremented each time STATE is updated.

SETFSTATE-COUNTER (NEW-VALUE OBJECT)

Counter incremented each time STATE is updated.

WORKER-AVAILABLE (OBJECT)

Condition signaled when a worker becomes idle.

WORKERS (OBJECT)

Vector containing all workers in the worker pool.

Undocumented

SETF%WORKERS (NEW-VALUE OBJECT)

VARIABLE

Public

Undocumented

*REX-PORT*

Private

*EVALS*

List containing an EVALUATION instance for each running computation.

*EVALS-LOCK*

Lock protecting access to *EVALS*.

*EVALUATION-ID*

Counter used to generate a unique ID for EVALUATION instances.

*EVALUATION-ID-LOCK*

Lock protecting access to *EVALUATION-ID*.

*LAST-FORM-EVALED*

The last form evaluated by the worker.

*LAST-RANDOM-STATE*

The value of *RANDOM-STATE* right before the worker started evaluating *LAST-FORM-EVALED*.

*LAST-REPEATED-EVAL-WORK-FUNCTION*

Thunk created by the current invocation of REPEATEDLY-EVALUATE to produce results for an EVAL-FORM-REPEATEDLY request on the master. This variable is useful for debugging.

*REPLAY-FORMS-COUNTS*

Mapping from worker pool IDs to the number of replay forms we have evaluated on this client for that pool.

*REPLAY-FORMS-COUNTS-LOCK*

Lock protecting access to *REPLAY-FORMS-COUNTS*.

*WORKER-POOLS*

Mapping from worker pool IDs to active worker pools.

*WORKER-POOLS-LOCK*

Lock protecting access to *WORKER-POOLS*.

CLASS

Public

WORKER-POOL

A pool of Swank workers to which Lisp forms can be sent for evaluation.

Private

ASYNC-EVAL

Data needed to process incoming asynchronous evaluation results.

CONNECT-INFO (OBJECT)

Information needed when connecting to a worker's Swank server.

EVALUATION

Stores the data needed to process incoming evaluation results.

REPEATED-EVAL

Stores the data needed to process an incoming repeated eval result.

WORKER

A remote Lisp running a Swank server.

CONSTANT

Private

+WORKER-RECONNECT-INTERVAL+

Seconds between attempts to reconnect workers.