copyright(c) 2000. Franz Inc
Introduction
Running AllegroServe
Starting the Server
start
Shutting Down the Server
shutdown
Publishing Information
publish-file
publish-directory
publish
Generating a Computed Response
with-http-response
with-http-body
get-request-body
header-slot-value
reply-header-slot-value
request-query
request-query-value
Request Object Readers and Accessors
request-method
request-uri
request-protocol
request-socket
request-wserver
request-raw-request
request-reply-code
request-reply-date
request-reply-headers
request-reply-content-length
request-reply-plist
request-reply-strategy
request-reply-stream
Form Processing
get-multipart-header
get-multipart-sequence
form-urlencoded-to-query
query-to-form-urlencoded
Authorization
get-basic-authorization
set-basic-authorization
password-authorizer
location-authorizer
Cookies
set-cookie-header
get-cookie-values
AllegroServe Request Processing Protocol
handle-request
standard-locator
unpublish-locator
authorize
failed-request
denied-request
process-entity
Client Functions
do-http-request
client-request
cookie-jar
make-http-client-request
read-client-response-headers
client-request-read-sequence
Proxy
Cache
Debugging
net.aserve::debug-on
net.aserve::debug-off
AllegroServe is a webserver written at Franz Inc. AllegroServe is designed to work with the htmlgen system for generating dynamic html, as one of the big advantages of a web server written in Common Lisp is the ability to generate html dynamically. In this document we'll consider the web server and dynamic html generation to be parts of the same product.
The design goals of AllegroServe are:
Running AllegroServe requires that you
We mention publish twice to emphasize that you can publish urls before and after you start the server.
The function net.aserve:start is used to start the server running.
(start &key port listeners
chunking keep-alive
server setuid setgid
debug proxy cache accept-hook ssl)
If no arguments are given then start starts a multi-threaded web server on port 80, which is the standard web server port. If you are running this on Unix then you can only allocate port 80 if you are logged in as root or have made Lisp a set-user-id root program.
The are quite a few keyword arguments to start, but in practice you only need be concerned with :port and :listeners. The arguments have the following meanings:
This shuts down the web server given (or the most recently started web server if no argument is passed in).
Once the server is started it will accept requests from http clients, typically web browsers. Each request is parsed and then AllegroServe searches for an object to handle that request. That object is called an entity. If an entity is found, it is passed the request and is responsible for generating and sending a response to the client. If an entity can't be found then AllegroServe sends back a response saying that that request was invalid.
Publishing is the process of creating entities and registering them in the tables scanned by AllegroServe after a request is read.
A request from an http client contains a lot of information. The two items that determine which entity will handle the request are
A request contains other information and while that information isn't used to determine which entity will handle the request it can be used by the entity handling the request in any way it sees fit.
The following functions create entities and specify which requests they will handle. An entity is distinguished by the path and host values passed to the particular publish function. When a publish is done for a path and host for which there is already an entity assigned, the old entity is replaced by the new entity.
(publish-file &key path host port file content-type class preload remove server)
This creates an entity that will return the contents of a file on the disk in response to a request. The url and file must be given, the rest of the arguments are optional.. The arguments have these meanings:
The function that handles requests for files will respond correctly to If-Modified-Since header lines and thus minimizes network traffic.
Example:
This will work on Unix where the password file is stored in /etc.
(publish-file :path "/password" :file "/etc/passwd" :content-type "text/plain")
(publish-directory &key prefix host port destination remove server)
This creates a mapping from all urls whose name begins with prefix to files stored in the directory specified by the destination. The host, port, remove and server arguments are as described above for publish. When a request comes in for which there isn't an entity that matches it exactly, AllegroServe checks to see if a prefix of the request matches the request. If so, and if the resulting entity is a directory-entity as created by this function, then it strips the prefix off the given request and appends the remaining part of the request to the destination string. It then publishes that (using publish-file and computing the content-type from the file type). Next that file-entity is made to handle the request in the normal manner.
We plan to make a number of refinements to publish-directory including the ability to specify how deep in the directory structure it will look and also restricting what kinds of files it will match.
(publish &key path host port content-type function class format remove server)
This creates a mapping from a url to a computed-entity, that is an entity that computes its response every time a request comes in. The path, host, port, remove, server and class arguments are as in the other publish functions. The content-type sets a default value for the response to the request but this can be overridden. The format argument is either :text (the default) or :binary and it specifies the kind of value that will be sent back (after the response headers, which are always in text). This value is only important if the response is generated in a particular way (described below).
The function argument is a function of two arguments: an object of class http-request that holds a description of the request, and an object of class entity that holds this entity which is handling the request. This function must generate a response to the http request, even if the response is only that the request wasn't found.
There are a variety of ways that a response can be sent back to the http client depending on whether keep-alive is being done, chunking is possible, whether the response is text or binary, whether the client already has the most recent data, and whether the size of the body of the response is known before the headers are sent. AllegroServe handles the complexity of determining the optimal response strategy and the user need only use a few specific macros in the computation of the response in order ot take advantage of AllegroServe's strategy computation
Here's a very simple computed response. It just puts "Hello World!" in the browser window:
(publish :path "/hello"
:content-type "text/html"
:function #'(lambda (req ent)
(with-http-response (req ent)
(with-http-body (req ent)
(html "Hello World!")))))
This example works regardless of whether the request comes in from an old HTTP/0.9 browser or a modern HTTP/1.1 browser. It may or may not send the response back with Chunked transfer encoding and it may or may not keep the connection alive after sending back the response. The user code doesn't have to deal with those possibilities, it just uses with-http-response and with-http-body and the rest is automatic. The html macro is part of the htmlgen package that accompanies AllegroServe. In the case above we are being lazy and not putting out the html directives that should be found on every page of html since most browsers are accommodating. Here's the function that generates the correct html:
(publish :path "/hello2"
:content-type "text/html"
:function #'(lambda (req ent)
(with-http-response (req ent)
(with-http-body (req ent)
(html
(:html
(:body "Hello World!")))))))
The function above generates: <html><body>Hello World!</body></html>.
The macros and functions used in computing responses are these:
(with-http-response
(req ent &key timeout check-modified response content-type)
&rest body)
This macro begins the process of generating a response to an http request and then runs the code in the body which will actually send out the response. req and ent are the request and entity objects passed into the function designated to compute the response for the request. timeout sets a time limit for the computation of the response. The default is the value of the variable *http-response-timeout* (which is initially 120 meaning 120 seconds). If check-modified is true (the default) then the last-modified time stored in the entity object will be compared against the if-modified-since time of the request and if that indicates that the client already has the latest copy of this entity then a not-modified response will be automatically returned to the client and the body of this macro will not be run. response is an object containing the code and description of the http response we wish to return. The default value is the value of *response-ok* (which has a code of 200 and a string descriptor "OK"). content-type is a string describing the MIME type of the body (if any) sent after the headers. It has a form like "text/html". If content-type isn't given here then the content-type value in the entity (which is set in the call to publish) will be used.
An http response consists of a line describing the response code, followed by headers (unless it's the HTTP/0.9 protocol in which case there are no headers), and then followed by the body (if any) of the response. with-http-response doesn't normally send anything to the client. It only does so when it determines that the if-modified-since predicate doesn't hold and that it must send back a not-modified response. Thus in the normal case it isn't enough, even if there no body to send, to just call with-http-response. This is where the with-http-body is used.
(with-http-body (req ent &key format headers external-format) &rest body)
This macro causes the whole http response to be sent out. The macro itself will send out everything except the body of the response. That is the responsibility of the code supplied as the body form of the macro. In cases where there is no body to the response being sent it is still necessary to call with-http-body so that the other parts of the response are sent out, e.g. at a minimum you should put (with-http-body (req ent)) in the body of a with-http-response
The format argument specifies whether the code in the body will want to write :text (e.g. write-char) or :binary (e.g. write-byte) when it writes the data of the body of the response. Based on the value of the format argument, AllegroServe will create the correct kind of response stream. If format is not specified here it will default to the value specified when publish was called to create the entity.
The headers argument is a list of conses, where the car is the header name (a keyword symbol) and the cdr is the header value. These headers are added to the headers sent as part of this response.
Within the body forms the code calls (request-reply-stream req) to obtain a stream to which it can write to supply the body of the response. The external-format of this stream is set to the value of the external-format argument (which defaults to :latin1-base). The variable *html-stream* is bound to the value of (request-reply-stream req) before the body is evaluated. This is done to for the convenience of users of the html macro.
Return the body of the request as a string. If there is no body the return value will be an empty string. The result is cached inside the request object, so this function can be called more than once while processing a request. The typical reason for there to be a body to a request is when a web browser sends the result of a form with a POST method.
(header-slot-value request header-name)
Return the value given in the request for the given header-name (a keyword symbol). If the header wasn't present in this request then nil will be returned. header-slot-value is a macro that will expand into a fast accessor if the header-name is a constant naming a known header slot
In older versions of aserve the header-name was a string..
(reply-header-slot-value request header-name)
Return the value associated with the header header-name in the reply sent back to the client. This function is setf'able and this is the preferred way to specify headers and values to be sent with a reply.
(request-query request &key uri post external-format)
Decode and return an alist of the query values in the request. Each item in the alist is a cons where the car is a string giving the name of the argument and the cdr is a string giving the value of the argument.
The query string is in one or both of two places:
request-query will by default look in both locations for the query string and concatenate the results of decoding both query strings. If you would like it to not check one or both of the locations you can use the :uri and :post keyword arguments. If uri is true (and true is the default value) then the query string in the uri is checked. If post is true (and true is the default value) and if the request is a POST then the body of the post form will be decoded for query values.
The external-format is used in the conversion of bytes in the form to characters.
A query is normally a set of names and values.
http://foo.com/bar?a=3&b=4 yields a query alist (("a"
. "3") ("b" . "4")).
If a name doesn't have an associated value then the value in the alist is the empty
string.
http://foo.com/bar?a&b=&c=4 yields a query alist (("a"
. "") ("b" . "") (c . "4"))
.
(request-query-value key request &key uri post test)
This combines a call to request-query to retrieve the alist of query keys and values, with a call to assoc to search for the specific key, and finally with a call to cdr to return just the value from the assoc list entry. The test argument is the function to be used to test the given key against the keys in the assoc list. It defaults to #'equal.
If the given key is not present in the query nil is returned. If the given key is present in the query but doesn't have an associated value then the empty string is returned.
The request object contains information about the http request being processed and it contains information about the response that is being computed and returned to the requestor. The following functions access slots of the request object. Those with names beginning with request-reply- are accessing the slots which hold information about the response to the request. When a function is listed as an accessor that means that it can be setf'ed as well as used to read the slot value.
(request-method request) - reader - a keyword symbol naming the kind of request, typically :get, :put or :post.
(request-uri request) - reader - a uri object describing the request. If the request contains a "Host:" header line then the value of this header is placed in the uri-host and uri-port slots of this uri object.
(request-protocol request) - reader - a keyword symbol naming the http protocol requested. It is either :http/0.9, :http/1.0 or :http/1.1.
(request-protocol-string request) - reader - a string naming the http protocol requested. It is either "HTTP/0.9", "HTTP/1.0" or "HTTP/1.1".
(request-socket request) - reader - the socket object through which the request was made and to which the response must be sent. This object can be used to determine the IP address of the requestor.
(request-wserver request) - reader - the wserver object describing the web server taking this request
(request-raw-request request) - reader - a string holding the exact request made by the client
(request-reply-code request) - accessor - the value describes the response code and string we will return for this request. See the value of the argument response in with-http-response for more information.
(request-reply-date request) - accessor - the date the response will be made (in Lisp's universal time format). This defaults to the time when the request arrived.
(request-reply-headers request) - accessor - an alist of some of the headers to send out with the reply (other headers values are stored in specific slots of the request object). Each entry in the alist is a cons where the car is a keyword symbol holding the header name and the cdr is the value (it is printed using the ~a format directive). Typically request-reply-headers isn't used, instead the headers to be sent are passed as the :header argument to with-http-body, or (setf reply-header-slot-value) is called.
(request-reply-content-length request) - accessor - the value to send as the Content-Length of this response. This is computed automatically by AllegroServe and thus a user program shouldn't have to set this slot under normal circumstances.
(request-reply-plist request) - accessor - this slot holds a property list on which AllegroServe uses to stores some lesser used information. The user program can use it as well.
(request-reply-strategy request) - accessor - the strategy is a list of symbols which describe how AllegroServe will build a response stream and will send back a response. More details will be given about the possible strategies at a future time.
(request-reply-stream request) - accessor - This is the stream to be used in user code to send back the body of the response. This stream must be used instead of the value of request-socket.
Forms are used on web pages in order to allow the user to send information to the web server. A form consists of a number of objects, such as text fields, file fields, check boxes and radio buttons. Each field has a name. When the user takes a certain action, the form data is encoded and sent to the web server. There are three ways that data can be sent to the web server. The method used is determined by the attributes of the <form> tag that defines the form
If you create a form with <form method="post" enctype="multipart/form-data"> then your url handler must do the following to retrieve the value of each field in the form:
It's important to retrieve all of the data sent with the form, even if that data is just ignored. This is because there may be another http request following this one and it's important to advance to the beginning of that request so that it is properly recognized.
Details on the functions are given next.
(get-multipart-header request)
This returns nil or the MIME headers for the next form field in alist form. If nil is returned then there is no more form data.
For an input field such as <input type="text" name="textthing"> the value returned by get-multipart-header would be
(("Content-Disposition"
(:param "form-data" ("name" . "textthing"))))
For an input field such as <input type="file" name="thefile"> the value returned by get-multipart-header would be something like
(("Content-Disposition"
(:param "form-data" ("name" . "thefile")
("filename" . "C://down//550mhz.gif")))
("Content-Type" "image/gif"))
Note that the filename is expressed in the syntax of the operating system on which the web browser is running. This syntax may or may not make sense to the Lisp pathname functions of the AllegroServe web server as it may be running on a totally different operating system.
(get-multipart-sequence request buffer &key start end external-format)
This retrieves the next chunk of data for the current form field and stores it in buffer. If start is given then it specifies the index in the buffer at which to begin storing the data. If end is given then it specifies the index just after the last index in which to store data.
The return value is nil if there is no more data to return, otherwise it is the index one after the last last index filled with data in buffer.
The buffer can be a one dimensional array of character or of (unsigned-byte 8). For the most efficient transfer of data from the browser to AllegroServe, the program should use a 4096 byte (unsigned-byte 8) array.
If the buffer is a character array then the data is converted from get-multipart-sequence's (unsigned-byte 8) array to characters using the given external-format (which defaults to :latin1-base).
get-multipart-sequence may return before filling up the whole buffer, so program should be sure to make use of the index returned by get-multipart-sequence.
In AllegroServe the information sent to the web server as a result of filling out a form is called a query. We store a query as a list of conses, where the car of the cons is the name (a string) and the cdr of the cons is the value (another string). When a query is transmitted by the web browser to AllegroServe it is sent as string using the encoding application/x-www-form-urlencoded. We provide the following functions to convert between the encoding and the query list:
(form-urlencoded-to-query string &key external-format)
Decodes the string and returns the query list. The default value for external-format is :latin1-base.
(query-to-form-urlencoded query &key external-format)
Encodes the query and returns a string. The default value for external-format is :latin1-base
Examples:
user(4): (query-to-form-urlencoded '(("first name" . "joe")
("last name" . "smith")))
"first+name=joe&last+name=smith"
user(5): (form-urlencoded-to-query "first+name=joe&last+name=smith")
(("first name" . "joe") ("last name" . "smith"))
user(6): (query-to-form-urlencoded
`(("last name" . ,(coerce '(#\hiragana_letter_ta
#\hiragana_letter_na
#\hiragana_letter_ka)
'string)))
:external-format :euc)
"last+name=%a4%bf%a4%ca%a4%ab"
user(7): (query-to-form-urlencoded
`(("last name" . ,(coerce '(#\hiragana_letter_ta
#\hiragana_letter_na
#\hiragana_letter_ka)
'string)))
:external-format :shiftjis)
"last+name=%82%bd%82%c8%82%a9"
user(8): (coerce
(cdr
(assoc "last name"
(form-urlencoded-to-query "last+name=%82%bd%82%c8%82%a9"
:external-format :shiftjis)
:test #'equalp))
'list)
(#\hiragana_letter_ta #\hiragana_letter_na #\hiragana_letter_ka)
You may want to restrict certain entities to be accessible from only certain machines or people. You can put the test for authorization in the entity response function using one of the following functions, or you can have the check done automatically by storing an authorizer object in the entity.
(get-basic-authorization request)
This function retrieves the Basic authorization information associated with this request, if any. The two returned values are the name and password, both strings. If there is no Basic authorization information with this request, nil is returned.
(set-basic-authorization request realm)
This adds a header line that requests Basic authorization in the given realm (a string). This should be called between with-http-response and with-http-body and only for response of type 401 (i.e. *response-unauthorized*). The realm is an identifier, unique on this site, for the set of pages for which access should be authorized by a certain name and password.
This example manually tests for basic authorization where the name is foo and the password is bar.
(publish :path "/secret"
:content-type "text/html"
:function
#'(lambda (req ent)
(multiple-value-bind (name password) (get-basic-authorization req)
(if* (and (equal name "foo") (equal password "bar"))
then (with-http-response (req ent)
(with-http-body (req ent)
(html (:head (:title "Secret page"))
(:body "You made it to the secret page"))))
else ; this will cause browser to put up a name/password dialog
(with-http-response (req ent :response *response-unauthorized*)
(set-basic-authorization req "secretserver")
(with-http-body (req ent)))))))
If an entity has an associated authorizer object, then before that entity's response function is run the authorizer is tested to see if it will accept or deny the current request. AllegroServe supplies two interesting subclasses of authorizer and users are free to add their own subclasses to support their own authorization needs.
The protocol followed during authorization is this:
This subclass of authorizer is useful if you want to protect an entity using the basic authorization scheme that asks for a name and a password. When you create this class of object you should supply values for the two slots:
| Slot Name | initarg | what |
| allowed | :allowed | list of conses, each cons having the form ("name" . "password") where any of the listed name password pairs will allow access to this page. |
| realm | :realm | A string which names the protection space for the given name and password. The realm will appear in the dialog box the browser displays when asking for a name and password. |
An example of it's use is the following where we allow access only if the user enters a name of joe and a password of eoj or a name of fred and a password of derf.
(publish :path "/foo"
:content-type "text/html"
:authorizer (make-instance 'password-authorizer
:allowed '(("joe" . "eoj")
("fred" . "derf"))
:realm "SecretAuth")
:function
#'(lambda (req ent)
(with-http-response (req ent)
(with-http-body (req ent)
(html (:head (:title "Secret page"))
(:body "You made it to the secret page"))))))
This authorizer class checks the IP address of the request to see if it is permitted access to the entity. The authorizer can specify a sequence of patterns and for each pattern a command of :accept (permit the access) or :deny (forbid the access). The first pattern that matches determines if the request is accepted or denied. If the pattern list is empty or if no pattern matches, then the request is accepted.
The single slot of an object of class location-authorizer is
| Slot Name | initarg | what |
| patterns | :patterns | a list of patterns and commands, where the syntax of a pattern-command is described below. |
A pattern can be
The example of using a location-authorizer only permits connections coming in via the loopback network (which occurs if you specify http://localhost/whatever) or if they come from one particular machine (tiger.franz.com). Note that we end the pattern list with :deny so that anything not matching the preceding patterns will be denied.
(publish :path "/local-secret-auth"
:content-type "text/html"
:authorizer (make-instance 'location-authorizer
:patterns '((:accept "127.0" 8)
(:accept "tiger.franz.com")
:deny))
:function
#'(lambda (req ent)
(with-http-response (req ent)
(with-http-body (req ent)
(html (:head (:title "Secret page"))
(:body (:b "Congratulations. ")
"You made it to the secret page"))))))
Cookies are name value pairs that a web server can direct a web browser to save and then pass back to the web server under certain circumstances. Some users configure their web browsers to reject cookies, thus you are advised against building a site that depends on cookies to work.
Each cookie has these components:
(set-cookie-header request &key name value expires domain path secure external-format)
This function should be called between the calls to with-http-response and
with-http-body. It can be called more than once. Each
call will cause one Set-Cookie directive to be sent to the web browser.
The name and value arguments should be given (and they
should be strings). They will be automatically encoded using the same encoding used
in urls (we call it uriencoding). The purpose of this encoding is to convert
characters that are either unprintable or those that have a special meaning into a
printable string. The web browser doesn't care about the name
and value, it just stores them and sends them back to the web server.
If you use the get-cookie-values function to retrieve
the cookie name and value pairs, then it will
automatically decode the uriencoding.
If the path argument isn't given, it will default to "/" which
will allow this cookie to match all requests.
If the domain argument isn't given then it will default to the host to
which this request was sent. If you wish to specify this you are only allowed to
specify a subsequence of the host to which this request was sent (i.e the name of the
machine running the webserver). The domain should have at
least two periods in it (i.e. ".foo.com").
expiration can be a lisp universal time or it can be the symbol :never
meaning this should never expire. If expiration isn't given or is nil
then this cookie will expire when the user quits their web browser.
secure should be true or false. Any non-nil value is interpreted as
true. The default value is false.
The external-format is used to convert bytes to characters. It
defaults to :latin1-base.
(get-cookie-values request &key external-format)
Return the cookie name and value pairs from the header of the request. Each name value pair will be in a cons whose car is the name and whose cdr is the value. The names and values will be decoded (in other words the decoding done by set-cookie-header will be undone). The external-format is used to convert bytes to characters. It defaults to :latin1-base.
We'll describe here the steps AllegroServe goes through from the time it receives a request until a response to that request has been sent back to the browser. We want the protocol to be open so that users can extend AllegroServe's behavior to suit their needs. However given that AllegroServe is a new program and will be undergoing extensive review from its users, we expect that the protocol will change. It shouldn't lose any of its current extensibility but the names and argument lists of generic functions may change.
When a client connects to the port on which AllegroServe is listening, AllegroServe passes that connected socket to a free worker thread which then wakes up and calls the internal function net.aserve::process-connection. If there are no free worker threads then AllegroServe waits for one to be available.
In each worker thread the variable *wserver* is bound to the wserver object that holds all the information about the webserver on which the connection was made (remember that one AllegroServe process can be running more than one webserver). process-connection reads the request from the socket (but doesn't read past the header lines). If the request can't be read within *read-request-timeout* seconds (currently 20) then the request is rejected. The request is stored in an object of class http-request. Next process-connection calls handle-request to do all the work of the request and then log-request to log the action of the request. Finally if the response to the request indicated that the connection was to be kept open rather than being closed after the response, then process-connection loops back to the top to read the next request.
(handle-request (req http-request)) [generic function]
This generic function must locate the entity to handle this request and then cause it to respond to the request. If there is no matching entity then handle-request must send a response back to the client itself. handle-request uses locators to find the entity (more on this below), and then if an entity is found and that entity has an authorizer, it calls authorize to see if this request is allowed to access the selected entity. If the entity passes the authorization then process-entity is called to cause the entity to respond to the request. process-entity returns true if it processed the entity, and nil if did not in which case the search continues for an entity. If there is no entity to respond then failed-request is called to send back a failure message.
A locator is an object used to map requests into entities. The value of (wserver-locators *wserver*) is a list of locator objects. handle-request calls
(standard-locator (req http-request) (loc locator)) [generic function]
on each successive locator in that list until one returns an entity object. AllegroServe has two built-in locator classes, locator-exact and locator-prefix, that are subclasses of locator. When you call publish or publish-file you are adding the entity to locator of class locator-exact found in the wserver-locators list. When you call publish-directory you are adding to the locator of class locator-prefix. Users are free to define new locator classes. Locators should define the standard-locator method as well as
(unpublish-locator (loc locator)) [generic function]
which if called should remove all published entities from the locator.
Let's return to handle-request. It has called standard-locator and found an entity. Next it checks to see if the entity has an authorizer value and if so calls
(authorize (auth authorizer) (req http-request) (ent entity)) [generic function]
The return value will be one of
If there is no authorizer for this entity then we just call process-entity. If there is no entity, then we call failed-request.
(failed-request (req http-request)) [generic function]
send back a response to the effect that the url request doesn't exist on this server.
(denied-request (req http-request)) [generic function]
send back a response to the effect that access to the requested url was denied.
(process-entity (req http-request) (ent entity)) [generic function]
Send back a response appropriate to the given entity. The macros with-http-response and with-http-body should be used in the code that sends the response.
AllegroServe has a set of functions that perform http client-side actions. These functions are useful in generating computed pages that reflect the contents of other pages. We also use the client-side http functions to test AllegroServe.
The client-side functions described in this section are exported from the net.aserve.client package.
The function do-http-request sends a request and retrieves the whole response. This is the most convenient function to use to retrieve a web page.
If you need more control over the process you can use the functions: make-http-request, read-client-response-headers and client-request-read-sequence.
(do-http-request uri
&key method protocol accept
content content-type query format cookies
redirect basic-authorization
keep-alive headers proxy user-agent external-format ssl)
Sends a request to uri and returns four values:
The uri can be a uri object or a string. The scheme of the uri must be nil or "http". The keyword arguments to do-http-request are
| Name | default | description |
|---|---|---|
| method | :get | The type of request to make. Other possible values are :put, :post and :head. :head is useful if you just want to see if the link works without downloading the data. |
| protocol | :http/1.1 | The other possible value is :http/1.0. Modern web servers will return the response body in chunks if told to use the :http/1.1 protocol. Buggy web servers may do chunking incorrectly (even Apache has bugs in this regard but we've worked around them). If you have trouble talking to a web server you should try specifying the :http/1.0 protocol to see if that works. |
| accept | "*/*" | A list of MIME types that are acceptable as a response to this request. The default is to accept anything. |
| content | nil | If the method is :put or :post then the request should include something to be sent to the web server. The value of this argument is either a string or a vector of type (unsigned-byte 8) which will be sent to the web server. See the query argument for a more convenient way to :post data to a form. |
| content-type | nil | A string which is to be the value of the Content-Type header field, describing the format of the value of the content argument. This is only needed for :put and :post requests. |
| query | nil | This is a query alist of the form suitable for query-to-form-urlencoded. If the method is a :get then the value of this argument is urlencoded and made the query string of the uri being accessed. If the method is :post then the query string is urlencoded and made the content of the request. Also the content-type is set to application/x-www-form-urlencoded. |
| format | :text | The body of the response is returned as a string if the value is :text or as an array of type (unsigned-byte 8) if the value is :binary. When the body is a string the external-format argument is important. |
| cookies | nil | If you wish the request to include applicable cookies and for returned cookies to be saved, then a cookie-jar object should be passed as the value of this argument. |
| redirect | 5 | If the response is a redirect (code 301, 302, 303), and the method is :get or :head, then if this argument is true (and, if an integer, positive), do-http-request will call itself to access the page to which the redirection is pointed. If redirect is an integer then in the recursiive call the valued passed for redirect will be one less than the current value. This prevents infinite recursion due to redirection loops. |
| basic-authorization | nil | If given, it is a cons whose car is the name and whose cdr is the password to be used to get authorization to access this page. |
| keep-alive | nil | If true then the web server will be told to keep the connection alive. Since do-http-request closes the connection after the request this option currently does no more than allow us to experiment with how a web server responds to a keep-alive request. |
| headers | nil | an alist of conses ("header-name" . "header-value") for additional headers to send with the request. |
| proxy | nil | the name and optionally the port number of a proxying web server through which this request should be made. The form is of the argument is "www.machine.com" or "www.machine.com:8000" if the web server is listening on port 8000 rather than 80. Proxying web servers are often used when clients are behind firewalls that prevent direct access to the internet. Another use is to centralize the page cache for a group of clients. |
| user-agent | nil | If given it specfies the value of the User-Agent header to be sent with the request. Some sites respond differently based on the user-agent they believe has made the request. The lack of a User-Agent header may cause a server to ignore a request since it believes that it is being probed by a robot. The value of user-agent can be a string or one of the keywords :aserve, :netscape or :ie in which case an appropriate user agent string is sent. |
| external-format | :latin1-base | This determines the socket stream's external format. |
| ssl | nil | If true then the connection is made using the Secure Sockets Layer protocol. If the uri uses the https scheme then ssl is assumed to be true and the ssl argument need not be specified. |
For example:
user(5): (do-http-request "http://www.franz.com")
"<HTML>
<HEAD>
<TITLE>Franz Inc: Allegro Common Lisp and Common Lisp Products</TITLE>
<BASE FONTFACE=\"helvetica, arial\" FONTSIZE=\"1\">
.....
"
200
(("content-type" . "text/html") ("transfer-encoding" . "chunked")
("server" . "Apache/1.3.9 (Unix) PHP/3.0.14")
("date" . "Mon, 24 Apr 2000 11:00:51 GMT"))
It's easy to use do-http-request to fill in form objects on a page. If the form has input elements named width and height then you can send a request that specifies that information in this way:
(do-http-request "http://www.foo.com/myform.html"
:query '(("width" . 23) ("height" . 45)))
The above assumes that the method on the form is "GET". If the method is "POST" then a similar call will work:
(do-http-request "http://www.foo.com/myform.html" :method :post
:query '(("width" . 23) ("height" . 45)))
Before we describe the lower level client request functions we will describe two classes of objects used in that interface.
A client-request object includes the information about the request and the response.
The public fields of a client-request that are filled in after a call to make-http-client-request are:
| Accessor | Description |
|---|---|
| client-request-uri | uri object corresponding to this request |
| client-request-socket | socket object open to the web server denoted by the uri |
| client-request-cookies | the cookie-jar object (if any) passed in with this request. |
After read-client-response-headers is called, the following fields of the client-request objects are set:
| Accessor | Description |
|---|---|
| client-request-response-code | the integer that is the response code for this request. The most common codes are 200 for Success and 404 for Not Found. |
| client-request-headers | an alist of header values in the response. Each entry is a cons of the form ("header-name" . "header-value"). The header names are all lower case. |
| client-request-protocol | A keyword symbol naming the protocol that the web server returned (which may be different that the protocol given in the request). A typical return value is :http/1.1 |
| client-request-response-comment | A string giving a textual version of the response code. The string is arbitrary and you should not depend on all web servers returning the same string for any given response code. |
A cookie-jar is a respository for cookies. Cookies are stored in a jar when a response from a client request includes Set-Cookie headers. Cookies from a jar are sent along with a request when they are applicable to the given request. We won't describe the rules for cookie applicability here, you need only know that if you use our client functions to access a site that uses cookies to implement persistence, then you should create a cookie-jar object and pass that same object in with each request. More information on cookies can be found here.
A cookie-jar is created with (make-instance 'cookie-jar).
(cookie-jar-items cookie-jar)
returns an alist of the cookies in the jar where each item has the form:
(hostname cookie-item ...)
The hostname is a string which is matched against the suffix of the name of the host in the request (that is, a hostname of ".foo.com" matches "a.foo.com" and "b.foo.com". ). The hostname should have at least two periods in it. The following cookie-item objects in the list all apply to that hostname. A cookie-item is a defstruct object and has these fields
| Accessor | Description |
|---|---|
| cookie-item-path | A string that must be the prefix of the path of the request for it to match. The prefix "/" matches all paths. |
| cookie-item-name | The name of the cookie. A string. |
| cookie-item-value | The value of the cookie. A string. |
| cookie-item-expires | A string holding the time the cookie expires [in a future release we may make this a universal time] |
| cookie-item-secure | true if this cookie should only be sent over a secure connection. |
(make-http-client-request
uri &key method protocol keep-alive
accept cookies headers proxy
basic-authorization query
content content-type content-length
external-format)
This function connects to the web server indicated by the uri and sends the request. The arguments are the same as those for do-http-request and are documented there. There is one additional argument: content-length. This argument can be used to set the content-length header value in the request. After setting the content-length the caller of make-http-client-request would then be responsible for sending that many bytes of data to the socket to serve as the body of the request. If content-length is given, then a value for content should not be given.
If make-http-client-request succeeds in contacting the web server and sending a request, a client-request object is returned. If make-http-client-request fails, then an error is signalled.
The returned client-request object contains an open socket to a web server, thus you must ensure that client-request object isn't discarded before client-request-close is called on it to close the socket and reclaim that resource.
After calling make-http-client-request the program will send the body of the request (if any), and then it will call read-client-response-headers to partially read the web server's response to the request.
(read-client-response-headers client-request)
This function reads the response code and response headers from the web server. After the function returns the program can use the client-request accessors noted above to read the web server's response. The body of the response (if any) has not been read at this point. You should use client-request-read-sequence to read the body of the response
(client-request-read-sequence
buffer client-request
&key start end)
This fills the buffer with the body of the response from the web server. The buffer should either be a character array or an array of (unsigned-byte 8). If given, start specifies the index of the first element in the buffer in which to store, and end is one plus the index of the last element in which to store.
The return value is one plus the last index in the buffer filled by this function. The caller of the function must be prepared for having the buffer only partially filled. If the return value is zero then it indicates an End of File condition.
AllegroServe can serve as an http proxy. What this means is that web clients can ask AllegroServe to fetch a URL for them. The two primary uses for a proxy server are
In order to run AllegroServe as a proxy server you should specify :proxy t in the arguments to the net.aserve:start function. With this specified AllegroServe will still act as a web server for pages on the machine on which AllegroServe is running. AllegroServe will act as a proxy for requests to other machines.
Each web browser has it's own way of specifying which proxy server it should use. For Netscape version 4 select the Edit menu, then Preferences... and then click on the plus sign to the left of Advanced. Then select Proxies and click on Manual Proxy Configuration and the click on View and specify the name of the machine running AllegroServe and the port number on which AllegroServe is listening. Then click OK on all the dialog boxes.
For Internet Explorer 5 select the Tools menu, and then Internet Options.. and then the Connections tab, and then LAN Settings. Click on Use a Proxy Server and then click on Advanced and specify the machine name and port number for AllegroServe. Then click on OK to dismiss the dialog windows.
The AllegroServe cache is a facilty in development. We'll describe here the current status of the code.
The cache consists of a memory cache and a set of zero or more disk caches. Items initially live in the memory cache and are moved to the disk caches when the memory cache fills up. Items enter the memory cache due to a page being accessed via the proxy server. Items in the disk cache move back to the memory cache if the data portion must be sent back to the requesting client (some requests can be answered without sending back the contents of the page and for these the item stays in the disk cache).
You specify the sizes of each cache. The disk caches will never grow beyond the size you specified but the memory cache can exceed the specified size for a short time. A background thread moves items from the memory cache to the disk caches and we will allow you to control how often that thread wakes up and ensures that the memory cache is within the desired contstaints.
When net.aserve:start is called you specify if you want caching and if so what sizes caches you want. A sample argument pair passed to net.aserve:start is
:cache '(:memory 10000000 :disk ("/tmp/mycache" 30000000) :disk (nil 20000000))
This says that the memory cache should be 10,000,000 bytes and that there should be two disk caches. One disk cache is the file "/tmp/mycache" and can grow to 30,000,000 bytes and the other cache will have a name chosen by AllegroServe and it can grow to 20,000,000 bytes. We should note here that one thing that distinguishes the AllegroServe caching facilty from that found in many other http proxy-caches is that AllegroServe uses a few large cache files rather than storing each cached item in a separate file in the filesystem.
A few other ways of specifying caching at startup is:
:cache t
This will create a memory case of the default size (currently 10 megabytes) and it will create no disk caches.
:cache 20000000
This will create a memory cache of 20,000,000 byte and no disk caches.
When caching is enabled we publish two links to pages showing cache information. This is useful during debugging and is likely to change in the future. The two pages are /cache-stats and /cache-entries.
Debugging entity handler functions is difficult since these are usually run on a separate lisp thread. Also AllegroServe catches errors in entity handler functions, thus preventing you from interactively diagnosing the problem.
You can put aServe in a mode that makes debugging easier with the net.aserve::debug-on function. Note that this is not an exported function to emphasize the fact that you are working with the internals of aServe.
(net.aserve::debug-on &rest debugging-features-to-enable)
We've classified the debugging features and given each a keyword symbol name. This function turns on those named features. If no arguments are given, then debug-on prints the list of debugging features and whether each is enabled.
(net.aserve::debug-off &rest debugging-features-to-disable)
This function turns off the given list of features.
The list of debug features are:
| :info | aServe prints information at certain places while doing its processing. |
| :xmit | aServe prints what it receives from and sends to the client. In some cases the body of a request or response will not be printed. |
| :notrap | When enabled, this prevents aServe from catching errors in entity handler functions. If an error occurs and you're running in an evironment where background processes automatically create new windows (such as the emacs-lisp interface) then you'll be given a chance to :zoom the stack and diagnose the problem. Note that if a timeout has been established to limit the amount of time that a certain step is allowed (and this is done by default) then the interactive debugging session will be aborted when the timeout is reached. |
Two pseudo debug features are :all and :log.. Specifying :all to debug-on or debug-off is the same as listing all of the debug features. Specifying :log is the same as specifying all features except :notrap.