URL_Fetcher (changes)

Showing changes from revision #4 to #5: Added | Removed | Changed

Programmer's Reference Manuals

» List of modules

»URL_Fetcher

The URL (Universal Resource Locator) module is a general purpose module for fetching data from various Internet services. The purpose of the module is to provide a uniform entry point into a set of “fetcher” protocols (e.g. FTP, HTTP, Gopher, NNTP, etc.), without the need for a client application to understand how that protocol works. This is done using a number of generalised URL SWIs. The fetcher protocols modules with which the URL module communicates, are called only by the URL module itself. The entry points into the protocol modules have similar names to the entry points into the URL module, but these are NOT the same, despite similarities.

Each client fetch occurs with in the context of a ‘session’. Each session is identified by a different session identifier. Client session identifiers are issued by the URL module upon request and remain valid until the client informs the URL module to discard the session. Subsequently, session identifiers may be re-issued by the URL module for new sessions. Only a single object fetch can be performed in any one given session. Sessions cannot be re-used by clients, even if a prior object fetch in that session has completed.

The typical client usage of the system is:

• Obtain a session identifier (SWI URL_Register)URL_Register)
• Start fetching an object (SWI URL_GetURL)URL_GetURL)
• Repeatedly, whilst multi-tasking if in the desktop environment:
  – Read blocks of data (SWI URL_ReadData)URL_ReadData)
  – Process that data
• Discard session (SWI URL_Deregister)URL_Deregister)

If an application decides it requires a premature termination (e.g. the user asked the application to quit whilst an object was being downloaded), then the application calls SWI URL_Stop immediately and then discards the session withURL_Stop immediately and then discards the session with SWI URL_Deregister. Typical clients, such as web browsers, will, most likely, have several sessions active concurrently.URL_Deregister. Typical clients, such as web browsers, will, most likely, have several sessions active concurrently.

The URL module uses its own session identifiers that are passed in many of the SWI interfaces to the protocol modules which are not those known to the client application – the URL module maintains its own private sessions into the protocol modules. Service calls are also provided to ease interaction between the URL module and the fetchers, mainly to inform other modules of the arrival or departure of a particular module.

Each protocol module accepts data and returns results as per the HTTP protocol. Thus any extra client data associated with a request (passed in R4 to SWI URL_GetURL) will take the format of a (possibly empty) set ofURL_GetURL) will take the format of a (possibly empty) set of HTTP headers, an empty line and then the data; and each response will start with an HTTP/1.1 Response-Line of the format: “HTTP/1.1 200 OK” followed by various headers identifying the content-type of the retrieved data, followed by an empty line, followed by the data itself.

Details

• SWI Calls
• Service Calls
• *Commands

Further information on how the URL_Fetcher module communicates with the protocol modules is found in the URL Fetcher API Specification. A start has been made in transferring this information to wiki pages. See URL_Fetcher Protocol module interface.