Efficient serving using flow control

Generally, different parts of Kamaelia systems run at different throughputs. You may be able to read file faster than you can serve it to your clients (because they or you have slow network connections). They may not be able to decompress images at the rate they can read their data into memory. By slowing down data sources to reflect the rate at which later components can process/transmit that data, we can save a lot of memory.
Unpausing of components upon recipient.recv is due to be added to Axon in the next few weeks. This is very important for HTTPServer as it allows it to handle bottlenecks efficiently.
Three ways a file could be sent over a network:

• Example A - without considering bottlenecks
  FileReader ---> TCPClient -> -> (slow network) -> -> TCPServer
  If the rate at which one can read a file from disk is faster than the rate at which the network can send (as it generally is), then the file ends up being stored in memory in TCPClient's inbox queue.


• Example B - considering bottlenecks but without unpausing upon recv
  RateAdaptiveFileReader ---> TCPClient -> -> (slow network) -> -> TCPServer
  
  Now let us assume that we have a file reader that won't read any more from disk until the number of chunks of data in its outbox goes below a certain value. If the file reader pauses when its outbox length reaches this value, having nothing to do, it won't be woken when messages are removed from its outbox, so most of the file will never be read. Thus the file reader cannot pause, so it has to poll its outbox repeatedly. Polling is a bad thing - it uses large amounts of CPU unnecessarily. This is how HTTPServer currently works.
  


• Example C - considering bottlenecks with unpausing upon recv
  RateAdaptiveFileReader ---> TCPClient -> -> (slow network) -> -> TCPServer
  As with Example B, except that the file reader can pause while it's waiting for the TCPClient to send stuff on. This is how HTTPServer is intended to be (and can be made to be with a few small changes).
  

HTTP server improvements and the release of GSoC 2006 work

My HTTP and BitTorrent components produced for Google Summer of Code 2006 are now in Kamaelia's main tree (/Code/Python/Kamaelia/ in subversion).

I'm auditing HTTP server components for security and efficiency to get it past the 'toy' stage of development. So far, I've begun setting limits for request sizes (not POST body sizes, just header and URI sizes). I have also improved standards-compliance by converting HTTP version numbers to tuples of the form (major, minor). This makes comparisons much easier than if they were stored as strings.

I have also augmented the CharacterFIFO (character queue) class that originated in Chunkifier with a "popline" method, allowing it to be used in HTTPParser. This character queue replaces the old, inefficient concatenation methods that were previously in use.

HTTP and streaming

I've made some big changes to my HTTP code. HTTPParser no longer waits for the end of a response/request before sending information about it on - it now follows the format:

• HEADER
• BODYCHUNK
• BODYCHUNK
• ...
• BODYCHUNK
• END

This allows memory to be saved and lower latency processing. As a result of this modification, my HTTP client now allows HTTP streaming.

My HTTP client has also been refactored to support this change - it is now split into a SingleShotHTTPClient which handles a single URL before terminating and a SimpleHTTPClient which uses the SSHC with a Carousel component to handle several requests one after another. I've also added a Kamaelia component called IcecastClient which can stream MP3 audio from a SHOUTcast/Icecast server. As the SHOUTcast protocol is almost identical to HTTP, this component subclasses SSHC.

Documentation will be my next focus.