Kamaelia 0.5.0 Released!

Kamaelia is an intuitive way to structure applications -- as a network of components which message each other. Much like Unix pipes implemented in Python. It was originally designed by BBC Research for rapid development of server software.

Here's a taster of what a Kamaelia application looks like: (built with the GUI)

And here's some equivalent Python code:

Pipeline(
  ConsoleReader(eol=""),
  SimpleHTTPClient(),
  SimpleFileWriter("downloadedfile.txt"),
).run()

Those 5 lines of Python give you a console-based HTTP downloading program (like wget or curl but with less options) using existing components.

• the console reader sends lines of text (URLs) the user enters to...
• the HTTP client, which fetches the associated page. It then forwards on the page data to...
  
• the file writer, which appends all messages it's sent to a file on disk.

It's as simple as that.

Version 0.5.0 is a major release - lots of functionality has been added from Google Summer of Code 2006. Key highlights of this release:

• BitTorrent support (using the official BitTorrent client) - includes preliminary 'streaming over BitTorrent' support. Lets you integrate P2P in your own Kamaelia applications.
  

  

• HTTP client and nascent seaside-style pure-python webserver
  
• OpenGL (e.g. the checkers/draughts board on the right)
• Strong DVB (freeview TV) support on Linux - including the foundations of a PVR
• Collaborative whiteboarding with audio (speex encoded) - draw and talk together over the internet
• Visual composition of Kamaelia systems - create and link components on screen, see the code produced (the screenshot near the top of the article)
  

For more information see the Kamaelia website. You can get a copy of Kamaelia and Axon from Sourceforge, together with most of the dependencies in the mega bundle. If you have any problems or questions, just pop along to #kamaelia on irc.freenode.net. No Python experience is needed to get started!

Curse of BitTorrent strikes again

Some oddness was noticed with my Torrent components after they were moved into the Kamaelia main tree - they no longer worked. TorrentTkGUI just sat there when started - it didn't even get as far as popping up a window. It turned out that BitTorrent expects a locale directory in ./ - a 'feature' added to the code base recently. We're now including the required English locale file where it is expected (as it is only 409 bytes) and considering removing the need for locale files as we only use BitTorrent for the backend.

Ubuntu 6.06 (Dapper) on my Inspiron 6400

Ubuntu Dapper Drake is working very nicely on my new laptop. I had to install the ATI proprietary drivers manually as the ones in the repositories are too old to support my Radeon X1300. I have written some brief notes on how to setup Ubuntu on a Inspiron 6400. Now I can finally break my code on a true SMP machine :).

Efficient serving using flow control

Generally, different parts of Kamaelia systems run at different throughputs. You may be able to read file faster than you can serve it to your clients (because they or you have slow network connections). They may not be able to decompress images at the rate they can read their data into memory. By slowing down data sources to reflect the rate at which later components can process/transmit that data, we can save a lot of memory.
Unpausing of components upon recipient.recv is due to be added to Axon in the next few weeks. This is very important for HTTPServer as it allows it to handle bottlenecks efficiently.
Three ways a file could be sent over a network:

• Example A - without considering bottlenecks
  FileReader ---> TCPClient -> -> (slow network) -> -> TCPServer
  If the rate at which one can read a file from disk is faster than the rate at which the network can send (as it generally is), then the file ends up being stored in memory in TCPClient's inbox queue.


• Example B - considering bottlenecks but without unpausing upon recv
  RateAdaptiveFileReader ---> TCPClient -> -> (slow network) -> -> TCPServer
  
  Now let us assume that we have a file reader that won't read any more from disk until the number of chunks of data in its outbox goes below a certain value. If the file reader pauses when its outbox length reaches this value, having nothing to do, it won't be woken when messages are removed from its outbox, so most of the file will never be read. Thus the file reader cannot pause, so it has to poll its outbox repeatedly. Polling is a bad thing - it uses large amounts of CPU unnecessarily. This is how HTTPServer currently works.
  


• Example C - considering bottlenecks with unpausing upon recv
  RateAdaptiveFileReader ---> TCPClient -> -> (slow network) -> -> TCPServer
  As with Example B, except that the file reader can pause while it's waiting for the TCPClient to send stuff on. This is how HTTPServer is intended to be (and can be made to be with a few small changes).
  

HTTP server improvements and the release of GSoC 2006 work

My HTTP and BitTorrent components produced for Google Summer of Code 2006 are now in Kamaelia's main tree (/Code/Python/Kamaelia/ in subversion).

I'm auditing HTTP server components for security and efficiency to get it past the 'toy' stage of development. So far, I've begun setting limits for request sizes (not POST body sizes, just header and URI sizes). I have also improved standards-compliance by converting HTTP version numbers to tuples of the form (major, minor). This makes comparisons much easier than if they were stored as strings.

I have also augmented the CharacterFIFO (character queue) class that originated in Chunkifier with a "popline" method, allowing it to be used in HTTPParser. This character queue replaces the old, inefficient concatenation methods that were previously in use.

Implicit pausing MiniAxon

Consider a version of Axon where microprocesses pause by default after each iteration of main. Such a setup encourages the writing of efficient components by making polling difficult (but not impossible - a clock/tick threadedcomponent could send a wakeup message every second). This removes the ideas of pausing and unpausing - activate() can be used instead of unpause().

I also have different ideas about trigging the activation/unpausing of components. The component should be able to specify events that wake it. For inboxes, the receipt of a message could wake the parent component, or not. For outboxes, the removal of a sent message from the recipient inbox (through recv()) could also wake the parent.

There are occasions when you would not want delivery to an inbox to wake the component:
Consider this component:

class Combiner(component):
  Inboxes = ["a", "b"]

  def __main__(self):
      while 1:
          while self.dataReady("a") and self.dataReady("b"):
              newmsg = [self.recv("a"), self.recv("b")]
              self.send(newmsg, "outbox")
          yield 1

It waits until both inbox A and inbox B have at contain one message each, takes the first (oldest) message from each, combines them into a two-element list and sends that list on.
If A contains messages, but B contains none, the Combiner is waiting for B. The addition
of messages to A does not merit waking the component - it won't be able to do anything more than it could do before.

Syntax such as:

self.wakeOn("b")

could be used to accomplish this selectiveness.

BlitScheduler - a new approach to using pygame with Kamaelia

Structure for blitting (using any 2d graphics library, but initially pygame):
- BlitScheduler component (which subclasses Scheduler) controls a single surface
- Visual components (e.g. line, sprite) are activated with a BlitScheduler component
- For each frame/refresh of the screen, BlitScheduler follows the following procedure:

1. Sort the visual components associated with it by z-order (so the backmost is first).
2. Send a message to the blit inbox of every visual component containing a reference to the shared surface (controlled by that BS)
3. Run the generator of each visual component in turn (by z-order)
4. Flip the surface (updating it to show the newly rendered surface).

Logo clone in Kamaelia

I'm making a simple LOGO clone - eventually I intend to allow users to telnet to a shared display and command their own turtle. So far I have a single console-controlled turtle that understands LEFT, RIGHT, FORWARD and BACK, but no lines are drawn.

Dell Core2 laptops out

Hurrah! Dell UK has finally started selling Core 2 laptops, just in time for me going to university. My new Inspiron 6400 should be with me in just over a week.