Use webpy to stream continuous data

Question

I want to stream continuous sensor data to a small web-socket using python and web.py. But I fail to update the data.

A minimum working example looks like this:

#!/usr/bin/env python
import web
from time import sleep

# 1. Data part, create some csv-string
streamString = "x,y\n"
streamString = streamString + "123,123\n"

# 2. webpy part, create a minimum server
urls = ('/', 'index')
class index:
    def GET(self):
        return streamString

if __name__ == "__main__":
    app = web.application(urls, globals())
    app.run()
    # simulate some sensor data that is read continuously
    i = 0 
    while True:
        newString = "{0},{1}\n".format(i, i+1)
        i = i + 2
        streamString = streamString + newString
        sleep(1)

This creates the app which displays the first entry (x,y\n123,123\n), but it fails to show the "live"-data (I assume this is as app.run() comes before the loop)

Any idea how I can make this work?

Answer 1

First, you're correct: your while True: is never executed because it's after the app.run().

The basic strategy is to make GET() a generator, using yield, rather than returning a string. For example:

class index:
    def GET(self):
      web.header('Transfer-Encoding', 'chunked')
      yield "Here we go!"
      i = 0
      while True:
          newString = "{0},{1}\n".format(i, i+1)
          i = i + 2
          sleep(1)
          yield newString

1. Use Transfer-Encoding 'chunked' to tell the browser not all the data is coming at once.
2. Yield only the new values (your example concatenated the strings).
3. Because you use yield, in the GET() all data returned by that GET must use yield (hence my yield 'Here we go!' example.)

It works fine, but your Browser will buffer the results: it will indeed receive the data a one-second intervals "forever", but the browser doesn't update on each reception. Chrome, for example, appears to wait until it's received an initial 1k of data. Then it makes the first write to the browser & then will update the window on each new reception (so it buffers only the first bit).

So, try the updated code and wait.

See also http://webpy.org/cookbook/streaming_large_files