drewp

Here's how to start from a linux box with python 2.7 and virtualenv, and end up with a running graphite web server and data collector daemon (carbon-cache). I use supervisord and nginx as well, but you don't need exactly those to make the setup work.

mkdir /new/dir

cd /new/dir

Fetch software

virtualenv .

bin/easy_install "gunicorn==0.13.4"
bin/easy_install "Django==1.3.1"
bin/easy_install "graphite_web==0.9.9"
bin/easy_install "carbon==0.9.9"

bin/easy_install "whisper==0.9.9"

bin/easy_install http://django-tagging.googlecode.com/files/django-tagging-0.3.1.tar.gz

Setup configs

cp ./lib/python2.7/site-packages/carbon-0.9.9-py2.7.egg/conf/carbon.conf.example ./carbon.conf
echo "TIME_ZONE = 'America/Los_Angeles'" > local_settings.py
ln -s ../../../../../local_settings.py ./lib/python2.7/site-packages/graphite_web-0.9.9-py2.7.egg/graphite/local_settings.py

mkdir -p storage/log/webapp

Run this once to init a sqlite database

GRAPHITE_STORAGE_DIR=storage/ DJANGO_SETTINGS_MODULE=graphite.settings bin/django-admin.py syncdb

It will say "you don't have any superusers defined. Would you like to create one now?" Say yes and put add minimal information. I'm not sure where this information ever shows up again.

Run these two daemons

GRAPHITE_CONF_DIR=. GRAPHITE_ROOT=. bin/carbon-cache.py --debug start

GRAPHITE_STORAGE_DIR=storage/ bin/gunicorn_django -b 0.0.0.0:9037 ./lib/python2.7/site-packages/graphite_web-0.9.9-py2.7.egg/graphite/settings.py

Here's a config for supervisord that runs those lines:

[program:graphite_carbon_2003]
directory=/opt/graphite/0.9.9
environment=GRAPHITE_CONF_DIR=.,GRAPHITE_ROOT=.
command=/opt/graphite/0.9.9/bin/carbon-cache.py --debug start
# --debug needed for fg mode, but now it logs every datapoint operation
stdout_logfile=/dev/null

[program:graphite_9037]
directory=/opt/graphite/0.9.9
environment=GRAPHITE_STORAGE_DIR=storage/
command=/opt/graphite/0.9.9/bin/gunicorn_django -b 0.0.0.0:9037 ./lib/python2.7/site-packages/graphite_web-0.9.9-py2.7.egg/graphite/settings.py

Serve the web page

Set your web server to proxy http://something.example.com/ to localhost:9037. The graphite pages are full of /root addresses, so it's very hard to serve graphite from anywhere but the top of its own (sub)domain. Serve http://something.example.com/content from the static files at ./lib/python2.7/site-packages/graphite_web-0.9.9-py2.7.egg/webapp/content

Here's an example config for nginx:

server {
  server_name graphite.example.com ;
  location / {
    proxy_pass http://localhost:9037/;
  }
  location /content {
    root /opt/graphite/0.9.9/lib/python2.7/site-packages/graphite_web-0.9.9-py2.7.egg/webapp;
  }

}

Add data

To track the number of files in your home dir over time, run this in a loop or cron:

echo demo.numFiles `ls -1 ~ | wc -l` `date +%s` | nc -q 0 localhost 2003

Galena is a python module that does the same thing.

checkvist.com is a nice todo list tool, with excellent multi-user and live-update support. Its visual design is fine, but not perfect for leaving your list open in the corner of your screen. There's plenty of whitespace all over and a few rows of menu bar that you'll get tired of looking at (or scrolling past).

Here is a way to get checkvist to be even tinier, so you can leave it open all the time. These instructions are for firefox. While I often use prism for projects like this, I couldn't for checkvist because of the chrome and extension hacks I need.

1. Make a new firefox profile called 'checkvist'. If you already have firefox running, use "firefox -no-remote -ProfileManager" to get the profiles window. You can run your new profile with "firefox -no-remote -P checkvist"

2. In the new profile, use the view menu to turn off the status bar and top bars. Install the Hide Menubar extension. Set your home page to your checkvist list page.

3. Quit firefox. Find your new profile's directory on disk. It'll be something like ~/.mozilla/firefox/p409yj34f.checkvist/. In the chrome/ subdirectory, make a new file called userContent.css:

#toolbar { display: none }
.topbar { display: none }
.main_div { margin: 0 !important }
li.task { padding: 0 !important; margin: 0 !important }
ul.topLevel li { font-size: 10pt }
div.coreDiv { padding: 0 !important; border: 0 !important; }
#h1Id { display: none }
.updateLine { font-size: 0 !important; padding-left: 10px !important;  }

html, body, .main_div { background: #8e8e8e !important }
.taskStatus a { color: #656565 }

(The last two lines are just to turn down the lights)

Now when you run "firefox -no-remote -P checkvist", you should get a really compact window for leaving open all the time. This makes a new incentive to get items completed, too, since an item that stays on-screen too long will burn in :)

I cleaned up my garage this weekend.

Cameras 1 and 2 were on two laptops running cheese set to take multiple pictures with a 60 second wait. I wish the photo count field went higher than 100. What's the design reason to limit the count to something so low? Mysterious. I'd file a bug if the world used distributed bug-trackers where I could just file it on my own machine with my own UI, not make an account and learn your UI. Consider this blog post to be an approximation of the DBTS.

Camera 3 was an old NTSC camera connected to a BT848 card with my own gstreamer/web camera software. I grabbed timelapse with that one with a line of zshell:

forever { curl -o`date "+%Y-%m-%d-%H_%M_%S"`.jpg http://10.1.0.21:9020/garage; sleep 60 }

I got about 2500 pics from each camera, but because of the way I stopped and started the 3 machines, the pics were far from in sync.

This python program makes a new directory of symlinks with a link for every minute after my fixed start time. If there's no frame for that minute, I make a link back to the last existing frame.

import os
from glob import glob

start = 1288461813

for cam in ['bench', 'dining', 'steps']:
    seen = set()
    maxMinute = 0
    for f in glob(cam+"/*.jpg"):
        t = os.path.getmtime(f)
        minutes = (t - start) // 60
        seen.add(minutes)
        maxMinute = max(maxMinute, minutes)
        try:
            os.symlink('../'+f, cam+'.renum/min_%04d.jpg' % minutes)
        except OSError, e:
            print "f=%s minutes=%s" % (f, minutes), e
    
    if 0 not in seen:
        os.symlink('min_%04d.jpg' % (min(seen)), cam+'.renum/min_0000.jpg')
    lastFrame = 0
    for minutes in range(1, maxMinute):
        if minutes not in seen:
            os.symlink('min_%04d.jpg' % (lastFrame), cam+'.renum/min_%04d.jpg' % minutes)
        else:
            lastFrame = minutes

I made mjpeg avi files out of those aligned sequences like this. This is also where I made everything 8 fps.

ffmpeg -r 8 -sameq -i dining.renum/min_%04d.jpg -vcodec mjpeg dining.avi

import Image, time, ImageFont, ImageDraw
start = 1288461813

font = ImageFont.truetype("/usr/share/fonts/truetype/ttf-mgopen/MgOpenCosmeticaBold.ttf", 50)

for minute in range(0, 2155):
    sec = start + minute * 60
    pretty = time.strftime("%A %H:%M", time.localtime(sec))
    print pretty

    img = Image.new("RGB", (400, 120))
    draw = ImageDraw.Draw(img)
    draw.text((0,0), "Garage cleanup", font=font)
    draw.text((0,60), pretty, font=font)
    img.save("time/min_%04d.jpg" % minute)

I had a tool that posted to twitter with a username and password that were saved in a neighboring file. Here is how I ported that setup to oauth. This procedure is not at all friendly to multiple users who would run your tool; it's only suitable if making a user account is something you do roughly once (like you used to do by pasting in the user+password into your config file).

Go to the new app page and make up some details. Application type is 'client'. No URLs matter.

Run this to get two new values:

% ipython
import oauth2, urlparse
consumer = oauth2.Consumer("consumer key goes here", "consumer secret goes here")
client = oauth2.Client(consumer)
resp, content = client.request("https://api.twitter.com/oauth/request_token")
request_token = dict(urlparse.parse_qsl(content))
print "%s?oauth_token=%s" % ("https://api.twitter.com/oauth/authorize", request_token['oauth_token'])

Visit that link, get a PIN.

token = oauth2.Token(request_token['oauth_token'], request_token['oauth_token_secret'])
token.set_verifier("pin goes here")
client = oauth2.Client(consumer, token)
resp, content = client.request("https://api.twitter.com/oauth/access_token", "POST")
access_token = dict(urlparse.parse_qsl(content))
{"oauth_token" : access_token['oauth_token'], "oauth_token_secret" : access_token['oauth_token_secret']}

Save those two values with the consumer key and consumer secret you got from twitter during registration.

All that was the one-off process for getting all the magic cookies. If you now want to talk to twitter from python, import this:

def makeOauthFilter(conf):
    consumer = restkit.util.oauth2.Consumer(conf['oauth_consumer_key'],
                                            conf['oauth_consumer_secret'])
    token = restkit.util.oauth2.Token(conf['oauth_token'],
                                      conf['oauth_token_secret'])
    return restkit.filters.oauth2.OAuthFilter("*", consumer, token,
              restkit.util.oauth2.SignatureMethod_HMAC_SHA1())

oauthFilter = makeOauthFilter(conf)
timelineJs = restkit.request(
     method="GET",
     url="http://api.twitter.com/1/statuses/home_timeline.json",
     filters=[oauthFilter],
     since_id=since_id).body_string()
for status in jsonlib.read(timelineJs, use_float=True):
...