drewp

Here's how to start from a linux box with python 2.7 and virtualenv, and end up with a running graphite web server and data collector daemon (carbon-cache). I use supervisord and nginx as well, but you don't need exactly those to make the setup work.

mkdir /new/dir

cd /new/dir

Fetch software

virtualenv .

bin/easy_install "gunicorn==0.13.4"
bin/easy_install "Django==1.3.1"
bin/easy_install "graphite_web==0.9.9"
bin/easy_install "carbon==0.9.9"

bin/easy_install "whisper==0.9.9"

bin/easy_install http://django-tagging.googlecode.com/files/django-tagging-0.3.1.tar.gz

Setup configs

cp ./lib/python2.7/site-packages/carbon-0.9.9-py2.7.egg/conf/carbon.conf.example ./carbon.conf
echo "TIME_ZONE = 'America/Los_Angeles'" > local_settings.py
ln -s ../../../../../local_settings.py ./lib/python2.7/site-packages/graphite_web-0.9.9-py2.7.egg/graphite/local_settings.py

mkdir -p storage/log/webapp

Run this once to init a sqlite database

GRAPHITE_STORAGE_DIR=storage/ DJANGO_SETTINGS_MODULE=graphite.settings bin/django-admin.py syncdb

It will say "you don't have any superusers defined. Would you like to create one now?" Say yes and put add minimal information. I'm not sure where this information ever shows up again.

Run these two daemons

GRAPHITE_CONF_DIR=. GRAPHITE_ROOT=. bin/carbon-cache.py --debug start

GRAPHITE_STORAGE_DIR=storage/ bin/gunicorn_django -b 0.0.0.0:9037 ./lib/python2.7/site-packages/graphite_web-0.9.9-py2.7.egg/graphite/settings.py

Here's a config for supervisord that runs those lines:

[program:graphite_carbon_2003]
directory=/opt/graphite/0.9.9
environment=GRAPHITE_CONF_DIR=.,GRAPHITE_ROOT=.
command=/opt/graphite/0.9.9/bin/carbon-cache.py --debug start
# --debug needed for fg mode, but now it logs every datapoint operation
stdout_logfile=/dev/null

[program:graphite_9037]
directory=/opt/graphite/0.9.9
environment=GRAPHITE_STORAGE_DIR=storage/
command=/opt/graphite/0.9.9/bin/gunicorn_django -b 0.0.0.0:9037 ./lib/python2.7/site-packages/graphite_web-0.9.9-py2.7.egg/graphite/settings.py

Serve the web page

Set your web server to proxy http://something.example.com/ to localhost:9037. The graphite pages are full of /root addresses, so it's very hard to serve graphite from anywhere but the top of its own (sub)domain. Serve http://something.example.com/content from the static files at ./lib/python2.7/site-packages/graphite_web-0.9.9-py2.7.egg/webapp/content

Here's an example config for nginx:

server {
  server_name graphite.example.com ;
  location / {
    proxy_pass http://localhost:9037/;
  }
  location /content {
    root /opt/graphite/0.9.9/lib/python2.7/site-packages/graphite_web-0.9.9-py2.7.egg/webapp;
  }

}

Add data

To track the number of files in your home dir over time, run this in a loop or cron:

echo demo.numFiles `ls -1 ~ | wc -l` `date +%s` | nc -q 0 localhost 2003

Galena is a python module that does the same thing.

I wanted pd, mpd, and libao-using programs all to be able to play sound on an EeePc 900A running Ubuntu 11.04.

pulseaudio wasn't working (the device name in pd was garbage bytes, and it would always fail after about a day). Plain alsa wasn't working, since I couldn't get multiple clients playing at once even with dmix. I finally got jack to work, which is pd's preferred solution anyway. Here are the steps that I think were needed:

• When you install jackd2, agree to the realtime priority stuff. Run "sudo dpkg-reconfigure -p high jackd2" if you need to be asked that question again. (original forum post)
• Make sure your user is in the 'audio' group.
• Write this to ~/.asoundrc (alsa docs)

pcm.jackplug {
	type plug
	slave { pcm "jack" }
}

pcm.jack {
	type jack
	playback_ports {
		0 alsa_pcm:playback_1
		1 alsa_pcm:playback_2
	}
	capture_ports {
		0 alsa_pcm:capture_1
		1 alsa_pcm:capture_2
	}
}

# not necessary if all clients can set their device to 'jackplug'
pcm.!default {
   type plug
   slave { pcm "jack" }
}

• Put the lines "default_driver=alsa" and "dev=jackplug" in /etc/libao.conf (libao docs)
• Arrange for "jackd -d alsa" to be running
• Give mpd.conf this pretty output block:

audio_output {
	type		"jack"
	name		"My jack Device"
}

• Test with "aplay -Djackplug some.wav" and "aplay some.wav" (the default device setting)
• Run qjackctl to see what's going on. It's like pavucontrol, for jack. You don't need to press its play button. Open the 'Connect' window to see current clients and connections.
• Run pdextended with -nrt -jack

The Pixxo USB WebCam, model A-MW2130, has a separate infrared filter that is very easy to remove. You just unscrew the outer case, unscrew the lens, and pop out the square filter piece. It looks kind of like the one in this demo.

The AW-M2130 works fine under linux with v4l2. Here are the controls:

% v4lctl -c /dev/video1 list
attribute  | type   | current | default | comment
-----------+--------+---------+---------+-------------------------------------
norm       | choice | (null)  | (null)  |
input      | choice | zc3xx   | zc3xx   | zc3xx
contrast   | int    |     128 |     128 | range is 0 => 256
Gamma      | int    |       4 |       4 | range is 1 => 6
Auto Gain  | bool   | on      | on      |
Light freq | choice | 50 Hz   | 50 Hz   | NoFliker 50 Hz 60 Hz
Sharpness  | int    |       2 |       2 | range is 0 => 3

The camera was $10 at Frys.

I have a headless box that isn't talking out its ethernet port. I wondered if it would network over usb. My Pre is on a NAT wifi network so I couldn't reach it directly.

1. Connect to pre with novaterm

root@palm-webos-device:/# ssh -f -N -n -R 9999:localhost:222 drewp@bigasterisk.com

2. On bigasterisk.com, ssh -p 9999 localhost

3. Unplug pre usb, take it to the computer for testing

4. On the new ssh connection, watch dmesg and ifconfig for action.

It might have been a bit easier to turn off pre wifi and ssh to it over the sprint network, although my connection would have been slower.

In an attempt to fix an nfs slowness, I tried upgrading my home network to nfs v4. This broke procmail and moinmoin.

Procmail fails on nfs4

I debugged the procmail issue. procmail makes new maildir files to write to by setting O_EXCL. A lot of the time, this leads to a file with random permission bits:

-r-----rwx 1 nobody nogroup 0 1970-01-22 09:34 1230111462.11015_0.dash
---------- 1 nobody nogroup 0 1970-01-22 09:35 1230111462.11019_0.dash
-r----xrwx 1 nobody nogroup 0 1970-01-22 09:36 1230111462.11023_0.dash
-r---w--wx 1 nobody nogroup 0 1970-01-22 09:37 1230111462.11027_0.dash

If the perms aren't right, procmail won't be able to write to its own file. For safety, procmail delivers the mail to $MAIL instead and lots this error:

procmail: Couldn't create or rename temp file ".../1230111469.11055_0.dash"

Moinmoin fails on nfs4

In my moinmoin install, logging in would lead to an exception page with no details except a single "None". I ran strace on the moinmoin server process, hoping to see if there were O_EXCL patterns. On my first try with strace, the login worked! Subsequent tries failed, and there were indeed calls like the following:

[pid  3774] open(".../moin/data/cache/__common__/session/g8va5i1tdovz0ej2ewk6mrx213alh90dv0jD4b.tmp", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE|O_NOFOLLOW, 0600) = -1 EACCES (
Permission denied)

So the moral is: don't use nfs v4 if you might have some programs using O_EXCL unless you have fixed this issue.

Testing O_EXCL on nfs4

Here's a test to see if you're getting scrambled perms. Run this with an NFS dir as the cwd:

python -c "import os; [os.open('pytest%s' % x, os.O_WRONLY | os.O_CREAT | os.O_EXCL| os.O_LARGEFILE, 0444) for x in range(10)]"

If all your files are -r--r--r-, you might be fine. My experience is that many of them would have random bits and dates near 1970.

Related

The NFS faq has something related, but it claims the problem is solved in kernel 2.6.5. I'm on 2.6.27.

This post about O_EXCL seems related, although I would expect I already have a patch from June 2007.

If opcontrol or oprof_start says this:

/etc/oprofile/daemonrc: 1: CHOSEN_EVENTS[0]=CPU_CLK_UNHALTED:100000:0:1:1: not found

just remove that daemonrc file. It's from an old run when the config lines were a little different. oprof_start will make a new one with a line like this:

CHOSEN_EVENTS_0=CPU_CLK_UNHALTED:100000:0:1:1

If you get firefox 3.1 and try to run it on a 64-bit ubuntu hardy install, you'll probably get this failure:

./firefox-bin: error while loading shared libraries: libdbus-glib-1.so.2: cannot open shared object file: No such file or directory

1. Install 'ia32-libs' with "sudo aptitude install ia32-libs" (or use the synaptic GUI)
   
2. Go get the i386 version of libdbus-glib and save the .deb file somewhere.
3. Run "file-roller /tmp/libdbus-glib-1-2_0.74-2_i386.deb".
4. Go into data.tar.gz -> . -> usr -> lib, and copy libdbus-glib-1.so.2.1.0 to /usr/lib32 (or right to your firefox 3.1 directory, if you want to avoid writing to /usr).
5. Run 'sudo ldconfig' to make the libdbus-glib-1.so symlink
   

Now you should be able to test firefox 3.1 and play some of the videos from here. The one called 'video embedded in svg' is impressive, although on my system I only got one video at a time and the audio sync drifted. See this screencast (or youtube) for what it should look like.

2009-09-02 Updated with comments from Krellan

I started a followup to my wildly successful Twisted python interface to MPD: a web version. The idea is that if you could GET and POST the various MPD commands, you could write cool music player user interfaces completely in javascript. And you could write lots of them, because they'd be so easy to make. You could make a mac desktop widget, an ipod touch version, and even players for use on non-apple products! The player UI would use AJAX to send commands back to the web server, which interfaces to mpd.

Need to control many mpd instances from one UI? Don't want to run php (like all the heavyweight mpd web clients use)? Like zero page reloads in your web UIs? Want to implement some fine-grained security scheme, where selected users can use selected mpd commands? Want to make mpd calls from unix, but your network only allows http (maybe through a proxy)? It's plain to see that there are between 'five' and 'countless' reasons to use mpd over http.

Progress appears here (because I haven't migrated this project to darcs yet): viewcvs

Someday, I hope to do (or find) the same thing for email. Why can't I write a new special-purpose webmail client every weekend? Why isn't my email box abstracted as a bunch of URLs I can GET (including folders, indexes, searches, etc), and why aren't my contacts represented as a bunch of URLs I can POST new email to?

server1% wc *
...
   8484374   56684310 2050292731 total

server1% du .
2007440	.

server1% du -h .
2.0G	.

I rsynced that dir to another server and did a quick check to make sure I had all the data:

server2% wc *
...
   8484374   56684310 2050292731 total

server2% du .
660872	.

server2% du -h .
646M	.

It took a quite bunch of digging and checking before I remembered server2 has a ZFS disk with compression turned on.

Here's a tiny demo of using the opencv face finder on a PIL image and displaying the result with pygame:

Source code

That's using the SWIG opencv interface (available on ubuntu from the python-opencv package), not the recently-released pycv (which I know nothing about).

My headtrack project is an attempt at something like Johnny Chung Lee's head tracking project. I tried opencv since I don't have a wiimote yet. My goal is to move my desktop windows according to my head position, so I can peek around a window or lean in to look at windows that are outside the normally-visible screen.

Meanwhile, the compiz hackers started a similar project.

I just missed the turning of the unix clock from 1,199,999,999 to 1,200,000,000. It happened last Thursday.

I remember the IRC party for the billennium- lots of people had big ascii art to roll by when their clock said we were going from 999,999,999 to 1,000,000,000 (timezones don't matter, but there are plenty of sync and delay issues on IRC). That was on 2001-09-08, a week which probably won't be remembered for the unix billennium.

1,300,000,000 is in about 3 years, but (aside from the billennium, which could have led to some breakage) these decmial holidays are just for entertainment. The year 2038 is another story, though. Read about the year 2038 bug or 2038bug.com.

I upgraded my laptop from ubuntu feisty to gutsy, and nothing got better. Several things broke. While trying to fix them, I improved my setup a little tiny bit.

The auto installer failed several times on cupsys. Then it had some exception while working on initramfs (!) and gave up. Somewhere during the upgrade, my keyboard broke and was typing nonsense. At least that corrected after the reboot.

On reboot, I seem to be on 7.10 despite the upgrade saying it failed. I reinstalled cupsys and a few other things. I used to use ndiswrapper and linux-wlan-ng for my two wifi adapters, but neither of those work now. The bcm43xx module seems to work, so I can run my worse-quality wifi adapter. linux-wlan-ng claims not to support kernel 2.6.22, so I guess I can't use my usb wifi card for now.

The gutsy version of the power applet started waking up every 30 seconds to tell me I unplugged, when really I was running on AC power and my flaky battery wasn't charging. (It starts charging after a few remove/replugs.)

I tried 'desktop effects' to see if compiz would work. It broke my desktop switching and added window shadows. I restarted X, and lost window borders but gained wobble windows. I never got desktop shadows back. http://ubuntuforums.org/showthread.php?p=3600201 worked for disabling Xgl, which was taking lots of CPU.

emacs wouldn't start. It said "No fonts match 'Monospace-10'". I found various matches for that phrase in google, but didn't see a solution.

Sick of gnome and its WMs, I installed fvwm2 which is what I use on all my desktops. I had it start up with gnome-panel to get the applets I was used to. I thought fvwm had a control for "don't let windows move into this panel's space", but I can't find it now. Anyway, running fvwm instead of metacity is the small improvement I performed today. The suspend button on gnome-panel now quits gnome-panel, so I made a different launcher for "gksudo /etc/acpi/sleep.sh". I'm not sure how to do suspend without requiring a sudo.

Over in fvwm (or maybe after enough X restarts), emacs started working again.

Then I discovered http://www.fvwm.org/doc/unstable/modules/FvwmTabs.html which looks like it might be great if I can stop tripping on its focus bugs.

---

Forgot to mention: the text-mode virtual consoles don't have logins anymore. That's sure to bite me when I'm doing some important demo for someone.

http://www.newt.com/debian/treo650.html is the best page about doing bluetooth hotsync and DUN with a sprint treo 650. The main trap on hotsync is that if you disobey and put in a hostname along with the host ip address, various things will stop working. Use "tcpdump -i ppp0" after ppp connects to debug. Also, treo camera pictures don't backup with "pilot-xfer -b .". I'm not sure how to make those backup, but you probably put them on an SD card anyway.

The DUN section of that page is for t-mobile, and I use sprint. My /etc/chatscripts/sprint is like this:

TIMEOUT 3
ABORT ERROR
ABORT BUSY
ABORT VOICE
ABORT "NO CARRIER"
ABORT "NO DIALTONE"
ABORT "NO DIAL TONE"
ABORT "NO ANSWER"
"" "ATZ"
"" "AT&FH0L3"
OK-AT-OK "ATDP#777"
TIMEOUT 75
CONNECT

The first timeout is small because the first AT command always times out for me, and the connection seems to work fine. On ubuntu, my /etc/ppp/peers/sprint starts with "/dev/rfcomm1".

I use 'pon sprint' to start the dialing instead of ifup.

I'm trying out http://www.plope.com/software/supervisor/ and it looks good so far. Part of my setup will be to have supervisord running on 4+ ubuntu boxes, and there's no ubuntu package (that I'm aware of). I'd like a package since that will make it incredibly easy to upgrade and downgrade the version on all my boxes all at once or to rebuild a box based mostly on the pkg list.

I found http://www.coffeebreaks.org/blogs/?p=65 which relies on http://asic-linux.com.mx/~izto/checkinstall/. checkinstall watches what files get written by a command (e.g. 'make install') and gathers those files into a package. checkinstall has a decent console UI for setting the simple metadata about a package. If I aliased 'sc' to 'sudo checkinstall', it would practically be EASIER to make packages out of everything I install than not to make them. It's ridiculous.

I'm not sure how to provide package dependencies to checkinstall (e.g. supervisor depends on python), so I guess that might require some more debian pkg tool fu. But I don't really need deps to solve my current problem.

My procmail runs every 2 min and gathers an mbox (over nfs) from one box and delivers maildir on another box. The procmail-runner uses lockfile-create/touch/remove to supposedly protect from repeated processing of the same mbox contents, but that lock failed somehow. Cron jobs piled up, and the same message got delivered dozens of times. Every message was going though spambayes, which helped throttle things.

So thanks, fdupes author and ubuntu packagers, for helping me get everything cleaned up in about 10 minutes. I just ran "fdupes -f .path.to.affected.folder/cur | xargs rm" a few times, slowed the procmail down until I feel like looking into that problem, and everything's fine again.

I used to use fslint a lot, but fdupes showed up first in a package search.

I'm also using delicious now (http://del.icio.us/drewpca) since bloglines screwed up and is hiding my hundreds of saved articles. Delicious looks like a better way to gather articles, and they're probably less likely to screw up on their core function than bloglines is to screw up one of its side functions.

Bloglines, if you at least send me the links to all the articles I had marked as keep-new, I'll post an update here saying that you did.

I upgraded from everybuddy to ayttm, but I wanted to bring my patch with me. This patch puts full times (except tzone) onto the display and log files, giving you a much better chance of analyzing them later. Also, I backslash-escape newlines and backslashes, so that the log records aren't ambiguous. This revised patch also includes a fix to a const error in yahoo.c, which I think the proper ubuntu package has its own fix for.

This was my first time working with ubuntu packages, and I learned almost nothing. To apply my patch, run something like this:

apt-get source ayttm
tar xvzf ayttm_0.4.6+26.orig.tar.gz
cd ayttm-0.4.6+26
patch -p 1 -i ../new_ayttm-0.4.6+26_log.diff
dpkg-buildpackage -nc -rfakeroot
cd ..
dpkg -i ayttm_0.4.6+26-1build1_i386.deb

The good logs will look like this:

20060411 23:20:13 drewperttu:  log test 2

Patch is here: http://bigasterisk.com/post/new_ayttm-0.4.6+26_log.diff

I upgraded from everybuddy to ayttm, but I wanted to bring my patch with me. This patch puts full times (except tzone) onto the display and log files, giving you a much better chance of analyzing them later. Also, I backslash-escape newlines and backslashes, so that the log records aren't ambiguous. This revised patch also includes a fix to a const error in yahoo.c, which I think the proper ubuntu package has its own fix for.

This was my first time working with ubuntu packages, and I learned almost nothing. To apply my patch, run something like this:

apt-get source ayttm
tar xvzf ayttm_0.4.6+26.orig.tar.gz
cd ayttm-0.4.6+26
patch -p 1 -i ../new_ayttm-0.4.6+26_log.diff
dpkg-buildpackage -nc -rfakeroot
cd ..
dpkg -i ayttm_0.4.6+26-1build1_i386.deb

The good logs will look like this:

20060411 23:20:13 drewperttu:  log test 2

Patch is here: http://bigasterisk.com/post/new_ayttm-0.4.6+26_log.diff

One host has a dual screen gfx card where one screen is my projector.

I want to send the mouse from my main workstation to the projected screen sometimes, but I don't want to slide it off the edge of my main workstation screen all day long, and I definitely don't want to mix the clipboards of my main workstation and the other screen on the projector's gfx card.

To make synergy not switch at screen edge and use a special key instead, I used this config:

section: options
  switchDoubleTap = 1 # try to disable most switching
  keystroke(F15) = switchInDirection(left)
end

where 'left' on each screen means to jump to the other screen. I couldn't find a way to disable clipboard copying, so I asked for one (which also lists my workaround).

I borrowed a 5.25" disk drive from someone at work to get the data off my very old disks. I put it in an empty computer along with a cdrom drive and booted the debian/bbc cd. trivial-net-setup found the ethernet card.

I made a no-password ssh key (ssh-keyget -t dsa) and appended it to another host's ~/.ssh/authorized_keys2 so I didn't have to type a password everytime I ssh to the other host.

Then, I simply inserted each disk and ran this:

cat /dev/fd1 | ssh drewp@otherhost "cat > disk"date +%s

These disks are from around 1985-1994, and about 4 of 16 failed to read. On a linux box, I mounted the images with the loop device and copied their files to my filesystem.

Some of the interesting files are in gwbasic's compressed form. I probably need to fire up actual gwbasic and use http://www.geocities.com/KindlyRat/BASICtoTEXT.html to get clean versions, unless I can find a decompiler.

I read through the source to strace tonight, and learned the following things:

• the ptrace syscall is what you use to set a child process to stop at every syscall (or at every instruction, which gdb might be doing)

• the strace function groups, such as trace=file ("trace all system calls which take a file name as an argument"), is done by a simple table http://cvs.sourceforge.net/viewcvs.py/strace/strace/linux/syscallent.h?rev=1.38&view=auto as opposed to by analyzing argument types or something

• trace=file outputted more functions than its documentation claims it should; this bug was fixed in 2005/06. I see that "strace -e trace=file echo" outputs some lines without filenames on my home boxes.

• the output is emitted by thousands of 'tprintf' statements through the code. Adding a mode to output as XML, for example, would be about as tough as parsing the current output, which is tough enough.

• there is a -o !pipe_output_to_this_program feature, which I didn't notice before

• a companion program 'strace-graph' attempts to parse the strace output to make a tree of the processes that ran (cool!), but it appears not to be maintained and produces no interesting output on two linux distros I tried