I started a new project using jabber and SIOC. It's a jabber bot that records sioc:Post entries in an RDF store.

I also used wokkel, twisted, and rdflib.

The idea is that you make a new bot account and have some users subscribe to it. Anything they say to the bot is logged (the diary part), and the bot will send out nag messages if it hasn't heard from anyone in a while. Each users' posts are broadcast to all the other users.

Kelsi and I tried it out, and I noticed an unintended usage pattern: when I posted something, she followed up with an elaboration. This is natural since she gets my message as a broadcast from the bot, and answering it makes a new post. Someday when I get around to finding or writing a SIOC viewer for this stuff, it would be nice to connect the entries that were made very close together.

Here's what the log looks like in n3:

      
@prefix : <http://rdfs.org/sioc/ns#> .
@prefix XML: <http://www.w3.org/2001/XMLSchema#> .
@prefix dc: <http://purl.org/dc/terms/> .

<http://example.com/forum>     :container_of  [
         a :Post;
         dc:created "2009-09-19T20:59:33.33-07:00"^^XML:dateTime;
         dc:creator "drewp@jabber.bigasterisk.com/Coccinella@dash";
         :content "good evening diary" ],
             [
         a :Post;
         dc:created "2009-09-19T22:18:00.12-07:00"^^XML:dateTime;
         dc:creator "drewp@jabber.bigasterisk.com/Coccinella@dash";
         :content "weather is looking fine" ] .

SIOC says to use sioc:has_creator which points to a sioc:User, but I haven't gotten all those connections done yet. The code also doesn't know what forum URI to use for the bot. The log store is only appended to, but the code currently rewrites the whole thing every time, which is a waste (and a risk!).

I'm thinking about adding a separate mode where each person writes his own local diary with one bot, and there's no broadcasting or sharing of messages between the users. In other words, it would be like you made separate bot accounts bot_user1, bot_user2, etc, except you wouldn't have to actually make all those accounts.

Source code

Something was wrong with the atom feed for one of my newsbruiser instances; and I think the working feed had some problems under bloglines. I'm trying to get off newsbruiser anyway, so I finally wrote an RDF exporter program for it. With an RDF version of the entries, it's really easy to make an atom feed (and hopefully I got it right).

My exporter program imports the actual newsbruiser code, fakes enough startup and config material to get going, and then adds statements about the notebook and entries to an rdflib graph. Man, does newsbruiser spend a ton of code on config and option nonsense.

My blog is currently 963 statements; and Kelsi's blog is 1011 statements. I just store them as NT files. They're about 300kB each.

The atom generator code uses rdflib but not newsbruiser. It's this SPARQL query:

  SELECT ?e WHERE {
          ?blog sioc:links_to ?e .
          ?e td:modified ?mod .
        } ORDER BY desc(?mod) LIMIT 5

with one more pattern if I'm trying to make a feed for a particular category. Then I use lxml's ElementMaker like the slides told me to.

I put up my first SPARQL endpoint, which is a standard way to allow queries about structured data. The dataset is a tiny collection of movie showtimes for the current day and for movies playing near my house.

I built it in response to this plea for more fresh data. My data is not especially exciting, but I did want to have a SPARQL endpoint for that project anyhow. Now I need to make the rest of the project use its own endpoint, to completely decouple the data-gathering code from the display code.

Here is a table of some other endpoints, although I think it's silly to try to maintain a wiki page of such things. That's what automatic search engines are for.

Even better is this recent slideshow that describes several SPARQL queries on interesting real-world data sources.

I went to SHDH at Sandbox Suites last night. Here I am talking with Joel about the signin system:

photo

I am trying to get the SHDH crew to use FOAF and RDF as the way to keep track of guests and their interests. FOAF includes one nice idea which is to treat email addresses sort of like passwords and only store their hashes. I wrote a demo of how to take an email address, hash it, use an RDF search engine to find documents with that hash, and search those documents for a full name to go with the email.

getNameFromEmail code and docs

Then I looked at CruiseControl for a while. It might be cool, but it unfortunately falls in the class of tools where the configuration is so hard, you might be able to use standard unix tools to get most of what you want a lot faster. Nagios is also like this (but munin does ok).

I won 3rd place in the SIOC Data Competition. One of the challenging things was finding the announcement email among spam that made it to my inbox:

Microsoft Windo (3.7K) Winning number................................YM/09788/60
National Lotter (1.7K) Winning Notice !
Breslin, John   (1.7K) Congratulations!  3rd Prize in the boards.ie SIOC Data Competition
admin@national- (8.9K) Final Winning Notification
vinyl907@netvig (2.6K) Email Address Have Won
united nations  (3.5K) PART-PAYMENT VALUED $8.3M UNITED STATES DOLLARS
UEFA LEAGUE PRO (3.2K) END OF THE YEAR AWARD!!!

Here's my entry, which has links to its own source code and the other tools I used.

This is a response to Download SPARQL results directly into a spreadsheet

So far you've motivated seeing the results of a query in a table and making a graph from them. I'd like to have both of those capabilities in a webapp. E.g. I should be able to embed a live graph in my own page like this:

<img src="http://sparqlgrapher.com/svg/example.com/query=SELECT+?date+?price+{...}">

Visiting my hypothetical sparqlgrapher.com directly would give you a UI to layout and customize the graph. When you're done, you'd take that url and embed it elsewhere (or just take a copy of the image, if you want a one-off).

I actually looked at the jena RDF API today, and I was interested to see how graph creation compares to rdflib's style, which is the one I normally use.

From the Jena introduction (minus the model setup and some comments):

String personURI    = "http://somewhere/JohnSmith";
String givenName    = "John";
String familyName   = "Smith";
String fullName     = givenName + " " + familyName;

Resource johnSmith
  = model.createResource(personURI)
         .addProperty(VCARD.FN, fullName)
         .addProperty(VCARD.N,
                      model.createResource()
                           .addProperty(VCARD.Given, givenName)
                           .addProperty(VCARD.Family, familyName));

An rdflib python port of that:

johnSmith    = URIRef("http://somewhere/JohnSmith")
givenName    = "John"
familyName   = "Smith"
fullName     = givenName + " " + familyName

graph.add((johnSmith, VCARD['FN'], Literal(fullName)))
name = BNode()
graph.add((johnSmith, VCARD['N'], name))
graph.add((name, VCARD['Given'], Literal(givenName)))
graph.add((name, VCARD['Family'], Literal(familyName)))

If I were making a new version of the rdflib API, here's what I'd consider:

1. Don't expose the strings of URIRefs very easily. It should be somewhat hard to examine or operate on a URIRef's string value. This is first to encourage good practices (no more "print u.split('/')[-1]" or "if 'foo' in u:") but more importantly to allow for optimizations in query engines. A backend should be able to return URIRefs containing its internal ids while a query is running, and then any URIs that make it to the result set can be looked up (if needed). Actually getting the string form of a URIRef would still be possible of course, but it might require an explicit method call. Most people's URIRef serializations are in __repr__ calls and output formats, so this change shouldn't be that noticable.
2. Literal can still be a string subclass; BNode should not be.
   
3. Like jena, don't require Literal() on strings unless they need a lang or datatype. People forget Literal() sometimes anyway, which rdflib sometimes handles. The rest of the time it corrupts your database. "hello" is pretty clearly the same as Literal("hello"), so I think it's fine to support "hello" the same way that rdflib now supports 5 to be a xsd:int literal. Another choice would be to error quickly on strings, which would be a good move if people were providing URIRefs as strings accidentally.
   
4. Support graph construction APIs with named methods, like graph.node(uri1).addProperty(pred1, "value1"). These are nice for new users since they bring terminology in quickly, and some of the condensed forms seem cool.  I don't like jena's 'createNode' method name, though, since as far as your graph is concerned, nothing got created. The fact that a java Node object was created is not important. Other possibilities:
   
   • graph.get(s).addEdge(p1, o1).addEdge(p2, BNode().addEdge(p3, o3))   # (the BNode makes a little temp graph for a moment)
   • graph.add(s.edge(p1, o1).edge(p2, o2))  # (another temp graph, as we evaluate the inner expression)
   I also obviously prefer 'edge' to 'property', since edges sound more like the free-form graph that we're making. What system would have a property whose value is another property? That's a perfectly natural RDF construct, but pred1.addProperty(pred2, pred3) doesn't look so natural. It's also less surprising that edges can be traversed both ways. Other systems with "properties" don't always support that, causing users to make redundant inverse properties where they think they might need to traverse backwards.
   

I played with a bunch of New Fangled Web Technologies and redid my home page. Almost everything is dynamically derived from data sources that I presumably keep up to date for other reasons. The foaf part and projects list part aren't done yet. I also haven't removed all the zope pages yet, unfortunately. (Zope turned out not to be a good system for making a low-maintenance site that lasts for 10+ years.)

I hope to have a DOAP document for each project, which will make them easy to list on my home page as well as other project-list systems.

I wrote this response to a thread on a mailing list, but I can't find anyplace where sourceforge has my reply online (I did receive it in an email). I would have expected it on this archived thread.

So here it is again, at a place I can link to.

On 21 Apr 2008, at 14:40, robl wrote:

>> SELECT * FROM pages WHERE page_title LIKE "Queen%Elizabeth"
>>
>> This would perform a case insensitive match on Queen(anything) 
>> Elizabeth
>> (at least in mySQL).
>>

...

>> Is there quick way to do what I want ?  Are there any indexes I could
>> apply to improve things (I have already created the indexes  
>> specified at
>> http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/ 
>> kidehen@openlinksw.com's%20BLOG%20%5B127%5D/1298)
>> ?
>>
>> Or do I need to create a conventional SQL table of resource names and
>> then do a SQL LIKE query on those ?
>>

You might also want to check out freebase. Here's the approach I'm about 
to attempt, myself. Start with a reconciliation query:

http://sandbox.freebase.com/dataserver/reconciliation/?name=Queen+Elizabeth&types=%2Fpeople%2Fperson&responseType=html
   - the reconciliation service handles misspellings and other variations
   - s/html/json/ for the machine readable version

Then look at the freebase page or perform a query:

http://www.freebase.com/view/en/elizabeth_ii_of_the_united_kingdom

That page has this link:

http://en.wikipedia.org/wiki/index.html?curid=12153654

On that page, we have

<a href="http://en.wikipedia.org/wiki/Elizabeth_II_of_the_United_Kingdom">article</a>

Maybe freebase can just hand us that link instead of the curid one. I 
haven't gotten to that part of my code yet. I don't know how often the 
last word of the freebase URI is in sync with the WP one, but that seems 
like it would be the least reliable. Following freebase's designated WP 
link is probably more robust.

Finally, take the wiki name, and make a dbpedia URI:

http://dbpedia.org/page/Elizabeth_II_of_the_United_Kingdom



You probably noticed that elizabeth_ii_of_the_united_kingdom wasn't the 
first result for 'Queen Elizabeth' of type /people/person. I'm not sure 
if freebase considers that a bad result page or not. The reconciliation 
service is new, so now's probably a great time to tell them how 
important good results are to you :)

Notes from the talks at Semantic Web: Are Scalable Graph Data Applications Possible?

I was looking forward to more Oracle demos and roadmap-type discussion, but instead the highlight was allegro.

jeff from oracle says:
fraud detection is using graphs
we're generating data faster than we can process it
business value comes from: reduce cost of operations; aid decision-making; improve the transparency of business operations (e.g. for businesses that need to meet regs)
nice slide on DB approaches, broken into disk/ram, native/layered, etc
siderean is a company doing in-mem, multiple machine storage

david from mulgara
key to web scaling is the late binding of address to resource. Allows the information mgmt technique of the web to scale well

the next gen mulgara version, which the team was meeting about this week in SF, will use lots of disk, perhaps 40G ram, and store 100B (?) triples

vertica:
SQL DBMS, focus on analytics
50+ customers: verizon, comcast, level3
came from the cstore project, MIT
MIT library catalog is rdf, 50M triples: Barton Dataset
uniprot protein dataset is 262M triples. vertica serves that dataset for public querying

jans aasman, franz inc:
23 years old company, 2 yr with a triple store
customers do 'event handling and activity recognition'
50 customers, plus free download
monterey aquarium doing Marine Metadata Interoperability Project
los alamos is studying who reads what publications, graph structures in readership
sun doing baetle, the bug tracking one
japan telecom KDDI is doing spam and fraud detection with allegro. they need to determine what is spam across their busy network. they create new spamassassin rules over time.
OFFIS using rdf for info about power grid usage
allegro loads 1e9 quads in 8 hours
has sesame interface
supports xml schema datatypes, e.g. range queries on dates. Literals can be stored as their own numbers
'social network analytics library' for degrees, cliques, group stats
quick loads from oracle for temporary dbs used for analytics (coming soon)
RDFS++ reasoner for the usual inferences
temporal reasoning (allen's temporal logic, for intervals)
their time/space handling helps with event search. one query involves a place and radius, person connections, other event details
police, e.g., need temporal reasoning
"homeland security is interested in every type of imaginable event"
"find all meetings that happened in december within 5 miles of berkeley that was attended by the most important person in Jans' friends and friends of friends"
they have a custom query language for their various datatypes and their capabilities, like (geo-box-around !geoname:Berkeley ?event 5 miles)
even 3 months of american phone call records is already petabytes
jans' thesis was about car driving behavior
GPS (maybe plus phone) leads to rich data about people- work, purchasing, etc

My question, which I didn't get to ask: How do the approaches compare in terms of latency for very small queries? Many of my queries are not batched together well, or my app needs to make a lot of decisions during the graph traversal.

Some goals for a better wiki system:

• HTML markup only! No restructured text, ad-hoc formats, etc. The lock-in is so awful in all of today's wikis. WYSIWYG editors are mature now- users should be able to pick WYSIWYG or full html.
  
• no training or external docs should be required to use all the features
• don't lose my scroll position when I want to edit some text I'm looking at
• encourage linking and creation of RDF data, probably with rdfa
• immediately exploit those links and metadata with easy searching/filtering/etc. Use FOAF data to limit my search to pages written by people I know, etc.
  
• keep the ownership clear (this is the opposite of some wiki philosophies). Spam cleanup should be very simple.
• encourage edits and especially annotations

A common case seems to be "add a new page and list it in some existing TOC section". Another one is "add a new section (paragraph or more) to this page". Editing words within an existing section that you didn't write, that might be rare.

I still like tinymce, although nelix_ isn't a fan.

Wikis that I use (that I'm trying to be better than) are: twiki, zwiki, confluence.

Related: rdf blog engine ideas

Related: Tom Morris - How to build the perfect wiki

Event: http://sdforum.org/index.cfm?fuseaction=Calendar.eventDetail&eventId=13012&nodeID=1

http://cvs.bigasterisk.com/viewcvs/room/sys.dpms?rev=1.1&view=auto

New program to watch whether my screens are powered on or DPMS-sleeping. I also track the idle time and the currently-focused window, since I happened to find code for those while I was working out the DPMS. The result is a little RDF graph:

@prefix _4: <http://dash/>.
@prefix _5: <http://bigasterisk.com/computerIdleState/power/>.
@prefix idle: <http://bigasterisk.com/computerIdleState/>.
 _4:console idle:focusClass "rxvt";
    idle:focusName "XTerm";
    idle:focusWindowName "drewp@dash:/my/proj/room <>";
    idle:lastNonIdle "1196593439.27"^^<http://www.w3.org/2001/XMLSchema#float>;
    idle:power _5:On.

(I know the URLs and date formats are poor right now)

The program should be easy to run if you're on X and you have rdflib, py2.5, and python-xlib.

This results format is part of my new plan to have each program regenerate entire graphs of whatever they measure. I'm thinking of sending the graphs around with jabber, using pubsub to send them only when they change. That would be unlike https://stpeter.im/?p=1328 which uses SPARQL as you'd expect.

For example, if the user (or DPMS timer) turned off the screen, the last triple in my example would change to the _5:Suspend node. Other listeners who have subscribed to the computerIdleState graph will get an updated version of it.

The reason I started tracking screen state was simply to measure how many hours my 300W monitors are on per month. Either this program, or some listener one, will have to log that data somewhere. Of course, there are obvious uses for logging idle time patterns too, and that measurement probably wants a bit more compression. (Example app: tell my friends on jabber that I'm out, but my average return time for Tuesdays is 9:45pm.)

I really have to move this old home automation project off CVS and onto darcs. I don't mind the conversion, but I want to keep at least some of the cvsweb urls working since I think I've pasted them into a lot of postings all over the web.

Updated: fixed FuXi link

I'm trying to do my home automation with RDF and reasoning. RDF is the unified way to write all the configurations, and I'm hoping to use a logic engine (maybe FuXi or Euler) to write the control systems. Hopefully those will make it easy for humans or computers to edit the setup.

I look forward to being able to ask an N3 proof system "why is the porch light on?" and having it tell me "the web said the sun has set by now, you tripped a motion sensor within the last 15 minutes, and there was no other light shining in this area, therefore I turned on the porch light".

Tonight I cobbled together the first working version of some home automation components talking RDF. A bluetooth dongle constantly searches for devices, and if it finds one, it states that [the bluetooth sensor] [senses] [the URI for the device]. Here's that program.

(BTW, avoid bluetooth chips by Integrated System Solution Corp and prefer ones by Cambridge Silicon Radio. The ISSC one I got has the lousy address 11:11:11:11:11:11 that's hard to change since I'm not using windows. Also, this bluetooth intro is really good.)

Next in my home automation system, a reasoning program hears about new statements and executes the right logic to produce more statements about what should happen. This program is a stub for now- it just turns the presence of my phone into a statement to power the door lock. But devices.n3 suggests what some of the logic might eventually look like.

Finally, an output program has been watching for statements about pins on the parallel port it controls. The reasoning program said to put power on bit2, so this program sets the output accordingly. On that pin is a circuit with an optoisolator, a triac, a transformer, and the electric strike that releases the door.

When the real logic is in place, the proof system should be able to say "I unlocked the door because someone friendly was nearby, because Drew is friendly and Drew carries a phone with the bluetooth address I saw".

Here's how to use tabulator to render a simple data table.

My test data might be a bit confusing since the terms overlap with tabulator terms. I'm trying to compare query runtimes of various queries on different databases. The result I'm trying to produce is a table showing how long each database took on each query.

Here's my mockup data in n3:

@prefix : <http://example.org/> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
:result rdfs:label "test result" .
:db rdfs:label "database" .
:time rdfs:label "elapsed time" .
<> :result
[a :Result; :query :q1; :db :rdflibBdb; :time ".5"],
[a :Result; :query :q2; :db :rdflibBdb; :time ".6"],
[a :Result; :query :q3; :db :rdflibBdb; :time ".7"],
[a :Result; :query :q1; :db :db2; :time ".8"],
[a :Result; :query :q2; :db :db2; :time ".9"],
[a :Result; :query :q3; :db :db2; :time ".11"]
.

Note the line which associates the 6 results with this document. Without that link, tabulator won't put the results in its outline.

I used cwm to create an XML version of that data, which you can view that data in tabulator with the following link. [Update: there was no need to convert; tabulator can read n3 thanks to a version of the cwm parser translated to js with pyjs!]

tabulator with mockup data

Tabulator has a query-building interface where you click on predicates and other nodes to constrain your result rows, but I couldn't figure out how to make the table I wanted. Instead, I used the SPARQL tab at the bottom and wrote my own query:

SELECT ?query ?bdb ?db2
WHERE
{
   ?v1 <http://example.org/db> <http://example.org/db2> .
   ?v0 <http://example.org/db> <http://example.org/rdflibBdb> .
   <http://bigasterisk.com/post-rdf/timing-results6.rdf> <http://example.org/result> ?v0 .
   ?v1 <http://example.org/query> ?query .
   ?v0 <http://example.org/time> ?bdb .
   ?v0 <http://example.org/query> ?query .
   ?v1 <http://example.org/time> ?db2 .
}

In english, that says "find queries with results for the two databases, and report their times in columns named after the databases". You can load tabulator with my datafile and that query together:

tabulator with mockup data and query

You have to click the radiobutton next to 'Query' to see the results.

Now I'll actually write my database benchmark, and I'll have it output result sets for each db. I should be able to combine the result sets together and display them in a table with the method described above. The biggest issue with abusing tabulator in this way is that I have to grow my query for each new database I test. Also, that query won't display a row unless it has results from all databases. It would be nice to have all cells optional, so I can still see a row if it only has a result from one database.

I discovered today that Unicode comes with an rdf symbol (almost): ༜

That's &#3868, "TIBETAN SIGN RDEL DKAR GSUM".

Use this if your font doesn't show the character.

Any proposal about allowing RDF literals as subjects, especially one that's for language purposes (in this case it's direction support), needs to address why RDF's current design has the exceptional 'language' attribute on literals. If your proposal is so good, why didn't RDF allow arcs from literals in the first place and avoid the langage/datatype special cases altogether?

I really know nothing about the direction support issue, but if it's one of the last few language-specific issues and it really ought to be separate from the 'langauge' attribute, I am inclined to prefer one more special case on literals than a total redo of the constraints on rdf graphs. My main concern with literals as subjects is that people will treat them like "casual" URIs that aren't universally unique.

http://gasuse.bigasterisk.com now runs the latest gasuse code, including some SVG line graphing. The data is fixed RDF (read from xml). Next comes authentication so I can start adding new records in the field from my cell phone.