drewp

I wrote some code in Python because I couldn't figure out the Scala. Later, I got the Scala version working, so now we can compare them. Sorry this is small. Make your browser text bigger, I guess.  

elastic = restkit.Resource("http://localhost:9200/main") class search(object): def GET(self): params = web.input() elasticResponse = json.loads(elastic.get( "doc/_search", payload=json.dumps({ "query" : { "filtered" : { "query" : {"prefix" : {"text" : params['q']}}, "filter" : {"term" : {"source" : params['source']}} } }, "highlight" : { "pre_tags" : [""], "post_tags" : [""], "fields" : {"text" : {"number_of_fragments" : 2}}, } })).body_string()) web.header("Content-Type", "application/json") ret = [] for hit in elasticResponse['hits']['hits']: s = hit['_source'] ret.append(dict(uri=hit['_id'], text=' '.join(hit['highlight']['text']), source=s['source'])) if 'view' in hit['_source']: ret[-1]['view'] = s['view'] return json.dumps(ret)

case class Highlight(text: List[String]) case class Source(source: String, view: Option[String]) case class Hit(_score: Double, _index: String, _id: String, _type: String, _source: Source, highlight: Highlight) case class Hits(hits: List[Hit]) case class HitsResponse(hits: Hits) ... get("/search") { val http = new Http val elasticResponse = parse(http.x(elastic / "_search" <<? Map("source -> compact(render( ("query" -> ("filtered" -> ( ("query" -> ("prefix" -> ("text" -> params("q")))) ~ ("filter" -> ("term" -> ("source" -> params("source")))) ) )) ~ ("highlight" -> ( ("pre_tags" -> List("")) ~ ("post_tags" -> List("")) ~ ("fields" -> ("text" -> ("number_of_fragments" -> 2))) )) ))) as_str)) val extracted = elasticResponse.extract[HitsResponse] compact(render( extracted.hits.hits.map(h => Merge.merge( ("uri" -> h._id) ~ ("text" -> h.highlight.text.mkString(" ")) ~ ("source" -> h._source.source), h._source.view match { case None => JNothing case Some(view) => ("view" -> view) }) ) )) }

In Scala, I'm using lift-json to make and parse JSON. Python is generally better for this since its own data structure syntax is already compatible with JSON. I tried for a while to make the Scala JSON parse code work without the case classes, but that quickly becomes a mess of JObject(List(JField(JValue(JString(... that's almost impossible to traverse. The 'extract' classes would surely be better if I was trying to attach behavior to them. Maybe a Python JSON lib can do the class-based extraction too-- I've never checked around.

The conditional attribute at the end, I didn't know how to write very clearly in Scala. Maybe there's some kind of orElse function that would have made it more readable.

The Scala one runs faster: 11ms average (jetty) vs 45ms average (gunicorn/WSGI). With 4 requests at a time: Scala 15ms vs 47ms. All responses were 3kB.