Data Munging with MongoDB Aggregation and Python

I am evaluating some named entity recognition systems for sorted.jobs , trying to improve our search by sorting the wheat from the chaff, and some of the initial results look encouraging -but just how encouraging? We need to do a bit of analysis to find out.

The most hopeful results come from the extraction of programming language and operating system entities from the text -see the figure below :

Entity Types

On to MongoDB

This table was generated from a MongoDb database using two collections entities and missed_entities entities contains the terms that the program found and missed_entities the ones that I though it missed. Of the ones it found it either got it right ('Hit'), wrong ('Miss') or it was a bit dubious ('Null'). To get the stats I used the new (to me) MongoDB aggregation operations, analagous to the SQL GROUP BY, HAVING, SUM &c.

You could do all this in the old MongoDB map/reduce way, but aggregation seems a bit more intuitive.

So to get the 'Hit', 'Miss' and 'Null' columns the Python code looks like :

entity_c.aggregate([{"$group": {"_id": {"etype" : "$_type", "hit_miss" : "$hit_miss"} , "count": {"$sum": 1}}}])

which returns me rows like :

{u'count': 55, u'_id': {u'etype': u'ProgrammingLanguage'ProgrammingLanguage', u'hit_miss': u'1'}}

{u'count': 2, u'_id': {u'etype': u'ProgrammingLanguage'}}

and nothing for the misses because there weren't any.

The hard work occurs in the $group where I create a compound '_id' field made up of the entity type field and entity hit_miss field and then count all the matching entities.

The Aggregation Pipeline

But we can also look at the terms that the recogniser missed :

 

Missed entities

Here we only want the entities for the given type ('ProgrammingLanguage') and we want them in order, our PyMongo aggregate call now becomes :

missed_c.aggregate([
    {"$match" : {"_type" : etype}},
    {"$group": {"_id": "$name", "count": {"$sum": 1}}},
    {"$sort" : {"count" : -1}}
  ])

We have extra terms : '$match' which filters the documents so we only consider those with the passed in type (etype) and '$sort' which orders by the count we generated in the group. MongoDB pipelines these performing the match then the group and then the sort before returning you the result.

Finally, looking at the results we can see that there are some casing issues, we can make everything the same case by adding in a '$project' :

    { "$project" : { "name":{"$toUpper":"$name"} } },

$project creates a new field (or overwrites an existing one in the pipeline, not the real one in the collection) in this instance I have told it to make all the names uppercase and we get :

Normalised entities

It doesn't matter where in the array you place any of theses terms MongoDB will sort out the ordering.

What does this tell us? Well in this case if I could persuade the tagger to recognise CSS and variants of terms it already knows with a number on the end I would get a big jump in the overall quality of the results.

References

Angular Aggregation manual pages.

Apache Logging to MongoDB Using a Named Pipe

Following on from the last article on remote logging where I collected all the logs into one place, I wanted to be able to query them -so that I can ask questions like "which pages take longer than  5 seconds to produce", 'how many of our redirects fail?" and the like.

I believe that there are already MySQL modules for rsyslogd, but I had written a script in Python to interpret logs and put them into MongoDB, so I wanted to use that.

The obvious solution to join the two programs together seemed to be to used a named pipe, a vaery basic type of interprocess communictation where one program chucks bits down the pipe and the other pulls them out.

Chucking them in is easy enough, firsty create your pipe with mkfifo , then  just change the line is the rsyslog server config from :

:programname, isequal, "apache2" /var/log/oneGiantHeapOfLogs.log
to
:programname, isequal, "apache2" |/tmp/logger_pipe



Reading from the pipe was pretty easy too as this Python snippet shows :

#Open the pipe
in_pipe = open(sys.argv[1], "r")
#Loop forever
while True :
  try:
    line = in_pipe.readline()[:-1]    #Supposedly makes this blocking
    if len(line) == 0 :                       #Happens in test
      time.sleep(5)
      continue


In short, open the pipe, loop forever reading lines. We want this to keep going so the program is wrapped in an exception handler to catch any log parsing errors.

Having got the input to the program it's just a problem or parsing that input -we can take a look at that in another post.

Creating a dashboard with Python, MongoDB and Highcharts.

We were asked to create views onto some of our development and deployment processes so that management and the teams could see how they were doing. We would have to take in and extract data from several different systems, for several different projects and display statistics and trend lines for those systems.

Choices and Decisions

 Some of the systems that we were going to monitor, such as subversion and anthill, weren't query-able and would push data to us, so we would need a data store. The facts that we weren't doing anything transactional (or critical), that the shape of the data may change for a source, and that it could, potentially, get quite large; lead us to MongoDB. There were a couple of other benefits too, the transition from SQL is supposed to be relatively painless, it uses javascript as it's scripting language, and the document structure is in BSON -which the languages drives return to you as JSON (which gives you the possibility of dumping it straight to the web page).


Highcharts -well it's just a damn fine javascript graphing package that's very well documented.

Python -just because we fancied trying something new really.

Having decided upon Python I wanted a really minimal MVC framework, as I felt we had enough new (to us) technologies on our plate, and settled upon Bottle. We set Python up to run in Apache using mod_wsgi, just because we are used to Apache, a more Python dedicated solution may have been better.

How did it all work out? 

Pretty well, once we'd got the hang of it. Mongo is fairly friendly, although getting your head around map/reduce takes a little time, and groupby queries are really syntactic sugar on that. You may want to take a look at MongoDB aggregation post -this is easier to use than map/reduce in many cases.

 Mongo sells itself as a document store and is schema- less. This is good and bad, good because things don't break when the data structures change, bad  because things don't break when the data structures change, so we had feeds dying where the feeding program should have got an exception, but didn't.

Python -is Python, some things, such as class auto loading we missed, other features such as list comprehensions we appreciated.

Bottle I really liked, I set up an MVC structure, all it really supports nativley is routing and views via SimpleTemplateEngine, but creating controller and model directories is easy enough although it's a bit of a bore having to load the classes manually. We ended up with one big template that had as it's arguments the output of partial templates for each widget and in some cases we were abble to extract a data structure from Mongo and pass it as a JSON object to the template without any translation or mapping in tween.

Issues

With a system like this the issues are all in the maintenance, we had perhaps half a dozen feeds from different products and teams, and any change in configuration at the product end would have a downstream effect on us. It was rare that all widgets were working at the same time. We did have a large number of unit tests written that could be run to narrow down a problem, but 90% of the time it was a change in another system, JIRA or svn, that would cause the problem.