Tuesday, 17 January 2012
Creating a dashboard with Python, MongoDB and Highcharts.
Choices and Decisions
Python -just because we fancied trying something new really.
Having decided upon Python I wanted a really minimal MVC framework, as I felt we had enough new (to us) technologies on our plate, and settled upon Bottle. We set Python up to run in Apache using mod_wsgi, just because we are used to Apache, a more Python dedicated solution may have been better.
How did it all work out?
Pretty well, once we'd got the hang of it. Mongo is fairly friendly, although getting your head around map/reduce takes a little time, and groupby queries are really syntactic sugar on that. You may want to take a look at MongoDB aggregation post -this is easier to use than map/reduce in many cases.
Mongo sells itself as a document store and is schema- less. This is good and bad, good because things don't break when the data structures change, bad because things don't break when the data structures change, so we had feeds dying where the feeding program should have got an exception, but didn't.
Python -is Python, some things, such as class auto loading we missed, other features such as list comprehensions we appreciated.
Bottle I really liked, I set up an MVC structure, all it really supports nativley is routing and views via SimpleTemplateEngine, but creating controller and model directories is easy enough although it's a bit of a bore having to load the classes manually. We ended up with one big template that had as it's arguments the output of partial templates for each widget and in some cases we were abble to extract a data structure from Mongo and pass it as a JSON object to the template without any translation or mapping in tween.
With a system like this the issues are all in the maintenance, we had perhaps half a dozen feeds from different products and teams, and any change in configuration at the product end would have a downstream effect on us. It was rare that all widgets were working at the same time. We did have a large number of unit tests written that could be run to narrow down a problem, but 90% of the time it was a change in another system, JIRA or svn, that would cause the problem.