GIGO - Check your data

A pothole?

One of the most important lessons I learnt in computing was Garbage In = Garbage Out, the GIGO law. When I got some less than perfect results from my pothole detector I took a look at the training data.

The data had been taken from a couple of DuckDuckGo searches, being lazy I had used terms from a notebook used to find birds in trees and just changed 'bird' to 'pothole' and 'tree' to 'road surface'. I displayed the first few images of each search and they looked reasonable, but then I took a closer look.

There were quite a few images of sunspots in the search, weird. I looked at my search string, there were three variations : 'pothole', 'pothole in the sun' and 'pothole in the shade'. The last two carried over from the bird search and I had left them in, what harm could it do? In this case it seemed to do quite a lot finding 'holes in the sun'  and also quite a few pictures of awnings, sun shades maybe?

Then there were some images that had come up 'randomly' in the search, like the one above, presumably mislabelled, or maybe on a page about potholes. Of the 140 files downloaded only around 40 were usable.

The road surface query produced much better results, in as much as all the photos were of road surfaces, the issue here was that many of them were variations on a theme.

Driving off into the sunset

This image has many strong features, the white lines grass either side of the road and a skyline, that the algorithm might learn to associate with 'road surface', whereas I just want it to learn about the asphalt, or the lack of. One image like this would be fine, but I felt 25% or more was too many.

To enhance the dataset, and perhaps tailor it to the UK country roads that I cycle around, I got on my bike and took more pictures of potholes and the road surface around them. I combined these with a selection of the downloaded images to get a better training set with around 30 images in each category.

I reran the model training and got a marginally better performance on the figures and a better fit with my validation images.

Further exploration in the next post.

Pothole Detection, without wrecking a wheel.

Is it or isn't it?

 

I decided to build and train a pothole image detector as part of following along with Fast Ai's Practical Deep Learning course. The reason I chose the pothole detector was that at a previous company a similar thing had been built from scratch, completely training it ourselves. Potholes are also something I come across very regularly on the British roads I cycle around. Having come off my bicycle once in the last year I am not in a hurry to repeat the experience.


The thing that immediately struck me was how quickly the model was trained and how easy it was to do. This is mainly because what we are doing is fine tuning an existing image model rather than trying to train one up from nothing, the base model already knows a lot about images before we start. The project where we trained the pothole detector from scratch took at least five weeks to get to something reliable, whereas I have trained something that seems to work in about five minutes. It's not quite a fair comparison as there was always also a need to slice videos into images and extract metadata from them, so there was more scope in the original project.


Fast AI is designed to make the development and training of models quick and easy. It wraps a lot of the complexities of the underlying libraries, and provides utilities that allow you to download images from an internet search. There are data types that are designed to provide categories and labels, test and training data for the model you are training or tuning. I am sure that there is much more in the Fast AI library that I haven't got to yet.


The other big difference between the earlier project, which was several years ago, is that we are able to use free compute and deployment resources to evaluate and tune the model. The earlier project used the boss’s video gaming machine because that had some GPUs in it and he went through Call of Duty withdrawal whilst training was done.


The specific tools we used were Jupyter Notebooks hosted by Kaggle, a frontend generated by Gradio and hosted at Hugging Face and the code basically came from chapter 1 and 2 of the Fast AI course. The end result of that is a public demonstration that you can see here:




So how did it do? I have only tried it out on one or two images downloaded from the internet and it performs well on those, but they might well be in the training set. I have tried it on one photograph of a pothole that I took and it got that wrong, possibly because it was more of a pot crack than a pothole. I'm planning to take more photographs and carry out more of a formal evaluation of how the model is doing, and maybe tune it again on images that are more relevant to the roads that I cycle along. Fast AI seems to provide some tools and utilities to do this, so look out for another exciting instalment.

Chat vs Search

 I had to add a couple of features to the UI on the MIR infrastructure router. Not really being a UI person I could either look them up, or I could ask ChatGPT. In the end I did both, how did it pan out?

Colour Picker

The first thing I wanted to add was a colour picker. MIR now has a lot of data layers for the UK and it is helpful to be able set the colours for these, the plan was to trigger a colour picker by clicking on a randomly coloured block that the layer has as a default.

I had a few goes at this but didn't get anything satisfactory here's an example below:

system_message = "You are an expert HTML, Javascipt and CSS coder." user_message = """ Create a colour picker to change the colour of an element in a web page. """

In some ways this gave me what I asked for, HTML, CSS and JS that would, probably, change the colour of an element. Unfortunately the colour picker is an HTML form field that you have to type a CSS RGB string into rather than an actual picker.

In other attempts I managed to get a basic JS colour picker working that consisted of three sliders that would change the RGB  parameters of a colour, again setting the colour of the element when you click the button. It potentially works but isn't very interactive.

This still wasn't ideal, so I tried a Google search for a JavaScript colour picker which led me to a site with a

collection of these and out of those I chose one called Swatchy.  This is where search truly wins over chat,

because you can actually see the colour pickers in action and choose one with functionality that you like. 

Modal

The other thing I wanted to add was a help modal, things have got a bit more complicated and help where you need it seemed to be a good idea.

Again I tried ChatGPT first these are the prompts that I came up with :

system_message = "You are an expert HTML, Bootstrap 5,Javascript and CSS coder."

user_message = """

Create a box to display some help text. It should open when a button with a help icon is clicked and close when a close button is clicked.

"""

This produced what looked like workable HTML, JavaScript and CSS to create a popup box, which I guess is what one would expect, but as I am using Bootstrap I don't need.

Altering the text to ‘create a modal’ did better, in that it just produced HTML and CSS to hide and show the box. It did also make use  of the ‘data-bs-dismiss’ Bootstrap attribute to dismiss  the modal, however the CSS was unnecessary as this is already included in Bootstrap.

 A quick search for ‘bootstrap 5 modal’ took me to the Bootstrap documentation and gave me a small amount of code that I could cut and paste into my site.  2-0 to search.

TL;DR

Search isn't dead yet, especially for anything visual. Search can give you options, writing good prompts is hard.

First try at developing a website with ChatGPT

Like everyone and his brother, I have been getting interested in chatGPT and Large Language Models in

general. A lot of development for a website can be pretty repetitive from one site to another and so I wanted

to see if chat GPT could make me more productive and allow me to concentrate on the more interesting

parts of the project.

Here at Msasa a pilot project came along where we wanted  to develop a website to show a client,

even though it may never be used. This seemed like an ideal use case for trying out  chatGPT. The website

we ended up with is called  MineralMarketplace, Take a look and see what we ended up with.

In preparation I took one of DeepLearning's short courses on prompt engineering for developers. If you haven't used chatGPT, or another LLM, then I recommend this, or something like it, to get you started.

The Experimant

Firstly I created a database schema for the site, using PGAdmin in my case.  I could have tried to construct prompts to get chatGPT to do this, which I'm sure it could, but it seemed as though it would be shorter and quicker to do it myself. It also gave me structured language to pass to the LLM from which it could generate other parts of the website.  If you prefer I'm sure you could do this either by creating objects or an Open API spec and then get chatGPT to generate a database scheme from that.

The next thing to try was to throwing the schema at chatGPT, and ask it to generate the CRUD code functions in Python to populate and query it. The good news was that chatGPT understood the schema,  the bad news was that it blew out the token limit  and couldn't generate much code. To solve this I asked it to generate CRUD code for each table in turn and that worked well. Following that I then asked it to generate me a flask web server and the end points for an API that would call the CRUD code and an Open API 3 spec for that code,  again these both worked well,  although I haven't evaluated the Swagger code to check for hallucinations. 

Since I am a relatively basic front end developer and this was just a demonstration site, I decided that I

would build the front end using Flask’s Jinja templating and Bootstrap. I found that developing the Bootstrap

templates didn't work very well and in the end I gave up and downloaded a theme from an open source

Bootstrap template site. What did work was generating the forms using the LLM, although the country

dropdown again exceeded the token limit.

The token limit is, more or less, the maximum amount of text you can pass into, and get back from,

the LLM.

My final attempt using AI for the site was to try image generators to generate the background images for the various minerals,  this didn't work at all and I gave up pretty quickly and search for free images that I could use.

Would I do it again?

I would, the LLM generated a lot of boilerplate for me that just worked, and I now have a set of prompts that I

could use to do the same thing again.

The fact that it failed to create the front end wasn’t actually a huge drawback, I am not a designer so it makes a lot of sense to tweak an existing theme rather than coming up with something from scratch. The MineralMarketplace theme was originally a property website. 

Example Prompt

Given the Postgres SQL definition defined inside <sql></sql>

For Python 3 and the psycopg2 library:

 

step 1: For each table defined by a 'CREATE TABLE' statement in the SQL definition create an Insert

function that inserts all non serial fields in the 'CREATE TABLE' statement and returns the value of  

the serial field. Also create Read and Delete functions using the serial field name as the key.  

The function names should include the name of the table.

step 2: Using Python Flask create an api for the functions defined in step 1

step 3: Create an OpenApi v3 specification for the api defined in step 2

SQL definition: <sql>

{sql}

</sql>

5 reasons why we are happy with AngularJS

I promised a general post about our experiences on sorted.jobs with AngularJS, so here's the view from ten thousand feet, rather than the 50 foot view of other posts.

1. Good for non-designers

In some ways this is more down to Bootstrap, but combine AngularJS, Bootstrap 3 and AngularUI you have a quick and easy way to produce a responsive and dynamic website.

2. Good for non programming web designers

Angular hides most of the Javascript so it makes for easy collaboration between a front end programmer and a designer. Angularjs directives mean that some of the clever bits can be presented to the designer as new HTML tags, which he can manipulate in the same way as other tags.

3. Very clear view seperation

Views live in templates, which are just HTML files, controllers  live in javascript controller files and never the twain shall meet. The actual controller function gets specified in an attribute to a <div> or other HTML tag -what could be easier (don't mention $scope)?

4. Decent documentation and community

The AngularJS site is pretty good, with a good tutorial, developer guide, and reference. Stackexchange is very active, Egghead.io is a good source of video tutorials although there are many others.  Manning has a couple of MEAP books; I have used AngularJS in Action, which is now nearly completed, the other is  AngularJS in Depth. Packt has a whole raft of AngularJS titles, I have used Mastering Web Application Development with AngularJS, I would guess the Manning books will be more up to date than the Packt one at the moment as they aren't formally published yet.

5. Testing tool.

Angular provides Protactor for end to end testing, this uses Selenium Webdriver to run Jasmine tests via standard web browsers. It is aware of Angular directives so you can run tests on Angular loops etc. It works well and the devlopers have been helpful when I found a problem.

That'll do for now, there's loads more and AngularJS has it's quirks, but I need coffee so that's another post.

Sorting and paging with AngularJS

We have two dashboard screens that make heavy use of AngularJS on sorted.jobs, one for candidates and one for recruiters. The recruiter screen in particular can end up with an awful lot of jobs on it, so the list needs to page, in addition it would be good to be able to sort by posting date, number of applicants, company name and so on.

Recruiter Dashboard

This is where we have got up to, not too happy about the UX/UI side, but it's workable. The paging is provided by the excellent AngularUI Booststrap library. Using it is pretty simple this is the AngularJS template directive ;

        <pagination total-items="active_jobs.total" ng-model="currentActivePage" ng-change="activePageChanged()"></pagination>

Pagination needs to know the size of the array it is paging over (active_jobs.total), has a variable to store the current page in (currentActivePage) and a function to call when you change the page - activePageChanged().

And here's the controller function :

  $scope.activePageChanged = function(i) {
    $scope.activePageStart = ($scope.currentActivePage -1) * $scope.itemsPerPage;
    $scope.activePageEnd = $scope.activePageStart + 10 ;
  }

As you can see all we are doing is changing the start end end points of the items we are viewing in the array; we have been a bit lazy and not passed itemsPerPage to the directive as we are using the default of ten.

The ng-repeat call looks like this :

      <div class='row' ng-repeat="job in active_jobs.hits.slice(activePageStart,activePageEnd)">

I did see pages offering a range filter but  array.slice() seems more direct.

In this example the whole array is passed from the backend to the front in one go (this is so it can be sorted in the browser), but you don't have to do it that way, other pages we have make a call to the back  end from within the activePageChanged() function  to get the next page of results.

Sorting

AngularJS provides the orderBy filter that will sort an array, the documentation is pretty good, the only real point to pick up is that to use it in our pagination example we need to call the controller version and not use the template filter version (this would only sort the array slice and not the whole array). So in the template we make a function call :

<p><a href="" ng-click="reverse=!reverse;active_order('_source.reference', reverse)">Reference<span class="glyphicon glyphicon-resize-vertical sort-col-header"></a></p>

We call the active order function with the column we want to sort by and the direction of sort. The  reverse=!reverse just twiddles the sort order.

We then need to set up the controller to use orderBy by injecting $filter :

function SomeCtrl ($scope, $location, $filter...

and using a bit of syntactic suger :

var orderBy = $filter('orderBy');

it is just a matter of defining the function :
           
  $scope.active_order = function(col, reverse) {
    $scope.active_jobs['hits'] = orderBy($scope.active_jobs['hits'], col, reverse);
  };        

and Robert is your father's brother.
 

Data Munging with MongoDB Aggregation and Python

I am evaluating some named entity recognition systems for sorted.jobs , trying to improve our search by sorting the wheat from the chaff, and some of the initial results look encouraging -but just how encouraging? We need to do a bit of analysis to find out.

The most hopeful results come from the extraction of programming language and operating system entities from the text -see the figure below :

Entity Types

On to MongoDB

This table was generated from a MongoDb database using two collections entities and missed_entities entities contains the terms that the program found and missed_entities the ones that I though it missed. Of the ones it found it either got it right ('Hit'), wrong ('Miss') or it was a bit dubious ('Null'). To get the stats I used the new (to me) MongoDB aggregation operations, analagous to the SQL GROUP BY, HAVING, SUM &c.

You could do all this in the old MongoDB map/reduce way, but aggregation seems a bit more intuitive.

So to get the 'Hit', 'Miss' and 'Null' columns the Python code looks like :

entity_c.aggregate([{"$group": {"_id": {"etype" : "$_type", "hit_miss" : "$hit_miss"} , "count": {"$sum": 1}}}])

which returns me rows like :

{u'count': 55, u'_id': {u'etype': u'ProgrammingLanguage'ProgrammingLanguage', u'hit_miss': u'1'}}

{u'count': 2, u'_id': {u'etype': u'ProgrammingLanguage'}}

and nothing for the misses because there weren't any.

The hard work occurs in the $group where I create a compound '_id' field made up of the entity type field and entity hit_miss field and then count all the matching entities.

The Aggregation Pipeline

But we can also look at the terms that the recogniser missed :

 

Missed entities

Here we only want the entities for the given type ('ProgrammingLanguage') and we want them in order, our PyMongo aggregate call now becomes :

missed_c.aggregate([
    {"$match" : {"_type" : etype}},
    {"$group": {"_id": "$name", "count": {"$sum": 1}}},
    {"$sort" : {"count" : -1}}
  ])

We have extra terms : '$match' which filters the documents so we only consider those with the passed in type (etype) and '$sort' which orders by the count we generated in the group. MongoDB pipelines these performing the match then the group and then the sort before returning you the result.

Finally, looking at the results we can see that there are some casing issues, we can make everything the same case by adding in a '$project' :

    { "$project" : { "name":{"$toUpper":"$name"} } },

$project creates a new field (or overwrites an existing one in the pipeline, not the real one in the collection) in this instance I have told it to make all the names uppercase and we get :

Normalised entities

It doesn't matter where in the array you place any of theses terms MongoDB will sort out the ordering.

What does this tell us? Well in this case if I could persuade the tagger to recognise CSS and variants of terms it already knows with a number on the end I would get a big jump in the overall quality of the results.

References

Angular Aggregation manual pages.

AngularJS ng-if and ng-show

We use AngularJS to do a lot of the front end on sorted.jobs, at some point I may do some posts on the pros and cons of AngularJS, but for now it's enough to say that -in conjunction with Bootstrap it makes producing a front end pretty easy for a back end developer like me.

Anyhoo one of the joys of Angular is how easy it is to produce a display that varies depending on the data provided to it (even dynamically -although that's not the subject of this post). When we started sorted.jobs the way to do this was via ng-show and ng-hide, which basically twiddles the CSS display property for the element to show or hide it. This works very well, but the element would always be fetched from the server, it just wouldn't appear on the screen.

Now we have ng-if, this will remove the element from the DOM altogether if the expression evaluates to false, and so, assuming it is some some of resource it won't fetch it from the server, thus speeding up the page.

Quick Example

ng-show :
<img ng-show="job.user_id && !job.system.logo_url" ng-src="/logo/{{job.user_id}}/{{job_id}}" class="img-responsive">

ng-if :
<img ng-if="job.user_id && !job.system.logo_url" ng-src="/logo/{{job.user_id}}/{{job_id}}" class="img-responsive">

The ng-show in this case has a particularly bad effect in that it will load, although hide, an image that isn't there, i.e. one with no user_id!

References :

ng-if in Angular Docs
Stackexchange -you may want to read this if unexpected things start happening with ng-if, like some other Angular commands it has a habit of creating child scopes.

AngularJS, Google Search and SEO

Our new site sorted.jobs is now in a stealthy, pre-launch mode, so it is time to start thinking about getting it into the various search engines -especially the big one.

AJAX

Our Job Posts are normally served as AngularJS views -which Google can't parse, since they are AJAX based (although interestingly it can render them with Google Fetch) . However, Google can be persuaded to fetch another version of the page and index that by including this meta tag in the head :

<meta name="fragment" content="!">

If the crawler sees this tag it will then resend the original request with ?_escaped_fragment_= tacked on the end (more here : https://prerender.io/js-seo/angularjs-seo-get-your-site-indexed-and-to-the-top-of-the-search-results/)

so
http://www.sorted.jobs/job_post/UX+Developer/ULKYpZTSSRqBGvGirp_yOQ 
would become
http://www.sorted.jobs/job_post/UX+Developer/ULKYpZTSSRqBGvGirp_yOQ?_escaped_fragment_=

on the server side in the handler I recognise the second form and render a page that is HTML only.

SEO

Since the user doesn't see this page  it doesn't need the full functionality of the original  and we also have the chance to tweak the page to do things like providing a more meaningful <title> and a <meta name=description> tag which Google can do things with as per this snippet post

 Testing and Webmaster Tools

One minor 'gotcha' that we found was with Webmaster tools, in as much as the stats aren't up to date -according to tools we have 0 pages indexed wheres a site specific google search (site:sorted.jobs) shows us  the ten pages we expect.

A second issue is that the 'Fetch' functionality within Webmaster Tools doesn't fire off the second request automatically -so you can't see the page Google would actually index from sorted.jobs on the Fetch results page, just the originally requested AngularJS page.

On the plus side if you 'Fetch' the page you can check that the <meta name="fragment" content="!"> is in the content, and if it is you can then 'Submit to Index' which will kick off Google's crawler on your page and put it into their index within minutes.

TDD - WTF?

Test Driven Development -good or bad? A friend went for an interview recently with a leading agile consultancy, and their mantra was 'we don't want documentation -we just want TDD'. I was mulling this over and I don't agree.

'Pull the switch' -but will it work?

Documentation -all bad?

There are two problems with documentation :

• It's always out of date.
• No one reads it.

Too much documentation is a bad thing -I once worked on a project where we had to document everything in pseudo code at the head of each file. It was disastrous, code reviews were done on the pseudo code, rather than the real thing, it was always out of date, and you wrote the same thing twice -but differently because you ran into language features that forced you away from the pseudo code. In an extreme case I thought of a better way of doing things and implemented that rather than the pseudo code, the boss wasn't happy, but it did stay in production for at least a decade. The pseudo code also  didn't stop people doing stupid things, like implementing their own version of malloc() -which leaked like a sieve.

On the other hand I worked on another system with no documentation at all, even the output was in Dutch, a language I don't understand at all. I didn't need tests to tell me it was failing -it broke completely, pretty much everywhere; but I could have done with something telling me what it was trying to achieve, especially since there was the possibility of a big bang if it all went wrong.

On the other hand good documentation addresses the 'why' rather than the 'what' and 'how' of the code. It should tell us what a function is for or why we chose a non-obvious way to implement something, or highlight a kludge.

Oh yes, RTFM -you know it makes sense.

Testing -all good?

Well yes, generally, although not without cost, test suites have to be developed and maintained and sod's law says that the one thing that breaks there won't be a test for, but that is when you create regression tests. So unit tests, acceptance tests, regression tests you need them all.

If there is a downside it is an excessive focus on test results. Development teams can have a habit of thinking that their job is done when all the tests pass; clients tend to get annoyed if the system still isn't doing what they think it should.  They can also fix failing tests without regard to the rest of the system, 'ok test 1234 went green, but now everything is 50% slower? You think that's acceptable?'

Test Drive Development then?

I think it can work as part of a development process, but it really only addresses the 'what'; as in, 'what is this code doing right now', passing or failing? It says nothing about why the bit of code is there or even how it works and how well it works, in the language of my youth -it's  very 'bottom up', an approach that was being maligned with good reason thirty years ago.

If you fit it into a larger process -say Agile Modeling or BDD, where you have ended up with user stories, tasks and all the rest, then TDD is a valid way to go, although personally I don't think it's any better than writing code and creating unit tests afterwards. I suppose the advantage here would be that with TDD you would create the tests, otherwise you may not, as with documentation there's always something more interesting to do, but it still seems to me to encourage looking at trees rather than the whole wood.

Would I use TDD?

I have given it a go on a couple of small projects, but in the end I fell back to coding first and writing the tests second, it just seems a much more natural way of doing things. I may refactor a lot at times, and this would mean rewriting many of the tests as well. I guess I like to get a prototype working and the go back and revise towards an optimal solution, at which point I'd put in the tests. How about you?

Analysing an Online Community

I had a look at carrying out network analysis of the relations in an online community for the Coursera Social Network Analysis class.

Original Bulletin Board Graph

The community was a UK based bulletin board of general chat, completely anonymised. I wanted to look at the relationship between topics and see what could be inferred from that, so in the graphs the nodes are threads and the edges are people who posted between threads. This post is mainly about the network analysis, another will look at nitty gritty of how I carried it out.

Tools

In short, Python and Gephi -more details later.

The Network

I ended up with a network of 79 nodes and 345 edges. The node labels were extracted keywords from the threads -edges were people who posted between threads.

Analysis

First question is there anything to look at? Any Structure? One way to tell is by constructing a random graph and comparing it's properties with those of our graph -this can be done, a little tediously, with Gephi.

Our graph has an average degree of 8.73, a shortest path length of 2.22 and an average clustering coefficient of 0.634. In comparison a random graph with same node and edge counts has a degree of ~4.4 a path length of 2.2 and the clustering coefficeient of 0.144 degree is considerably lower than our graph, so nodes are more connected and, although the path length is similar to a random graph the clustering is higher, indicating small world properties -which gives us something to look at.

There are a 3 super nodes in the graph :

Node	Betweenness
clarkson, people, one, bbc	1015
people, like, money, would	650
clarkson, one, public, sector	343.

People in the UK can probably guess the reason for the first and third nodes. The comment was so outrageous that perhaps most people felt that they had to say something about it, and abnormal relationships in the graph could be created. After removal of the clarkson nodes the graph is as below:

Clarkson free graph

Running Gephi Modularity with a resolution of 1.0 gives us 6 groups. Groups 4, 5 and 0 seem to be too diverse to say anything much about. Group 1 is mainly about sport, although its' highest degree node is about Apple. Group 3 is about UK domestic topics, Group 2 is more global centring around politics and economics.

Summary

We can see that there is some clustering of subjects when we relate them by people although we have not discovered any surprises on this small data set, as similar subjects cluster together. Next steps for this study would be too use a more sophisticated way of extracting the labels from the thread text, run against a larger data set and perhaps introduce the notion of time as an attribute, one could then perhaps look at 'contagion' of the network by a topic, the Clarkson posts might be interesting here. We could also invert the network and relate people by posts -but that seemed less interesting.

Fast Track to Mobile -Part Deux

This week we run up a couple of quick mobile pages and publish them on dropbox -not without it's idiosyncrasies- and install ADT the Android Development Toolkit.

A note on pronunciation -the dog jokes should be read in the voice of our late, lamented court jester -Spike Milligan.

Sencha App on Playbook

First off Sencha, I just copied the test page, updated the text and dropped it onto dropbox. The Sencha page looks reasonable to me -but it's just a test page. Looking at the source, it's a bit less friendly than jQuery Mobile for something this simple which implies there might be a slighly steeper learning curve. Time will tell.

jQuery Mobile page on Playbook

The jQuery Mobile page  also looks good, again it was just copied from the quick start. The code looks a lot more like a normal web page with the contents driven through named div blocks -view source on the pages and you'll see what I mean.

Dropbox had issues, or rather Chrome did, with using either 3rd party sources or mixing http & https requests -these were resolved by copying the css and javascript files into the Dropbox public folder so that they could be served from there.

Quick Delivery for Mobile

I have signed up for my friend Sam's Mobile Development Class, in the past I have built and run backends for mobile sites but have never had too much to do with the device side and his course seemed like a good chance to fill in the knowledge pothole. One of the assignments for week 1 was to write a 'contractual obligation' blog post -which is why you're suffering this; a second, more interesting, assignment was to watch this mobile dev introduction on Udemy.

Although it was for managers, shouldn't scoff I am one (sometimes),the set of lectures gave a good overview of some of the options for delivering mobile sites and apps.

Back in the day I was Head of Development for a company called Cantos, and made a now defunct mobile site for them, it broke when we put up the current web site , we did do a second version for the new Drupal site but it was never given the green light to go live, which is why I don't give a link to it.

When we were developing the mobile sites we did a lot of googling around looking for tools, guides and advice and I wish the Udemy video and the options it mentions were around then -we might have saved a lot of time and we would have been much more future proofed.

Tools like Cordova / PhoneGap offer a quick, cross-platform, entry to the market and Titanium a short cut to the native app experience. For the sites we built jQuery Mobile would have allowed us to offer a richer user experience to our customers. The video lectures discuss the pros and cons of each technology and makes suggestions about what to use when, including when you should go fully native. All in all 50 minutes well spent and I look forward to playing with all of them.

The power of the Matrix

Do you want to make your code run 100 times faster, at least in python? Try rewriting it in matrix form using numpy.

I came across this when implementing the Floyd Warshall Algorithm for finding all shortest paths in a graph as part of th Algorithms 2 course at Coursera. The guts of Floyd Warshall are 3 nested for loops that iterate over all the nodes, here's the Python code :


# v_sz = number of vertices
for k in range(1,v_sz) :
  for i in range(0,v_sz) :
    for j in range(0,v_sz) :
      new_len = A[k-1][i][k] + A[k-1][k][j]
      old_len = A[k-1][i][j]
      if new_len < old_len :
        A[k][i][j] = new_len
      else :
        A[k][i][j] = old_len


This takes around 30 minutes to run on my PC for the supplied graph of 1000 vertices.

Other people were getting much better times and the hint was given to try rewriting this in matrix form -took me a while but I had used Octave as part of the Machine Learning and Neural Nets courses, and together with the Numpy for Matlab Users page I got to :

for k in range(1,v_sz) :
  #tile() is the equivalent to repmat -copies a vector repeatedly into a matrix
  i2k = np.tile(M[0:v_sz][:,k],(1,v_sz))
  k2j = np.tile(M[k][:],(v_sz,1))
  M = np.minimum(M, i2k + k2j)

This runs in ~10 seconds, a good chunk of which will be loading in the data! Pretty impressive and there would be more to go as I think I can shave time off the data load.

What it shows is that numpy is heavily optimised, probably through use of underlying C libraries, for matrix computation and that it is always going to be worth looking to see if a set of for loops can be recast into vector and matrix manipulations instead.

Apache Logging to MongoDB Using a Named Pipe

Following on from the last article on remote logging where I collected all the logs into one place, I wanted to be able to query them -so that I can ask questions like "which pages take longer than  5 seconds to produce", 'how many of our redirects fail?" and the like.

I believe that there are already MySQL modules for rsyslogd, but I had written a script in Python to interpret logs and put them into MongoDB, so I wanted to use that.

The obvious solution to join the two programs together seemed to be to used a named pipe, a vaery basic type of interprocess communictation where one program chucks bits down the pipe and the other pulls them out.

Chucking them in is easy enough, firsty create your pipe with mkfifo , then  just change the line is the rsyslog server config from :

:programname, isequal, "apache2" /var/log/oneGiantHeapOfLogs.log
to
:programname, isequal, "apache2" |/tmp/logger_pipe



Reading from the pipe was pretty easy too as this Python snippet shows :

#Open the pipe
in_pipe = open(sys.argv[1], "r")
#Loop forever
while True :
  try:
    line = in_pipe.readline()[:-1]    #Supposedly makes this blocking
    if len(line) == 0 :                       #Happens in test
      time.sleep(5)
      continue


In short, open the pipe, loop forever reading lines. We want this to keep going so the program is wrapped in an exception handler to catch any log parsing errors.

Having got the input to the program it's just a problem or parsing that input -we can take a look at that in another post.

Creating a dashboard with Python, MongoDB and Highcharts.

We were asked to create views onto some of our development and deployment processes so that management and the teams could see how they were doing. We would have to take in and extract data from several different systems, for several different projects and display statistics and trend lines for those systems.

Choices and Decisions

 Some of the systems that we were going to monitor, such as subversion and anthill, weren't query-able and would push data to us, so we would need a data store. The facts that we weren't doing anything transactional (or critical), that the shape of the data may change for a source, and that it could, potentially, get quite large; lead us to MongoDB. There were a couple of other benefits too, the transition from SQL is supposed to be relatively painless, it uses javascript as it's scripting language, and the document structure is in BSON -which the languages drives return to you as JSON (which gives you the possibility of dumping it straight to the web page).


Highcharts -well it's just a damn fine javascript graphing package that's very well documented.

Python -just because we fancied trying something new really.

Having decided upon Python I wanted a really minimal MVC framework, as I felt we had enough new (to us) technologies on our plate, and settled upon Bottle. We set Python up to run in Apache using mod_wsgi, just because we are used to Apache, a more Python dedicated solution may have been better.

How did it all work out? 

Pretty well, once we'd got the hang of it. Mongo is fairly friendly, although getting your head around map/reduce takes a little time, and groupby queries are really syntactic sugar on that. You may want to take a look at MongoDB aggregation post -this is easier to use than map/reduce in many cases.

 Mongo sells itself as a document store and is schema- less. This is good and bad, good because things don't break when the data structures change, bad  because things don't break when the data structures change, so we had feeds dying where the feeding program should have got an exception, but didn't.

Python -is Python, some things, such as class auto loading we missed, other features such as list comprehensions we appreciated.

Bottle I really liked, I set up an MVC structure, all it really supports nativley is routing and views via SimpleTemplateEngine, but creating controller and model directories is easy enough although it's a bit of a bore having to load the classes manually. We ended up with one big template that had as it's arguments the output of partial templates for each widget and in some cases we were abble to extract a data structure from Mongo and pass it as a JSON object to the template without any translation or mapping in tween.

Issues

With a system like this the issues are all in the maintenance, we had perhaps half a dozen feeds from different products and teams, and any change in configuration at the product end would have a downstream effect on us. It was rare that all widgets were working at the same time. We did have a large number of unit tests written that could be run to narrow down a problem, but 90% of the time it was a change in another system, JIRA or svn, that would cause the problem.

Probablistic Deshredder -in Python

The second exercise in the ai class was to recompose a shredded message, this is a bit harder than the Caesar Cypher -although still do-able by Mk1 eyeball. Here's the message :

de|  | f|Cl|nf|ed|au| i|ti|  |ma|ha|or|nn|ou| S|on|nd|on
ry|  |is|th|is| b|eo|as|  |  |f |wh| o|ic| t|, |  |he|h
ab|  |la|pr|od|ge|ob| m|an|  |s |is|el|ti|ng|il|d |ua|c
he|  |ea|of|ho| m| t|et|ha|  | t|od|ds|e |ki| c|t |ng|br
wo|m,|to|yo|hi|ve|u | t|ob|  |pr|d |s |us| s|ul|le|ol|e
 t|ca| t|wi| M|d |th|"A|ma|l |he| p|at|ap|it|he|ti|le|er
ry|d |un|Th|" |io|eo|n,|is|  |bl|f |pu|Co|ic| o|he|at|mm
hi|  |  |in|  |  | t|  |  |  |  |ye|  |ar|  |s |  |  |.

We want to put the columns in the correct order, the rows already are ordered. I made one cheat optimisation, I assumed that the first row of the first column wasn't indented and that it would begin with a capital letter. Which gives me column 4 as the first column.

Rather than trying to match words, which looked hard as we don't know where the boundaries are, I decided to match n-grams (sequences of letters) and in the end I used trigrams, as I haven't yet found a decent table of quad-grams.

A few Python first for me here, reading in a file, cleaning it up on the way :

f = open('shredded.txt')

regex = re.compile('[,."]')
input_table = []
output_table = []
for str in  f:
  str = regex.sub(' ', str.strip("\n"))
  str = str.split('|')
  input_table.append(str)

Transposing a table (this is nice) :

col_table = zip(*input_table)

Then I created a Trigram class to return the probabilities of any given sequence, I used data from the Brown Corpus , there was one 'gotcha' here, the sequence " s " -the corpus assumes that "'s " is transformed into " s " and assigns it a very high probability, I decided to change this to the same as that for " t ".

Other than that it's a straight forward naive Bayes sum of probabilities, although I do it for both 'foward' trigram sequences; i.e for |Cl|nf| I find the probabilities for 'Cln' and 'lnf' and sum them.

So how does it do?

Claude Shannon founded ition  manfor 
theory  which is the bas      f is o 
probabilistic language mand   s odel 
of the code breaking methat    thods 
you would use to solve toblem prhis  
with the paper titled  Amaticahe Matl
Theory of Communication ished bl  pu 
in this year 

not too shabby, you could answer the question from this, it gets the first 12 columns dead right and bits of the others in the correct order as well.

Longer columns would increase the chances of success, but what about algorithm tweaks? Things to try :

• Find a 4-gram table
• Post process for words (I was put off this as my WordFreq doesn't include names)
• Process for spaces -there should only be one space between words, I have more as I have substituted spaces for punctuation. We could use this to detect 'illegal' orderings.

Caesar cypher solver in Python

An optional exercise in the ai class was to solve a Caesar cypher. The example given was pretty simple with only one shift, so the easiest way to solve it would be by inspection after printing out all 26 options.

But that isn't in the spirit of things, we want to solve it in an a.i. manner using probabilities, here's

the original cypher :

"Esp qtcde nzyqpcpynp zy esp ezatn zq Lcetqtntlw Tyepwwtrpynp hld spwo le Olcexzfes Nzwwprp ty estd jplc."

The first thing I noticed is that it has structure, if we assume that the punctuation has been left alone then we have a set of words. I can look up the probabilities of words in a table -I used the one from the British National Corpus:  ftp://ftp.itri.bton.ac.uk/bnc/all.num.o5

I created a Python class with a dictionary of words and probabilities (word_freqs) and a class method to return the probability of a word :

  @classmethod
  def word_prob(self,word):
    if word in self.word_freqs:
      return math.log(self.word_freqs[word] / self.corpus_freq)
    return math.log(1 / self.corpus_freq)

I take the log because, even with the correct, english, sentence, the probabilities soon get too small for simple arithmetic as they are multiplied together for each word in the sentence to get a 'sentence probability'

Storing and sorting the probabilities stumped me for a bit, but I eventually decided upon a list of tuples :

prob_map.append((prob, string2))


Which I sort, using the operator module, with :
 sorted(prob_map, key=operator.itemgetter(0), reverse=False):


This seems clunky compared to PHP, Perl or even C, but I'm still feeling my way around Python so I'm hoping that there's a better way.

To give me the answer :
(-7308.628573183837, 'the first conference on the topic of artificial intelligence was held at dartmouth college in this year.')

1956 BTW

There we go, a first bit of Natural Language Programming -which I have signed up for this term.

Machine Learning -Week 2 Multi Varient Linear Regression

Intoducing Octave

Fitted line graph -the blue lines is derived from the red crosses

 
If you got past the title you are doing well! More line fitting, but we're now into multi-dimensional space captain! Put simply this means that you have more parameters (different types) of information to match. So in the house price eample used in the course; the simple version matches area(sq ft) to the price of the house, the multivarient version might use area, number of rooms, and age to get you a more accurate fit. Fortunately, it works as you'd hope, in that you just add up the effects of the different terms, or in programming terms you just have two loops :

for i = 1 to number_of_houses
  for j = number_of things_i_know_about_a_house
    work_it_out()
  end
end

Cost function plot from Octave

Which leads nicely to Octave a programming language designed specifically for doing maths, it combines with gnuplot to draw pretty pictures of your data. The sort of data we are talking about now comes in tables -whether printed, or as databases tables or spreadsheets doesn't really matter, as a programmer you would normally bring this stuff in as arrays of whatever dimension and do the above. Octave, however understands matrices and tables intuitively and can manipulate them directly, the above might become :

number_of_houses * work_it_out(number_of things_i_know_about_a_house)

A lot simpler and, because Octave is built for this sort of job, much quicker.

Onwards and Upwards. The top graph represents the point of all this, if you know the area of your house you can estimate it's cost. The advantage of getting a computer to do this is that you can have enormous training sets -all house prices in the country, and do lots of subset plots -all semis, all semis in Suffolk etc.

AI - week 2 Bayes Networks, probably the best networks in the world

What are the chances of that?

After an easy introduction, via tree searching, last week, we're into the the thick of it with Bayes Networks and stochastic reasoning.

The good reverend Bayes

Bayes networks deal with uncertainty, they can answer questions such as -given that my cancer test was positive -what are the chances that I have cancer?  I'm not going to go into the details here, there are several resources on the web -but the answer is of the form

P(C|T) = ( P(T|C) . P(C) ) / P(T)

and likely to be lower than you think. P(C|T) is the chances of you having cancer given the test, P(T|C) is the chance of the test being positive if you have cancer -which I guess would come from the testing of the test, P(C) is the chance of you having cancer in general -which would come from actuarial tables or the like and P(T) is the chance of the test being positive whether o not you have cancer.

The course goes into more depth than this, showing how to reason from one test result to another, say, and how to chain probabilities.

All good stuff -but my brain aches.