GIGO - Check your data

A pothole?

One of the most important lessons I learnt in computing was Garbage In = Garbage Out, the GIGO law. When I got some less than perfect results from my pothole detector I took a look at the training data.

The data had been taken from a couple of DuckDuckGo searches, being lazy I had used terms from a notebook used to find birds in trees and just changed 'bird' to 'pothole' and 'tree' to 'road surface'. I displayed the first few images of each search and they looked reasonable, but then I took a closer look.

There were quite a few images of sunspots in the search, weird. I looked at my search string, there were three variations : 'pothole', 'pothole in the sun' and 'pothole in the shade'. The last two carried over from the bird search and I had left them in, what harm could it do? In this case it seemed to do quite a lot finding 'holes in the sun'  and also quite a few pictures of awnings, sun shades maybe?

Then there were some images that had come up 'randomly' in the search, like the one above, presumably mislabelled, or maybe on a page about potholes. Of the 140 files downloaded only around 40 were usable.

The road surface query produced much better results, in as much as all the photos were of road surfaces, the issue here was that many of them were variations on a theme.

Driving off into the sunset

This image has many strong features, the white lines grass either side of the road and a skyline, that the algorithm might learn to associate with 'road surface', whereas I just want it to learn about the asphalt, or the lack of. One image like this would be fine, but I felt 25% or more was too many.

To enhance the dataset, and perhaps tailor it to the UK country roads that I cycle around, I got on my bike and took more pictures of potholes and the road surface around them. I combined these with a selection of the downloaded images to get a better training set with around 30 images in each category.

I reran the model training and got a marginally better performance on the figures and a better fit with my validation images.

Further exploration in the next post.

Pothole Detection, without wrecking a wheel.

Is it or isn't it?

 

I decided to build and train a pothole image detector as part of following along with Fast Ai's Practical Deep Learning course. The reason I chose the pothole detector was that at a previous company a similar thing had been built from scratch, completely training it ourselves. Potholes are also something I come across very regularly on the British roads I cycle around. Having come off my bicycle once in the last year I am not in a hurry to repeat the experience.


The thing that immediately struck me was how quickly the model was trained and how easy it was to do. This is mainly because what we are doing is fine tuning an existing image model rather than trying to train one up from nothing, the base model already knows a lot about images before we start. The project where we trained the pothole detector from scratch took at least five weeks to get to something reliable, whereas I have trained something that seems to work in about five minutes. It's not quite a fair comparison as there was always also a need to slice videos into images and extract metadata from them, so there was more scope in the original project.


Fast AI is designed to make the development and training of models quick and easy. It wraps a lot of the complexities of the underlying libraries, and provides utilities that allow you to download images from an internet search. There are data types that are designed to provide categories and labels, test and training data for the model you are training or tuning. I am sure that there is much more in the Fast AI library that I haven't got to yet.


The other big difference between the earlier project, which was several years ago, is that we are able to use free compute and deployment resources to evaluate and tune the model. The earlier project used the boss’s video gaming machine because that had some GPUs in it and he went through Call of Duty withdrawal whilst training was done.


The specific tools we used were Jupyter Notebooks hosted by Kaggle, a frontend generated by Gradio and hosted at Hugging Face and the code basically came from chapter 1 and 2 of the Fast AI course. The end result of that is a public demonstration that you can see here:




So how did it do? I have only tried it out on one or two images downloaded from the internet and it performs well on those, but they might well be in the training set. I have tried it on one photograph of a pothole that I took and it got that wrong, possibly because it was more of a pot crack than a pothole. I'm planning to take more photographs and carry out more of a formal evaluation of how the model is doing, and maybe tune it again on images that are more relevant to the roads that I cycle along. Fast AI seems to provide some tools and utilities to do this, so look out for another exciting instalment.

Chat vs Search

 I had to add a couple of features to the UI on the MIR infrastructure router. Not really being a UI person I could either look them up, or I could ask ChatGPT. In the end I did both, how did it pan out?

Colour Picker

The first thing I wanted to add was a colour picker. MIR now has a lot of data layers for the UK and it is helpful to be able set the colours for these, the plan was to trigger a colour picker by clicking on a randomly coloured block that the layer has as a default.

I had a few goes at this but didn't get anything satisfactory here's an example below:

system_message = "You are an expert HTML, Javascipt and CSS coder." user_message = """ Create a colour picker to change the colour of an element in a web page. """

In some ways this gave me what I asked for, HTML, CSS and JS that would, probably, change the colour of an element. Unfortunately the colour picker is an HTML form field that you have to type a CSS RGB string into rather than an actual picker.

In other attempts I managed to get a basic JS colour picker working that consisted of three sliders that would change the RGB  parameters of a colour, again setting the colour of the element when you click the button. It potentially works but isn't very interactive.

This still wasn't ideal, so I tried a Google search for a JavaScript colour picker which led me to a site with a

collection of these and out of those I chose one called Swatchy.  This is where search truly wins over chat,

because you can actually see the colour pickers in action and choose one with functionality that you like. 

Modal

The other thing I wanted to add was a help modal, things have got a bit more complicated and help where you need it seemed to be a good idea.

Again I tried ChatGPT first these are the prompts that I came up with :

system_message = "You are an expert HTML, Bootstrap 5,Javascript and CSS coder."

user_message = """

Create a box to display some help text. It should open when a button with a help icon is clicked and close when a close button is clicked.

"""

This produced what looked like workable HTML, JavaScript and CSS to create a popup box, which I guess is what one would expect, but as I am using Bootstrap I don't need.

Altering the text to ‘create a modal’ did better, in that it just produced HTML and CSS to hide and show the box. It did also make use  of the ‘data-bs-dismiss’ Bootstrap attribute to dismiss  the modal, however the CSS was unnecessary as this is already included in Bootstrap.

 A quick search for ‘bootstrap 5 modal’ took me to the Bootstrap documentation and gave me a small amount of code that I could cut and paste into my site.  2-0 to search.

TL;DR

Search isn't dead yet, especially for anything visual. Search can give you options, writing good prompts is hard.

First try at developing a website with ChatGPT

Like everyone and his brother, I have been getting interested in chatGPT and Large Language Models in

general. A lot of development for a website can be pretty repetitive from one site to another and so I wanted

to see if chat GPT could make me more productive and allow me to concentrate on the more interesting

parts of the project.

Here at Msasa a pilot project came along where we wanted  to develop a website to show a client,

even though it may never be used. This seemed like an ideal use case for trying out  chatGPT. The website

we ended up with is called  MineralMarketplace, Take a look and see what we ended up with.

In preparation I took one of DeepLearning's short courses on prompt engineering for developers. If you haven't used chatGPT, or another LLM, then I recommend this, or something like it, to get you started.

The Experimant

Firstly I created a database schema for the site, using PGAdmin in my case.  I could have tried to construct prompts to get chatGPT to do this, which I'm sure it could, but it seemed as though it would be shorter and quicker to do it myself. It also gave me structured language to pass to the LLM from which it could generate other parts of the website.  If you prefer I'm sure you could do this either by creating objects or an Open API spec and then get chatGPT to generate a database scheme from that.

The next thing to try was to throwing the schema at chatGPT, and ask it to generate the CRUD code functions in Python to populate and query it. The good news was that chatGPT understood the schema,  the bad news was that it blew out the token limit  and couldn't generate much code. To solve this I asked it to generate CRUD code for each table in turn and that worked well. Following that I then asked it to generate me a flask web server and the end points for an API that would call the CRUD code and an Open API 3 spec for that code,  again these both worked well,  although I haven't evaluated the Swagger code to check for hallucinations. 

Since I am a relatively basic front end developer and this was just a demonstration site, I decided that I

would build the front end using Flask’s Jinja templating and Bootstrap. I found that developing the Bootstrap

templates didn't work very well and in the end I gave up and downloaded a theme from an open source

Bootstrap template site. What did work was generating the forms using the LLM, although the country

dropdown again exceeded the token limit.

The token limit is, more or less, the maximum amount of text you can pass into, and get back from,

the LLM.

My final attempt using AI for the site was to try image generators to generate the background images for the various minerals,  this didn't work at all and I gave up pretty quickly and search for free images that I could use.

Would I do it again?

I would, the LLM generated a lot of boilerplate for me that just worked, and I now have a set of prompts that I

could use to do the same thing again.

The fact that it failed to create the front end wasn’t actually a huge drawback, I am not a designer so it makes a lot of sense to tweak an existing theme rather than coming up with something from scratch. The MineralMarketplace theme was originally a property website. 

Example Prompt

Given the Postgres SQL definition defined inside <sql></sql>

For Python 3 and the psycopg2 library:

 

step 1: For each table defined by a 'CREATE TABLE' statement in the SQL definition create an Insert

function that inserts all non serial fields in the 'CREATE TABLE' statement and returns the value of  

the serial field. Also create Read and Delete functions using the serial field name as the key.  

The function names should include the name of the table.

step 2: Using Python Flask create an api for the functions defined in step 1

step 3: Create an OpenApi v3 specification for the api defined in step 2

SQL definition: <sql>

{sql}

</sql>

5 reasons why we are happy with AngularJS

I promised a general post about our experiences on sorted.jobs with AngularJS, so here's the view from ten thousand feet, rather than the 50 foot view of other posts.

1. Good for non-designers

In some ways this is more down to Bootstrap, but combine AngularJS, Bootstrap 3 and AngularUI you have a quick and easy way to produce a responsive and dynamic website.

2. Good for non programming web designers

Angular hides most of the Javascript so it makes for easy collaboration between a front end programmer and a designer. Angularjs directives mean that some of the clever bits can be presented to the designer as new HTML tags, which he can manipulate in the same way as other tags.

3. Very clear view seperation

Views live in templates, which are just HTML files, controllers  live in javascript controller files and never the twain shall meet. The actual controller function gets specified in an attribute to a <div> or other HTML tag -what could be easier (don't mention $scope)?

4. Decent documentation and community

The AngularJS site is pretty good, with a good tutorial, developer guide, and reference. Stackexchange is very active, Egghead.io is a good source of video tutorials although there are many others.  Manning has a couple of MEAP books; I have used AngularJS in Action, which is now nearly completed, the other is  AngularJS in Depth. Packt has a whole raft of AngularJS titles, I have used Mastering Web Application Development with AngularJS, I would guess the Manning books will be more up to date than the Packt one at the moment as they aren't formally published yet.

5. Testing tool.

Angular provides Protactor for end to end testing, this uses Selenium Webdriver to run Jasmine tests via standard web browsers. It is aware of Angular directives so you can run tests on Angular loops etc. It works well and the devlopers have been helpful when I found a problem.

That'll do for now, there's loads more and AngularJS has it's quirks, but I need coffee so that's another post.

Sorting and paging with AngularJS

We have two dashboard screens that make heavy use of AngularJS on sorted.jobs, one for candidates and one for recruiters. The recruiter screen in particular can end up with an awful lot of jobs on it, so the list needs to page, in addition it would be good to be able to sort by posting date, number of applicants, company name and so on.

Recruiter Dashboard

This is where we have got up to, not too happy about the UX/UI side, but it's workable. The paging is provided by the excellent AngularUI Booststrap library. Using it is pretty simple this is the AngularJS template directive ;

        <pagination total-items="active_jobs.total" ng-model="currentActivePage" ng-change="activePageChanged()"></pagination>

Pagination needs to know the size of the array it is paging over (active_jobs.total), has a variable to store the current page in (currentActivePage) and a function to call when you change the page - activePageChanged().

And here's the controller function :

  $scope.activePageChanged = function(i) {
    $scope.activePageStart = ($scope.currentActivePage -1) * $scope.itemsPerPage;
    $scope.activePageEnd = $scope.activePageStart + 10 ;
  }

As you can see all we are doing is changing the start end end points of the items we are viewing in the array; we have been a bit lazy and not passed itemsPerPage to the directive as we are using the default of ten.

The ng-repeat call looks like this :

      <div class='row' ng-repeat="job in active_jobs.hits.slice(activePageStart,activePageEnd)">

I did see pages offering a range filter but  array.slice() seems more direct.

In this example the whole array is passed from the backend to the front in one go (this is so it can be sorted in the browser), but you don't have to do it that way, other pages we have make a call to the back  end from within the activePageChanged() function  to get the next page of results.

Sorting

AngularJS provides the orderBy filter that will sort an array, the documentation is pretty good, the only real point to pick up is that to use it in our pagination example we need to call the controller version and not use the template filter version (this would only sort the array slice and not the whole array). So in the template we make a function call :

<p><a href="" ng-click="reverse=!reverse;active_order('_source.reference', reverse)">Reference<span class="glyphicon glyphicon-resize-vertical sort-col-header"></a></p>

We call the active order function with the column we want to sort by and the direction of sort. The  reverse=!reverse just twiddles the sort order.

We then need to set up the controller to use orderBy by injecting $filter :

function SomeCtrl ($scope, $location, $filter...

and using a bit of syntactic suger :

var orderBy = $filter('orderBy');

it is just a matter of defining the function :
           
  $scope.active_order = function(col, reverse) {
    $scope.active_jobs['hits'] = orderBy($scope.active_jobs['hits'], col, reverse);
  };        

and Robert is your father's brother.
 

Data Munging with MongoDB Aggregation and Python

I am evaluating some named entity recognition systems for sorted.jobs , trying to improve our search by sorting the wheat from the chaff, and some of the initial results look encouraging -but just how encouraging? We need to do a bit of analysis to find out.

The most hopeful results come from the extraction of programming language and operating system entities from the text -see the figure below :

Entity Types

On to MongoDB

This table was generated from a MongoDb database using two collections entities and missed_entities entities contains the terms that the program found and missed_entities the ones that I though it missed. Of the ones it found it either got it right ('Hit'), wrong ('Miss') or it was a bit dubious ('Null'). To get the stats I used the new (to me) MongoDB aggregation operations, analagous to the SQL GROUP BY, HAVING, SUM &c.

You could do all this in the old MongoDB map/reduce way, but aggregation seems a bit more intuitive.

So to get the 'Hit', 'Miss' and 'Null' columns the Python code looks like :

entity_c.aggregate([{"$group": {"_id": {"etype" : "$_type", "hit_miss" : "$hit_miss"} , "count": {"$sum": 1}}}])

which returns me rows like :

{u'count': 55, u'_id': {u'etype': u'ProgrammingLanguage'ProgrammingLanguage', u'hit_miss': u'1'}}

{u'count': 2, u'_id': {u'etype': u'ProgrammingLanguage'}}

and nothing for the misses because there weren't any.

The hard work occurs in the $group where I create a compound '_id' field made up of the entity type field and entity hit_miss field and then count all the matching entities.

The Aggregation Pipeline

But we can also look at the terms that the recogniser missed :

 

Missed entities

Here we only want the entities for the given type ('ProgrammingLanguage') and we want them in order, our PyMongo aggregate call now becomes :

missed_c.aggregate([
    {"$match" : {"_type" : etype}},
    {"$group": {"_id": "$name", "count": {"$sum": 1}}},
    {"$sort" : {"count" : -1}}
  ])

We have extra terms : '$match' which filters the documents so we only consider those with the passed in type (etype) and '$sort' which orders by the count we generated in the group. MongoDB pipelines these performing the match then the group and then the sort before returning you the result.

Finally, looking at the results we can see that there are some casing issues, we can make everything the same case by adding in a '$project' :

    { "$project" : { "name":{"$toUpper":"$name"} } },

$project creates a new field (or overwrites an existing one in the pipeline, not the real one in the collection) in this instance I have told it to make all the names uppercase and we get :

Normalised entities

It doesn't matter where in the array you place any of theses terms MongoDB will sort out the ordering.

What does this tell us? Well in this case if I could persuade the tagger to recognise CSS and variants of terms it already knows with a number on the end I would get a big jump in the overall quality of the results.

References

Angular Aggregation manual pages.