GIGO - Check your data

A pothole?

One of the most important lessons I learnt in computing was Garbage In = Garbage Out, the GIGO law. When I got some less than perfect results from my pothole detector I took a look at the training data.

The data had been taken from a couple of DuckDuckGo searches, being lazy I had used terms from a notebook used to find birds in trees and just changed 'bird' to 'pothole' and 'tree' to 'road surface'. I displayed the first few images of each search and they looked reasonable, but then I took a closer look.

There were quite a few images of sunspots in the search, weird. I looked at my search string, there were three variations : 'pothole', 'pothole in the sun' and 'pothole in the shade'. The last two carried over from the bird search and I had left them in, what harm could it do? In this case it seemed to do quite a lot finding 'holes in the sun'  and also quite a few pictures of awnings, sun shades maybe?

Then there were some images that had come up 'randomly' in the search, like the one above, presumably mislabelled, or maybe on a page about potholes. Of the 140 files downloaded only around 40 were usable.

The road surface query produced much better results, in as much as all the photos were of road surfaces, the issue here was that many of them were variations on a theme.

Driving off into the sunset

This image has many strong features, the white lines grass either side of the road and a skyline, that the algorithm might learn to associate with 'road surface', whereas I just want it to learn about the asphalt, or the lack of. One image like this would be fine, but I felt 25% or more was too many.

To enhance the dataset, and perhaps tailor it to the UK country roads that I cycle around, I got on my bike and took more pictures of potholes and the road surface around them. I combined these with a selection of the downloaded images to get a better training set with around 30 images in each category.

I reran the model training and got a marginally better performance on the figures and a better fit with my validation images.

Further exploration in the next post.

Pothole Detection, without wrecking a wheel.

Is it or isn't it?

 

I decided to build and train a pothole image detector as part of following along with Fast Ai's Practical Deep Learning course. The reason I chose the pothole detector was that at a previous company a similar thing had been built from scratch, completely training it ourselves. Potholes are also something I come across very regularly on the British roads I cycle around. Having come off my bicycle once in the last year I am not in a hurry to repeat the experience.


The thing that immediately struck me was how quickly the model was trained and how easy it was to do. This is mainly because what we are doing is fine tuning an existing image model rather than trying to train one up from nothing, the base model already knows a lot about images before we start. The project where we trained the pothole detector from scratch took at least five weeks to get to something reliable, whereas I have trained something that seems to work in about five minutes. It's not quite a fair comparison as there was always also a need to slice videos into images and extract metadata from them, so there was more scope in the original project.


Fast AI is designed to make the development and training of models quick and easy. It wraps a lot of the complexities of the underlying libraries, and provides utilities that allow you to download images from an internet search. There are data types that are designed to provide categories and labels, test and training data for the model you are training or tuning. I am sure that there is much more in the Fast AI library that I haven't got to yet.


The other big difference between the earlier project, which was several years ago, is that we are able to use free compute and deployment resources to evaluate and tune the model. The earlier project used the boss’s video gaming machine because that had some GPUs in it and he went through Call of Duty withdrawal whilst training was done.


The specific tools we used were Jupyter Notebooks hosted by Kaggle, a frontend generated by Gradio and hosted at Hugging Face and the code basically came from chapter 1 and 2 of the Fast AI course. The end result of that is a public demonstration that you can see here:




So how did it do? I have only tried it out on one or two images downloaded from the internet and it performs well on those, but they might well be in the training set. I have tried it on one photograph of a pothole that I took and it got that wrong, possibly because it was more of a pot crack than a pothole. I'm planning to take more photographs and carry out more of a formal evaluation of how the model is doing, and maybe tune it again on images that are more relevant to the roads that I cycle along. Fast AI seems to provide some tools and utilities to do this, so look out for another exciting instalment.

Chat vs Search

 I had to add a couple of features to the UI on the MIR infrastructure router. Not really being a UI person I could either look them up, or I could ask ChatGPT. In the end I did both, how did it pan out?

Colour Picker

The first thing I wanted to add was a colour picker. MIR now has a lot of data layers for the UK and it is helpful to be able set the colours for these, the plan was to trigger a colour picker by clicking on a randomly coloured block that the layer has as a default.

I had a few goes at this but didn't get anything satisfactory here's an example below:

system_message = "You are an expert HTML, Javascipt and CSS coder." user_message = """ Create a colour picker to change the colour of an element in a web page. """

In some ways this gave me what I asked for, HTML, CSS and JS that would, probably, change the colour of an element. Unfortunately the colour picker is an HTML form field that you have to type a CSS RGB string into rather than an actual picker.

In other attempts I managed to get a basic JS colour picker working that consisted of three sliders that would change the RGB  parameters of a colour, again setting the colour of the element when you click the button. It potentially works but isn't very interactive.

This still wasn't ideal, so I tried a Google search for a JavaScript colour picker which led me to a site with a

collection of these and out of those I chose one called Swatchy.  This is where search truly wins over chat,

because you can actually see the colour pickers in action and choose one with functionality that you like. 

Modal

The other thing I wanted to add was a help modal, things have got a bit more complicated and help where you need it seemed to be a good idea.

Again I tried ChatGPT first these are the prompts that I came up with :

system_message = "You are an expert HTML, Bootstrap 5,Javascript and CSS coder."

user_message = """

Create a box to display some help text. It should open when a button with a help icon is clicked and close when a close button is clicked.

"""

This produced what looked like workable HTML, JavaScript and CSS to create a popup box, which I guess is what one would expect, but as I am using Bootstrap I don't need.

Altering the text to ‘create a modal’ did better, in that it just produced HTML and CSS to hide and show the box. It did also make use  of the ‘data-bs-dismiss’ Bootstrap attribute to dismiss  the modal, however the CSS was unnecessary as this is already included in Bootstrap.

 A quick search for ‘bootstrap 5 modal’ took me to the Bootstrap documentation and gave me a small amount of code that I could cut and paste into my site.  2-0 to search.

TL;DR

Search isn't dead yet, especially for anything visual. Search can give you options, writing good prompts is hard.

First try at developing a website with ChatGPT

Like everyone and his brother, I have been getting interested in chatGPT and Large Language Models in

general. A lot of development for a website can be pretty repetitive from one site to another and so I wanted

to see if chat GPT could make me more productive and allow me to concentrate on the more interesting

parts of the project.

Here at Msasa a pilot project came along where we wanted  to develop a website to show a client,

even though it may never be used. This seemed like an ideal use case for trying out  chatGPT. The website

we ended up with is called  MineralMarketplace, Take a look and see what we ended up with.

In preparation I took one of DeepLearning's short courses on prompt engineering for developers. If you haven't used chatGPT, or another LLM, then I recommend this, or something like it, to get you started.

The Experimant

Firstly I created a database schema for the site, using PGAdmin in my case.  I could have tried to construct prompts to get chatGPT to do this, which I'm sure it could, but it seemed as though it would be shorter and quicker to do it myself. It also gave me structured language to pass to the LLM from which it could generate other parts of the website.  If you prefer I'm sure you could do this either by creating objects or an Open API spec and then get chatGPT to generate a database scheme from that.

The next thing to try was to throwing the schema at chatGPT, and ask it to generate the CRUD code functions in Python to populate and query it. The good news was that chatGPT understood the schema,  the bad news was that it blew out the token limit  and couldn't generate much code. To solve this I asked it to generate CRUD code for each table in turn and that worked well. Following that I then asked it to generate me a flask web server and the end points for an API that would call the CRUD code and an Open API 3 spec for that code,  again these both worked well,  although I haven't evaluated the Swagger code to check for hallucinations. 

Since I am a relatively basic front end developer and this was just a demonstration site, I decided that I

would build the front end using Flask’s Jinja templating and Bootstrap. I found that developing the Bootstrap

templates didn't work very well and in the end I gave up and downloaded a theme from an open source

Bootstrap template site. What did work was generating the forms using the LLM, although the country

dropdown again exceeded the token limit.

The token limit is, more or less, the maximum amount of text you can pass into, and get back from,

the LLM.

My final attempt using AI for the site was to try image generators to generate the background images for the various minerals,  this didn't work at all and I gave up pretty quickly and search for free images that I could use.

Would I do it again?

I would, the LLM generated a lot of boilerplate for me that just worked, and I now have a set of prompts that I

could use to do the same thing again.

The fact that it failed to create the front end wasn’t actually a huge drawback, I am not a designer so it makes a lot of sense to tweak an existing theme rather than coming up with something from scratch. The MineralMarketplace theme was originally a property website. 

Example Prompt

Given the Postgres SQL definition defined inside <sql></sql>

For Python 3 and the psycopg2 library:

 

step 1: For each table defined by a 'CREATE TABLE' statement in the SQL definition create an Insert

function that inserts all non serial fields in the 'CREATE TABLE' statement and returns the value of  

the serial field. Also create Read and Delete functions using the serial field name as the key.  

The function names should include the name of the table.

step 2: Using Python Flask create an api for the functions defined in step 1

step 3: Create an OpenApi v3 specification for the api defined in step 2

SQL definition: <sql>

{sql}

</sql>

Data Munging with MongoDB Aggregation and Python

I am evaluating some named entity recognition systems for sorted.jobs , trying to improve our search by sorting the wheat from the chaff, and some of the initial results look encouraging -but just how encouraging? We need to do a bit of analysis to find out.

The most hopeful results come from the extraction of programming language and operating system entities from the text -see the figure below :

Entity Types

On to MongoDB

This table was generated from a MongoDb database using two collections entities and missed_entities entities contains the terms that the program found and missed_entities the ones that I though it missed. Of the ones it found it either got it right ('Hit'), wrong ('Miss') or it was a bit dubious ('Null'). To get the stats I used the new (to me) MongoDB aggregation operations, analagous to the SQL GROUP BY, HAVING, SUM &c.

You could do all this in the old MongoDB map/reduce way, but aggregation seems a bit more intuitive.

So to get the 'Hit', 'Miss' and 'Null' columns the Python code looks like :

entity_c.aggregate([{"$group": {"_id": {"etype" : "$_type", "hit_miss" : "$hit_miss"} , "count": {"$sum": 1}}}])

which returns me rows like :

{u'count': 55, u'_id': {u'etype': u'ProgrammingLanguage'ProgrammingLanguage', u'hit_miss': u'1'}}

{u'count': 2, u'_id': {u'etype': u'ProgrammingLanguage'}}

and nothing for the misses because there weren't any.

The hard work occurs in the $group where I create a compound '_id' field made up of the entity type field and entity hit_miss field and then count all the matching entities.

The Aggregation Pipeline

But we can also look at the terms that the recogniser missed :

 

Missed entities

Here we only want the entities for the given type ('ProgrammingLanguage') and we want them in order, our PyMongo aggregate call now becomes :

missed_c.aggregate([
    {"$match" : {"_type" : etype}},
    {"$group": {"_id": "$name", "count": {"$sum": 1}}},
    {"$sort" : {"count" : -1}}
  ])

We have extra terms : '$match' which filters the documents so we only consider those with the passed in type (etype) and '$sort' which orders by the count we generated in the group. MongoDB pipelines these performing the match then the group and then the sort before returning you the result.

Finally, looking at the results we can see that there are some casing issues, we can make everything the same case by adding in a '$project' :

    { "$project" : { "name":{"$toUpper":"$name"} } },

$project creates a new field (or overwrites an existing one in the pipeline, not the real one in the collection) in this instance I have told it to make all the names uppercase and we get :

Normalised entities

It doesn't matter where in the array you place any of theses terms MongoDB will sort out the ordering.

What does this tell us? Well in this case if I could persuade the tagger to recognise CSS and variants of terms it already knows with a number on the end I would get a big jump in the overall quality of the results.

References

Angular Aggregation manual pages.

Analysing an Online Community

I had a look at carrying out network analysis of the relations in an online community for the Coursera Social Network Analysis class.

Original Bulletin Board Graph

The community was a UK based bulletin board of general chat, completely anonymised. I wanted to look at the relationship between topics and see what could be inferred from that, so in the graphs the nodes are threads and the edges are people who posted between threads. This post is mainly about the network analysis, another will look at nitty gritty of how I carried it out.

Tools

In short, Python and Gephi -more details later.

The Network

I ended up with a network of 79 nodes and 345 edges. The node labels were extracted keywords from the threads -edges were people who posted between threads.

Analysis

First question is there anything to look at? Any Structure? One way to tell is by constructing a random graph and comparing it's properties with those of our graph -this can be done, a little tediously, with Gephi.

Our graph has an average degree of 8.73, a shortest path length of 2.22 and an average clustering coefficient of 0.634. In comparison a random graph with same node and edge counts has a degree of ~4.4 a path length of 2.2 and the clustering coefficeient of 0.144 degree is considerably lower than our graph, so nodes are more connected and, although the path length is similar to a random graph the clustering is higher, indicating small world properties -which gives us something to look at.

There are a 3 super nodes in the graph :

Node	Betweenness
clarkson, people, one, bbc	1015
people, like, money, would	650
clarkson, one, public, sector	343.

People in the UK can probably guess the reason for the first and third nodes. The comment was so outrageous that perhaps most people felt that they had to say something about it, and abnormal relationships in the graph could be created. After removal of the clarkson nodes the graph is as below:

Clarkson free graph

Running Gephi Modularity with a resolution of 1.0 gives us 6 groups. Groups 4, 5 and 0 seem to be too diverse to say anything much about. Group 1 is mainly about sport, although its' highest degree node is about Apple. Group 3 is about UK domestic topics, Group 2 is more global centring around politics and economics.

Summary

We can see that there is some clustering of subjects when we relate them by people although we have not discovered any surprises on this small data set, as similar subjects cluster together. Next steps for this study would be too use a more sophisticated way of extracting the labels from the thread text, run against a larger data set and perhaps introduce the notion of time as an attribute, one could then perhaps look at 'contagion' of the network by a topic, the Clarkson posts might be interesting here. We could also invert the network and relate people by posts -but that seemed less interesting.

Probablistic Deshredder -in Python

The second exercise in the ai class was to recompose a shredded message, this is a bit harder than the Caesar Cypher -although still do-able by Mk1 eyeball. Here's the message :

de|  | f|Cl|nf|ed|au| i|ti|  |ma|ha|or|nn|ou| S|on|nd|on
ry|  |is|th|is| b|eo|as|  |  |f |wh| o|ic| t|, |  |he|h
ab|  |la|pr|od|ge|ob| m|an|  |s |is|el|ti|ng|il|d |ua|c
he|  |ea|of|ho| m| t|et|ha|  | t|od|ds|e |ki| c|t |ng|br
wo|m,|to|yo|hi|ve|u | t|ob|  |pr|d |s |us| s|ul|le|ol|e
 t|ca| t|wi| M|d |th|"A|ma|l |he| p|at|ap|it|he|ti|le|er
ry|d |un|Th|" |io|eo|n,|is|  |bl|f |pu|Co|ic| o|he|at|mm
hi|  |  |in|  |  | t|  |  |  |  |ye|  |ar|  |s |  |  |.

We want to put the columns in the correct order, the rows already are ordered. I made one cheat optimisation, I assumed that the first row of the first column wasn't indented and that it would begin with a capital letter. Which gives me column 4 as the first column.

Rather than trying to match words, which looked hard as we don't know where the boundaries are, I decided to match n-grams (sequences of letters) and in the end I used trigrams, as I haven't yet found a decent table of quad-grams.

A few Python first for me here, reading in a file, cleaning it up on the way :

f = open('shredded.txt')

regex = re.compile('[,."]')
input_table = []
output_table = []
for str in  f:
  str = regex.sub(' ', str.strip("\n"))
  str = str.split('|')
  input_table.append(str)

Transposing a table (this is nice) :

col_table = zip(*input_table)

Then I created a Trigram class to return the probabilities of any given sequence, I used data from the Brown Corpus , there was one 'gotcha' here, the sequence " s " -the corpus assumes that "'s " is transformed into " s " and assigns it a very high probability, I decided to change this to the same as that for " t ".

Other than that it's a straight forward naive Bayes sum of probabilities, although I do it for both 'foward' trigram sequences; i.e for |Cl|nf| I find the probabilities for 'Cln' and 'lnf' and sum them.

So how does it do?

Claude Shannon founded ition  manfor 
theory  which is the bas      f is o 
probabilistic language mand   s odel 
of the code breaking methat    thods 
you would use to solve toblem prhis  
with the paper titled  Amaticahe Matl
Theory of Communication ished bl  pu 
in this year 

not too shabby, you could answer the question from this, it gets the first 12 columns dead right and bits of the others in the correct order as well.

Longer columns would increase the chances of success, but what about algorithm tweaks? Things to try :

• Find a 4-gram table
• Post process for words (I was put off this as my WordFreq doesn't include names)
• Process for spaces -there should only be one space between words, I have more as I have substituted spaces for punctuation. We could use this to detect 'illegal' orderings.

Caesar cypher solver in Python

An optional exercise in the ai class was to solve a Caesar cypher. The example given was pretty simple with only one shift, so the easiest way to solve it would be by inspection after printing out all 26 options.

But that isn't in the spirit of things, we want to solve it in an a.i. manner using probabilities, here's

the original cypher :

"Esp qtcde nzyqpcpynp zy esp ezatn zq Lcetqtntlw Tyepwwtrpynp hld spwo le Olcexzfes Nzwwprp ty estd jplc."

The first thing I noticed is that it has structure, if we assume that the punctuation has been left alone then we have a set of words. I can look up the probabilities of words in a table -I used the one from the British National Corpus:  ftp://ftp.itri.bton.ac.uk/bnc/all.num.o5

I created a Python class with a dictionary of words and probabilities (word_freqs) and a class method to return the probability of a word :

  @classmethod
  def word_prob(self,word):
    if word in self.word_freqs:
      return math.log(self.word_freqs[word] / self.corpus_freq)
    return math.log(1 / self.corpus_freq)

I take the log because, even with the correct, english, sentence, the probabilities soon get too small for simple arithmetic as they are multiplied together for each word in the sentence to get a 'sentence probability'

Storing and sorting the probabilities stumped me for a bit, but I eventually decided upon a list of tuples :

prob_map.append((prob, string2))


Which I sort, using the operator module, with :
 sorted(prob_map, key=operator.itemgetter(0), reverse=False):


This seems clunky compared to PHP, Perl or even C, but I'm still feeling my way around Python so I'm hoping that there's a better way.

To give me the answer :
(-7308.628573183837, 'the first conference on the topic of artificial intelligence was held at dartmouth college in this year.')

1956 BTW

There we go, a first bit of Natural Language Programming -which I have signed up for this term.

AI - week 2 Bayes Networks, probably the best networks in the world

What are the chances of that?

After an easy introduction, via tree searching, last week, we're into the the thick of it with Bayes Networks and stochastic reasoning.

The good reverend Bayes

Bayes networks deal with uncertainty, they can answer questions such as -given that my cancer test was positive -what are the chances that I have cancer?  I'm not going to go into the details here, there are several resources on the web -but the answer is of the form

P(C|T) = ( P(T|C) . P(C) ) / P(T)

and likely to be lower than you think. P(C|T) is the chances of you having cancer given the test, P(T|C) is the chance of the test being positive if you have cancer -which I guess would come from the testing of the test, P(C) is the chance of you having cancer in general -which would come from actuarial tables or the like and P(T) is the chance of the test being positive whether o not you have cancer.

The course goes into more depth than this, showing how to reason from one test result to another, say, and how to chain probabilities.

All good stuff -but my brain aches.

A.I. -Week 1, the Prologue

In which the first video goes live and we meet Pros. Thrun and Norvig.

They had a bit of a struggle but Stanford have got the introductory lectures up on their site (via YouTube). The profs. seem affable and there's nothing too scary in the first video series -mainly just definition of terms and the illustration of some problems- but the reading list promises a hard climb.

Stanford seem to be encouraging these courses to be social -and for the videos to be viewed communally, turns out that there's a group in London so I'll trot along and take a look.


With apologies to Lurcio

Experiments With A Simple Evolutionary Algorithm (1)

In Richard Dawkin's 'The Blind Watchmaker'he describes a program that he built to mutate a random string into a given target string ('Methinks it is like a Weasel'), so I thought I'd try and build one myself. In version one I have three core functions fit(), breed() and make_child().  In each generation one breeds by making a number of children, the fittest of which is returned to be the parent of the next generation. Here are the guts of the functions :

fit(string1, string2)
compare the strings and return the difference between them as a number. 0 is a perfect fit. In version 1 the difference is the alphabetic distance between each individual character in each string :

        for i in range(len(t)):
                f = f + abs(t[i] - s[i])

make_child(parent)
randomly changes some of the characters in the parent string and returns the new string as the child :
.         for i in range(len(parent)):
                if(random.random() > 0.9):
                        child.append(random.choice('abcdefghijklmnopqrstuvwxyz '))
                else :
                        child.append(parent[i])


breed(parent)
create a number of children, compare them with each other and the parent for fitness and return the best to be used as the parent for the next generation. You don't need code for this -it's trivial.

How does it do?
It works! See update.


Questions and Improvements

•  How good is the measure of fitness?
• How good is make_child?
• Is there an ideal combination of generations, offspring and 'variation'
• Is only having 1 parent OK?

Based on nothing at all, I think that make_child has the most fundamental problem, I think that it should be tweaked to take into account current fitness of the parent. As the fit gets better then the changes should get smaller, as any organism that changed a lot between generations is quite likely to 'evolve' right out of its niche.

Update

Well, I was right, make child did have a fundamental problem. I thougth I was calling a random() function but instead seemed to be instantiating (or perhaps defining) some sort of random object -at least the Python interpreter wasn't complaining, this meant that all the strings were randomized every time and there was no generational inheritance. Fixed this and it worked.