GIGO - Check your data

A pothole?

One of the most important lessons I learnt in computing was Garbage In = Garbage Out, the GIGO law. When I got some less than perfect results from my pothole detector I took a look at the training data.

The data had been taken from a couple of DuckDuckGo searches, being lazy I had used terms from a notebook used to find birds in trees and just changed 'bird' to 'pothole' and 'tree' to 'road surface'. I displayed the first few images of each search and they looked reasonable, but then I took a closer look.

There were quite a few images of sunspots in the search, weird. I looked at my search string, there were three variations : 'pothole', 'pothole in the sun' and 'pothole in the shade'. The last two carried over from the bird search and I had left them in, what harm could it do? In this case it seemed to do quite a lot finding 'holes in the sun'  and also quite a few pictures of awnings, sun shades maybe?

Then there were some images that had come up 'randomly' in the search, like the one above, presumably mislabelled, or maybe on a page about potholes. Of the 140 files downloaded only around 40 were usable.

The road surface query produced much better results, in as much as all the photos were of road surfaces, the issue here was that many of them were variations on a theme.

Driving off into the sunset

This image has many strong features, the white lines grass either side of the road and a skyline, that the algorithm might learn to associate with 'road surface', whereas I just want it to learn about the asphalt, or the lack of. One image like this would be fine, but I felt 25% or more was too many.

To enhance the dataset, and perhaps tailor it to the UK country roads that I cycle around, I got on my bike and took more pictures of potholes and the road surface around them. I combined these with a selection of the downloaded images to get a better training set with around 30 images in each category.

I reran the model training and got a marginally better performance on the figures and a better fit with my validation images.

Further exploration in the next post.

Pothole Detection, without wrecking a wheel.

Is it or isn't it?

 

I decided to build and train a pothole image detector as part of following along with Fast Ai's Practical Deep Learning course. The reason I chose the pothole detector was that at a previous company a similar thing had been built from scratch, completely training it ourselves. Potholes are also something I come across very regularly on the British roads I cycle around. Having come off my bicycle once in the last year I am not in a hurry to repeat the experience.


The thing that immediately struck me was how quickly the model was trained and how easy it was to do. This is mainly because what we are doing is fine tuning an existing image model rather than trying to train one up from nothing, the base model already knows a lot about images before we start. The project where we trained the pothole detector from scratch took at least five weeks to get to something reliable, whereas I have trained something that seems to work in about five minutes. It's not quite a fair comparison as there was always also a need to slice videos into images and extract metadata from them, so there was more scope in the original project.


Fast AI is designed to make the development and training of models quick and easy. It wraps a lot of the complexities of the underlying libraries, and provides utilities that allow you to download images from an internet search. There are data types that are designed to provide categories and labels, test and training data for the model you are training or tuning. I am sure that there is much more in the Fast AI library that I haven't got to yet.


The other big difference between the earlier project, which was several years ago, is that we are able to use free compute and deployment resources to evaluate and tune the model. The earlier project used the boss’s video gaming machine because that had some GPUs in it and he went through Call of Duty withdrawal whilst training was done.


The specific tools we used were Jupyter Notebooks hosted by Kaggle, a frontend generated by Gradio and hosted at Hugging Face and the code basically came from chapter 1 and 2 of the Fast AI course. The end result of that is a public demonstration that you can see here:




So how did it do? I have only tried it out on one or two images downloaded from the internet and it performs well on those, but they might well be in the training set. I have tried it on one photograph of a pothole that I took and it got that wrong, possibly because it was more of a pot crack than a pothole. I'm planning to take more photographs and carry out more of a formal evaluation of how the model is doing, and maybe tune it again on images that are more relevant to the roads that I cycle along. Fast AI seems to provide some tools and utilities to do this, so look out for another exciting instalment.

Analysing an Online Community

I had a look at carrying out network analysis of the relations in an online community for the Coursera Social Network Analysis class.

Original Bulletin Board Graph

The community was a UK based bulletin board of general chat, completely anonymised. I wanted to look at the relationship between topics and see what could be inferred from that, so in the graphs the nodes are threads and the edges are people who posted between threads. This post is mainly about the network analysis, another will look at nitty gritty of how I carried it out.

Tools

In short, Python and Gephi -more details later.

The Network

I ended up with a network of 79 nodes and 345 edges. The node labels were extracted keywords from the threads -edges were people who posted between threads.

Analysis

First question is there anything to look at? Any Structure? One way to tell is by constructing a random graph and comparing it's properties with those of our graph -this can be done, a little tediously, with Gephi.

Our graph has an average degree of 8.73, a shortest path length of 2.22 and an average clustering coefficient of 0.634. In comparison a random graph with same node and edge counts has a degree of ~4.4 a path length of 2.2 and the clustering coefficeient of 0.144 degree is considerably lower than our graph, so nodes are more connected and, although the path length is similar to a random graph the clustering is higher, indicating small world properties -which gives us something to look at.

There are a 3 super nodes in the graph :

Node	Betweenness
clarkson, people, one, bbc	1015
people, like, money, would	650
clarkson, one, public, sector	343.

People in the UK can probably guess the reason for the first and third nodes. The comment was so outrageous that perhaps most people felt that they had to say something about it, and abnormal relationships in the graph could be created. After removal of the clarkson nodes the graph is as below:

Clarkson free graph

Running Gephi Modularity with a resolution of 1.0 gives us 6 groups. Groups 4, 5 and 0 seem to be too diverse to say anything much about. Group 1 is mainly about sport, although its' highest degree node is about Apple. Group 3 is about UK domestic topics, Group 2 is more global centring around politics and economics.

Summary

We can see that there is some clustering of subjects when we relate them by people although we have not discovered any surprises on this small data set, as similar subjects cluster together. Next steps for this study would be too use a more sophisticated way of extracting the labels from the thread text, run against a larger data set and perhaps introduce the notion of time as an attribute, one could then perhaps look at 'contagion' of the network by a topic, the Clarkson posts might be interesting here. We could also invert the network and relate people by posts -but that seemed less interesting.

Machine Learning -Week 2 Multi Varient Linear Regression

Intoducing Octave

Fitted line graph -the blue lines is derived from the red crosses

 
If you got past the title you are doing well! More line fitting, but we're now into multi-dimensional space captain! Put simply this means that you have more parameters (different types) of information to match. So in the house price eample used in the course; the simple version matches area(sq ft) to the price of the house, the multivarient version might use area, number of rooms, and age to get you a more accurate fit. Fortunately, it works as you'd hope, in that you just add up the effects of the different terms, or in programming terms you just have two loops :

for i = 1 to number_of_houses
  for j = number_of things_i_know_about_a_house
    work_it_out()
  end
end

Cost function plot from Octave

Which leads nicely to Octave a programming language designed specifically for doing maths, it combines with gnuplot to draw pretty pictures of your data. The sort of data we are talking about now comes in tables -whether printed, or as databases tables or spreadsheets doesn't really matter, as a programmer you would normally bring this stuff in as arrays of whatever dimension and do the above. Octave, however understands matrices and tables intuitively and can manipulate them directly, the above might become :

number_of_houses * work_it_out(number_of things_i_know_about_a_house)

A lot simpler and, because Octave is built for this sort of job, much quicker.

Onwards and Upwards. The top graph represents the point of all this, if you know the area of your house you can estimate it's cost. The advantage of getting a computer to do this is that you can have enormous training sets -all house prices in the country, and do lots of subset plots -all semis, all semis in Suffolk etc.

Machine Learning - Intro and Gradient Descent

Week 1 - In which we meet Prof. Ng and experience the delights of cost functions and Linear Regression with One Variable.

This is my first week on the Stanford University machine learning course and so far so good -as the man said when he jumped off Nelson's column. Prof. Ng seems to be a pretty decent lecturer who doesn't over estimate the ability of his student and there are plenty of examples and explanations to drive the points home. The lectures come as videos with a few embedded questions to keep you awake, and they are, thankfully, split into bite sized chunks of 10 -15 minutes.

In the intro we get introduced to terminology -the difference between supervised and unsupervised learning. Supervised learning requires a training set, the main  example given is of house price history vs. house size; unsupervised is just trying to make inferences from a bunch of data -say clustering news stories. We are also given the difference between regression (line fitting, trending) problems and classification ones (sorting into buckets, true/false &c.).

Thence we arrive at Linear Regression with One Variable -which is line fitting on a 2D graph basically. The maths arrives, but not too brutally -calculus shows its head, but fortunately you don't have to understand the whole of calculus to use the little bit we want, so all good.

As well as the videos there are also online review questions -which I believe contribute to the final score. The good news is that you not only can, but are actively encouraged to, retake them until you get a perfect score. The real benefit of this is that they become a learning tool rather than just a test, there are answers given as well as just scores and I think that they have helped me understand what's going on rather better than I otherwise would.

I'm now most of the way through the optional linear algebra review -but I won't post on that.