TDD - WTF?

Test Driven Development -good or bad? A friend went for an interview recently with a leading agile consultancy, and their mantra was 'we don't want documentation -we just want TDD'. I was mulling this over and I don't agree.

'Pull the switch' -but will it work?

Documentation -all bad?

There are two problems with documentation :

• It's always out of date.
• No one reads it.

Too much documentation is a bad thing -I once worked on a project where we had to document everything in pseudo code at the head of each file. It was disastrous, code reviews were done on the pseudo code, rather than the real thing, it was always out of date, and you wrote the same thing twice -but differently because you ran into language features that forced you away from the pseudo code. In an extreme case I thought of a better way of doing things and implemented that rather than the pseudo code, the boss wasn't happy, but it did stay in production for at least a decade. The pseudo code also  didn't stop people doing stupid things, like implementing their own version of malloc() -which leaked like a sieve.

On the other hand I worked on another system with no documentation at all, even the output was in Dutch, a language I don't understand at all. I didn't need tests to tell me it was failing -it broke completely, pretty much everywhere; but I could have done with something telling me what it was trying to achieve, especially since there was the possibility of a big bang if it all went wrong.

On the other hand good documentation addresses the 'why' rather than the 'what' and 'how' of the code. It should tell us what a function is for or why we chose a non-obvious way to implement something, or highlight a kludge.

Oh yes, RTFM -you know it makes sense.

Testing -all good?

Well yes, generally, although not without cost, test suites have to be developed and maintained and sod's law says that the one thing that breaks there won't be a test for, but that is when you create regression tests. So unit tests, acceptance tests, regression tests you need them all.

If there is a downside it is an excessive focus on test results. Development teams can have a habit of thinking that their job is done when all the tests pass; clients tend to get annoyed if the system still isn't doing what they think it should.  They can also fix failing tests without regard to the rest of the system, 'ok test 1234 went green, but now everything is 50% slower? You think that's acceptable?'

Test Drive Development then?

I think it can work as part of a development process, but it really only addresses the 'what'; as in, 'what is this code doing right now', passing or failing? It says nothing about why the bit of code is there or even how it works and how well it works, in the language of my youth -it's  very 'bottom up', an approach that was being maligned with good reason thirty years ago.

If you fit it into a larger process -say Agile Modeling or BDD, where you have ended up with user stories, tasks and all the rest, then TDD is a valid way to go, although personally I don't think it's any better than writing code and creating unit tests afterwards. I suppose the advantage here would be that with TDD you would create the tests, otherwise you may not, as with documentation there's always something more interesting to do, but it still seems to me to encourage looking at trees rather than the whole wood.

Would I use TDD?

I have given it a go on a couple of small projects, but in the end I fell back to coding first and writing the tests second, it just seems a much more natural way of doing things. I may refactor a lot at times, and this would mean rewriting many of the tests as well. I guess I like to get a prototype working and the go back and revise towards an optimal solution, at which point I'd put in the tests. How about you?

Mobile Implementation Plan

It's almost time to start coding our  mobile  app! But we need an implementation plan.

The plan is -there is no plan. Or as we used to say diving -plan the dive, change the plan! Which is why I spent half an hour in the Port Napier sitting in a corridor waiting for the mud to settle on one occasion, and spent 30 minutes another time getting a singularly dull view of the bottom of the hull. Both times we missed the excellent dive through the middle of it.

Mucked it up twice and it's so big!

Both these times proper application of the 5 Ps would have resulted in a good dive, or even the ability to tell left from right on the second dive. That's Proper Planning Prevents Poor Performance to the uninitiated, for the initiated there's a sixth P -but this is a family blog.

Time for a plan then?

Well no, not a formal one anyway, it's easy to start planning too soon. I was working at GEC when the Nimrod AEW project collapsed, fortunately not for that bit, but £1 billion, in 1980s money! Part of the problem, I was told at the time, was the early specification of the hardware, and requirements that changed.

Anyway we're all Agile and Scrum now, we don't have plans we have backlogs, we don't have delivery dates we have sprints. So, in the spirit of Agile we'll start a product backlog and a sprint backlog although the latter will be pretty flaky as I don't know how much time I'll have to spend on this.

Or should we go Lean?

If I was in the day job, leading the team, we'd have a bunch of meetings to thrash out the product backlog with the rest of the business, playing by MoSCoW rules. Next we'd get together as a team, do the poker planning and work out what would go into the sprint. Since we're a self-managing team people would take ownership of tasks (unless I decided otherwise -that's what they pay me for.) and we'd disappear into our sprint.

But you know what? It's just me, I think we'll just have a list. Since we're at an early stage and just getting used to the technology we can start with :

1. Get camera working from within app.
2. Save image.
3. Upload image (somewhere).

and that gets us to the holy grail of the Minimal Viable Product!

Mobile Scanners - The Competition

In this post we take a look at the competition. One of the drawbacks of the app stores is that there is an app for everything and sure enough it took seconds to find a couple.

CamScanner

Camscanner Screenshot

Camscanner in play store

• Scanning good, easy to use.
• I don't like the layout of the home page.
• Cloud integrated with their own cloud storage
• PDF generated
• no OCR, or if there is it's well hidden
• Website so-so, nice and simple to the extent that it's using the back button for navigation, seems like it's a bit of a Beta.
• App is missing confirmation here and there, e.g. pressed sync nothing happened. It did sync but I couldn't tell without looking on the web site.

 

Handy Scanner

Handy Scanner Screenshot

Handy scanner in play store

• Basic scanning and sharing.
• PDF generated
• No organisation of documents
• No cloud storage.
• Simple but limited. 

Both of these apps do a decent job of capturing the image and doing some basic alignment and cleanup. With both you can easily share the document, that's the point at which HandyScanner stops. Camscanner carries on with some basic groupings, but they don't seem very helpful, you can add notes which is useful and missing from my app, but you can only tag on the website.

I think that this leaves me room for better document organisation and tagging and also for introducing OCR. That said, if you use Google Drive as storage it will attempt to OCR the document, although it failed with both my test examples.

Should I carry on, or give up and wait until Google runs our whole lives? Why is it not surprising their data centre software is called Borg, Orwell could only dream.

Mobile Design Patterns

despite my rant, I am here to design and build, (or rather build and design, since I'm a techie and not a fluffie) a mobile app so it would be good to look at some navigation patterns for it, with a view to stealing them.

We were pointed to http://www.mobiledesignpatterngallery.com as a source of inspiration and asked to pick out 3 relevant ideas.

Home Screen

How about this for a home screen?

• This roughly matches my idea for a home screen.
• It might be better to split search from capture in the way notification and search are split here.
• Use of colour and icons is good, even if the colours themselves are a bit insipid.
• The numbers against the icons could represent documents stored against a tag for me.

Tabbed Screen

Listings

I hadn't thought of this, but it's a good way of expanding the desktop beyond the screen.

• Could be good for listing tags and the documents belonging to them
• Limited number of tags across the top but could swipe
• Scroll down documents
• We already had concepts of place and time in the original design

Search Results

Filtered search

I thought this might do for the search results :

• Icons would be tags allowing you to filter the search
• Would anyone use it?
• Techies love search driven sites.
• Users only use Google.
• Is it better than the foursquare screen, which could be adapted to the same job
• Is the overlay too big, although what's underneath isn't what the user wants to see?

 Answers on a postcard please

• Do users use the search in a mobile app? They don't use site specific search on the web.
• Is swiping and scrolling going to make users of the tabbed screen seasick?

It's just an effing phone -get over it!

I was supposed to write a blog post about mobile design sites -but when I looked at them my heart just died, these mobile designers have less interesting lives than trainspotters -a black turtleneck is the new blue anorak.

It wasn't cool when I bought it

As you can see from my phone, I may not be the best person to write about mobile design, it's old, it's bust, I spent 0s arranging my desktop and the only reason the background isn't the default is that my daughter changed it. In fact the only up to date thing about it is that Evernote got hacked last week.

What do I use my phone for ( compulsory list coming up ) :

1. Making phone calls -your phone really should do this -please take note Apple.
2. Sending texts, the soft keyboard is actually good at this, predictive text is brain dead in this version, the one on the playbook is much better.
3. Browsing the web -of course not, it's slow and far too small, that's what tablets are for.
4. Email ditto desperation only.
5. Reading a book ditto.
6. Listening to music -could do, but don't.
7. Checking train times -something useful, at last! You can find out when your train should have been here.
8. Stalking (sorry social media), not since the restraining order.
9. Navigation -was good but the GPS has never been the same since I went up Snowden with the phone in an open pocket. It was Wales, it was raining, no need for point 10.
10. Checking the weather -well I am British (and fiercely proud of it!)
11. Angry Birds -was amusing for 30s. now please fire the lot into an erupting volcano. I am over 12.

So I went looking at these mobile design sites :

Tech mobile blogs:

• http://beautifulpixels.com/
• http://flyosity.com/blog/

Design Inspiration:

• http://www.mobile-patterns.com/
• http://inspired-ui.com/
• http://www.lovelyui.com/
• http://pttrns.com/

They are Dull with a capital Duh, just lots of screen shots of mobiles with apps open in them, compare that to what the trainspotter sees :

I think that's my brother!

So my media-bespectacled brethren, throw off your turtlenecks, embrace the nutty slack and get yourselves down to the nearest sidings before it's too late. You've probably got longer than you think, see point 7 above.

Of course there is

Should this post have raised any issues, or upset you in any way please feel free to comment below, there are charities which can help.

Balsamiq Mockups

Following on from last week's paper exercise we now get to create some working wire frames using Balsamiq. Balsamiq is a pretty good tool for doing this and can now export pdfs with links between mockups -which gives you a prototype you can drive. The only downside is that it's expensive for just playing around with, at $12 / month, although it's cheaper if you download the tool -or use the GoogleDrive plugin.




This is a minimal working version of the app -enough to allow uploading, storage and retrieval. Although I didn't carry out the heuristic exercise -maybe at the next meeting of the Likely Lads- I did find a few obvious holes, such as the lack of a home screen. In addition when designing this I noticed some inconsistencies in the interface that I will need to go back and fix.

Following Sam's feedback on the navigation, the usual Android nav. is 'assumed' -which means I was too lazy to link up the back button and the home button, I also forgot to link up the trash.

On the choice of icons vs. a one liner list, I decided that I could get more icons on the small phone home screen. In the actual app the home screen would preferentially display a category of items that the user picks, and perhaps the actual icon could change based on a primary tag?

In the real world I'd probably do both and A/B test, or have it configurable.

Storyboarding and Prototyping the Mobile Filer

Problem Definition

In last week's need finding post I came up with the, not particularly original, idea of a mobile filer, this week it's time to prototype that. The first thing we want is a point of view, a very high level problem definition :

I want to record or copy documents and other objects and file them so that I can easily find them.

Having a definition, we flesh things out with a couple of storyboards, at this stage we are still in the 'fail early' mindset -trying to get something that looks like it will work.

Storyboarding

I created two storyboards, showing the flow through the application.

First Storyboard

In the first storyboard the user organises his files on a computer, this gets around the challenge of the small screen size  and the poor interface of the phone.

Second storyboard

In the second version we accept the limitations of the mobile device, on a tablet these wouldn't be so bad, and do the organising there. This means that we do the job in one go.

Ideally we would be able to use stuff we already know such as the date and time, or the location or type of image, or stuff we can work out -perhaps via OCR to carry out much of the classification automatically.

Paper Prototypes

Ideal Input System

 The ideal input system above may look a bit facetious (moi?), but it serves to illustrate a point, if this system was smart enough this is how we would like the input to work.

Likely Input System

A more likely system lets the user choose between our two storyboards above, if he presses 'upload' then the image will be filed immediately, and he will probably have to sort it out later. Otherwise the user can tag the image -default tags are time stamp and location.

Finding Notes

To retrieve documents the user selects a category of note and drills into it. I have just realised that I need a way to move notes between categories, so there would be an edit screen in here too.

There is also an opportunity for a simpler output system, which I had better sketch up now.

Search Based Interface

In this version  the user gets a set of important (flagged?, latest?) notes up front and access to the rest of the documents is driven through search.


If I get a chance I'll put up a video demonstration -but life is getting in the way.

Needfinding

The task was to observe some people and derive a product by finding a need that they had.Unfortunately most of the people that I interact with regularly aren't suitable subjects -I had an idea based around homework, but understandably schools won't let you in to observe the pupils; the other idea was based around the elderly but similar problems arise and, in addition, there comes a stage where the new technology you are going to be picking up  is a Zimmer and not an iPhone.

Capturing a document, need to share it

In my interactions with these two groups I did observe some tasks that I was carrying out, so I decided to use these for my needs. Both groups get documents that need to be dealt with, say homework for one and hospital appointments for the other, this generates a lot of paper, people need copies and all the usual. I found myself taking photos of these, as did my daughter -who promptly deleted it highlighting the need for off-phone storage, with our phones. So far so good, but the filing is where it falls down.

Document upload and filing -there has to be a better way.

The document has to be got off the phone, one way or another, then filed somewhere locally or remotely and found again when you need it.

Needs and Goals

Here's a brainstorming list I came up with :

1. Quick
2. Simple
3. Easy to use
4. Documents are easy to find
5. Categories
6. Tags
7. Date/time stamps
8. Location stamps
9. Safe
10. Secure
11. On hand -don't have to find a computer to access.
12. Shareable
13. Feature extraction for filing/tagging
14. OCR for filing/tagging
15. Social coz you have to
16. Calender integration

Inspirations

Five things that might help define the solution :

The fridge wall of information

Important household information is posted in a known, obvious and prominent place.

Filing Wallet

The old school definition of organised, either alphabetically or, as this one is, 'tagged'.

Filing Clerk

Someone to get the piece of information you want. They may have their own system specifically tailored to where they work. Slobber extra.

It should be, one click is a bit of an ask but we need to get as close as possible.

Shiny, the app should look sleek and crafted -which will be a challenge for me.

Analysing an Online Community

I had a look at carrying out network analysis of the relations in an online community for the Coursera Social Network Analysis class.

Original Bulletin Board Graph

The community was a UK based bulletin board of general chat, completely anonymised. I wanted to look at the relationship between topics and see what could be inferred from that, so in the graphs the nodes are threads and the edges are people who posted between threads. This post is mainly about the network analysis, another will look at nitty gritty of how I carried it out.

Tools

In short, Python and Gephi -more details later.

The Network

I ended up with a network of 79 nodes and 345 edges. The node labels were extracted keywords from the threads -edges were people who posted between threads.

Analysis

First question is there anything to look at? Any Structure? One way to tell is by constructing a random graph and comparing it's properties with those of our graph -this can be done, a little tediously, with Gephi.

Our graph has an average degree of 8.73, a shortest path length of 2.22 and an average clustering coefficient of 0.634. In comparison a random graph with same node and edge counts has a degree of ~4.4 a path length of 2.2 and the clustering coefficeient of 0.144 degree is considerably lower than our graph, so nodes are more connected and, although the path length is similar to a random graph the clustering is higher, indicating small world properties -which gives us something to look at.

There are a 3 super nodes in the graph :

Node	Betweenness
clarkson, people, one, bbc	1015
people, like, money, would	650
clarkson, one, public, sector	343.

People in the UK can probably guess the reason for the first and third nodes. The comment was so outrageous that perhaps most people felt that they had to say something about it, and abnormal relationships in the graph could be created. After removal of the clarkson nodes the graph is as below:

Clarkson free graph

Running Gephi Modularity with a resolution of 1.0 gives us 6 groups. Groups 4, 5 and 0 seem to be too diverse to say anything much about. Group 1 is mainly about sport, although its' highest degree node is about Apple. Group 3 is about UK domestic topics, Group 2 is more global centring around politics and economics.

Summary

We can see that there is some clustering of subjects when we relate them by people although we have not discovered any surprises on this small data set, as similar subjects cluster together. Next steps for this study would be too use a more sophisticated way of extracting the labels from the thread text, run against a larger data set and perhaps introduce the notion of time as an attribute, one could then perhaps look at 'contagion' of the network by a topic, the Clarkson posts might be interesting here. We could also invert the network and relate people by posts -but that seemed less interesting.

Ubuntu Cross-Platform Mobile Setup

I had 3 candidates for a cross-plattform development framework, Cordova, Appcelerator and Corona not knowing which to choose I looked at all three.

Corona

I liked the look of this framework, but it doesn't work with Linux (although it could be possible to make Corona work with Ubuntu with a little hacking.). It's Java based (remember 'write once, run anywhere'?) so I don't know why they don't fix it, but that's one down.

Cordova

Cordova on Linux

There aren't instructions for Linux -but the Mac ones looked reasonable so I thought that I'd give it a go. I already had Android Developer Toolkit installed so I adjusted the paths accordingly, ran the create script, and it fell flat on it's arse. Not a good start. Did the obvious and fixed the permissions, still no joy. Had a look at the script and several lines ended with '&> /dev/null' -hmm. I write scripts that go '1>/dev/null 2>&1' so I assume that is what these mean. Maybe the other syntax is a Mac thing, but on my machine it ends up with processes being backgrounded that shouldn't be as the script then tries to remove files that aren't downloaded yet. I deleted the  '&> /dev/null'  and 'create' generates a project.

Fire up the ADT , based on Eclipse (oh well), open up the project as described in the instructions, Eclipse hides it from me, eventually find the right button. Attempt to deploy the script ADT dies horribly, there are a few libcurses5 errors that I have been ignoring. Turns out the Android SDK is 32 bit, wheras I have a beefy 64 bit machine, 'apt-get install lib32ncurses5 lib32stdc++6' sorts this out for me. I fire up ADT redeploy and hey presto!

Appcelerator

Titanium on Linux

They make me sign up, and then immediately want an update, some smart marketeer on the job counting the number of fields that we'll fill in at once. Having done that it's off to the Titanium Quick Start Guide.

Next we download, unzip and run Titanium Studio (again Eclipse based), so far so good, the first thing it wants to do is update itself and it runs into a problem with nodeJS. Fortunately this is documented and after following a couple of links I run :

   sudo add-apt-repository ppa:chris-lea/node.js
   sudo apt-get update
   sudo apt-get install nodejs npm

and we are off to the races.


I was able to point the SDK at the one in ADT, which saved a bit of time, although I also installed Android 2.2 as that's what my ancient phone uses, and then ran through the rest of the Quick Start with the result above. The emulator takes forever to fire up, but Eclipse looks a lot more rational than it does in Cordova.

Brainstorming and Prototyping

This week's design exercise for the HPU Mobile course was around the beginnings of the process of delivering a mobile app -getting a concept and making a prototype.  A good introduction to this is in Scott Klemmer's HCI course.

Brainstorming

The challenge was to come up  with 20 things to do on your mobile whilst waiting in line, a crack brainstorming team of myself and the trainee woman was assembled and this is what we came up with.

1.     Gambling  (bingo, roulette, fruit machine)
2.     Vouchers (did this one for real)
3.     Find someone to have a coffee with
4.     Browse Comics
5.     Listen to Jokes (Professor?)
6.     Local News & Events
7.     Watch a clip (trailer, sports, music vid)
8.     Gossip
9.     Update Shopping List
10.     Take a virtual tour.

Language learning -about item of shopping?     Play a game     Read Magazine (e.g. Hello! guess who?)     Diet program linked to your shopping     Styling app -beauty products on your photo     Test the Kids app. (for mums in the line)     Recipe for item of shopping.     Queue switcher -the other queue is always faster.     Interactive Story     This has been deleted for reasons of taste, let's say dating app.

Prototyping

The next step was to build two paper prototypes. Firstly, I chose the Queue Switcher - it's simple to explain, it's a bit like a game (come to think of it there could  be a collaborative Queue Racer version) and I haven't seen one before. Secondly, it was the Recipe maker -because when I am standing in the supermarket staring vacantly at the food shelves I could use one. It is one of the perennial problems of family life -what am I going to cook tonight?

It seems the Queue switcher does need a bit more explanation after all, it's based around the perceived wisdom that the other queue always moves faster than the one you are in. The app lets you track that and suggests when you should move queues, based on their relative speed -of course we all know that as soon as you leave your original queue it speeds up! Thinking about it now we could add in a bit of social allowing you to share the outcome and a bit of gamification -you too can be a 'Quintessential Queuer'!

Queue Switcher

Recipe App

User Feedback

Next step was to try out the prototypes on some users. I wasn't really up for buttonholing strangers in the street so I picked on the family :

Entering an ingredient

Generally they got the idea pretty well, but one thing I did learn is that it helps to screen your testers : My wife has never used even a feature phone; and the daughter doesn't really shop for food.

Recipe Dropdown

Insights Gained

1. Better window titling to flag up what each screen is for.
2. Error messaging would be needed.
3. More interior navigation -having picked a recipe all the ingredients may not be available, so the ability to go back to the recipe list or the recipe search would be useful.
4. Predictive typing would speed things up.
5. The shopping list feature could just be a list, it was felt that the ability to tick items off, or remove items was unnecessary.

Fast Track to Mobile -Part Deux

This week we run up a couple of quick mobile pages and publish them on dropbox -not without it's idiosyncrasies- and install ADT the Android Development Toolkit.

A note on pronunciation -the dog jokes should be read in the voice of our late, lamented court jester -Spike Milligan.

Sencha App on Playbook

First off Sencha, I just copied the test page, updated the text and dropped it onto dropbox. The Sencha page looks reasonable to me -but it's just a test page. Looking at the source, it's a bit less friendly than jQuery Mobile for something this simple which implies there might be a slighly steeper learning curve. Time will tell.

jQuery Mobile page on Playbook

The jQuery Mobile page  also looks good, again it was just copied from the quick start. The code looks a lot more like a normal web page with the contents driven through named div blocks -view source on the pages and you'll see what I mean.

Dropbox had issues, or rather Chrome did, with using either 3rd party sources or mixing http & https requests -these were resolved by copying the css and javascript files into the Dropbox public folder so that they could be served from there.

Quick Delivery for Mobile

I have signed up for my friend Sam's Mobile Development Class, in the past I have built and run backends for mobile sites but have never had too much to do with the device side and his course seemed like a good chance to fill in the knowledge pothole. One of the assignments for week 1 was to write a 'contractual obligation' blog post -which is why you're suffering this; a second, more interesting, assignment was to watch this mobile dev introduction on Udemy.

Although it was for managers, shouldn't scoff I am one (sometimes),the set of lectures gave a good overview of some of the options for delivering mobile sites and apps.

Back in the day I was Head of Development for a company called Cantos, and made a now defunct mobile site for them, it broke when we put up the current web site , we did do a second version for the new Drupal site but it was never given the green light to go live, which is why I don't give a link to it.

When we were developing the mobile sites we did a lot of googling around looking for tools, guides and advice and I wish the Udemy video and the options it mentions were around then -we might have saved a lot of time and we would have been much more future proofed.

Tools like Cordova / PhoneGap offer a quick, cross-platform, entry to the market and Titanium a short cut to the native app experience. For the sites we built jQuery Mobile would have allowed us to offer a richer user experience to our customers. The video lectures discuss the pros and cons of each technology and makes suggestions about what to use when, including when you should go fully native. All in all 50 minutes well spent and I look forward to playing with all of them.

The power of the Matrix

Do you want to make your code run 100 times faster, at least in python? Try rewriting it in matrix form using numpy.

I came across this when implementing the Floyd Warshall Algorithm for finding all shortest paths in a graph as part of th Algorithms 2 course at Coursera. The guts of Floyd Warshall are 3 nested for loops that iterate over all the nodes, here's the Python code :


# v_sz = number of vertices
for k in range(1,v_sz) :
  for i in range(0,v_sz) :
    for j in range(0,v_sz) :
      new_len = A[k-1][i][k] + A[k-1][k][j]
      old_len = A[k-1][i][j]
      if new_len < old_len :
        A[k][i][j] = new_len
      else :
        A[k][i][j] = old_len


This takes around 30 minutes to run on my PC for the supplied graph of 1000 vertices.

Other people were getting much better times and the hint was given to try rewriting this in matrix form -took me a while but I had used Octave as part of the Machine Learning and Neural Nets courses, and together with the Numpy for Matlab Users page I got to :

for k in range(1,v_sz) :
  #tile() is the equivalent to repmat -copies a vector repeatedly into a matrix
  i2k = np.tile(M[0:v_sz][:,k],(1,v_sz))
  k2j = np.tile(M[k][:],(v_sz,1))
  M = np.minimum(M, i2k + k2j)

This runs in ~10 seconds, a good chunk of which will be loading in the data! Pretty impressive and there would be more to go as I think I can shave time off the data load.

What it shows is that numpy is heavily optimised, probably through use of underlying C libraries, for matrix computation and that it is always going to be worth looking to see if a set of for loops can be recast into vector and matrix manipulations instead.