Chris Thorne

I’ve been reading 2 interesting books recently.

“The Lean Startup” by Eric Ries is a great book about how to do product management for a new product. In his book, he talks about the development cycle: Build Measure and Learn.

An illustration of the cycle of Build Measure Learn

- Build, Measure Learn. A new products goal is to repeat the build measure learn cycle as quickly as possible. Build a new feature, Measure the user response to that feature – did they want it? Did they use it in the way you expected them to?

If you measured the right things, you’ll know more about what users want. For each feature you can now improve the feature , keep it as is, or discard it. As you learn more about what users really want from your product, you are better able to predict which new features to build.

The quicker you successfully complete a build measure learn cycle, the closer you are to having a successful product.

- Your assumptions about what the audience wants are probably wrong. What a relief! Seriously, it becomes easier to reach agreement between stakeholders, if you able to admit that nobody knows what the right feature to build is. We can hopefully make an educated guess, but that is all. Until its built and you see users using it, no-one knows for sure.

I know from my own experience that a design always changes following user testing. You think you’ve delivered a great design, and there’s always something that needs to improve because users don’t want it, or don’t know how to use it.

Eric Ries recommends a book called ‘The Other Side Of Innovation‘ (written by Vijay Govindarajan and Chris Trimble) which is a great book for those of us trying to improve innovation within an existing (large) organisations.

In ‘The Other Side Of Innovation’ the authors convinced me that in order to complete the measure and learn part of the cycle you need to create ‘hypotheses of record’.

School science projects taught us about hypotheses, so this is quite an easy concept to understand.

Creating a hypotheses of record for a product

A hypothesis for product development needs to be focused on assumptions about user behavior. It also needs to be derived from the core business goals. In many businesses, the business is trying to provide a product or service that users want. By providing something that users want, the business can sell more products or services to the users.  The business goal is to get more revenue and profit, which it achieves by selling more products and services to users.

In the BBCs case, in my opinion, the business goal is to increase audience reach and engagement so that when a user is asked to purchase a TV license (which funds the BBC) they feel it represents excellent value for money. There are of course additional aims, notably to educate, inform and entertain, but without achieving reach and engagement, the organisation cannot hope to achieve these subsequent aims.

An example hypothesis for the new BBC homepage might be:

Hypothesis: Given the user experience of the new BBC homepage we expect most users to use the carousel buttons at the top of the page.

What does success looklike? If, with time, more visitors (as a proportion of the total visitors) to www.bbc.co.uk are using the carousel buttons then the design works as well as we hoped. As a subsequent success we would expect to see more clicks through to items within the carousel. Each click through to an item in the carousel represents an increase in engagement. If we see a low proportion of users or a decreasing proportion of users engage with the carousel over time, then we think the carousel design needs to change.

Other parts of a products design can similarly be considered – For a particular feature, what is the user behavior we expect, if that feature works? Which of our goals is this feature going to improve? If we can’t agree with a stakeholder about which business goal this feature will improve, perhaps this feature isn’t as needed as we thought it was!

Writing down a set of hypotheses and how each hypothesis will be tested, forms a “Hypotheses of record”. Once a hypothesis is tested, and the metrics obtained, then we can check our records to see if our hypothesis is correct.

Priorisation of hypotheses

Some of our assumptions are high risk and high cost (ie we don’t know if the audience want this new feature, but in order to deliver this feature it costs a lot of money). These high risk high cost features are the hypotheses we should focus on testing first. If the hypothesis about the user is wrong, we’ve spent less money developing it if we test it sooner.

[Testing a feature can be just interviewing users with a paper illustration of the feature and asking them if this feature is something they would use? Would it make them likely to use the product more often?]

Hypotheses of record in my experience

Recently ive been trying to apply this in my work (building the new bbc.co.uk/radio).

The combination of the ideas above has been very interesting.

In my experience, the conversation between stakeholders and development team can get stressful – particularly when it comes to agreeing requirements.

Typically person A insists that they need this feature, and person B insists that they don’t need that feature, and there is very little data with which to make either case.

Either person might persuade the other that their case is valid, and therefore agree the new functionality required.

However, the conversation and persuasion can be hard, complex and slow.

A “Hypotheses of Record” allows us to complete our measure and learn parts of the Build Measure Learn cycle: We can:

* Deduce what we need to measure for the hypothesis we are testing.

* Agree what success or failure results look like. We’re not going to base our decisions on a snapshot of results from our testing. We are looking for a trend in results over time.

* and learn from the results through proving or adjusting our hypotheses.

In the BBC homepage example, we might see only a few users navigating with the carousel when it launches, but if the number of users navigating the carousel is increasing significantly with time, we might say that the new carousel feature succeeds as a design. If we see the opposite trend, we might say that the carousel design is not working.

Each time we prove or disprove a hypothesis we have more accurate information, which helps us decide what features users want, and therefore what to build next.

This framework focuses stakeholders and product management on the assumptions in the current approach, and the desired user behavior if the product is a success.

These are very very powerful points to focus on – in my experience they shift the conversation from a sometimes delicate negotiation, to a more easily agreed route forward, where testing and evidence from the audience will be used to determine improvements to the product. When stakeholders and product management disagree about a feature, we can still agree the test and the user behavior we want to achieve, and therefore find a positive way forward.

Its still early days in applying this approach, but trying to “Build Measure and Learn”, and creating a “Hypothesis of Record” is proving very valuable in focusing effort and gaining agreement between product management and stakeholders.

An image of part of the BBC Proms website

Update October 2011: we’re submitting The Proms website to this year’s Webby Awards:http://www.bbc.co.uk/proms

A week ago we relaunched The BBC Proms website. This website and content management system was developed by Magnetic North, in collaboration with the BBC. I was lead information architect on the project.

Key features of the new website

Andrew Downs, editorial lead on the BBC Proms website development, adds:

In 2011 we rebuilt the BBC Proms website from the ground up.
We wanted to do a more streamlined and focused job of meeting, and exceeding, the online expectations of both the broadcast and live event audiences for the World’s greatest classical music festival.

The new website offers easier ways to book tickets, listen and watch, and reveals a wealth of in-depth content from the BBC archive as well as shorter, light-touch briefings directly related to each concert programme.

Key features from before the Proms season is underway:
Browse, book and listen by composer, performer or category
Discover more about the music with links on every event page to the Radio 3 online archive of programmes about works and composers.

For 150 of the works performed we offered preview clips of the music right up to start of the concert.
Sign up for the newsletter.
Search the archive with details of every Prom since 1895
Play the Proms quiz
Mobile-optimised version of the site.

When the season is underway:
Listen live in High Definition Sound, or listen to or watch Proms broadcasts on demand for 7 days.
Audio and video highlights of each week’s concerts in the Proms Showcase.
In-depth programme notes for each concert
Relive and share your favourite moments with photos, videos, reviews and comments on site or via Facebook and Twitter.
Go behind the scenes with the Blog and Q&A with the Director of the Proms
Download interviews with key performers in our weekly podcasts or download a daily 4 minute briefing on a key work from each event.

On TV press Red Button to watch Maestrocam with explanatory commentary.

Background to the design

We began redesigning the proms in October last year, with some user research work conducted by Engine.

One of the deliverables Engine produced was a set of Personas for each of the target audiences. There were 3 types of audience defined, which I summarise as follows:

• The casual enthusiast - This person is interested in the Proms but doesn’t feel confident in their knowledge of the music. They are looking for guidance and reassurance.

• The existing Radio 3 audience – Knowledgeable and articulate about classical music. Knows what they want, and wants to get to it quickly and easily.

• The expert enthusiast - Highly knowledgeable and articulate, but looking to be further educated, to learn more about the music they are choosing.

We also did some user testing of 2010s proms site, and user interviews, with users who were classical music enthusiasts (conducted by Foviance). In some of these interviews, users talked of how they would ’swot up’ before going to a concert, by listening to recordings, so they really knew the music they were going to listen to before they attended the concert.

Core objectives from the audience research

The above research gave us confidence when setting objectives for the redesign. We found out that when talking about and selecting classical music, the audience almost always talks about liking music from a particular composer. They don’t tend to talk about liking a particular piece, until they have framed it with ‘I like <composer name>’.

This encouraged us to make selecting a prom based on the composers name a core piece of the navigation (hence its presence in the top navigation).

Discover the music

The discover the music panels used throughout the site came out of our audience research. One of our business objectives is to increase the number of listeners to radio 3. The expert and classical enthusiast personas also caused us to develop the ‘Discover the music’ panels that are seen throughout the site. On each event page, the ‘Discover The Music’ panel is used to highlight relevant content, based upon the music being played at that event.

It helped that Radio 3 were given permission by the BBC trust to make more use of their fabulous archive of music programmes. For example, Radio 3’s Discovering Music programme involves a presenter/conductor showing the audience a well known piece of classical music, and performing parts of it with an orchestra. An amazing resource for users looking to ’swot up before they go to a concert’!

Starting the redesign – Content Management

The new site is built on the BBCs latest infrastructure, which allowed us to build a content management system. As usual, I took a domain driven approach, which meant that we started by defining ‘What things are in the proms?’ and ‘How are these things related?’.

You can see the final domain model we derived below:

The Proms Domain Model

Notice how this is not very specific to the Proms. A good domain model is about the domain, not the particular brand or project. What we have here is an approach for building any music events based website. So, our content management system is reusable across any music events site.

The concepts shaded in light blue are concepts that have a controlled vocabulary of terms – for example there are approximately 80 terms that can be used to define a performers role in a performance. They can have the role of ‘guitar’ but not of ‘guitarist’. This helps us when building aggregations of performers. For example, our Composers A-Z shows all contributors with a role of ‘Composer’.

Performance Flags allow us to provide emphasis to a particular performance – for example if it is the world premier, or the UK permier. This obviously helps the user notice certain performances more than others.

Similarly, a contributor can be flagged as someone that Radio 3 supports through its ‘New Generation Artists’ programme.

Where to put the effort in a Content Management System?

One of the strategic decisions to take when defining a CMS is about where to put the effort. The approach we took in this CMS is all about the reuse of the content. A lot of the effort in this CMS goes into populating and curating the ‘Works’ information.The reason for this is that the Works information can be reused in subsequent years. For example, editorial staff can associate an audio preview with a work. Next year, when that work is performed, the audio preview for that work will be already populated. We know that in classical music, a lot of the works are regularly performed, so putting effort in here makes strategic sense.

Contributor information is also reusable in subsequent years, so adding related links for a particular contributor is efficient use of effort.

A final example of careful Content Management design is that because many of the classical music composers are dead, we can reuse analyses of their music over many years. So, by associating an analysis with a work, we can reuse that analysis each time the Proms performs that work. Episodes of the ‘Discovering Music’ programme are each an example of an audio analysis which we can reuse each time that work is performed.

The easy alternative would be to put a lot of effort into creating beautiful event pages, but without contributor, work and performance concepts, this effort would have been very hard to reuse in subsequent years.

The website navigation

One of the navigation elements I’m most pleased with is the consistent event navigation on (almost) every page:

Event navigation is used on almost every page to help the user navigate the site easily.

The launch and audience reaction:

It has been particularly pleasing to me to see the audience reaction. A significant change from previous years was that we decided to build ‘a page per prom’. This lets users talk about, link to, blog about and generally point at an individual prom. They have!

It’s been great to see on twitter the number of users linking to parts of the proms site. For example:

These and many other coversations vindicate the strategy of ‘one URL for one thing’ which we followed on this project.

There have also been some great articles talking about their favourite proms. And more great articles like this.

I’m looking forward to seeing how ‘discover the music’ panels get used – I’m hoping that this feature on the new site encourages the users to listen to Radio 3, and then to get to a prom and hear a great classical music performance.

It’s been a while since I did a blog post. I thought I would write about one of my favourite projects of the last few months.

Quizmaster is a new content management system for the BBC which is specifically focused on multichoice quizzes. Using it, content producers at the BBC can quickly develop and deploy a quiz. The quizmaster platform publishes to facebook, flash, ajax and mobile (Ajax and J2ME) platforms, and is fully accessible.

The project started in March 2009, with a small (but perfectly formed?) company called Kite.

[you can also read their take on the quizmaster platform.]

Back in March we were asked by BBC Radio 2 to develop a multiplatform quiz for the Ken Bruce show, an online version of his ‘Popmaster’ quiz.

As the BBCs Lead Information Architect on this project, and part of the ‘External Projects and Integration Consultancy’ team, it was my job to shepherd the project through to launch. It’s quite a simple looking quiz, but arriving at the final result was a long, and surprisingly convoluted journey.

I can’t take much of the credit for it really (most of that goes to Kite!) but there are bits I’m particularly pleased about.

1. It is fully accessible.

We tested the different versions of the quiz with a wide variety of audience. This included testing with blind and partially sighted users (result: we increased the default amount of time you get to make an answer, and increased the vertical distance between the hot spots for answer buttons in the design). Kite also rebuilt the flash version to be more accessible using tab/enter keys).

The vertical distance between answers increased as a result of user testing

We also did testing with autistic users (result: Some of the comments the presenter makes were made less negative, since autistic users we tested had difficulty recognising them as humour and got upset by them. This is something that I was surprised about at the time, although with hindsight it seems obvious – it’s one of those results that makes me love user testing!

Other interesting bits – the options screen, accessed from the quiz home page, and its impact on the quiz (again trying to keep it accessible).

[But not everything can be made accessible]

The last point about accessibility is the timer. We could have taken the timer out for some parts of the audience (for example if you have difficulty reading the answers within the time limit). However, it wouldn’t be the same game.

Taking it out would fundamentally change the offer to the audience, and the core principal of the BBC when it comes to accessibility is to offer as many people as possible the same experience.

Hopefully, with this project, we achieved that. I hope that the audience with accessibility needs enjoy the resulting experience. If not, let me know, and I’ll try and get quizmaster adjusted.

2. The design is flexible, the technology is reusable.

The same technology platform has now produced many different ‘multichoice quizzes’ for different parts of the BBC, all of them pretty quick and easy to produce, and fully accessible from the start. It would be easy to build a one-off product that delivers a small amount of functionality to bbc.co.uk. I’m pleased that we designed and built a product which has been reused many times across the BBC.

Here are some examples of quizmaster in use:

• Popmaster – the original, flash version.
• Popmaster – the ajax version (more accessible for screen readers)
• Popmaster mobile versions (j2me and ajax)
• Popmaster on facebook

• Glastonbury 40th Anniversary Quiz – Flash version here
• Radio 3 Proms Quiz- Flash version here

And an interesting extension of the original version – the 5live world cup shootout quiz (with a tournement/football match structure).

This version of the game was playable on iPhone and Android mobile phones.

An example question screen from Radio 3's Proms quiz

The Evolution of the Quizmaster

You’ll notice some subtle changes through the different versions, as we try to refine the experience. For example, the proms and glastonbury versions score points for each tenth of a second that is left on the clock when the player answers correctly. In popmaster we scored for each second. This resulted in a large number of players having the same score (not good). The 10th of a second scoring gives much more variations in scores.

Social integration

My favourite way to play these quizzes is on facebook, because it’s really easy to invite your friends, and I think its when comparing scores with friends that it becomes truly fun! [even if like me, you're rubbish, so score really badly!]

My favourite user journey is ‘Send to a friend’  from the home screen (after you’ve decided to play with sound or not), or as I called it, the ‘I forgot to send this quiz to my mate, but I like it and I don’t want to play the whole game before being able to tell my mate about it’. That button (and rather wordy journey) is one I’ll take credit for . I think its actually quite important in improving the chance that the quiz can ‘go viral’ amongst internet users.

Let me know what you think.

On the 18th of January the BBC launched ‘A History of the World in 100 objects‘, a series of 100 programmes, narrated by Neil MacGregor, Director of the British Museum, focusing on 100 objects from the British Museum’s collection, travelling through two million years time to retell the history of humanity through the objects we have made.

A history of the world - example homepage

I was the Lead Information Architect on this project. I thought you might be interested in how and why some of the decisions were made .

The project was built and delivered by goodtechnology in collaboration with the BBC. There were 3 of us in the user experience team. Yasser Rashid (Creative Director) , Tom Spalding (Senior Design) and myself (Lead Information Architect). There is an excellent post here, about the design process we (and other colleagues) went through for this project.

For the ‘A History of the world’ website, the BBC and British Museum decided they wanted to showcase man made objects. 350 museums from around the UK contributed objects, and the central function of the site is to encourage users to explore those objects, and contribute their own objects they care about.

I’m proud that in 3 months, we built a moderation system, an editing/uploading system, and a public facing website. Given the complexity of the site, I’m pleased with what we achieved.

This site is domain driven (hooray)

I have been in the department a couple of years, and have swallowed the domain driven design pill, together with the ‘Cool URIs don’t change’ pill and the semantic web pill. So, when it came to defining the information architecture for this site, I wanted to apply domain driven design. You can see the domain model I sketched below:

A sketch of the domain model

The websites core journeys are:

1. Browse the objects contributed until you find one of interest.
2. Examine an object of particular interest in detail (this is done on the object page).
3. Where an object has a broadcast episode associated with it, users can listen to that objects programme whilst examining the object.

For journey 1, the flash-based ‘object explorer’ can be used, or an HTML based ‘basic explorer’ can be used. Each of these explorers allows the user to set filters, reducing a large number of objects down to just a few. Each filter is represented on the domain model. This allows us to work out what user experiences are possible, what data is required, and the relationship between the filters. If we’d had time, we could also have used the domain model to calculate persistent URLs.

Only 2 filters at once.

To try to ensure that the user is never presented with ‘no objects’ we decided to only allow the setting of a maximum of 2 filters at once. Given the hundreds of objects, it is therefore more likely that the user will be presented with some objects to explore, once they have set filters.

For journeys 2 and 3, we created an object page for each object in the site.

An example object page from 'A History of the world'

Our Object page URLs are designed to last 200 years.

The Cool URIs don’t change paper (by Tim BL) suggests that we should design URIs as though they won’t change for 200 years. It’s a point of view I try to subscribe to. Unfortunately, due to time constraints, it wasn’t possible to apply this principle as purely as I would have liked. However, for journeys 2 and 3, we were able to ensure that each objects page was published at a persistent (’cool’) URL.

For example, the “Mummy of Hornedjitef ” has a URL of http://www.bbc.co.uk/ahistoryoftheworld/objects/sogITE3FSKStlk12qd2W3w.

This can be generalised to:

www.bbc.co.uk/ahistoryoftheworld/objects/:object where ‘:object’ is a GUID.

I guess this is an example of practicing what I preach, since I’ve talked about persistent URL design before (as have my colleagues).

We could have published an object page to:

www.bbc.co.uk/users/:user/objects/:object , but the person who uploaded the object may want to change their name (:user). By using a GUID for the object, we also allow the name (and all other taxonomy attributes) to change without changing the URL of that object. In this way, I hope, Ive designed a URL that can last 200 years without changing .

By using a GUID we are also better able to scale the site – we can allow many users to upload their objects concurrently, using multiple servers at the same time,  safe in the knowledge that they will not be allocated the same URL as another object, because the GUID is pretty much guaranteed to be random and unique.[OK, I'm believing what the software engineers tell me, and also what wikipedia tells me, but hopefully its true!].

Search Engine Optimisation

By ensuring the URLs don’t change over time, we can ensure that more people can link to the content, which boosts that contents google ‘page rank’. I’m pleased to see that when users search for a particular object in google, the object pages appear high in search results:

But still much to improve…..

There is lots to improve on the site, and we will be striving to make improvements over the coming weeks, but I hope you enjoy the site. I know I’ve particularly enjoyed the fact that the site seems to be attracting local newspapers to write about local history - and some of the objects users have chosen to upload have been genuinely interesting to me .

Building the moderation, admin, upload and public facing site in 3 months was an awesome challenge. The team involved were truly great to work with. The good technology team, the radio 4 team, my UX+D colleagues, and many others. I thoroughly enjoyed it.

Update: 10th Feb 2010: A History of the World has just been awarded ‘Website of the week’ by New Media Age magazine. Cool! Full review here.

One of the aspects I found most interesting this year was the number of Information Architects who are no longer Information Architects. They started off as one of them, and became a ‘Product Manager’. Christina Wodtke – a ‘big cheese’ in the Information Architecture community, was one of the founders of the Information Architecture Institute. Yet, look at her ‘linked in’ profile today and it says she is ‘Principal Product Manager’ at linked in.

One of the primary reasons seems to be that if you’re just an information architect in an organisation, the usability (one of the things you care about as an IA – don’t you?) of your work can be cut. Too often we see project managers decide that when something has to be cut, the something they choose to cut is the usability.

In an agile project process, it is easy to claim that you’ve answered a user story – ‘The user can add blah to a web page’. At the end of a sprint, the project manager can tick the box. Unfortunately, what isn’t documented is that when the sprint started slipping, the usability of the feature was compromised. You’ve shipped ‘lipstick on a pig’. Yes you can tick a box that say’s you’ve completed a user story, but your target users find the new feature hard to use, frustrating, and in some cases, they refuse to use that new feature that was supposed to have been built for them.

It is one of my frustrations that, in an age where companies like Apple show that prioritizing usability throughout the business can make you significant profits, project managers continue to see it as a ‘nice to have’ a ‘well, if we have time.’

What’s the answer?

Well, I have 2 suggestions:

1. In Agile sprints, user stories should include usability statements. For example ‘A user can add feature blah to the website within 30 seconds of starting the add feature blah process’. Perhaps this should be the second sprint – the first being ‘A user can add feature blah to a website’?

2. User experience professionals could, like Christina Wodtke, become product owners.

If you’re a product owner (or CEO of Apple) you get a lot more influence over what gets implemented.

As I suggest to my IA colleagues next time the usability of their product gets crippled by a project manager/scrum master, climb the greasy management pole…..when you call the shots, you can ensure usability is prioritised.