As part of the ongoing redesign of the Cumbria County Council website using LocalGov Drupal, we have been looking at the highways section.
As a content designer, one of the first things I do when starting to look at the content, is to examine:
- the data (for example, the number of page views, average time spent on page, percentage of people who view a page and immediately leave)
- the structure
- how easy it is to find the content, through search and through navigation
- how easy it is to read and understand the content
There are a number of tools available to help with this process, one of which is tree testing. Two of us worked on this tree testing process:
- Ben Hills-Jones, Content Designer
- Kayo Hayashi, User Researcher
What is tree testing?
Tree testing helps to assess the ‘findability’ of items within a website. The test presents users with a stripped down structure of the information architecture (IA) and it evaluates how easily users are able to find content based solely on the categorisations/ page labels.
In order to run this test we used an online user experience platform called Optimal Workshop. We decided to use this platform for tree testing as it provides a comprehensive breakdown on user pathways, but also useful visualisations on overall performance. The tree test can also be embedded into a wider survey so that we wouldn’t have to send out the research to respondents in two parts.
Metrics that are measured in tree testing can include:
- whether users were able to complete the task (success or fail)
- how long it took them
- how efficient they were (direct or indirect)
The performance on these metrics can tell us which page labels or parts of the information architecture users are struggling with.
What we evaluated using tree testing
One of the key pieces of work I wanted to evaluate and validate was my proposed new information architecture (IA) for the new ‘Streets, roads and pavements’ section.
Using a tool called Miro, I mapped out the current structure of the section, as well as the proposed new structure. Miro is an online collaborative canvas that is perfect for this task.
I also used Miro to detail proposed changes in the way we referred to certain terms. For more on how I used Miro, see my blog post ‘Using collaborative tools to redesign highways webpages’.
How we used tree testing to validate the IA
Creating a new IA can mean significant changes to how content is structured, and it’s important to validate it with real people. This ensures the changes are truly user focused.
Tree testing is perfect for this as it:
- is relatively quick to set up
- can be done remotely online
- can capture a large number of responses quickly
- can be completed using real user stories
The tree test was done online with residents in Cumbria. The new streets, roads and pavements structure was mapped out, using the new headings. People were asked a number of questions, based on predetermined user stories. Basing the questions on user stories allows us to make sure that what we are asking meets the needs of various groups of people that use the site.
They were then asked to click through to where they thought the relevant content would be.
Our tree testing questions
We asked 8 questions based on these user stories:
- I am a resident, planning to make a long journey across Cumbria, and want to look at weather conditions on the road
- I am a solicitor and I want to find out how to apply for a local authority search on a property that my client is buying
- I am a commuter and I want to know if my route to work has been gritted
- I am a business owner and I want to find out if I need a permit to have a skip placed outside my building
- I am a resident and I want to report a pothole in the pavement
- I am a parent and I want to find out if there are any roadworks that will affect my journey to school
- I am a resident and am thinking of buying an electric vehicle. I want to know if I can get a charge point installed outside my home
- I am a resident and I want to know if my road is due to be resurfaced
In total 228 people finished the test.
IA tree testing results
Across all user stories, 56% of respondents were able to identify the correct answer and 91% of answers were chosen without backtracking. The results also showed us the performance for each user story and we could also see the pathways that each of the users took. From this we could start to see where and potentially why users are taking the wrong turn.
According to Optimal Workshop, a success score of around 80% or more is considered a good score for a task. Therefore, we were keen to locate the pain points within the IA and explore ways to improve the findability of relevant content. We mapped out the proposed IA and ‘pietrees’ on a new Miro board so that we could trace where users were getting lost or hesitating.
A variety of solutions were methodically generated for each of the user journeys and we discussed how we could improve signposting, header labelling and rerouting users to help them find what they’re looking for efficiently.
Changes made post testing
Based on the results of the tree test, we made some changes to the site. We:
- added a prominent call the action in the ‘Road maintenance’ section, pointing people to the weather cameras, as some people had expected to find it here
- changed the section name of ‘Local authority searches’ to ‘Local authority highways searches’, to be specific that the searches are highways related
- updated the summary text of the ‘Road maintenance’ section to include ‘winter maintenance and gritting’, as it was missing from this section
- decided to include a prominent call to action button in the winter months in the ‘Road maintenance’ section, pointing to gritting, as it is a key task doing the winter
- linked up the weather cameras map with the gritting map, as some people thought you might find out about gritting by looking at weather cameras
- added a link to ‘Skips, scaffolding and hoarding permits’ from the ‘Permits to place things in the street’ section, as some people thought it would be in this section
- added a prominent call to action on the main landing page, pointing to ‘How to report a problem’, as this is a key task
- linked up the ‘Yearly programme of work’ to the ‘Roadworks’ page, as some people thought roadworks would be in this section
- added a link to ‘Electric vehicles’ from ‘Streets permits and licences’, as many people thought you needed a licence to install a charge point
- added a link to the ‘Road dressing programme’ from the ‘Roadworks’ section
How we used the survey to validate the new terminology
As well as validating the structure, tree testing is also useful in validating the choice of words you use to describe those sections and pages. When undertaking a qualitative analysis of the content, a number of words were identified that were not written in plain English and seemed to be jargon.
We therefore labelled the new IA with the new terms, allowing us to see if people made sense of the labels and understood where they were being asked to click.
The terms we were interested in testing were:
- ‘Footways’ (old) vs ‘Pavements’ (new)
- ‘Highways’ (old) vs ‘Streets, roads and pavements’ (new)
- ‘Carriageways’ (old) vs ‘Roads’ (new)
We used these new terms in the test and the results showed there was no ambiguity in where to click. We also tested these terms in a survey with the same people who conducted the tree test.
The results
The results from our survey were:
- ‘Street, roads and pavements’ was preferred by 48% of people
- ‘Highways’ was preferred by 44%
- An alternative term was preferred by 7%
- ‘Pavements’ was preferred by 67% of people
- ‘Footways’ was preferred by 31%
- An alternative term was preferred by 2%
- ‘Roads’ was preferred by 76% of people
- ‘Carriageways’ was preferred by 22%
- An alternative term was preferred by 2%
These results validated our assumptions that the new terms are preferred. The closest result was between that of ‘street, roads and pavements’ and ‘highways’. Street, roads and pavements is our preferred term, and it got the higher number of votes, so we decided to change it from highways. As the section is currently called highways, and has been for many years, we felt that this could have influenced people into choosing that term, as they are simply more used to it. Whereas the new term is a more accurate description of the content in that section.
As with all these changes, if after publication we get feedback that people do not understand a particular term, we would consider switching it back or switching it to a different term.
The benefits of tree testing
Tree testing can be quick, easy and a relatively cost effective way of validating any assumptions you may have about:
- information architecture
- terminology
It is also useful if any stakeholders you are working with have a difference of opinion of any changes. By asking real people who use the site, you are making sure the focus of the site is truly around the user. Any discrepancies between where content sits and how it is named can then be resolved by asking the users and making amends accordingly.