Which VR usability heuristics should I use?
Now that we’re in a calm moment for VR (especially after the 2016 wave), it’s probably a good time to review the different sets of VR-related usability heuristics out there. From a selfish perspective, I want a list of 5–20 defining heuristics that I can use to guide my design process, and also to provide usability feedback in a structured manner. If you’ve never used heuristics before, you’re missing out (read the first section here). They can become a taxonomy for everyone on the team to discuss usability issues, and they help your team stay objective. It’s time I found this for my VR work.
Here are the criteria that I’m using for this evaluation:
- Good coverage: Ideally, this is a one-stop-shop for reviewing the usability of a VR app. The heuristics cover both VR-specific, as well as non-VR-specific usability principles that still apply.
- Approachable: The heuristics can be quickly taught to a non-designer, so that we can use them in a conversation. They’re also brief enough for me to glance at while I’m designing.
- Hardware agnostic: Different headsets afford different interactions, and new headsets are coming every year. While not perfect, Nielsen’s heuristics managed to survive the internet’s multitude of design cycles, so let’s try to get some heuristics on the book that can work for at least 5 years.
- Validated: When Nielsen created his heuristics, he didn’t just create a list and throw them out into the wild — he actually presented details on how he built the list using actual research data.
The lists I’ll be reviewing:
YouTube UX Research team’s heuristics
Heuristic Evaluation for Virtual Reality Systems (page 2)
By Rabia Murtza, Stephen Monroe, and Robert J. Youmans
Good Coverage: 3/5
Approachable: 5/5
Hardware agnostic: 2/5
Validated: 2/5
The YouTube UX Research team developed this list of 9 heuristics based off of a survey of common VR usability issues — a majority being students at the Savannah College of Art and Design. The team acknowledges the flaws in this approach, but nevertheless, it does contain a solid foundation. Not to mention, the heuristics are pertinent to today’s common VR usability issues. I especially appreciate that heuristic #9 basically says “don’t just rely on this list” by referencing Nielsen’s heuristics (although I’d of course prefer that they looped in some of the most relevant of Nielsen’s heuristics into this list).
Unfortunately, the team’s goal here was to create a list that combines hardware and software, so it’s notable that there are only 4–5 heuristics here that a commonplace VR designer can take action on.
Sutcliffe & Gault’s heuristics
Heuristic evaluation of virtual reality applications (page 3)
By Alistair Sutcliffe and Brian Gault, 2004
Good Coverage: 3/5
Approachable: 5/5
Hardware agnostic: 5/5
Validated: 2/5
Sutcliffe & Gault’s list of heuristics is both concise and self-explanatory. Much of the list is relevant for a VR designer/developer today, and it has great coverage of common VR usability issues. Because this list is relatively concise, you’ll be able to post them up on the wall and glance at them as you see fit. It’s also short enough to teach to a developer, in order to make these heuristics a regular part of your vocabulary. It’s also great that this list contains a (just a few) heuristics that can be applied outside of VR, such as with “Support for learning”.
Unfortunately, the methodology for creating the list is a bit opaque — they describe some of the academic sources from which they received inspiration, but it appeared to be a subjective process.
Joseph Gabbard’s usability guidelines
A Taxonomy of Usability Characteristics in Virtual Environments (page 16–35)
Good Coverage: 5/5
Approachable: 1/5
Hardware agnostic: 3/5
Validated: 3/5
Joseph Gabbard’s thesis contains the most thorough list I’ve come across. It’s a collection of 195 guidelines broken down into 4 categories. He developed this list by surveying papers, conferences, online searches, and general discussions with academics in the field. Using his guidelines, you’ll be sure to catch usability issues that you may not have been looking out for. Gabbard’s guidelines also cover usability issues that are not specific to VR, which I believe is really important (see SysInfo1-SynSinfo9 on page 24).
The depth of this list is great for experienced VR designers, but problematic for newcomers (he never intended to make it short, anyways). Each guideline is also relatively brief, which means that some are a bit confusing. You may have to do some additional reading on each guideline if you’re not deep in the VR space. The guidelines also cover hardware considerations, and use-case specific considerations such as collaboration. My ideal list would nix these, but that’s a personal preference. Just be sure to cross out the guidelines that aren’t relevant to your app.
Olga Ivanova’s heuristics
Good Coverage: 3/5
Approachable: 3/5
Hardware agnostic: 4/5
Validated: 1/5
Note: I know Olga personally, although we’ve never discussed heuristic evaluations prior to publishing. I came across her work when doing my own online research.
Ivanova’s list of heuristics provides great coverage of VR-specific usability issues, although it’s quite long at 64 heuristics. The heuristics are practical (especially for modern VR apps), and are largely geared towards VR app designers. She’s also developed a tool that you can use to jot down notes under each heuristic, which makes it easier to get going with the list.
It’s important to note that she makes no claim as to the scientific validity of the list. She’s an independent VR designer, and it’s up to you to decide if it covers your bases. In addition, the heuristics are occasionally too prescriptive, or unclear. Additional explanation on her methodology for creating this list, as well as more details on each heuristic would be helpful.
Which list should I use?
To be honest, none of these lists quite fits the bill for what I’m looking for. I think it’s important to be able to hand VR designers/developers a single, comprehensive, but manageable list of heuristics that they can use without additional research. While all great in their own ways, none of them checks all of the boxes.
But, that’s just where I stand.
After all, this isn’t a science, it’s an art. Even if the methodologies for creating these lists appears to be rigorous and academic, those methodologies are still extremely subjective. So, thank you to all of the researchers and designers who are pushing the field. Be sure to browse through their work, and pick one list that you think you’ll be able to use repetitively and consistently. The most important thing is to use something to help guide your critical thinking and discussions.
Let me know how it goes! Follow me on Twitter @cgallello and shoot me a DM.
😎,
Chris
