Usability Testing: The Basics and Limitations

Case Study for NOON Lighting System

Usability testing is a very important step in the iteration process for any product. It helps to provide information about how a product works practically, and can help define possible improvements. A team of other designers and I went through testing the NOON Smart Light technology installed in the UVU’s UX Lab to see how the product currently works, and what improvements could be made. Throughout while we personally walked through the system to see how to set it up ourselves so we could see how others do, we ran into some issues that potentially caused issues to making a complete usability test, but we did find results that we expected.

Background on NOON

NOON Home, Inc. is a company that was established in 2015, and launched their first smart lighting system in 2017. This is a smart home lighting system that allows the user to manage the lights in their house using smart switches and mobile devices. The system works with what is called a Room Director that controls all the other extension switches in the room. The director can be controlled via the touch screen on the room director itself, and from mobile devices alike. The Room Director, and each extension switch controls its own set of lights that it is installed with.

Limitations to Creating a Complete Usability Test

We first tested the walk through of setting up the lighting system, making the different switches talk with the Room Director, and other things with the NOON app on our smartphones and tablets so we could create a usability test for others to follow. We had a couple of things we wanted to figure out:

• How to connect the Room Director to the light switches
• How to control the lights with the switches
• How to control lighting with the app itself
• Understanding how to set up certain lighting modes with the app

As we were doing a blind walkthrough of how to use the system ourselves, we ran into a few roadblocks. We then pulled up the directions on how to set up the director and the extension switches, but we were unable to get things connected. After going through all the troubleshooting we could find, which wasn’t much, we concluded that this could be a hardware issue, which is what the troubleshooting we found suggested. We looked up to see if we could potentially fix the hardware issue, but it was recommended that this would be the work of an electrician. We didn’t have access to one, so we were unable to see if this was what was actually the issue.

Defining Usability and Goals

We went through testing the app and the switches again with the knowledge that the system may have not been installed correctly, while still keeping in mind that the issues we were experiencing may be due to us not being able to figure it out correctly. We then decided to go through what we would think would define usability with what we were able to do so far. These were the factors we listed:

• Understanding what the director and extension switches are
• Using the app to connect directly to the lighting system is clear and easy
• Each step of the setup on the app is easy to understand
• How to add an extension to the director is understandable

In addition, we made a list of several different goals of our test:

• test the ease of setting up NOON lighting after it has been already set up
• reveal confusing parts of the experience
• identify any bugs or issues with the app

These goals were set in place to help verify what we gathered in our initial run throughs of the lighting system, and to see if there was more to uncover. We figured it was good to get a better view of what is consistently going wrong so we can know where it can be improved the most.

Testing the System with Participants

We had a test group of students at UVU with the age range of 20 to 30, and we had 4 participants in total. We split the participants in two groups, with each being given a little background in NOON Lighting. For group 1, we had prepared a list of walk through instructions we made ourselves to see if they were able to go through the process of setting up the lights as per instructions that NOON lighting had. For group 2, they were given no walkthrough.

We also had a list of tasks we gave each group:

1. Open the Noon App and login

• After logging in, connect to the Wi-Fi
• On the next screen select “open plan”
• Then add a director and pair it
• After pairing, select “No” on the next screen
• Select “Get Started” On the next screen
• Select “Recessed” for the type of ceiling light
• Select how many lights and go to next screen
• Click “Auto” for the system to set bulb type
• Wait for lights to be tested then click next
• Then click “Done Adding Extensions
• Test the different lighting settings

2. Go add an extension by:

• Click the settings icon
• Click “Rooms”
• Click “Open Plan”
• Click “Add a Noon Switch”
• Click the plus button
• Select “Office Area”
• Follow instructions on screen
• Test the lights on the director and the extension.

3. Post Test-Questions:

• Was the NOON app easy to use and understand?
• What was confusing about using NOON Lighting?
• What was confusing about the test?
• Would you buy this product for yourself?

Results of Group 1, Users with the Walkthrough:

They had an easier time going through the app, and were able to set up the director to control one row of lights. Both users were not able to get the director set up to work completely though, and were unable to set up the director with the switches to control the other lights in the room. They reported that they were both confused on how long to hold down switches to pair them with the director, and said it would have been nice to have in-app help as well.

Results of Group 2, Users with no Walkthrough:

They both had a hard time figuring out how to work with the director. They reported that the app was confusing, especially when it came to connecting the director to the app. They also had issues knowing how to connect the director to the switches. We also learned through this walk through that an iPad’s app and the iPhone’s app work differently and some usability is lost with the iPad, as there isn’t a way to add extension switches to the director with an iPad.

Recommendations & Action Items

We concluded that the NOON Lighting system with just our initial tests had a lot of improvement, especially when it comes to having users use the lighting system when they were not the ones that initially installed it. While the app’s design is sleek, and the wall lights are really modern looking, the pain points such as not understanding how to make anything connect stand out a lot. From our test, while some people enjoyed it’s usage, they were unable to use the lighting system.

Because the switches did not pair with the director at all, we were wondering if there was any possibility that the hardware of the switches themselves were set up properly, so in that case, an electrician would need to take a look at it to see if it was set up correctly in the first place. And if that was not the case, there needs to be more troubleshooting guides available in general. There also would need to be a rework of the app so the iPad and iPhone apps are similar.

The results from the usability test did confirm that the NOON Lighting system was not easy to use, so there would need to be a bit of a rework as far as verification goes, so that a user knows what’s not working and why, and can get instructions on how to fix the issues. Currently, the website and the app don’t provide many of those much needed troubleshooting instructions.

Final Thoughts: What I Would Do Differently with this Test

If we were to do this again, I would suggest getting more participants, and maybe asking instructors, or others that may have knowledge about the original installation to see if we could get things connected properly for a full usability test. I would also suggest reworking the questions to not “lead the witness.” Unfortunately, due to rushing the testing process, we did not work on the questions as much as we should have. All in all, we were able to get results we expected out of the usability test we conducted.

Chelsea Hansen is a student in the Digital Media program at Utah Valley University, Orem Utah, studying Interaction & Design. The above article relates to the Smart Home Case Study project in the DGM 1240 Course and representative of the skills learned.