Test the user experience, not the feature list

I started this post yesterday. But with Steve Job's resignation, please note that his work exemplifies what this post promotes. Cliff Kuang wrote in Fast Company Design, "Steve Jobs may not be the greatest technologist or engineer of his generation. But he is perhaps the greatest user of technology to ever live..." A Jobsian focus on the user experience is an excellent aspiration for any software engineer.

An important aspect of any product's development is quality assurance. Essential to that is the reviewer's mindset. This is a cliché in macro approaches; of course a "quality mindset" should inform all product creation phases. But even in the micro of quality assurance as discrete phase, mindset matters.

Unfortunately, the mistake of quality assurance as feature testing is a predominant mindset. Features have to work but what matters most is that customers enjoy the product.

I give my software QA staff two primary guidelines:
1. Test using a script.
2. Do everything the way the customer would.

A customer mindset informs these simple instructions. It is their purpose and selects their phrasing.

First, it’s important to use a test “script”. Don't call it a test regimen or a protocol. Sure, project management primers will note that unplanned “click everything” testing is bound to miss something; documentation ensures greater test coverage and improves review. But beyond that, "script" matters.

The testing script should not be thought of as simply a plan. A script allows testers to become actors performing the role of customer. Don't test features, make sure that the user has a positive experience. To do that, act the part of the customer.

The second instruction is to do everything the way a customer would. The script needs to be authentic. Act the customer with heart.

For example, pre-release software is often transferred as a soft disk image, not as a physical CD or DVD. But if customers are buying a disk, then that ISO can’t be tested unless burned to a physical disk. A soft ISO on a desktop will load more quickly than a physical disk. This can mean that a painfully long disk spin time won’t get caught. Even the simple act of installing software demands a customer focus if it is effective.

Another, cautionary example: Early in my career I was an entry level tester of online educational assessment software. The directions were clear. Create a user account, create tests, then try them as a student. Everything worked as specified, the feature set was good to the letter, and I had yet lacked experience to question the testing regimen. Management was happy with my performance.

I later learned that unfortunately, the vendor's marketing department was very unhappy with this perfectly functioning product. Even though the features worked as specified, the product didn’t work. Rather, customers wanted different features. There were problems supporting teachers with multiple classes. The password management system was also underdeveloped.

The product had perfectly functioning features, but a lack of focus on the customer experience hampered the product. The whole team should have been focused on customer experience, not features. Now that I lead a team myself, I try to make it so.

Avoid shipping perfectly functioning software that is unusable. Don’t test features. Examine the customer experience.