How we tested
Our tests were not the performance-related analyses typically found in product reviews. We weren't interested in aspects such as processor speed or battery life.
Rather, Tom Thornton, senior research scientist at Perceptive Sciences, and his team focused on the usability of the hardware. Perceptive Sciences developed nine tasks for these ultrathin laptops and then examined how quickly and easily the testers completed those tasks. A total of 20 people were recruited to participate in the tests -- 11 men and nine women. Of those, half were students and half were business users who travel anywhere from once a month to once a week.
Usability testing is part science, part art. That's why the results are a combination of the objective -- the time it took to complete specific tasks and the success rate at completing those tasks -- and the subjective impressions of the testers.
For each given task, each participant tested two of the devices while observed by a Perceptive Sciences staffer. Half of the time, they tested one of the two laptops first; the other half of the time they started with the other laptop. The purpose of this approach was to negate any advantage or disadvantage of the order in which the devices were tested. After the tests were done, the researchers interviewed each of the testers to get their more subjective reactions.
The testers were mixed in terms of which operating systems they used in their everyday life: Eleven were PC-only users, seven used both PCs and Macs, and two were Mac-only users. None of the users were familiar with the specific laptops they were testing.
In fact, the tests were designed to keep the operating system out of the mix as much as possible, although Thornton acknowledged that it was impossible to avoid that issue entirely. Mac users were bound to feel more comfortable with the Mac OS, and PC users would tend to be more comfortable with Windows. The MacBook Air came with the latest version of the Mac OS X, dubbed Leopard, while both the Lenovo X300 and the Toshiba Portege R500 were equipped with Windows Vista Business.
What we tested
Even though we couldn't try out every sort of task that people would use these notebooks for, Perceptive Sciences came up with a representative mix for their testers to try.
1. Take the computer out of the box, plug it in and turn it on.
There were no clear-cut winners in this out-of-the-box test -- and each laptop had a glitch. Thornton said that the ThinkPad X300's battery wasn't initially connected to the laptop, which took testers additional time to correct. Some testers were confused about whether to use the extra power cord extension that was included in the Air's box, which also took a bit of time to sort out. The Toshiba Portege R500 had the longest boot time, and "people were frustrated by that," Thornton said.
2. Connect to the wireless network.
The MacBook Air and Lenovo performed similarly, with testers taking, on average, about 2 minutes 30 seconds to connect. The Toshiba took, on average, about 3 minutes 45 seconds.
"People had more errors [with the Toshiba] as far as going to the wrong place [in the interface] to connect," said Emily Swinkels, research scientist at Perceptive Sciences. "Also, the Toshiba has a wireless switch on the side. One user accidentally turned that off."
