Beyond the Drag Race Approach

The process that led to the creation of DHCAT began when engineers in Intel’s Performance Benchmarking and Competitive Analysis made two key observations. The first was self-evident: usage models for home media PCs are evolving rapidly as new types of media, content sources and playback devices proliferate. Second, the conventional drag-race approach (faster-is-better) to digital home platform evaluation has run out of gas.

The problem has been that the speed-focused approach to platform evaluation alone didn’t fully or accurately predict people’s experiences using those platforms. Furthermore, tests that focus on a small number of media functions didn’t capture the diverse capabilities of devices that often handle television, radio, video, photo and music content from many different sources and in many different media formats. Quantifying platform performance was challenging enough with conventional computing applications, but for emerging usage areas such as digital home there were simply no tools available that could provide meaningful measurements.

But the need for such tools has been urgent and growing more so. Hardware OEMs, software developers, analysts and consumers all needed a more sophisticated toolset and methodology for evaluating platform goodness.

This tool would need to adhere to these four tenets:

Rule One: Create methodologies to measure and report user experience quality that is rigorous, repeatable, and meaningful to users. The digital home is about seamless delivery of entertainment content; the success of that delivery should be the primary measure of platform goodness.

Rule Two: Measure speed only where it determines or strongly influences user experience quality.

Rule Three: Report results in terms of capabilities – what the system can, and can’t do well.

Rule Four: The tool and the tests it runs should be based upon recognized open industry standards.

Early last year, a development team at Intel® began meeting to tackle the problem of designing an assessment tool for digital home PC platforms that would follow this more experiential approach. While this approach may seem simple enough, actually implementing it was anything but simple.