At the time of their formation, leaders of these organizations made a key observation: old methods of benchmarking were just not keeping up with the times. Similarly, as noted in past entries, it is my belief that today’s tried and true methods of benchmarking, although still accurate and valid in the technical sense, are beginning to lose their connection with the realities faced by our industry’s customers. Some even argue that this gap began to appear years ago.

In the benchmarking community we often rejoice at incremental performance improvements and communicate these in terms of fractional gains. Drawing a half-dozen more frames a second in a game with average frame rates in the hundreds is often cause for celebration. In many cases, especially true in real-time applications, these speedups communicated in fractional improvements do not significantly enhance user experience. Models for translating these fractional improvements into what I call quantum units of user perception are painfully lacking in most common benchmarks. Partly as a result of this, end-users -- our ultimate real target audience -- have become numb to much of our industry’s progress. We owe it to ourselves, our industry and our customers to improve this. Can we turn the clock back to the early days of computer benchmarking and reinvigorate the creativity and verve with which founders of SPEC and BAPCo approached similar issues?

We asked the same question a while back and began looking into how and where we could course-correct in our own capacity. Without delving into all the details (those are here), results of our investigations led us to development and subsequent release of Intel® DHCAT, a comprehensive and scientific attempt at quantifying computer performance in terms of perceived user value and significance. With Intel® DHCAT we entered the era of benchmarking 3.0, and hope to once again change the course of benchmarking to keep it relevant to the people who matter most: end-users. You could say “we’re standing on the shoulders of giants” who invented the 1.0 and 2.0 versions. Kaivalya’s spirit lives on...




*Other names and brands may be claimed as the property of others.