Toward the end of the 80’s and beginning of the 90’s several industry groups were formed. Their common goal was to develop standardized tests for measuring computer performance in more meaningful ways than had been done in the past. Standard Performance Evaluation Corporation (SPEC* – at some point I think they called themselves a “cooperative”) was formed in 1988. It mainly consisted of workstation vendors who were later joined by other industry players and academics. They, along with others, such as TPC* and BAPCo*, aimed to take a more scientific and application-based approach towards developing benchmarks.

In some ways, these organizations also fostered the concept of cooperative benchmark development, and the net result of these efforts was quite positive. Starting from about early 90’s, a stream of benchmarks from these organizations began to appear in the industry. These benchmarks slowly established a new methodology, and became industry standards. I personally was involved in formation of BAPCo* and led the design and development of its first product, SYSmark* 92 – Side note Trivia: SYSmark* 92 came in a box containing 21 3.5 inch diskettes!

The key trends in these developments were the use of real applications in a standard way, scientific design, and more importantly, that they were developed cooperatively by all involved. It was a good answer for the issues facing the benchmarking world at the time. It continues to be one the most sustainable approaches.

Today, organizations like BAPCo* and SPEC* continue to create application-based scientifically designed benchmarks like SYSmark* and the SPEC* CPU benchmarks. These reflect many of the industry's best benchmarking practices, and are the culmination of the work of many talented computer scientists. And yet, given the increasingly media-centric direction the PC industry is taking, we need something to meaningfully capture not just performance, but delivered experience. More on that in the next installment.

*Other names and brands may be claimed as the property of others.