Why AMD’s decision to pull the plug on their support for BapCo’s SYSmark benchmark matters

For many years, the PC industry created products that were pretty straight forward and relied on a central CPU to drive almost all of its computational functions. But since there was a great deal of competition among microprocessor vendors as well as PC makers to try and differentiate their products, the need arose for a set of benchmarks to deliver a consistent view of how these products performed.

So, BapCo’s SYSMark benchmark testing program emerged as one of the more important benchmarking programs that evolved in the 1990’s to serve this purpose. But as we headed into a this decade, PC’s were being asked to do a lot more then basic word processing, spreadsheets and relatively simple multimedia computing. In fact, PC applications began to demand more graphics based functionality in even mainstream applications which included desktop PC games, processing of various media formats and new ways to integrate imaging into everyday applications.

Given all of these new changes in processing demands and the addition of a GPU into the performance equation, the need for new benchmarks that recognizes this new age of computing is needed. This is at the heart of why AMD has backed away from supporting BapCO’s SYSmark program. SYSmark was based on older computing performance models and a company like AMD, who was a major supporter of this benchmark in the past and BapCo could not agree on updated testing criteria.

Of course, there is a lot more to this entire subject and AMD’s CMO, Nigel Dessau has posted AMD’s perspective on this issue that I include below. And I have asked fellow analyst Rob Enderle of the Enderle Group, a well know authority on performance based computing, to weigh in on why this matters. First up is Nigel’s blog post, reprinted by permission, also read Rob Enderle’s analysis here..

Voting for Openness-By Nigel Dessau

AMD has a long history of supporting open standards; if you have any doubt just look at our support for OpenCL. And this support extends to active involvement with open industry consortia that likewise promote open standards. The beauty of open standards is that they are just that – open. Open to analysis, open to improvement and open to criticism.

AMD has for some time been a member of BAPCo, an industry organization that promotes, among other things, a benchmark known as SYSmark. In the past year or so AMD, with openness and transparency, has tried to explain why we believe this benchmark is misleading with respect to today’s commonplace applications − about a year ago I published a blog designed to explore this. If you work for a company that believes in transparency and integrity – and I do – then you have to take a stand and speak up when something is wrong.

BAPCo’s response to this blog was a threat to expel AMD from the consortium.
The heart of our complaint is this: the SYSmark benchmark is not only comprised of unrepresentative workloads (workloads that ignore the importance of heterogeneous computing and, frankly, favor our competitor’s designs), but it actually generates misleading results that can lead to very poor purchasing decisions, causing governments worldwide to historically overspend somewhere in the area of approximately $8B!

Now you’re starting to see why this is relevant to you (presuming you’re a taxpayer).

Good Intentions, Bad Results

AMD decided to do what we believed was the right thing for the industry and our customers, so we continued to work within BAPCo to try to get the next-generation benchmark, SYSmark12 (“SM2012”), right. Our hope was to effect change so that it would be open, transparent and processor-neutral. We got workloads included that represent the things you and I actually do in a day (instead of 35,000 line spreadsheets!).

But the question remained: what weighting would BAPCo ultimately give to the real-world workloads − since it is this weighting that defines the actual benchmark scores.

Unfortunately, our good intentions were met with an outcome that we believe does a disservice to the industry and our customers. We weren’t able to effect positive change within BAPCo, and the resulting benchmark continues to distort workload performance and offers even less transparency to end users. Once again, BAPCo chose to ignore the opportunity to promote openness and transparency.

1. While SM2012 is marketed as rating performance using 18 applications and 390 measurements, the reality is that only 7 applications and less than 10 percent of the total measurements dominate the overall score. So a small class of operations across the entire benchmark influences the overall score.
2. In fact, a relatively large proportion of the SM2012 score is based on system performance rated during optical character recognition (OCR) and file compression activities − things an average user will rarely if ever do.
3. And SM2012 doesn’t represent the evolution of computer processing and how that evolution is influencing average users’ experience. SM2012 focuses only on the serial processing performance of the CPU, and virtually ignores the parallel processing performance of the GPU. In particular, SM2012 scores do not take into account GPU-accelerated applications that are widely used in today’s business environments.

There are more things that AMD objects to in SM2012, like the excessive wall clock time consumed by its installation and execution. But this explanation will hopefully help you understand why, ultimately, we couldn’t look in the other direction.
Moving Forward…to Openness

So how can AMD stay in BAPCo? Simply put, we can’t. We have resigned from BAPCo and asked that our name and logo be removed from marketing materials promoting SM2012.

Now I hear some of you asking, “Isn’t this really just about the long-running antagonism between AMD and your competitor?”
No, it’s not.

1. It’s about fairness. Fairness to consumers and business users, to governments and other organizations that make purchasing decisions based on benchmarks, and, in the case of SYSmark, needlessly overspend because of it.
2. It’s about relevance. Because do you want to buy a system based on an outdated approach to measuring performance? Don’t you want your system’s performance measured against relevant measures like HTML5 or GPU acceleration? And shouldn’t a benchmark that measures PC performance be relevant to other devices that are likely in your life (if you’re reading this blog I think it’s safe to presume you use an array of devices – I do). Benchmarks should measure the way people engage with their devices today – not stick to a formula more appropriate for the last millennium.
3. And it’s about openness. Because you, and IT purchasing managers, should know what a benchmark represents and what the score really means to how the device will be used. That’s being set free.

And this is why we are exploring the options to encourage an alternative consortium, one that will deliver unbiased, representative benchmarks and promote more transparency for our industry. We are committed to working with likeminded companies that want to give consumers and business users an accurate, honest measure of what they can expect from their PCs and mobile devices. And what if ultimately we don’t “win” on these new benchmarks? Well, if the work is done with openness and transparency and results in a useful benchmark, we will make our case and let the market decide.
That’s all we have been asking for from BAPCo
.
My hope is you, and the market, will vote for openness.

Nigel Dessau is Senior Vice President & Chief Marketing Officer for AMD. His postings are his own opinions and may not represent AMD’s positions, strategies or opinions. Links to third party sites are provided for convenience and unless explicitly stated, AMD is not responsible for the contents of such linked sites and no endorsement is implied.