Tech.pinions

For many years, the PC industry created products that were pretty straight forward and relied on a central CPU to drive almost all of its computational functions. But since there was a great deal of competition among microprocessor vendors as well as PC makers to try and differentiate their products, the need arose for a set of benchmarks to deliver a consistent view of how these products performed.

So, BapCo’s SYSMark benchmark testing program emerged as one of the more important benchmarking programs that evolved in the 1990’s to serve this purpose. But as we headed into a this decade, PC’s were being asked to do a lot more then basic word processing, spreadsheets and relatively simple multimedia computing. In fact, PC applications began to demand more graphics based functionality in even mainstream applications which included desktop PC games, processing of various media formats and new ways to integrate imaging into everyday applications.

Given all of these new changes in processing demands and the addition of a GPU into the performance equation, the need for new benchmarks that recognizes this new age of computing is needed. This is at the heart of why AMD has backed away from supporting BapCO’s SYSmark program. SYSmark was based on older computing performance models and a company like AMD, who was a major supporter of this benchmark in the past and BapCo could not agree on updated testing criteria.

Of course, there is a lot more to this entire subject and AMD’s CMO, Nigel Dessau has posted AMD’s perspective on this issue that I include below. And I have asked fellow analyst Rob Enderle of the Enderle Group, a well know authority on performance based computing, to weigh in on why this matters. First up is Nigel’s blog post, reprinted by permission, also read Rob Enderle’s analysis here..

Voting for Openness-By Nigel Dessau

AMD has a long history of supporting open standards; if you have any doubt just look at our support for OpenCL. And this support extends to active involvement with open industry consortia that likewise promote open standards. The beauty of open standards is that they are just that – open. Open to analysis, open to improvement and open to criticism.

AMD has for some time been a member of BAPCo, an industry organization that promotes, among other things, a benchmark known as SYSmark. In the past year or so AMD, with openness and transparency, has tried to explain why we believe this benchmark is misleading with respect to today’s commonplace applications − about a year ago I published a blog designed to explore this. If you work for a company that believes in transparency and integrity – and I do – then you have to take a stand and speak up when something is wrong.

BAPCo’s response to this blog was a threat to expel AMD from the consortium.
The heart of our complaint is this: the SYSmark benchmark is not only comprised of unrepresentative workloads (workloads that ignore the importance of heterogeneous computing and, frankly, favor our competitor’s designs), but it actually generates misleading results that can lead to very poor purchasing decisions, causing governments worldwide to historically overspend somewhere in the area of approximately $8B!

Now you’re starting to see why this is relevant to you (presuming you’re a taxpayer).

Good Intentions, Bad Results

AMD decided to do what we believed was the right thing for the industry and our customers, so we continued to work within BAPCo to try to get the next-generation benchmark, SYSmark12 (“SM2012”), right. Our hope was to effect change so that it would be open, transparent and processor-neutral. We got workloads included that represent the things you and I actually do in a day (instead of 35,000 line spreadsheets!).

But the question remained: what weighting would BAPCo ultimately give to the real-world workloads − since it is this weighting that defines the actual benchmark scores.

Unfortunately, our good intentions were met with an outcome that we believe does a disservice to the industry and our customers. We weren’t able to effect positive change within BAPCo, and the resulting benchmark continues to distort workload performance and offers even less transparency to end users. Once again, BAPCo chose to ignore the opportunity to promote openness and transparency.

1. While SM2012 is marketed as rating performance using 18 applications and 390 measurements, the reality is that only 7 applications and less than 10 percent of the total measurements dominate the overall score. So a small class of operations across the entire benchmark influences the overall score.
2. In fact, a relatively large proportion of the SM2012 score is based on system performance rated during optical character recognition (OCR) and file compression activities − things an average user will rarely if ever do.
3. And SM2012 doesn’t represent the evolution of computer processing and how that evolution is influencing average users’ experience. SM2012 focuses only on the serial processing performance of the CPU, and virtually ignores the parallel processing performance of the GPU. In particular, SM2012 scores do not take into account GPU-accelerated applications that are widely used in today’s business environments.

There are more things that AMD objects to in SM2012, like the excessive wall clock time consumed by its installation and execution. But this explanation will hopefully help you understand why, ultimately, we couldn’t look in the other direction.
Moving Forward…to Openness

So how can AMD stay in BAPCo? Simply put, we can’t. We have resigned from BAPCo and asked that our name and logo be removed from marketing materials promoting SM2012.

Now I hear some of you asking, “Isn’t this really just about the long-running antagonism between AMD and your competitor?”
No, it’s not.

1. It’s about fairness. Fairness to consumers and business users, to governments and other organizations that make purchasing decisions based on benchmarks, and, in the case of SYSmark, needlessly overspend because of it.
2. It’s about relevance. Because do you want to buy a system based on an outdated approach to measuring performance? Don’t you want your system’s performance measured against relevant measures like HTML5 or GPU acceleration? And shouldn’t a benchmark that measures PC performance be relevant to other devices that are likely in your life (if you’re reading this blog I think it’s safe to presume you use an array of devices – I do). Benchmarks should measure the way people engage with their devices today – not stick to a formula more appropriate for the last millennium.
3. And it’s about openness. Because you, and IT purchasing managers, should know what a benchmark represents and what the score really means to how the device will be used. That’s being set free.

And this is why we are exploring the options to encourage an alternative consortium, one that will deliver unbiased, representative benchmarks and promote more transparency for our industry. We are committed to working with likeminded companies that want to give consumers and business users an accurate, honest measure of what they can expect from their PCs and mobile devices. And what if ultimately we don’t “win” on these new benchmarks? Well, if the work is done with openness and transparency and results in a useful benchmark, we will make our case and let the market decide.
That’s all we have been asking for from BAPCo
.
My hope is you, and the market, will vote for openness.

Nigel Dessau is Senior Vice President & Chief Marketing Officer for AMD. His postings are his own opinions and may not represent AMD’s positions, strategies or opinions. Links to third party sites are provided for convenience and unless explicitly stated, AMD is not responsible for the contents of such linked sites and no endorsement is implied.

This article is a guest contribution from Rob Enderle, Principal Analyst, The Enderle Group.

This week AMD pulled the plug on their participation with BAPCo’s SYSmark benchmarking project. While there has been a bit of drama related to this with some folks blaming Intel for being too heavy handed and others blame AMD for being too thin skinned, the reality is that both companies are on different paths now and that, as a result, a collaborative common benchmark no longer makes any sense. Let me explain.

SYSmark

SYSmark is a benchmark that was designed to measure PCs largely the way they were in the 1990s, heavily using office applications and as largely standalone work centers. Back then graphics were largely reserved for gamers and SYSmark was about business. Intel and AMD were on the same CPU centric path and neither had any real strength in graphics which were added from companies like ATI and NVIDIA after the fact and only on high end systems generally not targeted at business.

This is a world defined by Intel, who remained throughout, vastly larger and better funded than AMD and AMD played the role, albeit involuntarily, as backup vendor to Intel. Still it was a good business until the two companies stopped being socket compatible and that one move changed the impression that it was very difficult for AMD to beat Intel to impossible.

AMD was simply overmatched.

Changing the Game

So AMD changed the game; realizing that both companies were very weak when it came to graphics they took a huge gamble and bought ATI who had been struggling against NVIDIA but was better matched to that company than AMD was to Intel. This move set AMD back a bit against Intel as they integrated the two technologies but they ended up with a dramatically different part, called Fusion, which is a hybrid of the technology they had and the technology they bought.

This part was focused on where applications seemed to be going, to something we were calling GPU computing, and as a hybrid it was designed to bridge between the past and future. In short AMD no longer agreed with Intel with regard to how people were going to use their PCs and this put SYSmark at risk.

The Death of SYSmark

You see a collaborative benchmark is only good if the two companies providing the technology can agree on how to measure it. Once they disagree the benchmark is done. Just like you wouldn’t benchmark a sports car to a truck, if the two products are fundamentally different it makes no real sense to use a common benchmark against them. In fact, people weren’t using their PCs in the same way there were in the 90s anyway. Few are using spreadsheets anymore or local databases as these have given way to hosted and cloud based remote applications. Movies are being streamed and increasingly applications are calling on the graphics side of the PC to render, transcode, or even run highly parallel new applications.

SYSmark needed to go through a dramatic change anyway but Intel and AMD, being on different paths, no longer could agree on what that change was and that disagreement killed it.

Wrapping Up: The End of an Era

We’ve really reached the end of another personal computer era; the web, cloud services, GPU computing, and a huge shift in focus to hardware that is better connected, lighter and has longer battery life has forever changed the world that was into the world that will be. The death of SYSmark is no one’s fault, Intel isn’t being evil and AMD isn’t being unreasonable. The firms changed, the market changed, and a common benchmark between the vendors simply made no more sense.

This is a guest contribution from Jack Gold the founder and principal analyst at J.Gold Associates, LLC an information technology analyst firm based in Northborough, MA, covering the many aspects of business and consumer computing and emerging technologies. Learn more about J.Gold Associates here.

The market seems to think that that the folks at ARM and its licensees (TI, Nvidia, Qualcomm, Marvel, Apple, et. al.) are on the verge of attacking Intel where it is most susceptible – the PC and server space. Indeed, ARM is making inroads with low power designs, and has a virtual monopoly on mobile devices. But the path to PCs and Servers is a very different path than smartphones and tablets. And clearly, Intel doesn’t think it can afford to concede any territory, which is why it is pushing back hard on the mobile “heartland” of ARM. So let’s step back and see what Intel has going for it vs. the ARM ecosystem.

Many observers have a bias towards ARM and are discounting Atom’s potential for success in phones and tablets, I think that Atom really does have a chance to succeed and thrive. Not perhaps in the current version, but in the next generation of chips Intel will launch in the next 6-12 months. And I believe that Intel will stay very far ahead of ARM in the race for PCs and even high end tablets. Why? Here are some reasons.

First, Intel’s huge investment in processing technology is not putting it at a disadvantage as some have suggested. Actually it’s the other way around. The ARM camp is relying on the foundries to make the process improvement investment for them. But after they’ve matched Intel’s recent investment of $15B or so, they still will have to recoup that investment, and that will mean higher chip costs to the fabless chip vendors (no free lunch here). At the end of the day, process advantage does matter. It’s how Moore’s Law has remained in play, and process advantage means higher performing chips at lower power and eventually lower cost (as yields increase). And Intel’s recent development of 22nm and 3-D transistors means its lead is increasing and has a two year (or more) advantage on the competition.

Second, the conversation comparing ARM to Intel usually turns to RISC vs. CISC. I thought we settled that argument years ago with Transmeta and MIPS before that. But I guess not. The bottom line is that with more complex systems that have increasingly complex computing requirements, longer and more complex instruction sets improve performance. This is what Microsoft
found out years ago when it suspended development of Windows on RISC. Yes, they now say they will have Windows 8 running on ARM. But the question remains, what version and what features? There is no doubt in my mind that the highest end and more performance oriented versions of Windows will remain focused on the x86 architectures. And don’t forget that ARM isn’t even on 64 bits yet. Imagine a server with a large database running on a 32 bit RISC architecture compared to a full featured 64 bit CISC version. So as functions get more complex, specialized instructions and HW additions give x86 (including Atom) an advantage unless ARM adds the same HW and SW extensions.

The third issue is compatibility. There is a perception that ARM is compatible across platforms and vendors, and clearly its not. As a result, look at the upgrade problem being faced by older devices in the market, and even among devices from the same manufacturer. In fact, different licensed versions of the ARM architecture have incompatibilities. And deep licensees (e.g., Qualcomm) are building their own architecture that is supposed to be compatible with other vendors’ chips (but is it?). ARM fragmentation is an issue usually not discussed. But it is real no less.

Finally many think that Intel is a chip company, and forget that it has tens of thousands of SW engineers on staff. This allows it to create the best compilers in the industry, and to optimize ports to its platform well beyond what others can do. And WindRiver gives Intel incredible breath in tools and designs. Many perceive Google’s commitment to Android on Atom as lukewarm, but Intel has invested considerable resources to port and optimize Android for its platforms, albeit a bit late. And even though current WP7 doesn’t run on Atom, it is quite likely that WP Next (e.g. Windows 8) could easily do so if there is OEM demand, which there well may be especially in the tablet space. Finally, now that Intel has McAfee in its stable, it is very likely to create industry-leading HW-enabled security features that users will find appealing and competitors will have trouble duplicating.

Of course MeeGo remains a sore spot for Intel, especially after Nokia’s rejection of the OS for its devices. It is not clear MeeGo will ever get out of the niche markets it now is targeting. But clearly some vendors see it as an alternative to Android’s (and Apple’s) hegemony, especially in emerging markets. So while it may never achieve the huge numbers of units that its competitors will ship, it will nevertheless have a credible niche to exploit. But Intel is not riding MeeGo as its only path to success.

Bottom Line: For Intel, it’s about the ecosystem. As code gets more complex, it’s increasingly difficult to produce and manage, especially across multiple platforms. This is problematic for both OS developers and ISVs who want to port their apps (look at how many versions of Android apps there are, and not just for OS versions numbers, but also for different devices from different manufacturers). Intel’s x86 consistency is a strong point and fragmentation plays to its strength. Certainly I’m not signally the death knell for ARM. But those who minimize Atom’s future potential are making a mistake.