AMD Will Not Endorse BAPCo's SYSmark 2012 Benchmark, BAPCo Responds
AMD today announced that it will not endorse the SYSmark 2012 Benchmark (SM2012), which is published by BAPCo (Business Applications Performance Corporation). Along with the withdrawal of support, AMD has resigned from the BAPCo organization.
"Technology is evolving at an incredible pace, and customers need clear and reliable measurements to understand the expected performance and value of their systems," said Nigel Dessau, senior vice president and Chief Marketing Officer at AMD. "AMD does not believe SM2012 achieves this objective. Hence AMD cannot endorse or support SM2012 or remain part of the BAPCo consortium."
BAPCo, which stands for Business Applications Performance Corporation, is an industry organization that "develops and distribute a set of objective performance benchmarks based on popular computer applications and industry standard operating systems." BAPCo's current members include: ARCIntuition, Dell, Hewlett-Packard, Hitachi, Intel, Lenovo, Microsoft, Samsung, Seagate, Sony, Toshiba and others.
"The SYSmark benchmark is not only comprised of unrepresentative workloads (workloads that ignore the importance of heterogeneous computing and, frankly, favor our competitor's designs), but it actually generates misleading results that can lead to very poor purchasing decisions, causing governments worldwide to historically overspend somewhere in the area of approximately $8B!" Dessau added.
According to AMD, "while SM2012 is marketed as rating performance using 18 applications and 390 measurements, the reality is that only 7 applications and less than 10 percent of the total measurements dominate the overall score. So a small class of operations across the entire benchmark influences the overall score."
"A relatively large proportion of the SM2012 score is based on system performance rated during optical character recognition (OCR) and file compression activities − things an average user will rarely if ever do," AMD added.
In addition, SM2012 focuses only on the serial processing performance of the CPU, and virtually ignores the parallel processing performance of the GPU. In particular, SM2012 scores do not take into account GPU-accelerated applications that are widely used in today?s business environments.
AMD said that it would only endorse benchmarks based on real-world computing models and software applications, and which provide useful and relevant information. AMD believes benchmarks should be constructed to provide unbiased results and be transparent to customers making decisions based on those results. Currently, AMD is evaluating other benchmarking alternatives, including encouraging the creation of an industry consortium to establish an open benchmark to measure overall system performance.
AMD also encourages anyone wanting more details about the construction and scoring methodology of the SM2012 benchmark to contact BAPCo.
Responding to AMD's decision, BAPCo released the following statement:
"Advanced Micro Devices (AMD) was, until recently, a long standing member of BAPCo. We welcomed AMD?s full participation in the two year development cycle of SYSmark 2012, AMD?s leadership role in creating the development process that BAPCo uses today and in providing expert resources for developing the workload contents. Each member in BAPCo gets one vote on any proposals made by member companies. AMD voted in support of over 80% of the SYSmark 2012 development milestones, and were supported by BAPCo in 100% of the SYSmark 2012 proposals they put forward to the consortium.
BAPCo also notes for the record that, contrary to the false assertion by AMD, BAPCo never threatened AMD with expulsion from the consortium, despite previous violations of its obligations to BAPCo under the consortium member agreement.
BAPCo is disappointed that a former member of the consortium has chosen once more to violate the confidentiality agreement they signed, in an attempt to dissuade customers from using SYSmark to assess the performance of their systems. BAPCo believes the performance measured in each of the six scenarios in SYSmark 2012, which is based on the research of its membership, fairly reflects the performance that users will see when fully utilizing the included applications."
According to reports, Nvidia and VIA have also quit BAPCo, leaving Intel as the only semiconductor maker still in the consortium.
BAPCo, which stands for Business Applications Performance Corporation, is an industry organization that "develops and distribute a set of objective performance benchmarks based on popular computer applications and industry standard operating systems." BAPCo's current members include: ARCIntuition, Dell, Hewlett-Packard, Hitachi, Intel, Lenovo, Microsoft, Samsung, Seagate, Sony, Toshiba and others.
"The SYSmark benchmark is not only comprised of unrepresentative workloads (workloads that ignore the importance of heterogeneous computing and, frankly, favor our competitor's designs), but it actually generates misleading results that can lead to very poor purchasing decisions, causing governments worldwide to historically overspend somewhere in the area of approximately $8B!" Dessau added.
According to AMD, "while SM2012 is marketed as rating performance using 18 applications and 390 measurements, the reality is that only 7 applications and less than 10 percent of the total measurements dominate the overall score. So a small class of operations across the entire benchmark influences the overall score."
"A relatively large proportion of the SM2012 score is based on system performance rated during optical character recognition (OCR) and file compression activities − things an average user will rarely if ever do," AMD added.
In addition, SM2012 focuses only on the serial processing performance of the CPU, and virtually ignores the parallel processing performance of the GPU. In particular, SM2012 scores do not take into account GPU-accelerated applications that are widely used in today?s business environments.
AMD said that it would only endorse benchmarks based on real-world computing models and software applications, and which provide useful and relevant information. AMD believes benchmarks should be constructed to provide unbiased results and be transparent to customers making decisions based on those results. Currently, AMD is evaluating other benchmarking alternatives, including encouraging the creation of an industry consortium to establish an open benchmark to measure overall system performance.
AMD also encourages anyone wanting more details about the construction and scoring methodology of the SM2012 benchmark to contact BAPCo.
Responding to AMD's decision, BAPCo released the following statement:
"Advanced Micro Devices (AMD) was, until recently, a long standing member of BAPCo. We welcomed AMD?s full participation in the two year development cycle of SYSmark 2012, AMD?s leadership role in creating the development process that BAPCo uses today and in providing expert resources for developing the workload contents. Each member in BAPCo gets one vote on any proposals made by member companies. AMD voted in support of over 80% of the SYSmark 2012 development milestones, and were supported by BAPCo in 100% of the SYSmark 2012 proposals they put forward to the consortium.
BAPCo also notes for the record that, contrary to the false assertion by AMD, BAPCo never threatened AMD with expulsion from the consortium, despite previous violations of its obligations to BAPCo under the consortium member agreement.
BAPCo is disappointed that a former member of the consortium has chosen once more to violate the confidentiality agreement they signed, in an attempt to dissuade customers from using SYSmark to assess the performance of their systems. BAPCo believes the performance measured in each of the six scenarios in SYSmark 2012, which is based on the research of its membership, fairly reflects the performance that users will see when fully utilizing the included applications."
According to reports, Nvidia and VIA have also quit BAPCo, leaving Intel as the only semiconductor maker still in the consortium.
