Benchmarking the cloud

20 Jun 2012

SPEC forms new body to test and compare the performance of cloud services.

For companies shopping for cloud computing services today, it can be difficult, if not impossible, to accurately compare the performance of one provider’s offering with that of another provider.

This new group, OSGCloud, will operate under the organisation’s Open Systems Group (OSG) and its work will include input from companies such as AMD, Dell, IBM, Intel, Oracle, Red Hat and VMware. Long-time SPEC benchmark developer Yun Chao will help the effort, as well members from the SPEC Research Cloud group.

“Cloud computing is on the rise and represents a major shift in how servers are used and how their performance is measured,” said Rema Hariharan, chair of OSGCloud. “We want to assemble the best minds to define this space, create workloads, augment existing SPEC benchmarks and develop new cloud-based benchmarks.”

While cloud performance takes into account many of the same characteristics as current SPEC benchmarks - throughput, response time and power, for example - new metrics such as elasticity, defined as how quickly a service can adapt to changing customer needs, add a new dimension to the benchmarking effort, the organisation said.

“A cloud benchmark’s primary and subordinate metrics [will] reflect important considerations such as agility/elasticity, durability, response time, throughput reliability, density and variability,” it said. “Other metrics will be documented and included as considerations necessary for comparison, but impractical or too variable to measure. These might include provisioning interval, durability, reliability, power and price.”

As with other SPEC benchmarks, vendors of cloud software and services could use the benchmarks to measure and market their own offerings.

In fact, the group is targeting three types of users for the workloads and benchmarks it will create: hardware and software vendors providing products that enable cloud services; cloud providers offering infrastructure-as-a-service (IaaS), platform-as-a-service (PaaS) and software-as-a-service (SaaS); and business customers who could use benchmark results to help them select cloud providers.

According to Hariharan, OSGCloud is off to a “great start” in creating guidelines for benchmarks with clearly defined, standardised metrics. The group has already developed a 50-page report [www.spec.org/osgcloud] that details its objectives, benchmark considerations, characteristics of a cloud benchmark, and tools for creating metrics. That said, she “would like to see wider participation, especially from cloud providers and users.”

Related articles

CSA UK & Ireland shuts up shop

Atos joins forces with EMC and VMware to create new cloud company

BYOD raises stakes and concerns for IT

Windows 8: where business information meets consumer devices

A six-stage cloud health check