Instead of measuring the average performance of a static system at maximum load, it is more logical to evaluate other characteristics – the ability of cloud services to adapt to changing loads (performance versus cost). The additional metric should also cover the reliability and security of these services in the event of individual node failures (as well as outages of data centers completely).
Important points to remember before testing
- distributed computing technologies are developing rapidly and there are many incompatible implementations;
- the cloud IaaS market is young and at the stage of development, experimentation;
- the user does not have direct contact with the provider’s equipment and interacts with the rented system resources remotely through the web interface;
- professional testing software packages cost a lot of money.
Practice and tools for testing cloud infrastructures
The basic requirements for tests for clouds are as follows: they should take into account the modern specifics of distributed computing and remote user work via the Internet on resources that are provided on demand in automatic mode, with the ability to scale in both directions.
Testing the performance of individual subsystems (processors, memory, disks) is clearly not enough for a comprehensive assessment of the cloud (in contrast to similar tests of desktop computers). But for the very initial check of IaaS health, they are still used.
Economic viability for business is one of the main factors driving the development of cloud computing. And this includes the availability of the service, and stable operation during load fluctuations, and security in the public cloud (including backup), and much more. Something that is not always estimated by performance numbers, but really greatly affects the comfortable and predictable use of the service by the user. Thus, when assessing the parameters of the cloud, you should not limit yourself to a formal test using even the most powerful software.
The minimum (and sufficient, in most cases) testing scenario before starting work with the cloud may be the following list:
- measuring the performance of virtual machines in the cloud;
- performing a test of the speed of data transfer between infrastructure nodes in the cloud;
- checking the availability and scalability, elasticity of the cloud service at different times of the day;
- testing the functionality of backup and data recovery;
- Estimation of the response time of the provider’s technical support to a request.
For professional research and performance evaluation of cloud resources, you can offer a choice of several alternative test software packages:
- SPEC Cloud (TM) IaaS 2016 Benchmark;
- Hewlett Packard Enterprise StormRunner Load;
- SOASTA CloudTest;
- SmartBear LoadComplete;
- Spirent Blitz;
- SendGrid Loader.
These and other similar packages require serious training, high user qualifications and are commercial (some of the listed software packages are also offered in a demo version with a limited validity period and reduced features).
In any case, you need to have a carefully thought out plan – a scenario for preparing and launching measurements, collecting and receiving results (benchmarks). In this case, the concept of SUT (System Under Test, system in a state of testing) is used – a set of mandatory components for starting a measurement script. This set includes those components of the cloud infrastructure, the parameters of which are of interest to the user and all the others, the functionality of which cannot be dispensed with during testing and the parameters of which must be known in advance. In addition, the software package contains drivers that simulate the workload for the IaaS infrastructure under test.
In conclusion, briefly about the main thing (when testing a public cloud)
When preparing for cloud testing, discuss with the technical support of your cloud service provider if you intend to run this or that test on their resources (hardware and software). This will allow you to get qualified advice in advance, avoid misunderstandings due to possible incompatibility of the provider’s hardware resources / virtualization system with the test software package you have chosen for your project.