| Abstract | Performance-intensive software, such as that found in high-perfo-rmance computing systems and distributed real-time and embedded
systems, increasingly executes on a multitude of platforms and user
contexts. To ensure that performance-intensive software meets its
quality of service (QoS) requirements, it must often be fine-tuned to
specific platforms/contexts by adjusting many (in some cases hun-
dreds of) configuration options. Developers who write these types
of systems must therefore try to ensure that their additions and mod-
ifications work across this large configuration space. In practice,
however, time and resource constraints often force developers to
assess performance on very few configurations and to extrapolate
from these to the entire configuration space, which allows many
performance bottlenecks and sources of QoS degradation to escape
detection until systems are fielded.
To improve the assessment of performance across large config-
uration spaces, we present a model-based approach to develop-
ing and deploying a new distributed continuous quality assurance
(DCQA) process. Our approach builds upon and extends the Skoll
environment, which is developing and validating novel software QA
processes and tools that leverage the extensive computing resources
of worldwide user communities in a distributed, continuous man-
ner to significantly and rapidly improve software quality. This pa-
per describes how our new DCQA performance assessment process
enables developers to run formally-designed screening experiments
that isolate the most significant options. After that, exhaustive ex-
periments (on the now much smaller configuration space) are con-
ducted. We implemented this process using model-based software
tools and executed it in the Skoll environment to demonstrate its ef-
fectiveness via two experiments on widely used QoS-enabled mid-
dleware. Our results show that model-based DCQA processes im-
proves developer insight into the effect of system changes on per-
formance at an acceptable cost.
|
