A FRAMEWORK FOR MEASURING QUALITY OF MODELS: EXPERIENCES FROM A SERIES OF CONTROLLED EXPERIMENTS

Controlled experiments in model-based software engineering, especially those involving human subjects performing modeling tasks, often require comparing models produced by experiment subjects with reference models, which are considered to be correct and complete. The purpose of such comparison is to assess the quality of models produced by experiment subjects so that experiment hypotheses can be accepted or rejected. The quality of models is typically measured quantitatively based on metrics. Manually defining such metrics for large modeling languages is often cumbersome and error-prone. It can also result in metrics that do not systematically consider relevant details and in turn may produce biased results. In this paper, we present a generic quality measurement framework to automatically generate quality metrics for MOF-based metamodels (M2 level) (e.g., the UML metamodel), which in turn can be used to measure quality of models (M1 level) (instances of the MOF-based metamodels). A conceptual model is provided to formally describe the main concepts and their relationships of the framework. The definition of the framework is formally specified and the transformation algorithm is also presented, which are therefore implemented as a prototype tool. We applied our tool to automatically obtain quality metrics for UML state machine diagrams and successfully applied them in a controlled experiment. The automatically generated metrics for UML class and sequence diagrams were compared with two sets of manually constructed metrics for UML class and sequence diagrams applied in two controlled experiments. Results show that it is more efficient and systematic to define quality metrics with the tool than doing it manually.