Scenario-based Automated Evaluation of Test Traces of Autonomous Systems

Testing the robustness and safety of autonomous systems (like domestic or manufacturing robots) is a challenging task since these systems can make decisions on their own depending on their environment. We proposed a model based testing approach to capture the context and basic safety-related behavioural requirements of such systems, and to generate test data representing stressful contexts. During the execution of these tests in a real or simulator based test environment, the captured test traces shall be checked by comparison with the requirements in order to detect the violation of any requirement in each situation. In this paper we analyse this test evaluation problem and propose a method that can be used for efficient comparison.