Testing real-time vehicular systems challenges the tester to design test cases for concurrent and sequential input events, emulating unexpected user and usage profiles. The vehicle response should be robust to unexpected user actions. Sequence Covering Arrays (SCA) offer an approach which can emulate such unexpected user actions by generating an optimized set of test vectors which cover all possible t-way sequences of events. The objective of this research was to find an efficient nonfunctional sequence testing (NFST) strategy for testing the robustness of real-time automotive embedded systems measured by their ability to recover (prove-out test) after applying sequences of user and usage patterns generated by combinatorial test algorithms, considered as "noisy" inputs. The method was validated with a case study of an automotive embedded system tested at Hardware-In-the-Loop (HIL) level. The random sequences were able to alter the system functionality observed at the prove-out test. The outputs were affected in different ways such as unexpected events, lead-lag phase shift, and shifted parameter values. The method is effective to find concerns and efficient due to the reduction in the number of test cases compared to the full factorial alternative. The method is also efficient because it only requires the observation of the recoverability of the system at the prove-out test, without the need of determining the system response to the noisy combinatorial inputs. The NFST method has been recommended as a testing standard for our partner vehicle manufacturer for testing electronic and software systems applicable to all vehicle variants.
Testing real-time vehicular systems challenges the tester to design test cases for concurrent and sequential input events, emulating unexpected user and usage profiles. The vehicle response should be robust to unexpected user actions. Sequence Covering Arrays (SCA) offer an approach which can emulate such unexpected user actions by generating an optimized set of test vectors which cover all possible t-way sequences of events. The objective of this research was to find an efficient nonfunctional sequence testing (NFST) strategy for testing the robustness of real-time automotive embedded systems measured by their ability to recover (prove-out test) after applying sequences of user and usage patterns generated by combinatorial test algorithms, considered as "noisy" inputs. The method was validated with a case study of an automotive embedded system tested at Hardware-In-the-Loop (HIL) level. The random sequences were able to alter the system functionality observed at the prove-out test. The outputs were affected in different ways such as unexpected events, lead-lag phase shift, and shifted parameter values. The method is effective to find concerns and efficient due to the reduction in the number of test cases compared to the full factorial alternative. The method is also efficient because it only requires the observation of the recoverability of the system at the prove-out test, without the need of determining the system response to the noisy combinatorial inputs. The NFST method has been recommended as a testing standard for our partner vehicle manufacturer for testing electronic and software systems applicable to all vehicle variants.
