Unit-testing and Integration Testing for Large or Medium-Complexity System Models in SIMULINK

1
votes

PROBLEM

I have a fairly large Hydro-Pneumatic system model composed of roughly 20-25 different subsystems. Each of the subsystem is then composed of digital logic, edge delay blocks, and gateway to external output ports (real-world outputs). Additionally some of the small building blocks are legacy C code imported as S-functions in order to maximise cost-benefit ratio. Each of the subsytem model has been designed in SIMULINK with the use of basic blockset i.e. no add-on commercial blocksets (e.g. aerospace, simscape, simMechanics, etc.) used.

The main issue is that I don't have sufficient knowledge of testing models in SIMULINK. I know that there is a bespoke unit testing framework that Mathworks provides for testing and verification. The trouble is that I am not quite sure how this will be relevant to my domain. Additionally, my subsystems are quite complex themselves and unit testing each of them is a nightmare. BUT if black-box testing with inputs and outputs is the only way, so be it and I am happy to accept that.

Although my question lies around testing large-scale and complex systems in SIMULINK, my aim is to get suggestions from experienced SIMULINK users who have done it in the past and will be doing it a lot in the future. I have done unit testing in SIMULINK before, but all those tests where not even near the vicinity of my correct design.

Any help will be appreciated!!

UPDATE SINCE @PHILGODDARD'S COMMENTS

I forgot to mention above that I am familiar with the HIL and PIL. However, those are valid only when you have your target real-time platform available for loop testing. What if someone wants to do a soft real-time testing?

I am waiting to go through the Mathworks large-scale system testing Webinar in a day or two. Hopfully, I can get some better answers/suggestions through that?

UPDATE SINCE @AM304's COMMENTS

To clarify the context a bit further, we are modelling all the interacting systems in software without involving any physical devices i.e. the modelling and simulation of all the systems in software with the outputs being displayed in operator/instructor terminals..... e.g. we have electrical, air conditioning, and hydro-pneumatic systems working together, but we have modelled them in software. therefore, when we simulate it, all the necessary signals are generated from the behaviour of the software models, no actual hardware/physical device is involved in delivering those behavioural outputs.

3
unit-testingtestingsimulinkmodelinglarge-scale
The first thing I'd do is break the model up (if possible) into separate models; combining them at the highest level using Model blocks. This will facilitate writing test cases (i.e. developing separate test models) for each of the individual components. Unfortunately there's no magic bullet for creating the test models (but the tone of your post seems to indicate that you are aware of this). – Phil Goddard
@PhilGoddard Do you think HIL or PIL is an acceptable, but quite expensive answer for testing large-scale system? – ha9u63ar
@PhilGoddard And also, if you look at my problem description, the entire system is broken down into subsystem models. Those require testing - Isn't that what you are suggesting too? Or am I missing your point? – ha9u63ar
I took your original wording to mean that you were using generic subsystems, not that you were using Model blocks to break it up into separate models. Your original post doesn't mention targeting hardware, so I wouldn't have thought about using HIL or PIL approaches. I'd still opt for breaking the model up using Model blocks; then having separate models set up as test harnesses for each of the components; and writing MATLAB code to run/exercise the test harnesses with different sets of test inputs. This is essentially a roll your own version of some of the Mathworks testing tools. – Phil Goddard
Are you based in the UK? If so, it might be a good idea to go to the MATLAB Expo (matlabexpo.com/uk) and you can talk directly to the developers, in particular the guy giving this talk: matlabexpo.com/uk/2013/embracing-complexity.html – am304

3 Answers

2
votes

I would look into Simulink Design Verifier and Simulink Verification&Validation. Specifically, they offer features such as automated test generation and coverage analysis, which I think are applicable to your problem.

As a side note...

With regards to the matlab unit-testing framework which you mentioned in your post, this is basically a MATLAB implementation of the xUnit framework. So it obviously comes in handy if you want to be able to apply xUnit design patterns to tests of MATLAB code.

The framework can be used to write tests for a Simulink model, but keep in mind that you have to be able to write the tests in MATLAB code. So this just means making use of the Simulink command-line interface (e.g. commands like set_param and sim) to setup and exercise your model, then use the verification methods of the framework (e.g. verifyEqual) to compare actual and expected results.

0
votes

C-code as well as Simulink models can be tested using TPT. There is no difference in the test development as well as in the test assessment but in the test execution environment. For testing C-code one can decide if the C-code shall be tested in the so called SiL-mode in Simulink where the C-Code is embedded in Simulink as so called S-function. The generation of the S-function can be done automatically by m-scripts dependent of the code generator.

For the Simulink Coder may use the following MATLAB commands to force SiL:

set_param(<testFrameName>,'RTWSystemTargetFile','rtwsfcn.tlc');
set_param(<testFrameName>,'RTWTemplateMakefile','rtwsfcn_default_tmf');
rtwbuild(<subsystem to be tested>);

For TargetLink the script uses the TargetLink commands “tl_built_host” and “tl_set_sim_mode”:

tl_build_host('Model', <testFrameName>, 'TlSubsystems', <subsystem to be tested>);
tl_set_sim_mode('Model', <testFrameName>, 'TlSubsystems', <subsystem to be tested>, 'SimMode', 'TL_CODE_HOST');

Alternatively in TPT C-code can be tested in two other ways that do not require Simulink but some manual programming and compiling.

The first alternative is using the so called EXE-platform where a test harness is built in C-code and to be compiled by the user by its own compiler. The generation of the test harness is supported in the EXE-platform configuration dialog in TPT.

The second alternative is the so called FUSION platform which is a co-simulation environment. The FUSION is an open architecture where user can adapt their own system under test C-code using a well-defined and documented API. The c-code together with the API forms a so called node that can simulated at the FUISON platform as single node or together with other nodes. In case of automatic code generation nevertheless most user use MiL and SiL testing in Simulink because it can be done fully automatic including Back to back regression testing between MiL and SiL.

Why should one test C-code instead of models? The reason is that mostly the functional development is done in floating point notation while the implementation in the target ECU is to be done in fix-point notation. The process of scaling and the fix-point calculation is at least for TargetLink done during code generation. Thus the floating point (MiL) results are to be compared with the fix-point implementation (SiL) because from the experience the scaling introduces many errors.

If 100% statement or condition coverage there is a feature called TASMO in TPT. TASMO tries to generate test cases automatically to fulfill a maximum of coverage of a Simulink or TargetLink model. The algorithm is based on optimization and is search based. Please note that automatic test case generation should not replace functional test cases. The coverage goals can be checked by using code coverage tools like CTC++, V&V toolbox or TargetLink that brings its own coverage measurement. By looking at the coverage results the user may decide on its own how to stimulate the blind spots and can identify dead code.

I’m one of the TPT developer. For further information about TPT you can visit our website.

0
votes

MathWorks changed it's V&V product offering in R2017b (September 2017). I recommend taking a look at the following links to see how they can help with unit and integration level testing of designs in Simulink/Stateflow.

  1. Simulink Test - provides an environment to test your Simulink/Stateflow design
  2. Simulink Coverage - produces coverage metrics to determine how much of your design has been tested
  3. Simulink Design Verifier - automatically generates tests to meet functional or coverage criteria

There is an overview page on the MathWorks website that provides additional detail on the overall offering for verification, validation, and test.