Unit Testing of CloverETL Graphs

CloverDX Platform
1

We have been using CloverETL for a few months now. We have a 3 user license and are creating graphs for a number of purposes.

We have graphs that perform transformations using CTL. What is the best way to test these graphs?
What are the best practices for testing graphs in CloverETL?

I am really looking for something like Unit Testing, so that I can change a graph and then run through the tests and verify that it works as expected. As with Unit Testing, this will work great when there is a graph that has not been touched in months but that needs an update or fix.

Thanks for any suggestions.

2

Hi Paul,

There is couple of options:

  1. Weakest is checking number of records passing important edges of graph. This can be easily done using JobFlows (Server Corporate and higher) - component http://doc.cloveretl.com/documentation/ … graph.html provides statistic for executed graph in output mapping. In Designer, you have to count records on edges (e.g. by http://doc.cloveretl.com/documentation/ … egate.html) and store them somewhere. Then compare referential counts with recorded.

  2. Stronger variant (DATT/CTL) requires comparing expected graph results with recorded. In this case you compare real graph outputs - CSV, XLSX, DB, … Checking can be done using another graph taking 2 inputs and providing diff log.

  3. There is also possibility to write audit log during graph execution and then compare with referential using common diff tools.

Also, because you have paid licenses, you can use commercial support, which has much better reaction times than public forum. Just send your questions to support@cloveretl.com

Please let me know if you have further questions.

3

This is the approach that has worked for me so far:

This solution seems to work for the graphs that I develop that take database or flat file inputs and transform those to database or XML outputs. This solution would not necessarily work with lookups.

After every Reader, I put a SimpleGather and add a UniversalDataReader. Before every Writer, I put a SimpleCopy and add a UniversalDataWriter.

I add a parameter called TESTCASE.

The UniversalDataReader and UniversalDataWriter read and write test input and output files that have TESTCASE in the name.

For each test case, I create input data and expected output data. I disable the database readers/writers and enable the test readers/writers.

Then I run a Testing graph that calls my graph multiple times each with a different TESTCASE value. The Testing graph compares the output to the expected output.

When I am done, I disable the test readers/writers and enable the database readers/writers.

It is not an elegant solution, but it meets my needs. I am confident that I can modify my graph and then rerun the tests to ensure that I have not broken anything.