Require implementing logging information

CloverDX Platform
1

Hi,

I am implementing a solution where the source of data is csv file and destination is sql/oracle database after some mapping, joins and transformation.
My requirement is to log/record the total number of rows processed, how many succeeded and how many are failed. So is there any inbuilt logging feature in CloverETL Designer which I can use to accomplish my goal or do I need to write custom logic ?

PS: I am using CloverETL Designer 3.5.0

Regards
Gurdeep

2

Hi Gurdeep,

From your post, I am not able to see what type of information you need to get, since the graph may fail on several places due to invalid data. So here are the answers to your possible questions. If you want to:

  • get count of all rows (records) that have not been read from a CSV file, you can set a maximum acceptable number of errors in UniversalDataReader. These records will be sent to the second output port. Then you can simply use Aggregate component and apply its function „Count“. On the output port of this component you will get the information you needed.

  • get number of records that were not inserted into your database table, you may simply use the first DBOutputTable’s output port the same way as I described above example above. Again, do not forget to enter maximum number of errors.

  • want to check your data somewhere in the middle of your graph, you may use Validator component. This component is very powerful in terms of the rules you can apply to your data. For more information on Validator, please refer to http://www.cloveretl.com/blog/cloveretl … validator/ or to our documentation (http://doc.cloveretl.com/documentation/ … dator.html).

Apart to the options above, if you run the graph as part of some larger transformation (orchestrated by a JobFlow), you might take into consideration getting your tracking data (such as record counts on all input/output ports etc.) from our ExecuteGraph/ExecuteJobFlows components (http://doc.cloveretl.com/documentation/ … graph.html) - these JobFlow components are distributed as part of CloverETL Server Corporate.

Hope this helps.

3

Thanks for your reply.

How can I log errors which occurs during ExtHashJoin or Reformat or any other component which does not support maximum numbers of error support. Basically I want logging at every step of transformation.

will Appreciate your help.

4

Hi,

Alternatively to Jan’s suggestions you can also:

* Use http://doc.cloveretl.com/documentation/ … ecopy.html to fork arbitrarily edge and then http://doc.cloveretl.com/documentation/ … egate.html for statistic calculation; then store result into file
* In CLT2 there is possibility to print logs via printLog function (http://doc.cloveretl.com/documentation/ … -ctl2.html)

I hope this helps.

5

Thanks for your reply.

What I want to do is that when any transformation fails then I need to log the record number and the error occurred with that record into a separate file or database table so that I can refer to the log that which records are failed during transformation.

probably I can put try catch block in transformation section and log the error in catch block. but I am helpless that how can i write the data into separate file with error detail. can you send me a sample code snippet which help me out.

6

Hi,

Your description is unfortunately too generic to answer. There are many scopes:

  1. Catch error on graph:

When there is error causing graph to stop, you can detect this in parent graph. This can be easily done using JobFlows available in commercial editions of CloverETL Server http://doc.cloveretl.com/documentation/ … graph.html . In OpenSource edition you can use http://doc.cloveretl.com/documentation/ … graph.html . Both this components have ability to report error of child graph.

  1. Catch on component:

This was already discussed by Jan and me. Error handling depends on component, please check its documentation.

  1. Catch in CTL2 code:

Please see http://doc.cloveretl.com/documentation/ … -ctl2.html . You can then send (http://doc.cloveretl.com/documentation/ … alues.html) such error to second port of Reformat and store into file.

If you combine all tips from our previous answers you should be able to handle arbitrary error.

I hope this helps. If not, please send us example. Graph with marked points of failure, in best case.