Errors and rejected records

CloverDX Platform
1

Hi,

I’m testing cloverETL latest release and I don’t understand how graph execution is logged. For example when running a simple graph which load few records on a single column table, if records are too long and “max errors” param is set to high value then “Execution of graph successful !” is returned while no records were inserted in destination table. On the other hand, when “max errors” param is set to “1”, “Execution of graph failed !” is returned but there is no message about reason of failure except “Maximum # of errors exceeded…” So do you know how it is possible to have more details on errors and rejected records? Thank you very much

Mat

2

Thank you, I was talking about maxErrors parameter of DB_OUTPUT_TABLE component but your message helped me a lot ! now, I’m abble to:

  1. “catch” rejected records by using output port of DB_OUTPUT_TABLE component;
  2. “catch” logging messages by using my own logger (and its appender) to create a report;

In that case, with any logging level, errors were never logged. So I added some simple logging message in catch block following “preparedStatement.executeUpdate/Batch()” in DBOutputTable.java so that it could report errors and records number.

So I have new question(s): Is it a good approach? Do you think it could be interesting to add this few lines of logging code in a future release? (I know its time consumer cause a lot of records may be rejected but it can help users to correct theirs ETL jobs quickly)

Thanks

PS: Now we are logging something like this:
[WARN ] rec#23 - ERROR: value too long for type character varying(20)
[WARN ] rec#35 - ERROR: value too long for type character varying(20)
…

3

Hello. I am not sure which parameter “max errors” do you mean? I predict you use Universal data reader and its attribute maxErrorCount. By these circumstances the described behaviour sounds correct. Found errors should be logged in info level (that could be little bit misleading and now already fixed) of log4j output.

OtaSanek