Clover Batch problem - DBOutputTable

[enrique.g]

Hey guys,

I’m running an ETL graph where I use a DBOutputTable with an Oracle DB.
When I use batch mode, if one record gets rejected, it is sent to Port0. Not only is that record sent to Port0, but also all the other records in the batch are sent to that port too. This is mentioned early in a thread: viewtopic.php?f=4&t=4570&p=7199

Problem here is that records sent to Port0 are not inserted in the database at all. Is there a workaround to this problem (while keeping the batch as I’m trying to insert over a millon records?
Any help would be greatly appreciated.

Thanks in advance!

Enrique

[slechtaj]

Hi Enrique,

It seems the batch processing does not work as designed. We have created new tickets in our issue tracking system for this issue. Please refer to these tickets to keep yourself updated:
https://bug.javlin.eu/browse/CL-2700
https://bug.javlin.eu/browse/CL-2702

For now, you may take into consideration using the following workaround:

• Create a graph that genrates external file that contains SQL DML query.

• Enrich these statements with LOG ERROR clause in order to make sure that all records will be handeled by Oracle database. (for more information please refer to: http://www.oracle.com/technetwork/issue-archive/2006/06-mar/o26performance-096310.html or http://www.oracle-base.com/articles/10g/dml-error-logging-10gr2.php)

• Execute the PL/SQL external script using DBExecute.

• You may also pass these scripts through edge instead of using external file, but you need to make sure the FIELD_LIMIT_SIZE (http://doc.cloveretl.com/documentation/UserGuide/index.jsp?topic=/com.cloveretl.gui.docs/docs/cloveretl-default-settings.html) is approriate to length of the scripts.