CloverETL is now CloverDX - Learn Why

Looking for a better error output

Support/help with CloverETL (4.9) and CloverDX (5.0 or newer) implementation problems

jcatoe
Posts: 5
Joined: Wed Jan 17, 2018 4:50 pm

Looking for a better error output

Postby jcatoe » Fri Mar 09, 2018 11:13 pm

I'm trying to solve an issue for our analysts.

Every time we read in a file, there will inevitably be a couple of rows that don't make it through and they are currently getting deposited into a separate csv.

The issue is that the row that causes the DataReader_Error is not in the output, just a record id. So now the analysts have to go find the csv with the error output, collect all of the record ids that didn't make it through, locate the original spreadsheets, and then those specific rows in these massive spreadsheets.

My question is: is there a way to send the actual row to the csv so they don't have to hunt through all of these files and rows just to discover that it was appropriately discarded.

Please and thank you!

the_goat
Posts: 21
Joined: Fri Dec 01, 2017 6:16 am

Re: Looking for a better error output

Postby the_goat » Mon Mar 12, 2018 2:23 pm

If I understand correctly, you do not have this output from FlatFileReader/UniversalDataReader? Or you're talking about very different component here?

FFR_error.PNG
FFR_error.PNG (54.31 KiB) Viewed 867 times

jcatoe
Posts: 5
Joined: Wed Jan 17, 2018 4:50 pm

Re: Looking for a better error output

Postby jcatoe » Mon Mar 12, 2018 6:34 pm

Interesting, is that only for FlatFile and Universal because most of my graphs use the SpreadsheetDataReader and it gives me:

Code: Select all

<Record fieldDelimiter="|" name="SpreadsheetDataReader_Error" recordDelimiter="\n" type="delimited">
<Field name="recordID" trim="true" type="integer"/>
<Field name="file" type="string"/>
<Field name="sheet" type="string"/>
<Field name="fieldIndex" trim="true" type="integer"/>
<Field name="fieldName" type="string"/>
<Field name="cellCoords" type="string"/>
<Field name="cellValue" type="string"/>
<Field name="cellType" type="string"/>
<Field name="cellFormat" type="string"/>
<Field name="message" type="string"/>
</Record>


The cell value exists but it's not always easy to tell what actually happened with that row.

the_goat
Posts: 21
Joined: Fri Dec 01, 2017 6:16 am

Re: Looking for a better error output

Postby the_goat » Tue Mar 13, 2018 1:25 pm

Oh, thought we're talking about CSV files here. Let me think about a solution here, should be workaroundable. What kinds of errors those usually are? Conversion ones?

bartonv
Posts: 96
Joined: Wed May 03, 2017 12:10 pm

Re: Looking for a better error output

Postby bartonv » Tue Mar 13, 2018 3:07 pm

Hello Jcatoe,
there are a couple of different options that you can choose in this situation. However, they can be summed up in these 2 general approaches basically:
  1. Converting the XLS(X) file into a CSV file and then reading the CSV file using the FlatFileReader component (as demonstrated by the_goat in the screenshot).
  2. Reading the XLS(X) file as if all the fields were string-type fields and then using the CloverETL functions and features to validate and convert those fields into their respective types if possible.
Please note that these options would make sense only if the errors you are dealing with are purely conversional (for example, when trying to read a string-type field as integer). At this point, I believe it would be most beneficial for an efficient and effective resolution of this case if you either attached an example graph that tries to read an XLS(X) file or at least if you listed those various types of errors that you get when reading the XLS(X) file using the SpreadsheetDataReader component.
Kind regards,
---
Vladimir Barton
CloverCARE Support
CloverDX

Visit us online at http://www.cloverdx.com