How to validate file structure against meta data

CloverDX Platform
1

I am wondering what the best way is to validate that a file structurally matches the metadata. Is there a way to determine if:

1. The number of columns in a CSV actually match the number in the metadata description
2. If the column headers in a CSV match the column headers in the metadata description
3. If column headers are present in a CSV file
4. Generate an exception/notification of some sort if one of the above is FALSE

Thanks in advance…

-Martin

2

Hello Martin,

Regarding your questions:

1. You have to set in the metadata exactly the same number of fields as a number of columns you expect in your CSV file. If you have in your CSV file different number of columns than what is defined in the metadata, it will cause error.

2. and 3. You can access dynamically in the CTL to the record’s fields, for more details please see the documentation:
http://doc.cloveretl.com/documentation/UserGuide/index.jsp?topic=/com.cloveretl.gui.docs/docs/field-access-functions-ctl2.html

4. You can throw your own error in the CTL by the function

raise_error(string)

This function takes one string argument and throws out error with the message specified as the argument.

I prepared for you sample demo of your desired functionality (points 2, 3 and 4), so please take a look into the sample project. It demonstrates reading CSV file, parsing first line and checking headers of columns. If the CSV file has not headers of columns named as the fields of metadata or if the CSV file has columns in different order, the error is thrown. There is a sample CSV file located in data-in/data.csv.

I hope this sample will help you to find your solution.