Using Complex Data Reader to select Metadata based on fields

CloverDX Platform
1

Unfortunately my data comes in a few different formats as shown below:

1: A|B|{blank}|C|D
2: A|B|C|D
3: A|B|C|D|{blank}|{blank}

The data is the same for all of them, but with some extra fields stuck in. I have made metadata for each case and am trying to use Complex Data Reader to to route it to the correct Reformat before I Concatenate it and continue the processing. I am running into problems, as I have never used Advanced Data Reader and am not adept at java.

I was using this guide: http://cloveretl.wordpress.com/tag/data-integration/
but my data doesn’t have a field at the front to tell me what metadata to use, only the number of fields tells me that.

Can anyone provide assistance to me regarding this?

P.S. I keep getting an error message saying “Dynamic java compilation is not allowed (USE_DYNAMIC_COMPILER variable in DefaultProperties is false)” how do I fix this?

2

Hi,

if I understand your question correctly, you would just like to remove certain characters/strings from records. Particularly you would like to get rid off {someChars}. For this you do not need to use ComplexDataReader, which is designed to read heterogeneous structured data. All you need to do is to read the whole line into one field and using replace() function remove aforementioned strings. It may look like this:

$out.0.row = replace($in.0.row,"\\|\\{.*?\\}","");

Where the first argument is the input field that you want to update, the second one contains escaped regular expression and the last argument contains the value that should be used to replace the strings that match the previous regular expression.

The example is attached.

Hope it helps.

3

Actually, it’s that I am dealing with two types of metadata. One of them has 21 fields and the other has 20 fields. The extra field is right in the middle of them all.

You can tell based on the file name which type of data it will be, and I am trying to use ComplexDataReader to sort between the two and appropriately select the correct metadata.

4

Hi,

there are multiple ways to do this. In this case, I would probably go the way of checking what type of file I am processing and depending on that run appropriate graph. However, if you would like to use ComplexDataReader, you should configure the selector that will use the metadata of which regular expression matches the input record. I have prepared for you an example that consumes two types of metadata.

  • The first one has four string fields delimited by |

  • The second one has four string fields delimited by the same delimiter

See the selector tab in ComplexDataReader in the attached project for more information. For more information about ComplexDataReader, please refer to our documentation: http://doc.cloveretl.com/documentation/ … eader.html

Hope this helps.