CSV file with non-standard delimiters and quote characters

CloverDX Platform
1

What’s the best way to process a CSV file with non-standard delimiters and quote characters?

In my case the delimiter is \u0014 and the quote character is \u00fe . I’m in the metadata editor now in the designer, and I’m trying to specify \u0014 as the delimiter, and it’s not splitting properly. There also doesn’t seem to be a way to specify a quote character.

Thanks!

2

Hello Andy,
you can set such delimiter with ctl: set field delimiter as `‘\u0014’` - all, that is between `` characters is interpreted as ctl code.
Unfortunately only single and double quote characters are recognized as quotation by Clover.
But you can use following workaround to parse the data:

  • read full record as one string field

  • translate \u00fe to " in Reformat:

function transform() {  
	$0.field1 := translate($0.field1,'\u00fe','"');  
}
  • send translated field to Data Reader - use port:$0.field1:discrete in fileURL attribute. Remember to set eofAsDelimiter=“true” on output metadata