Data Reader with Comma at EOL

Support/help with CloverETL (4.9) and CloverDX (5.0 or newer) implementation problems

gtateco
Posts: 2
Joined: Fri Jul 01, 2011 11:03 pm

Data Reader with Comma at EOL

Postby gtateco » Fri Jul 01, 2011 11:13 pm

I'm trying to read a .csv with the Universal Data Reader that has a comma a /r and /n for each end of line(EOL). I'm not able to get the reader to go past the first record, it always fails on the last field. Here is the error message.

Parse error: The size of data buffer for data record is only 12288. Set appropriate parameter in defautProperties file. when parsing record #1 field blank

I've tried setting the last field to ,\r\n for the delimiter and tried creating a bogus blank field with just \r\n. Both return the same error which seems kind of strange to me.

Here is a sample of the data.

0,13.65,13.65,"","","","","","Nordrhein-Westfalen",

Any help would be great, thank you!

Glenn

teebag
Posts: 2
Joined: Mon Jul 04, 2011 12:52 pm

Re: Data Reader with Comma at EOL

Postby teebag » Tue Jul 05, 2011 8:47 am

You need to add a 10 field to your metadata, type integer, delimiter \r\n

Name Type Delimiter
Field1 Integer ,
...
Field8 String ,
Field9 String ,
Field10 integer \r\n

gtateco
Posts: 2
Joined: Fri Jul 01, 2011 11:03 pm

Re: Data Reader with Comma at EOL

Postby gtateco » Tue Jul 05, 2011 5:56 pm

Thanks for the idea, that worked great. Of course I had two issues which was compounding the issue.

The other issue is I'm having trouble reading the file directly from the customer. It is coming from a Linux box, probably output from some sort of scripting language. The format is UTF-8 and although it reads fine in the preview the Java engine has issues with it, ex. you can't see the field preview when editing the metadata. If I open on my windows computer in VIM and copy and paste it to a new file it works great. That workaround is OK for now, but this is a daily process that needs to be automated.

Once again, any ideas appreciated!

Glenn

jurban
Posts: 163
Joined: Fri Jul 20, 2007 9:25 am

Re: Data Reader with Comma at EOL

Postby jurban » Thu Jul 07, 2011 9:27 am

Hi Glenn,

could you please describe your problem in more details please? Do I understand it correctly that on the same computer the preview works OK (when creating metadata) but it doesn't work when editing metadata? Do you see some error message?

On first thought I would expect some problems with line delimiters - can you try just \n or just \n?

Best regards,
Jaro
Jaroslav Urban
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com

avackova
Posts: 841
Joined: Fri Jul 20, 2007 9:28 am

Re: Data Reader with Comma at EOL

Postby avackova » Tue Jul 12, 2011 11:42 am

Hello Glenn,
the problem is probably with line breaks. To make your graph working with both, linux and windows ends of line, you can use alternative delimiter \n\\|\r\n (for full information see Changing and Defining Delimiters)
Agata Vackova
Javlin a.s.
[email protected]


cron