multiplying output

Support/help with CloverETL (4.9) and CloverDX (5.0 or newer) implementation problems

schabluk
Posts: 28
Joined: Sun May 25, 2008 6:58 pm

multiplying output

Postby schabluk » Fri Aug 22, 2008 9:39 am

In every Delimiter Data Reader that I'm using for reading flat files i'm getting endless loop. For ex. I have flat file with 30 rows, but on output it gives milions of rows! What's wrong with my env?


INFO [main] - Checking graph configuration...
DEBUG [main] - Opening input file D:/ETL/validator/data-in/file.DAT
DEBUG [main] - Reading input file D:/ETL/validator/data-in/file.DAT
INFO [WatchDog] - Thread started.
INFO [WatchDog] - Running on 2 CPU(s) max available memory for JVM 65088 KB
INFO [WatchDog] - [Clover] Initializing phase: 0
DEBUG [WatchDog] - initializing edges:
DEBUG [WatchDog] - all edges initialized successfully...
DEBUG [WatchDog] - initializing nodes:
DEBUG [WatchDog] - Opening input file D:/ETL/validator/data-in/file.DAT
DEBUG [WatchDog] - Reading input file D:/ETL/validator/data-in/file.DAT
DEBUG [WatchDog] - DELIMITED_DATA_READER0 ...OK
DEBUG [WatchDog] - TRASH0 ...OK
INFO [WatchDog] - [Clover] phase: 0 initialized successfully.
INFO [WatchDog] - Starting up all nodes in phase [0]
DEBUG [WatchDog] - DELIMITED_DATA_READER0 ... started
DEBUG [WatchDog] - TRASH0 ... started
INFO [WatchDog] - Sucessfully started all nodes in phase!
INFO [WatchDog] - ---------------------** Start of tracking Log for phase [0] **-------------------
INFO [WatchDog] - Time: 22/08/08 10:35:09
INFO [WatchDog] - Node Status Port #Records #KB Rec/s KB/s
INFO [WatchDog] - ----------------------------------------------------------------------------------
INFO [WatchDog] - DELIMITED_DATA_READER0 RUNNING
INFO [WatchDog] - %cpu:0.98 Out:0 1738759 237720 -4 -1
INFO [WatchDog] - TRASH0 RUNNING
INFO [WatchDog] - %cpu:0.83 In:0 1738759 237720 -4 -1
INFO [WatchDog] - ---------------------------------** End of Log **--------------------------------
INFO [WatchDog] - ---------------------** Start of tracking Log for phase [0] **-------------------
INFO [WatchDog] - Time: 22/08/08 10:35:14
INFO [WatchDog] - Node Status Port #Records #KB Rec/s KB/s
INFO [WatchDog] - ----------------------------------------------------------------------------------
INFO [WatchDog] - DELIMITED_DATA_READER0 RUNNING
INFO [WatchDog] - %cpu:0.97 Out:0 3521796 481495 363440 49689
INFO [WatchDog] - TRASH0 RUNNING
INFO [WatchDog] - %cpu:0.82 In:0 3521801 481496 363441 49689
INFO [WatchDog] - ---------------------------------** End of Log **--------------------------------

dpavlis
Posts: 182
Joined: Sat Mar 10, 2007 8:12 pm

Postby dpavlis » Fri Aug 22, 2008 10:20 am

Try to use UniversalDataReader - that is a successor to DelimitedDataReader which is able to handle more problems in imput data.

Could you send/post also the .grf file which is causing the problem ?

Nonetheless, your problem is quite odd. We have not seen this behavior before with this component.

schabluk
Posts: 28
Joined: Sun May 25, 2008 6:58 pm

Postby schabluk » Fri Aug 22, 2008 11:04 am

Same story with Universal Data Reader.

Graph: http://www.sendspace.com/file/lcfs8w

schabluk
Posts: 28
Joined: Sun May 25, 2008 6:58 pm

Postby schabluk » Fri Aug 22, 2008 11:25 am

This is for sure environment problem (probably eclipse). 1/10 times graph works fine.

schabluk
Posts: 28
Joined: Sun May 25, 2008 6:58 pm

Postby schabluk » Sat Aug 23, 2008 2:24 pm

Furthermore, the output file includes only those values, which are Autofilling. Non of the "in file" data is proceeded to output.

jausperger
Posts: 19
Joined: Wed Apr 04, 2007 2:44 pm

Postby jausperger » Thu Sep 04, 2008 8:43 am

I checked up your graph file and I didn't find a metadata file. I suppose you have autofilling for all metadata. Ie:

<Record fieldDelimiter="|" name="recordName1" recordDelimiter="\n" type="delimited">
<Field auto_filling="source_name" name="field1" type="string"/>
</Record>

Then you are right. The clover should throw an exception because there isn't a metadata description how to read the data.

jausperger
Posts: 19
Joined: Wed Apr 04, 2007 2:44 pm

Postby jausperger » Thu Sep 04, 2008 9:05 am

If you want to get only file name I suppose to use SystemExecute component. Something like dir -... > fileNames.txt and then use a reader.