Multiplying output

CloverDX Platform
1

In every Delimiter Data Reader that I’m using for reading flat files i’m getting endless loop. For ex. I have flat file with 30 rows, but on output it gives milions of rows! What’s wrong with my env?

INFO [main] - Checking graph configuration…
DEBUG [main] - Opening input file D:/ETL/validator/data-in/file.DAT
DEBUG [main] - Reading input file D:/ETL/validator/data-in/file.DAT
INFO [WatchDog] - Thread started.
INFO [WatchDog] - Running on 2 CPU(s) max available memory for JVM 65088 KB
INFO [WatchDog] - [Clover] Initializing phase: 0
DEBUG [WatchDog] - initializing edges:
DEBUG [WatchDog] - all edges initialized successfully…
DEBUG [WatchDog] - initializing nodes:
DEBUG [WatchDog] - Opening input file D:/ETL/validator/data-in/file.DAT
DEBUG [WatchDog] - Reading input file D:/ETL/validator/data-in/file.DAT
DEBUG [WatchDog] - DELIMITED_DATA_READER0 …OK
DEBUG [WatchDog] - TRASH0 …OK
INFO [WatchDog] - [Clover] phase: 0 initialized successfully.
INFO [WatchDog] - Starting up all nodes in phase [0]
DEBUG [WatchDog] - DELIMITED_DATA_READER0 … started
DEBUG [WatchDog] - TRASH0 … started
INFO [WatchDog] - Sucessfully started all nodes in phase!
INFO [WatchDog] - ---------------------** Start of tracking Log for phase [0] **-------------------
INFO [WatchDog] - Time: 22/08/08 10:35:09
INFO [WatchDog] - Node Status Port #Records #KB Rec/s KB/s
INFO [WatchDog] - ----------------------------------------------------------------------------------
INFO [WatchDog] - DELIMITED_DATA_READER0 RUNNING
INFO [WatchDog] - %cpu:0.98 Out:0 1738759 237720 -4 -1
INFO [WatchDog] - TRASH0 RUNNING
INFO [WatchDog] - %cpu:0.83 In:0 1738759 237720 -4 -1
INFO [WatchDog] - ---------------------------------** End of Log **--------------------------------
INFO [WatchDog] - ---------------------** Start of tracking Log for phase [0] **-------------------
INFO [WatchDog] - Time: 22/08/08 10:35:14
INFO [WatchDog] - Node Status Port #Records #KB Rec/s KB/s
INFO [WatchDog] - ----------------------------------------------------------------------------------
INFO [WatchDog] - DELIMITED_DATA_READER0 RUNNING
INFO [WatchDog] - %cpu:0.97 Out:0 3521796 481495 363440 49689
INFO [WatchDog] - TRASH0 RUNNING
INFO [WatchDog] - %cpu:0.82 In:0 3521801 481496 363441 49689
INFO [WatchDog] - ---------------------------------** End of Log **--------------------------------

2

Try to use UniversalDataReader - that is a successor to DelimitedDataReader which is able to handle more problems in imput data.

Could you send/post also the .grf file which is causing the problem ?

Nonetheless, your problem is quite odd. We have not seen this behavior before with this component.

3

Same story with Universal Data Reader.

Graph: http://www.sendspace.com/file/lcfs8w

4

This is for sure environment problem (probably eclipse). 1/10 times graph works fine.

5

Furthermore, the output file includes only those values, which are Autofilling. Non of the “in file” data is proceeded to output.

6

I checked up your graph file and I didn’t find a metadata file. I suppose you have autofilling for all metadata. Ie:

Then you are right. The clover should throw an exception because there isn’t a metadata description how to read the data.

7

If you want to get only file name I suppose to use SystemExecute component. Something like dir -… > fileNames.txt and then use a reader.