Hi, I’m facing some issue while trying to read the flat file lets say csv which is of 47.5 GB in total size , trying to read this file and filter on based on some condition and loading into respective database table. But im not able to read the flat file for at least 30 million rows because of the file worth of having 800+ million rows. I have tried increasing the heap size from initial to 16384 and max size to 60000 but still there is no progress, the graph is erroring out to space issue, how can i resolve this? if anyone had hint let me know
Hi Balaji,
Clover normally processes data in a streaming way - in your case, it would be reading your CSV row by row, filtering each row and then immediately sending to DB for writing.
If it fails due to space issues, it should not be memory issues.. Could be that if you run this in designer, then each edge connecting components run in debug mode and captures every record flowing through it for debugging purposes.
A quick thing you can try if you are executing this from Designer is to go to:
Run→Run Configurations→your_graph_name and check “Disable edge debugging” to make sure not debug data are stored.
But you should put here the console output with the error Clover displays and ideally also your transformation graph (the .grf file) so it can be further analyzed.
David