Flat file size allowed in CloverETL

CloverDX Platform
1

What is the maximum size of a flat file we can use in CoverETL to perform ETL operations in performance regard.
For say I have several million records in a flat file and i just want to extract some limited number of rows. Can someone provide me an ideal scenario.

Thanks
Chandu

2

Hi chanduBI,

FlatFileReader should be capable of reading upwards of millions of records, now we are talking about performance it depends of the structure of the data (ex. number of columns), but it should process the records in a reasonable amount of time. The second part of your question, FlatFileReader allows to skip a number of records. You can do this by adding the number of skipped records by double clicking on FlatFileReader - Advanced - Number of skipped records. If the process of skipping records is different then I would recommend using other transformation components (ex. ExtFilter, Reformat, etc.)