Split large data set when extracting data from source

Support/help with CloverETL (4.9) and CloverDX (5.0 or newer) implementation problems

marcosnels
Posts: 1
Joined: Mon Feb 01, 2016 3:59 pm

Split large data set when extracting data from source

Postby marcosnels » Mon Feb 01, 2016 4:01 pm

Hi,

We have a folder with a lot of files. We cannot run a graph and extract all the files at once, because our temp folder is not that big. Is it in the UniversalDataReader possible to split the incoming data in sets? So that the temp folder is deleted after every set.

Thnx.

Marco

slechtaj
Posts: 192
Joined: Wed Aug 15, 2012 8:18 am

Re: Split large data set when extracting data from source

Postby slechtaj » Wed Feb 03, 2016 8:00 pm

Hi Marco,

You should use Jobflows (comes with CloverETL Corporate Server) and File operations components (ListFiles) for this.
ListFiles component can produce the list of files available for data transformation and then using ExecuteGraph, you can execute the graph for every single of the files from the list - this way you will get always one graph processed at a time.
Jan Slechta
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com

How to speed up communication with CloverCARE support


cron