Split large data set when extracting data from source

CloverDX Platform
1

Hi,

We have a folder with a lot of files. We cannot run a graph and extract all the files at once, because our temp folder is not that big. Is it in the UniversalDataReader possible to split the incoming data in sets? So that the temp folder is deleted after every set.

Thnx.

Marco

2

Hi Marco,

You should use Jobflows (comes with CloverETL Corporate Server) and File operations components (ListFiles) for this.
ListFiles component can produce the list of files available for data transformation and then using ExecuteGraph, you can execute the graph for every single of the files from the list - this way you will get always one graph processed at a time.