I am relatively new to CloverETL and is looking for some solution to the below problem:
- I am reading data from a .dat file (field delimiter being ‘|’) one field being modified date.
- Need to incrementally load data based on the modified date field. Example if 2 records get modified/added in the input file I need to have them in a separate extract file, say, incremental.dat
I have gone through the forums/user guide and came across incremental file/incremental key.However it would be great if anyone gives me step by step solution to this.
I am using UniversalDataReader from an input file called customer_input.dat
I prepared for you simple example of incremental reading. In this example there is used incremental reading from input file (based on date field). When you run the graph for the first time, all records will be stored into output file called incremental.dat. The second run of the graph will read only new lines from input file (please add some new lines into the input file for test).
However this works only if a new record is added to the input file. What if the attribute values of the existing records gets modified? In this case using incremental file/incremental key is not identifying the modified records. Could you please provide any example for the same?
Unfortunately, it is not possible to use incremental reading from file when existing records were modified. The incremental reading mechanism does not recognize these changes.
You have to read a whole file and search for changed records. You can, for example, compare new content of your input file with a previous copy of this file (you can use, for example, DataIntersection component).