# Process only unprocessed files

**URL:** <https://forum.cloverdx.com/t/process-only-unprocessed-files/765>\
**Category:** CloverDX Platform\
**Created:** [October 20, 2010, 12:00am UTC](https://forum.cloverdx.com/t/process-only-unprocessed-files/765 "2010-10-20T00:00:00Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![megreddy](https://avatars.discourse-cdn.com/v4/letter/m/58f4c7/32.png) [@megreddy](https://forum.cloverdx.com/u/megreddy)\
**Post date:** [October 20, 2010, 12:00am UTC](https://forum.cloverdx.com/t/process-only-unprocessed-files/765/1 "2010-10-20T00:00:00Z")

</div>

Is it possible to have a UniversalDataReader (or any Reader for that matter), point to a file folder and read in all the contained files? I know you can highlight multiple files, but can it be set up for a folder whose contents are periodically updated?

Additionally, can you do incremental reading where the reader only looks at files that have been added since the last run of the graph? Here I assume that there is a timestamp in either the filename or a data field, which can be saved to a separate “key” file.

Thanks in advance for the help!

---

<div class="post-metadata">

**Author:** ![avackova](https://avatars.discourse-cdn.com/v4/letter/a/65b543/32.png) [@avackova](https://forum.cloverdx.com/u/avackova)\
**Post date:** [October 21, 2010, 8:54am UTC](https://forum.cloverdx.com/t/process-only-unprocessed-files/765/2 "2010-10-21T08:54:00Z")

</div>

Hello,  
to point a Reader to a group of files, use ? and \* wild-cards. So for reading all the files from the selected folder, your _fileURL_ attribute should look like _fileURL=“my\_folder/\*”_.  
For reading only the files, that have not been read yet use incremental reading feature: set _incrementalKey_ and _incrementalFile_ attributes, eg.:

```auto
incrementalFile="${DATATMP_DIR}/inc_my_grf.prop"
incrementalKey="my_reader"

```

After the graph execution, the file specified in _incrementalFile_ attribute contains the names of the files with numbers of records, that have been read. So in the next graph execution there are read only new files or the records, that has been added to existing files since the last execution. But such reading is, of course, slower then reading without any checking, so if it is possible rather remove read files (eg. with [SystemExecute component](http://www.cloveretl.com/documentation/UserGuide/topic/com.cloveretl.gui.docs/docs/sysexecute.html)), than use incremental reading.

---

<div class="post-metadata">

**Author:** ![megreddy](https://avatars.discourse-cdn.com/v4/letter/m/58f4c7/32.png) [@megreddy](https://forum.cloverdx.com/u/megreddy)\
**Post date:** [October 21, 2010, 2:23pm UTC](https://forum.cloverdx.com/t/process-only-unprocessed-files/765/3 "2010-10-21T14:23:32Z")

</div>

Didn’t realize you could use wildcards in the GUI… I see that using SysExecute to tag the files as “read” and/or delete them is much faster than incremental reading. Thanks for the help!