Directory/file listing

Support/help with CloverETL (4.9) and CloverDX (5.0 or newer) implementation problems

golomeen
Posts: 21
Joined: Mon Jan 17, 2011 10:21 am

Directory/file listing

Postby golomeen » Thu Nov 24, 2011 2:57 pm

Hello all!

Is there any way to scan a directory for a /new/ files to process them then with readers. Excluding old files is my problem, i only describe context.
I want to omit a step when user picks a file from a date-based shared directory and puts it to certain place in Clover project
Can I?
Thanks!

golomeen
Posts: 21
Joined: Mon Jan 17, 2011 10:21 am

Re: Directory/file listing

Postby golomeen » Fri Nov 25, 2011 11:48 am

i've found one
to write a java class with java.io support for DataGenerator
never ever thought it would be so easy

so can you tell me the way to debug a java transformation class in graph?
and is there any method to set the generated record count in runtime (i.e. class) - if i need a list of files?

jurban
Posts: 163
Joined: Fri Jul 20, 2007 9:25 am

Re: Directory/file listing

Postby jurban » Thu Dec 01, 2011 5:39 pm

Hello,

currently it's not possible to set the number of generated records in runtime. Luckilly there's a workaround. Set the number of records to be generated by the DataGenerator to a number that you think will always be larger than will really want to generate (number of listed files in your case). Then in the generator code, first send the filenames to the output as separate records. When you run out of filenames, return RecordGenerate.SKIP for all following requests to generate output. With this the generator will generate exactly the number of records as is the number of filenames. It's a bit innefective at the end because of the skipping, but skipping is very fast so it shouldn't be a problem.

We didn't try debugging of the Java transformation, but it should work if you use remote debugging, see:
http://www.eclipsezone.com/eclipse/forums/t53459.html
In the above article, you can see command line arguments that need to be passed to the executed graph (via its Run Configuration) and then how to configure the Eclipse debugger. One important thing - you'll need to use external Java tranformations, i.e. write the Java transformation in separate file (in "trans" directory) and specify its classname in DataGenerator configuration.

However, we're looking into improving debugging of Java transformations - please see issue CLD-3092 in out bugtracker (bug.javlin.eu).

Best regards,
Jaro
Jaroslav Urban
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com


cron