Still working with CloverETL and encountered another problem I can’t resolve (already looked into past forum posts): I have multiple record stores (e.g., rs1, rs2,…, rs1000), and what I need to do is to run a graph (the same graph) sequentially for all these record stores, through changing the record store name in “Record Store Reader” component. The graph basically stores the RS data in a data domain. Could you help me how to achieve this? Basically I need a loop to rerun the graph each time through changing the “Record Store Instance” property at each iteration.
Thanks!
Hello p_swiss,
If all files are in the same directory, you can use wildcard. E.g. C:\myfiles\*
If they are not in the same directory, you have several options:
A) A reader with several paths with wildcards, one per directory, separated by semicolons.
B) A few SystemExecute components with ls command and some filter if necessary. The result is sent to reader component via input port.
C) CloverETL Server with FileEventListeners. If some file appears, it can be processed by a graph.
D) CloverETL Server with jobflows. There is a ListFiles component and a graph can be started for each file found by ListFiles.
E) If you have some list of files, you can read it and sent to reader component via input port this way: port:$0.field1:source.
Best regards,
I guess I should have been more precise here: I am using Oracle Endeca Information Discovery 3.0, which is shipped with CloverETL as the integrator. In this version we have this “Record Store Reader” component, which is part of the “Discovery” package of CloverETL, and it basically reads some crawled data. It takes no files as input, neither does it have an input port, but only a “Record Store Instance” in the properties, which is the name of the crawled instance (stored in some directory through some data files, only readable by Record Store Reader, but I doubt if a wild card would work here as I specify no path or anything!). So what I need is to replace this property (or maybe in general, any property specific to a component) at each run of the graph, and run the graph sequentially.
I am sorry but this is outside the scope of CloverETL. We don’t have enough information about Oracle Integrator to give any advices here. Please, contact your Oracle support. Thank you for your understanding.
Best regards,
I’ve not tried to do exactly what you are attempting, but you might be able to edit the Source tab for the graph to specify the RecordStoreReader instances dynamically. I’ve only just seen that this can be done (e.g. to define dynamic metadata with varying number of ports).
If you click on the Source tab, you should find an entry for your RSR something like the following:
If you investigate the documentation and posts for creating dynamic metadata this may help you create a dynamic recordstore reader.