Hello,
I have a similar case to another post, but it is a little different. http://forum.cloveretl.com/viewtopic.php?f=4&t=4576. Or, I just can not get the finer points down.
Here is the generic version of my case:
I have a graph (transformer.grf) which does some transform on file_1.txt and writes out file_1_finished.txt
I want to do this same execution on file_2.txt through file_100.txt.
In transformer.grf I’ve made an internal variable called NUM. and I’ve set these into the input and output paths for my reader/writers (eg file_${NUM}.txt) .
I have made a wrapping graph (wrapper.grf) which reads in a list of graphs(graph_list.txt) and passes them to a run graph component. I want to pass a different value in for NUM for every entry in my graph_list.txt. This would be easy, except that I am running on Server and therefore not starting a new JVM.
graph_list.txt example if I was in a new JVM:
graph/transformer.grf,-P:NUM=1
graph/transformer.grf,-P:NUM=2
…
graph/transformer.grf,-P:NUM=100
In ikulman’s response, the first option seems allow me to pass a value from wrapper.grf to transformer.grf, but that value doesn’t change. The second option requires a new JVM which I can’t do since I’m using server.
I hope there is some missing element that I’m not seeing.
All the best,
Marc
Hi Marc,
Would it be possible for you not to use RunGraph component, but specify different file names using an input port of a reader component (most readers have that port)?
See a simple graph in a sceenshot. I have connected a DataGenerator component, which generates URLs with use of this CTL expression:
“${DATAIN_DIR}/file” + num2str(sequence(FileNbrSequence).next()) + “.txt”
where FileNbrSequence is a custom defined sequence with both, start and step, set to 1. You can easily define your sequence is the view Outline ->Sequences → context menu Sequences → Create New Sequence.
I attach a graph for illustation (it reads files file1.txt, file2.txt and file3.txt are writes them back to file.out.txt.
I have tested it on a Clover server and it works fine for me.
Pavel Simecek
Javlin
Unfortunately, this wouldn’t do what I need. Your suggestion merges the output from all files.
file_1_in.txt + file_2_in.txt > file_out.txt
I need a distinct output foreach file.
file_1_in.txt > file_1_out.txt
file_2_in.txt > file_2_out.txt
Hi Marc,
Yes, for this case some non-trivial modifications to my previous graph are needed:
(1) metadata of a link out-going from a reader component must additionally contain a field with an input file name. For this field you must set autofilling to source_name.
(2) Then the file number must be extracted with use of Reformat component with transforms a file path like “c:\data\file_55.txt” to a string value “55”. The following CTL formula does the job:
pop(find($0.filename, "[0-9][0-9]*"))
(3) An incomming link to a writer component than contains data to write plus an additional field with file number. With use of partitioning you can write records to different file according to an attached file number. File partitioning can be set up this way:
support_file_in_out_sequence.png
I hope this will solve your problem, since I do not see any way to solve it with use of RunGraph component. We will surely make enhancements to the way of executing other graphs on a server. For now, I would use the solution I proposed here.
The graph with a proof of the concept is attached.
Best regards,
Pavel Simecek
Pavel,
Thank you very much for you reply. I had to finish this before i left for a winter holiday. I ended up making a perl script to just write the 100 graphs that perform the tasks. I do look forward to seeing if I can do this in another programmatic way with server in the future.
All the best,
Marc
