Dynamic file name in Reader?

CloverDX Platform
1

Hi
I would like to dynamically set the filename for a reader node (XLSReader), based on a file name extracted from another data source.
From what I can see, it doesn’t look like there is a way to do this with the standard XLSReader component, and even if I could do this I’m not sure how to construct an ‘iterating’ graph.
Basically the situation I have is that I need to read up to ‘n’ instances of a spreadsheet (each of the same format), and aggregate the data in the output. I don’t know until run time how many is ‘n’ and what they are named.

ideas?

thanks

cliff

2

These are some great ideas - thanks for taking the time to respond.

I think we’ll start with the simple script iteration, and then look at extending the XLSReader to accept multiple files. Our files should be named in a predictable pattern - so using the wild-card file matching for a multi-file reader should work. I’ll look at the other readers for examples.

thanks

cliff

3

Hello Cliff,
unfortunately clover is not suitable to iterate some part of the transformation graph, nevertheless there are some ways to solve your task. Most of reader components in the latest clover engine release 2.1 are available to read from multifile defined by list or file mask (i.e. ‘data1.txt;data2.txt;data3.txt’ or ‘data*.txt’), unfortunately XSLReader is not, but this extension is not hard to implement. But if you realy don’t know anything about input files, you could use ability to set fileURL attribute by parameter. In the first graph you can for example generate through the use of Structure writer component parameter file:

XLS_TO_READ=data1.xls;data2.xls;data3.xls

And second graph use this file to setup fileURL of your reader (this solution of course supposes extension in XSLReader for multi-file reading).

Only solution without changes of clover engine is in using some shell script, which implements required iteration outside clover:

runGraph <graph_generates_input_file_name_list>
for i in `cat filenames`; do
runGraph -D:FILEURL=$i <graph_concatenating_data_to_the_one_destination(use CloverWriter)>
done

runGraph <read(use CloverReader)_all_data_and_aggregate>

Don’t hesitate and ask me in case any question.

OtaSanek