SystemExecute on consecutive records

Hello,

I’m new to CloverETL, struggling to do the following: I have an input stream of multiple records, and I need to run a SystemExecute FOR EACH RECORD and generate an output based on that, where the System Command is a function of a string field of the corresponding input record. I tried to do this by connecting the reader to the input port of SystemExecute; however, this does not read consecutive input records… Any suggestions how I can do it?

Example of commands I want to run:

python myprogram.py arg1
python myprogram.py arg2
python myprogram.py arg3

where arg1 arg2 arg3… are the values of a field of my input stream.

Thanks a lot!

Hi, p_swiss,

I think very similar problem is already solved here: viewtopic.php?f=4&t=5241

Please, take a look on it and let me know with any questions.

Best regards,

Hi Lobus,

Thanks a lot for your reply. I had actually seen this post. In the solution I see it’s mentioned all input records are serialized and sent to SystemExecute as STDIN. But what I want is a different System Command to be executed for each record (the script depending on the field). Kind of a loop over the records… If I’m wrong, could you please explain in a bit more details how I can treat my example (arg1 arg2 arg3 … each for one different record). Sorry if I’m being a bit dummy here :slight_smile:

Thanks!

Hi p_swiss,

You can use approach described in mentioned topic:

1. in loop parse lines out of whole stdin (line will be equal to one serialized record)
2. if line/records contains more fields, parse them out (using fields delimiter used in metadata)

Inside of the loop use if-statements (using parsed fields in conditions) selecting proper commands to execute. Syntax of parsing&loop&if highly depends on your OS and used shell script interpreter.

Sample of stdin:


script1.py|42
script2.py|66

Parsed line:


script1.py|42

Parsed fields:


"script1.py" and "42"

I hope this helps.

Thank you so much for your reply. I finally got how to do it with a loop in a shell script, works like charm :slight_smile:

Thanks a lot!