Date in file URL or file

CloverDX Platform
1

I am looking to write out log files that contain listing of rows at various points in my flow, initial extract, filtered, rejected, etc.

What I want to do is ideally write out one log file per run, with the date in the name of the log file. Is this possible?

If not, I’d like to write out the date into the log file, so I know the date of the run that is associated with the rows.

Is this possible with the StructuredDataWriter?

Thanks…

2

Hello Bernie,
your aim can be achieved with partition feature, which is supported with most of the Writers (see Partitioning Output into Different Output Files). If your data contains current time stamp (if not, you can add it, when reading input data with reader_timestamp auto filling function), let say in field current_date, your Writer should have following settings:

  • fileURL=“${DATAOUT_DIR}/output_#.log”

  • partitionFileTag=“keyNameFileTag”

  • partitionKey=“current_date”

When using StructerWriter, you specify if you want the current_date field in output file in mask attribute; when using UniversalDataWriter, you can specify the filed in excludeFields attribute, if you don’t want to store the date in output file.