I have a file that I need to access through a universal data reader where the file is updated every Thursday and I would like to be able to access the file on any other day without the need to specify the specific date for “last thursday” within a parameter (which I’m currently doing).
So, I’ve set up a data generator which figures out what the date is for last thursday and sends it through the output port to the data reader.
Here is the data generator code:
//#CTL2
string last_thursday;
integer day_of_week;
integer days_from_thurs;
// Generates output record.
function integer generate() {
day_of_week = str2integer(date2str(today(), "joda:e"));
if(day_of_week > 4) days_from_thurs = day_of_week - 4;
else days_from_thurs = day_of_week + 4;
last_thursday = date2str(dateAdd(today(), -days_from_thurs, day),"MMddyyyy");
$out.0.last_thursday_filename = "zip:(ftp://${FTP_USERNAME}:${FTP_PASSWORD}@DOMAIN.COM/FOLDER/FILE"+last_thursday+".ZIP)#FILE"+last_thursday+'.CSV';
return OK;
}
This is generating the correct filename that works perfectly within the ‘File URL’ box of the data reader if I enter the generated filename in the box manually, but when passed through the input port it will not download the file and process its data.
The generated filename is: zip:(ftp://username:password@DOMAIN.COM/FOLD … 222013.ZIP)#FILE08222013.CSV
I’ve tried setting the Port and field to all of the following and it doesn’t work:
port:$0.last_thursday_filename:discrete
port:$0.last_thursday_filename:source
port:$0.last_thursday_filename:stream
I get the following which might mean something?
ERROR [WatchDog] - Node NODENAME finished with status: Component has finished and input port 0 still contains some unread records.
How do I get the dynamically generated file-name to pass into the data reader and have it actually download and process the file?
FYI - I’ve tried doing this through a dictionary but I couldn’t get that to work either.
FYI - I wish CloverETL had the ability to generate a date based on something like the following: str2date(‘last thursday’,“MMddyyyy”); PHP has this functionality and I use it all the time. 
With this function I can simply use that within the file-name like: zip:(ftp://username:password@DOMAIN.COM/FOLDER/FILE`str2date(‘last thursday’,“MMddyyyy”)`.ZIP)#FILE`str2date(‘last thursday’,“MMddyyyy”)`.CSV