Append the number of records in a partitioned file

CloverDX Platform
1

Hi all,

I am using a StructuredDataWriter with a limit of 1000 records per file. I would like to append the number of records contained in the partitioned file as a footer (ie, 1000 for the first files and ‘n’ for the last one).
Is there a simple way to obtain this result?

2

Hello,
I haven’t found an easy way to do that. My solution is as follows:

  • in a transformation component that is before the Writer (it can be a component you have just in your graph or put a Reformat before the Writer) store the number of files you will need and the number of records in the last file:
int fileNumber = 1;  
int counter = 0;  
// Transforms input record into output record.  
function transform() {  
        .  
        .  
        .  
	if (counter == ${full_file}){  
		counter = 1;  
		fileNumber++;  
	}else{  
		counter++;  
	}  
        .  
        .  
        .  
}  
// Called after the component finishes.  
function finished() {  
	dict_put_str('num_files',to_string(fileNumber));  
	dict_put_str('num_rec',to_string(counter));  
}

where full_file parameter stores number of records for partition (1000)

  • add a new phase to your graph after the records (without any footer) are written to the files:
    DataGenerator → StructureWriter
    DataGenerator code:
boolean initialized = false;  
int index = 0;  
int num_files;  
function initInternal(){  
	num_files = str2num(dict_get_str('num_files'));  
}  
function setOutput(lastFile) {  
	$number := iif(lastFile, str2num(dict_get_str('num_rec')), ${full_file});  
}  
function generate(){  
	if (!initialized){  
		initInternal();  
		initialized = true;  
	}  
	if (index < num_files) {  
		index++;  
		setOutput(index == num_files);  
		return ALL;  
	}  
	return SKIP;  
}

The output from DataGenerator contains the number of records you need to append to the files created before.
Now configure StructerWriter:[list:153gja1z][*:153gja1z]fileURL - like in the Writer, that stores data records

  • append=true

  • recordsPerFile=1
    [/*:m:153gja1z][/list:u:153gja1z]