Generate something like a batch key for records

CloverDX Platform
1

Hi there,

Is there a way to do the following using transformation in CloverETL (Eg. Reformat, etc…)?

- Create a new field for each record based on their “rownum”, i.e. generate a new field with something like a “batch_key” so I can tag the first 5K records to be batch_key=1, 2nd 5K records to be batch_key=2,etc… and so on. Would like to do this in CloverETL and not in the input source(s).
- The goal here is to be able to use the batch_key field as the key to a denormalizer component to group/denormalize records into buckets/sets.

Thanks!
Peter

2

Hi Peter,

you can do following within Reformat:


//#CTL2

//current bucket nr
integer bucket = 0;

//passed records since last bucket nr increment
integer passed = 0;

function integer transform() {
	//test bulk size
	if (passed>10) {

		//reset counter
		passed = 0;
		
		//new bucket
		++bucket;
	}
	
	//copy input record to output
	$out.0.* = $in.0.*;
	
	//enrich by bucket nr
	$out.0.bucket = bucket;

	++passed;

	return ALL;
}

I hope this helps.