Generate Dynamic Metadata from files

CloverDX Platform
1

Hi all,

I am appreciate your time to read this. Here is my case.

I want to use CloverETL to process a file which upstream stream produces. Normally, upstream will let me know if they want to added new values into the file.

However, if the upstream forgot to tell me and they make the changes to the file. It will break my system.

Is any way we can dynamic generate the metadata and use it in CloverETL?

thanks,

derrick

2

Hello, Derrick,

Metadata are just XML as the rest of the graph. You can externalize it into some .fmt file and open the file with text editor to see it’s internal structure. You can create this file dynamically with CloverETL graph and use it in some other graph, using parameters. Let me know if you want to know more about this topic.

If you use database as your input of data, you have also the second option. See http://doc.cloveretl.com/documentation/ … adata.html

Best regards,

3

@imriskal: Please do elaborate on the solutions you suggest. Thanks!!

4

Hello,

Example of simple metadata structure you can see here:

<?xml version="1.0" encoding="UTF-8"?>
<Record fieldDelimiter="|" name="myMetadata" recordDelimiter="\r\n" type="delimited">
	<Field name="firstname" type="string"/>
	<Field name="lastname" type="string"/>
	<Field name="age" type="integer"/>
</Record>

You can generate in your graph this .fmt XML file using XMLWriter component, the metadata files are typically stored in the “meta” directory. The structure of your metadata depends on your specific needs.

If you need more detailed answer, please tell us your desired usage of dynamic metadata.

Best regards,

5

Hello, Derrick,

Metadata are just XML as the rest of the graph. You can externalize it into some .fmt file and open the file with text editor to see it’s internal structure. You can create this file dynamically with CloverETL graph and use it in some other graph, using parameters. Let me know if you want to know more about this topic.

If you use database as your input of data, you have also the second option. See http://doc.cloveretl.com/documentation/ … adata.html

Best regards,

“imriskal”

Hello,

Can you please provide more information on how to perform following?

“You can create this file dynamically with CloverETL graph and use it in some other graph, using parameters”

Regards,
Jigar

6

Hi Jigar,

Please see the attached graph. It is using a Normalizer component with some CTL2 functions to create the metadata from the input record. After the metadata is created it is converted into a XML format and written to the HDD by the XMLWriter component. Using this file is as easy as using any other externalized metadata. Once it’s linked to your project metadata, it will be up to date as long as your creation graph keeps overwriting the same metadata file.

If you need to use various metadata, you can pass its path in a parameter via e.g. RunGraph component.

Best regards,