Generate Dynamic Metadata from files

Support/help with CloverETL (4.9) and CloverDX (5.0 or newer) implementation problems

derrickwan
Posts: 1
Joined: Tue Apr 23, 2013 11:25 pm

Generate Dynamic Metadata from files

Postby derrickwan » Tue Apr 23, 2013 11:40 pm

Hi all,

I am appreciate your time to read this. Here is my case.

I want to use CloverETL to process a file which upstream stream produces. Normally, upstream will let me know if they want to added new values into the file.

However, if the upstream forgot to tell me and they make the changes to the file. It will break my system.

Is any way we can dynamic generate the metadata and use it in CloverETL?

thanks,

derrick

imriskal
Posts: 397
Joined: Wed Aug 15, 2012 8:18 am

Re: Generate Dynamic Metadata from files

Postby imriskal » Wed Apr 24, 2013 9:08 am

Hello, Derrick,

Metadata are just XML as the rest of the graph. You can externalize it into some .fmt file and open the file with text editor to see it's internal structure. You can create this file dynamically with CloverETL graph and use it in some other graph, using parameters. Let me know if you want to know more about this topic.

If you use database as your input of data, you have also the second option. See http://doc.cloveretl.com/documentation/ ... adata.html

Best regards,
---
Lubos Imriska
CloverCARE Support
CloverDX

Visit us online at http://www.cloverdx.com

Calder
Posts: 1
Joined: Tue Sep 23, 2014 7:14 pm

Re: Generate Dynamic Metadata from files

Postby Calder » Tue Sep 23, 2014 7:17 pm

@imriskal: Please do elaborate on the solutions you suggest. Thanks!!

novakj
Posts: 54
Joined: Wed Sep 04, 2013 11:55 am

Re: Generate Dynamic Metadata from files

Postby novakj » Wed Sep 24, 2014 4:13 pm

Hello,

Example of simple metadata structure you can see here:

Code: Select all

<?xml version="1.0" encoding="UTF-8"?>
<Record fieldDelimiter="|" name="myMetadata" recordDelimiter="\r\n" type="delimited">
   <Field name="firstname" type="string"/>
   <Field name="lastname" type="string"/>
   <Field name="age" type="integer"/>
</Record>

You can generate in your graph this .fmt XML file using XMLWriter component, the metadata files are typically stored in the "meta" directory. The structure of your metadata depends on your specific needs.

If you need more detailed answer, please tell us your desired usage of dynamic metadata.

Best regards,
Jiri Novak
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com

jd_surat
Posts: 1
Joined: Tue Aug 30, 2016 6:11 am

Re: Generate Dynamic Metadata from files

Postby jd_surat » Tue Aug 30, 2016 6:15 am

imriskal wrote:Hello, Derrick,

Metadata are just XML as the rest of the graph. You can externalize it into some .fmt file and open the file with text editor to see it's internal structure. You can create this file dynamically with CloverETL graph and use it in some other graph, using parameters. Let me know if you want to know more about this topic.

If you use database as your input of data, you have also the second option. See http://doc.cloveretl.com/documentation/ ... adata.html

Best regards,


Hello,

Can you please provide more information on how to perform following?

"You can create this file dynamically with CloverETL graph and use it in some other graph, using parameters"

Regards,
Jigar

cholastal
Posts: 135
Joined: Tue Sep 01, 2015 1:22 pm

Re: Generate Dynamic Metadata from files

Postby cholastal » Tue Aug 30, 2016 1:42 pm

Hi Jigar,

Please see the attached graph. It is using a Normalizer component with some CTL2 functions to create the metadata from the input record. After the metadata is created it is converted into a XML format and written to the HDD by the XMLWriter component. Using this file is as easy as using any other externalized metadata. Once it's linked to your project metadata, it will be up to date as long as your creation graph keeps overwriting the same metadata file.

If you need to use various metadata, you can pass its path in a parameter via e.g. RunGraph component.

Best regards,
Attachments
create_metadata.grf
(4.32 KiB) Downloaded 331 times
testing_data.csv
(25 Bytes) Downloaded 220 times

---
Lukas Cholasta
CloverCARE Support
CloverDX

Visit us online at http://www.cloverdx.com


cron