Working With Multiple Files

CloverDX Platform
1

I’m trying to replicate this process with CloverETL:

- download multiple zip files from a website
- unzip the text files within; each has the same name as the zip file they’re in
- the text files get converted into tables within a Postgres database
- then some tables get joined together

My question is: do I need to create multiple paths for each file or can they be transformed together without explicitly referencing individual files?

I have successfully been able to download all of the zipped files from the website by reading in a CSV of filenames and mapping it through the HTTPConnector, but from there, I don’t know how to handle the different metadata for each file without working with each file separately.

I’m hesitant to work with them separately because there are 20-30 files and new ones get generated each year so it would require yearly maintenance that I may not be here for and it would get unwieldy quickly.

I was hoping that http://forum.cloveretl.com/viewtopic.php?t=3622 would help me, but it was written in CLT1, and I haven’t been able to replicate it successfully.

Thank you for the help!

2

Are we talking about some BLOBs or CLOBs in that database? Or as I suspect, you need to have them parsed into fields/columns and then put to the database in some sort of format?

3

Columns and tables. And we really just drop all of the tables and recreate them from the text files for all but one table.

4

I created an example of how you can achieve your use case. I used a combination of Jobflows and Graphs to generate the metadata, and the database tables dynamically based on the data in each file and then inserted the data based on the table name.