MySQLBulkWriter to return newly generated ids

Support/help with CloverETL (4.9) and CloverDX (5.0 or newer) implementation problems

Nismon8er
Posts: 10
Joined: Thu Apr 01, 2021 7:51 pm

MySQLBulkWriter to return newly generated ids

Postby Nismon8er » Wed Apr 21, 2021 3:20 pm

Is there any way to have MySQLBulkWriter to return the ids of the newly added records.

So for example I have a table like so:

| id | name | last | company | position |

and I have a flat file that has data like so:

^^Joe^^Johnson^^Company ABC^^Manager^^
^^Mike^^Mills^^Company 123^^Staff^^
^^Dale^^Dwells^^Company ABC^^Staff^^


So I have my bulk writer setup to read it with the ^^ delimeter and it inserts the records perfectly

| id | name | last | company | position |
| 5 | Joe | Johnson | Company ABC | Manager |
| 6 | Mike | Mills | Company 123 | Staff |
| 7 | Dale | Dwells | Company ABC | Staff |


Is there anyway that the bulkwriter can return to me a flatfile like this

5
6
7

or maybe

5,6,7

?

jandikovae
Posts: 63
Joined: Fri Nov 04, 2016 8:51 am

Re: MySQLBulkWriter to return newly generated ids

Postby jandikovae » Mon May 10, 2021 11:07 am

Hi,

The way the MySQLBulkWriter and DatabaseWriter are designed, the components can only send to the output the values that have not been loaded correctly to the database. Also, as I understand your situation, the ID column here is being produced by the database itself (auto-increment field or so), is that right? Therefore the component would need to read from the database after the data has been loaded. If that is fine with you and you believe it is still beneficial for you to read the data afterward, you might want to implement the so-called "incremental reading" and use DatabaseReader to get your newly created records. Having this function set in a reader component, with each run the last read value is stored in a file therefore the reader then knows what is the next value that hasn't been read before.

I have created an example graph for your review. In the beginning, I am simply loading some records to the database. Then I am reading from the database using the "incremental reading" function (in a different phase). If you would like to run the graph as well, you might need to set up the same database table and change the connection setups, therefore I am also attaching a screenshot of my database table. Please review mainly the SQL query in the DatabaseReader as there is one increment-related amendment (it is "where `test`.`autotable`.`ID` > #key1" in my case) and then the Incremental file and Incremental key has to be set as well.

Best Regards,
Eva
Attachments
table_structure.PNG
table_structure.PNG (17.19 KiB) Viewed 310 times
IncrementalReading.grf
(3.09 KiB) Downloaded 18 times
---
Eva Jandikova
CloverCARE Support
CloverDX

Visit us online at http://www.cloverdx.com

jandikovae
Posts: 63
Joined: Fri Nov 04, 2016 8:51 am

Re: MySQLBulkWriter to return newly generated ids

Postby jandikovae » Tue May 18, 2021 2:06 pm

Hi there,

Although I am aware of the fact that you are mainly interested in the MySQLBulkWriter, I have realized that I have misled you regarding the usual DatabaseWriter. The function that you are describing is, in fact, available when writing to the database using the DatabaseWriter.

Therefore in addition to my previous post, I am attaching an example of using the so-called "Returning statement" in SQL query. In the graph attached, you should only change the Connection to some MySQL testing database available and after that, the graph is fully runnable. It will create a testing table in your database and then show you how it works when writing the values into a table with AUTO_INCREMENT column in it.

The main part of this solution is the SQL query within the DatabaseWriter:

Code: Select all

INSERT INTO \`myUsers\` (\`firstname\`, \`lastname\`)
VALUES (?, ?) RETURNING $inserted_id:=AUTO_GENERATED


You can find more details on the documentation page regarding the DatabaseWriter.

Best Regards,
Eva
Attachments
ReturningStmnt.grf
(3.73 KiB) Downloaded 17 times
---
Eva Jandikova
CloverCARE Support
CloverDX

Visit us online at http://www.cloverdx.com


cron