Refreshing a data warehouse

CloverDX Platform
1

Hi All.
I’ve just created a very simple graph, using cloverGUI, that creates a DW.
All this graph does is extract data from Ms Access Databases, sort them, perform a very simple transformation and finally loads it into a mySQL DB.

My questions are:

  1. How do I refresh the data warehouse without having to load all data once again?

  2. I think that’s suposed to be done using the transformation class. My ideia would be to keep an xml file with the last field loaded from access and when doing new extractions compare with that value. how do I do that? build a java class?

thanks

2

Just used your approach and it works quite well!
Not other question:

When unloading data from mySQL i get the following warning:
WARN [DB_INPUT_TABLE1] - SQLException when closing statement
java.sql.SQLException: Can’t call commit when autocommit=true
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:914)
at com.mysql.jdbc.Connection.commit(Connection.java:2248)
at org.jetel.connection.SQLDataParser.close(SQLDataParser.java:277)
at org.jetel.component.DBInputTable.run(DBInputTable.java:194)
at java.lang.Thread.run(Unknown Source)

Is this a bug from the driver?
How can I solve this?
Also when validating the query on the DB Input i get the following message:
Query invalid: Streaming result set com.mysql.jdbc.RowDataDynamic@b6994f is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.

but then i get results on the query. been “googling” about this and seems like a bug from J connector! any solutions on this?

Best Regards

Lucas

3

Hi !

One option would be to use DataIntersection component or simply a Join component.
You start unloading all data from MS Acess and MySQL at the same time - sort them or make sure you user “order by” clause.

I assume your sorce data has some kind of unique key which is also propagated to target. Uset that key to find new records in flow from MS Access which have no corresponging records in MySQL - such records should be then save to DB. But make sure you do it in separathe phases - don’t read data from DB and save data there at the same time.

David.