Executing Graph multiple times

CloverDX Platform
1

Hi,

I’ve followed the guibe to embedding Clover (2.9.3) http://wiki.cloveretl.com/doku.php?id=embedding_clover but am having problems executing my graph multiple times. It will execute fine the 1st time but it goes into a loop the second time. I have tried using both


  //graph loading
  TransformationGraph graph = TransformationGraphXMLReaderWriter.loadGraph(in, runtimeContext.getAdditionalProperties());

  //engine initialization
  EngineInitializer.initGraph(graph, runtimeContext);

  IThreadManager threadManager = new SimpleThreadManager();
  WatchDog watchDog = new WatchDog(graph, runtimeContext);
  threadManager.executeWatchDog(watchDog);

  sleep(5000);

  threadManager.executeWatchDog(watchDog);

and


  //graph loading
  TransformationGraph graph = TransformationGraphXMLReaderWriter.loadGraph(in, runtimeContext.getAdditionalProperties());

  //engine initialization
  EngineInitializer.initGraph(graph, runtimeContext);

  runGraph.executeGraph(graph, runtimeContext);

  sleep(5000);

  runGraph.executeGraph(graph, runtimeContext);

This is what is in the logs continually on the second execution:

DEBUG [WatchDog] - incoming msg:null
INFO [WatchDog] - ---------------------** Start of tracking Log for phase [0] **-------------------
INFO [WatchDog] - Time: 17/06/10 12:23:44
INFO [WatchDog] - Node Status Port #Records #KB Rec/s KB/s
INFO [WatchDog] - ---------------------------------------------------------------------------------
INFO [WatchDog] - DATA_READER2 RUNNING
INFO [WatchDog] - %cpu:… Out:0 3 1 0 0
INFO [WatchDog] - MYSQL_DATA_WRITER0 FINISHED_OK
INFO [WatchDog] - %cpu:… In:0 3 2 0 0
INFO [WatchDog] - REFORMAT0 RUNNING
INFO [WatchDog] - %cpu:… In:0 3 1 0 0
INFO [WatchDog] - Out:0 3 2 0 0
INFO [WatchDog] - ---------------------------------** End of Log **--------------------------------

any ideas?
Cheers.

2

Hello,
you have to call graph.reset() method between particular graph execution:

			
graph = TransformationGraphXMLReaderWriter.loadGraph(new FileInputStream(file), runtimeContext.getAdditionalProperties());
EngineInitializer.initGraph(graph, runtimeContext);
IThreadManager threadManager = new SimpleThreadManager();
WatchDog watchDog = new WatchDog(graph, runtimeContext);
futureResult = threadManager.executeWatchDog(watchDog);
Result result =  futureResult.get();
graph.reset();
futureResult = threadManager.executeWatchDog(watchDog);
result =  futureResult.get();
graph.free();
3

Yes I tried that but it still goes into a loop when the 2nd threadManager.executeWatchDog(watchDog); is called. This also happens when I tried your example.
The only way I could get it to work was to add a sufficient sleep time between executeGraph() calls.


Future<Result> result = runGraph.executeGraph(graph, runtimeContext);
sleep(5000);
graph.reset();
result = runGraph.executeGraph(graph, runtimeContext);

Adding the sleep() using the threadManager method did not work, it still looped.

4

I have updated the related wiki page

http://wiki.cloveretl.com/doku.php?id=embedding_clover#embedding_clover

Please follow the example, I hope it helps.

In short:

  1. you don’t need to ‘reset’ the graph anymore
  2. you need to create a watchdog instance for each separete graph run

Let me know if the example on wiki page is not comprehensive. More detailed piece of code is also available at our junit test org.jetel.graph.ResetTest.

5

All mentioned above is totally right from release 3.0 which is not still public :slight_smile: But you can download it from our svn trunk, it should be already pretty stable. We are going to release it in few days.

For older clover release I still recommend to be inspired by org.jetel.graph.ResetTest junit test (should be part of each source distribution).

Let me know if you don’t have it. Martin