Executing ETL graphs parallely from java

CloverDX Platform
1

Hi *

I am having a problem running the same graph from java code parallely. Is it a known behaviour of ETL or i must take care of few things before i run the graphs at the same time.

Another question in this regard would be , how the graph runs are handeld from java, i mean is

graphReader.read(new FileInputStream(graphFile));

is a singelton implementation or a new instance is made everytime this method is called.

Attached is the exception that i am getting … Actually i am reading an XML file and after transforming the data i write it to either database or to a file. for that i am using partition component.

Any help / advice will be appreciated…


14 Feb 2008 16:22:39,766 - ERROR [WatchDog] org.jetel.graph.Phase       DB_OUTPUT_TABLE0 ...FAILED !
Connection with ID: Connection0 isn't instance of the DBConnection class.
        at org.jetel.component.DBOutputTable.init(DBOutputTable.java:321)
        at org.jetel.graph.Phase.init(Phase.java:161)
        at org.jetel.graph.runtime.WatchDog.executePhase(WatchDog.java:562)
        at org.jetel.graph.runtime.WatchDog.run(WatchDog.java:164)
14 Feb 2008 16:22:39,767 - ERROR [WatchDog] org.jetel.graph.runtime.WatchDog !!! Phase finished with error - stopping graph run !!!
2

Hi, could you clear up your problem? I’ve understand that when you run graph it works properly, but if you run 2 graphs at once you get the error. Is that right? Or you get error always?

3

Hi

Let me rephrase the problem that i am having.

I have an ETL graph to import an XML file in the database. I run this graph through a java class, which i have taken from the example code of ETL library and have changed according to my need. All works fine as far as i execute it for one file at a time. The problem comes when my application finds two or more files in a folder and try to execute both of them at the same time. That means, two instances of the java class through which i execute the graph file, try to import two files at the same time.

I hope this time i have explained the problem more precisely. If it is still not clear, plz let me know and i will try to explain it again…

thanx and regards

4

Could you show your java code?

5

here is my code through which i execute the ETL graph.


private boolean execute(String dataFile, String metaData, String graphFile) {
		boolean flag = true;
		Defaults.init();
		Plugins.init("c:/plugins");

		DataRecordMetadata metadataIn = null;
		DataRecordMetadataXMLReaderWriter reader = new DataRecordMetadataXMLReaderWriter();

		try{
			metadataIn = reader.read(new FileInputStream(metaData));
		}catch(IOException ex){
			flag = false;
			throw new RuntimeException(ex);
		}

		if (metadataIn==null){
			flag = false;
			try {
				throw new RuntimeException("No INPUT metadata");
			} catch (Exception e) {
				
			}
		}

		TransformationGraph graph = new TransformationGraph();
		TransformationGraphXMLReaderWriter graphReader = new TransformationGraphXMLReaderWriter(graph);

		try {
			// set few properties in graph ...

			graphReader.read(new FileInputStream(graphFile));
		} catch (FileNotFoundException e) {
			flag = false;
			throw new RuntimeException(e);
		} catch (XMLConfigurationException e) {
			flag = false;
			throw new RuntimeException(e);
		} catch (GraphConfigurationException e) {
			flag = false;
			throw new RuntimeException(e);
		}

		if(!graph.init()){
			flag = false;
			logger.error("Graph initialization failed !");
		}

		if (!graph.run().equals(Result.FINISHED_OK)){
			flag = false; 
			logger.error("Failed starting all nodes!");
		}
	

	return flag;
}
6

I don’t see problem in your code (the more you don’t use dataFile in the body of the method), but if you want run graph for different input files it would be better to do graph as global variable or set fileURL attribute as “multi file” (http://wiki.clovergui.net/doku.php?id=components:readers#readersfileurl)

7

Hi,
Please help me that whether this graph file name is the .grf file or the xml of the graph…i am getting error while runing graph from java

8

Hi,
Please help me that whether this graph file name is the .grf file or the xml of the graph…i am getting error while runing graph from java

“mishra.hanu@yahoo.co.in”

Hi,
I’m not sure if I understand your question. A Clover graph is stored in a .grf file, which contains an XML of the graph.

Jaro