Clover graphs are stuck for a long time

CloverDX Platform
1

Hi

I am facing an issue where my graph execution clover are stuck for a long time and I don’t see any logs which gives details on why the graphs are stuck for a long time.

Now if I wait for some more time then I am getting an error message saying:

error executing graph
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:691)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1371)
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
	at com.cloveretl.server.c.b.b.executeWatchDog(Unknown Source)
	at com.cloveretl.server.c.h.f(Unknown Source)
	at com.cloveretl.server.c.h.a(Unknown Source)
	at com.cloveretl.server.c.h.executeGraph(Unknown Source)
	at com.cloveretl.server.c.j.executeGraph(Unknown Source)
	at com.cloveretl.server.c.b.c.a(Unknown Source)
	at com.cloveretl.server.c.b.c.executeGraph(Unknown Source)
	at com.cloveretl.server.c.b.c.executeGraphSync(Unknown Source)
	at org.jetel.component.RunGraph.runGraphThisInstance(RunGraph.java:521)
	at org.jetel.component.RunGraph.runSingleGraph(RunGraph.java:418)
	at org.jetel.component.RunGraph.execute(RunGraph.java:263)
	at org.jetel.graph.Node.run(Node.java:453)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:722)

Please help me how to debug this issue? Is there any logs that I need to check other than server logs to see why the graph is stuck? I am using Weblogic server.

2

Hi chaitanya_r,

Please verify if you are not hitting OS related limit “max user processes” - on Linux counts also threads. You can see the limit on Linux by command:

ulimit -u

Default limits are usually low and will prevent you to create let’s say more than 1024 threads per Java process.

Another possible reason of this exception and your slow graph execution could be too much concurrently running graphs.

For more detailed answer, could you please specify some details regarding your environment? For example, operating system, CloverETL Designer/Server version, JVM memory settings etc.

3

Thanks a lot, increasing the max user processes on my Linux box has fixed my issue.