CloverDX 5.2.0 - Too many open files

Support/help with CloverETL (4.9) and CloverDX (5.0 or newer) implementation problems

jusman
Posts: 10
Joined: Tue Apr 14, 2015 12:45 am

CloverDX 5.2.0 - Too many open files

Postby jusman » Tue Jun 11, 2019 8:07 pm

Hi there,

We recently upgraded to CloverDX 5.2.0 server (from CloverETL 4.0.4) and since then have been having a problem with "Too many open files". The scenario is: a specific graph (Users.Get.grf) fails after about 30 minutes of running, returning this error message. At this point, the worker process is practically useless, because every graph would fail after this. Meanwhile, the server keeps accepting requests from clients, not knowing that the worker is unable to process anything.

Is there a config setting to restart the worker when this occurs? I noticed that when the problem is with memory, the worker is restarted automatically and everything keeps going. Can we do the same thing for open files? Or at least, for when the graphs have been consistently failing for the last x runs.

If there's nothing like that, do you have guys have some groovy code for the Universal Listener, to watch for this stuck condition? Or, how do you handle this "always fail" condition?

Thanks so much for your help!
Jus

From the run log:

Code: Select all

2019-06-11 03:29:11,006 ERROR 80957 [WatchDog_80957] Component [Get User Data:GET_USER_DATA] finished with status ERROR.
 Execution of graph './graph/Users.Get.grf' failed!
  Component [GETUserRoles:GETUSER_ROLES] finished with status ERROR. (In0: 3972 recs, Out0: 3971 recs)
   Too many open files
2019-06-11 03:29:11,007 ERROR 80957 [WatchDog_80957] Error details:
org.jetel.exception.JetelRuntimeException: Component [Get User Data:GET_USER_DATA] finished with status ERROR.
   at org.jetel.graph.Node.createNodeException(Node.java:655)
   at org.jetel.graph.Node.run(Node.java:620)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)
Caused by: org.jetel.exception.JetelRuntimeException: Execution of graph './graph/Users.Get.grf' failed!
   at org.jetel.component.RunGraph.logError(RunGraph.java:546)
   at org.jetel.component.RunGraph.runGraphThisInstance(RunGraph.java:539)
   at org.jetel.component.RunGraph.runSingleGraph(RunGraph.java:407)
   at org.jetel.component.RunGraph.execute(RunGraph.java:283)
   at org.jetel.graph.Node.run(Node.java:580)
   ... 3 more
Caused by: org.jetel.exception.JetelRuntimeException: Component [GETUserRoles:GETUSER_ROLES] finished with status ERROR. (In0: 3972 recs, Out0: 3971 recs)
 Too many open files
   ... 7 more

dpavlis
Posts: 182
Joined: Sat Mar 10, 2007 8:12 pm

Re: CloverDX 5.2.0 - Too many open files

Postby dpavlis » Fri Jun 14, 2019 10:47 pm

Please, contact CloverDX support directly - at support@cloverdx.com as they may need additional info to solve this for you.
David Pavlis
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com

bartonv
Posts: 125
Joined: Wed May 03, 2017 12:10 pm

Re: CloverDX 5.2.0 - Too many open files

Postby bartonv » Tue Jun 25, 2019 2:08 pm

This request is being addressed directly by CloverCARE Support outside of this Forum. However, worth mentioning is that there is a known issue in the 5.2.0 version of CloverDX (demonstrated by the "Too many open files" error) which can occur when using AdoptOpenJDK 11 with the CloverDX Server. This problem has been fixed in the upcoming 5.3 release of our software. More information about this issue can be found in our JIRA.
---
Vladimir Barton
CloverCARE Support
CloverDX

Visit us online at http://www.cloverdx.com

jusman
Posts: 10
Joined: Tue Apr 14, 2015 12:45 am

Re: CloverDX 5.2.0 - Too many open files

Postby jusman » Wed Jun 26, 2019 8:11 pm

Thanks a bunch guys, Vladi!
Jus