Hi,
We have a graph that has been running along just fine. Today it started throwing BufferOverflowException. We bumped both
Data.DATA_RECORDS_BUFFER_SIZE and Data.MAX_BUFFERS_ALLOCATED but still getting the errors. The number of records has not changed significantly from when the graph ran fine. What are other possibilities? Could it be one bad data record that is causing this problem? Thanks
11-02 13:20:06,485[ WatchDog] WatchDog ERROR 124553 Node SORT_GOREMAL finished with status: ERROR caused by: null
11-02 13:20:06,485[ WatchDog] WatchDog ERROR 124553 Node SORT_GOREMAL error details:
java.nio.BufferOverflowException
at java.nio.Buffer.nextPutIndex(Buffer.java:495)
at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:250)
at org.jetel.util.bytes.ByteBufferUtils.encodeLength(ByteBufferUtils.java:146)
at org.jetel.graph.DirectEdge.eof(DirectEdge.java:343)
at org.jetel.graph.Edge.eof(Edge.java:439)
at org.jetel.graph.Node.closeAllOutputPorts(Node.java:854)
at org.jetel.graph.Node.broadcastEOF(Node.java:865)
at com.opensys.cloveretl.component.FastSort.b(Unknown Source)
at com.opensys.cloveretl.component.FastSort.execute(Unknown Source)
at org.jetel.graph.Node.run(Node.java:425)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Update… It turns out that the changes we made to defaultProperties were not picked up by the clover engine. I believe the file being used by clover engine is in a different path. I’ll try to edit there.
CloverETL uses default engine properties file unless specified otherwise. In order to do so, in CloverETL designer go to menu Run | Run Configurations and on Arguments tab specify Program argument
-config engine_configuration_file
The engine_configuration_file is a full path or path relative to project.
Jan,
Thanks. We would really like to make a change to the defaultProperties file used on the server but since that is proving difficult I will ask about using the –config switch on the Arguments tab of Run Configurations. I have done this successfully in Clover ETL Designer and tested on a graph I’m using to recreate our problem. However, the graph we are having problems with is running on the server. It is scheduled to run on a regular basis on the server. How can we set the –config switch such that it will persist on the server and be used for all scheduled runs of our graph??
When I logon to the server there is a “Monitoring” link. Clicking on this link brings me to a page with a number of tabs. The “Clover Engine properties” tab shows all the values that are being used by the engine. The “Clover Server properties” tab has no data associated with engine.config.file. See attached. Also, for the –config switch question, is it only possible to use from within designer?
On the server where we are running clover I see three copies of defaultProperties in the following paths:
E:\Archives\CloverETL\cloverETL.src.rel-3-0-0\cloverETL\cloveretl.engine\src\org\jetel\data
E:\Archives\CloverETL\cloverETL.src.rel-2-9-4\cloverETL\cloveretl.engine\src\org\jetel\data
E:\Apps
We are running 3.0.0 so I doubt the 2.9.4 file is used at all. I can change these files but any change is not being picked up by the server…I have tried.
Thanks
Hello,
Any thoughts on the above post? Where might we find the config file being used by our server?? Also, regarding the second part of the question. Is it possible to use the -config switch for a graph that is running scheduled on the server?
You need to set path to your custom engine configuration file into the engine.config.file server property. How to do that? I suggest 2 solutions. First is simpler, second is cleaner, I guess.
Alter the engine.config.file property of the default config.properties server configuration file that comes with the Clover Server clover.war distribution. It’s located inside WEB-INF directory inside your clover.war file (which is actually ordinary zip file). Then redeploy the modified clover.war file on your application server. Check out the “Clover Server properties” tab then if the engine.config.file property has desired value.
In case you don’t want to change the clover.war file, you need to somehow specify custom Server configuration file instead of the default config.properties from your clover.war Server distribution, see http://server-demo-ec2.cloveretl.com/cl … urces.html on possible ways how this is done.
Tom,
Thanks. I have tried a number of things without much success. My apologies but this is going to be a long post.
The link you provided mentions :
[tomcat_home]/conf/Catalina/localhost/clover.xml
We are running on tomcat but I did not find the above file. I did find a file named ROOT.xml in path
E:\Apps\apache-tomcat-6.0.29\conf\Catalina\localhost\
I added :
I restarted tomcat server and looked at the Clover Engine Properties tab on the server. It was still showing the old vale for Record.MAX_RECORD_SIZE (12288).
Next I searched for a clover.xml file and found one in path :
E:\Apps\apache-tomcat-6.0.29\webapps\ROOT\WEB-INF\classes\com\cloveretl\server\nodemanager\
So I tried making changes to this file and restarted server but again the Clover Engine Properties tab is showing the original value.
Next I created a custom config file with one line:
Record.MAX_RECORD_SIZE=24576
And in E:\Apps\apache-tomcat-6.0.29\webapps\ROOT\WEB-INF\config.properties
I changed
engine.config.file=
to
engine.config.file=E:\Apps\apache-tomcat-6.0.29\webapps\ROOT\WEB-INF\custconfig.properties
This change was reflected on the Clover Engine Properties tab but it was now the only property displayed on that tab. I suspect that all the default engine properties were still intact but I could not be sure. Maybe you could confirm?? Is it possible to have default, plus custom properties displayed on the tab?
Next I tried adding all the properties that had previously been displayed on the Clover Engine Properties tab to my custom config file. Like this:
My graph started to get errors like :
Unclosed group near index 19 (?<!\)`(.*?)(?<!\)` ^
I suspect that some of the properties would need to be in double quotes? But I was not sure so I restored config.properties to look like : engine.config.file=
I also tried changing web.xml as mentioned in the linked page you provided but the Clover Engine properties tab again showed the original value.
Here is my question. Can I trust that my custom engine.config.file will only affect the properties defined in the custom file? All others will be unchanged defaults, correct? If I create a custom file is there some way to see all the settings since they are not being displayed on the Clover Engine Properties tab? thanks
yes, we can confirm that the custom engine.config.file will only affect the properties defined in itself, and the other properties will keep their defaults. The fact that you’re seeing only your custom properties is a bug, I’ve reported it in our bugtracker (see https://bug.javlin.eu/browse/CLS-607).
