Fast Sort BufferOverflowException

Support/help with CloverETL (4.9) and CloverDX (5.0 or newer) implementation problems

pro7
Posts: 42
Joined: Fri Aug 05, 2011 6:54 pm

Fast Sort BufferOverflowException

Postby pro7 » Wed Nov 02, 2011 9:35 pm

Hi,
We have a graph that has been running along just fine. Today it started throwing BufferOverflowException. We bumped both
Data.DATA_RECORDS_BUFFER_SIZE and Data.MAX_BUFFERS_ALLOCATED but still getting the errors. The number of records has not changed significantly from when the graph ran fine. What are other possibilities? Could it be one bad data record that is causing this problem? Thanks

11-02 13:20:06,485[ WatchDog] WatchDog ERROR 124553 Node SORT_GOREMAL finished with status: ERROR caused by: null
11-02 13:20:06,485[ WatchDog] WatchDog ERROR 124553 Node SORT_GOREMAL error details:
java.nio.BufferOverflowException
at java.nio.Buffer.nextPutIndex(Buffer.java:495)
at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:250)
at org.jetel.util.bytes.ByteBufferUtils.encodeLength(ByteBufferUtils.java:146)
at org.jetel.graph.DirectEdge.eof(DirectEdge.java:343)
at org.jetel.graph.Edge.eof(Edge.java:439)
at org.jetel.graph.Node.closeAllOutputPorts(Node.java:854)
at org.jetel.graph.Node.broadcastEOF(Node.java:865)
at com.opensys.cloveretl.component.FastSort.b(Unknown Source)
at com.opensys.cloveretl.component.FastSort.execute(Unknown Source)
at org.jetel.graph.Node.run(Node.java:425)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

pro7
Posts: 42
Joined: Fri Aug 05, 2011 6:54 pm

Re: Fast Sort BufferOverflowException

Postby pro7 » Thu Nov 03, 2011 3:56 pm

I changed to an ExtSort but I still get the error.

pro7
Posts: 42
Joined: Fri Aug 05, 2011 6:54 pm

Re: Fast Sort BufferOverflowException

Postby pro7 » Thu Nov 03, 2011 6:32 pm

Update.... It turns out that the changes we made to defaultProperties were not picked up by the clover engine. I believe the file being used by clover engine is in a different path. I'll try to edit there.

julrych
Posts: 36
Joined: Mon Dec 13, 2010 6:52 pm

Re: Fast Sort BufferOverflowException

Postby julrych » Thu Nov 03, 2011 8:25 pm

CloverETL uses default engine properties file unless specified otherwise. In order to do so, in CloverETL designer go to menu Run | Run Configurations and on Arguments tab specify Program argument

Code: Select all

-config engine_configuration_file

The engine_configuration_file is a full path or path relative to project.
Jan Ulrych
Javlin Inc.
[email protected]

pro7
Posts: 42
Joined: Fri Aug 05, 2011 6:54 pm

Re: Fast Sort BufferOverflowException

Postby pro7 » Fri Nov 04, 2011 3:16 pm

Jan,
Thanks. We would really like to make a change to the defaultProperties file used on the server but since that is proving difficult I will ask about using the –config switch on the Arguments tab of Run Configurations. I have done this successfully in Clover ETL Designer and tested on a graph I’m using to recreate our problem. However, the graph we are having problems with is running on the server. It is scheduled to run on a regular basis on the server. How can we set the –config switch such that it will persist on the server and be used for all scheduled runs of our graph??

Thanks

mzatopek
Posts: 76
Joined: Fri May 11, 2007 9:49 am

Re: Fast Sort BufferOverflowException

Postby mzatopek » Mon Nov 07, 2011 5:02 pm

The clover engine configuration file can be changed even on clover server, see 'engine.config.file' property at http://server-demo-ec2.cloveretl.com/clover/docs/list-of-properties.html.
Martin Zatopek
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com

pro7
Posts: 42
Joined: Fri Aug 05, 2011 6:54 pm

Re: Fast Sort BufferOverflowException

Postby pro7 » Mon Nov 07, 2011 5:27 pm

When I logon to the server there is a “Monitoring” link. Clicking on this link brings me to a page with a number of tabs. The “Clover Engine properties” tab shows all the values that are being used by the engine. The “Clover Server properties” tab has no data associated with engine.config.file. See attached. Also, for the –config switch question, is it only possible to use from within designer?

On the server where we are running clover I see three copies of defaultProperties in the following paths:
E:\Archives\CloverETL\cloverETL.src.rel-3-0-0\cloverETL\cloveretl.engine\src\org\jetel\data
E:\Archives\CloverETL\cloverETL.src.rel-2-9-4\cloverETL\cloveretl.engine\src\org\jetel\data
E:\Apps

We are running 3.0.0 so I doubt the 2.9.4 file is used at all. I can change these files but any change is not being picked up by the server....I have tried.
Thanks
Attachments
clover_server_prop.JPG
clover_server_prop.JPG (30.14 KiB) Viewed 15311 times

pro7
Posts: 42
Joined: Fri Aug 05, 2011 6:54 pm

Re: Fast Sort BufferOverflowException

Postby pro7 » Mon Nov 14, 2011 5:48 pm

Hello,
Any thoughts on the above post? Where might we find the config file being used by our server?? Also, regarding the second part of the question. Is it possible to use the -config switch for a graph that is running scheduled on the server?

Thanks

tkramolis
Posts: 8
Joined: Tue Nov 15, 2011 2:53 pm

Re: Fast Sort BufferOverflowException

Postby tkramolis » Tue Nov 15, 2011 5:22 pm

Hi,

You need to set path to your custom engine configuration file into the engine.config.file server property. How to do that? I suggest 2 solutions. First is simpler, second is cleaner, I guess.

  1. Alter the engine.config.file property of the default config.properties server configuration file that comes with the Clover Server clover.war distribution. It's located inside WEB-INF directory inside your clover.war file (which is actually ordinary zip file). Then redeploy the modified clover.war file on your application server. Check out the “Clover Server properties” tab then if the engine.config.file property has desired value.
  2. In case you don't want to change the clover.war file, you need to somehow specify custom Server configuration file instead of the default config.properties from your clover.war Server distribution, see http://server-demo-ec2.cloveretl.com/cl ... urces.html on possible ways how this is done.

Best regards,
Tom

Javlin a.s.

tkramolis
Posts: 8
Joined: Tue Nov 15, 2011 2:53 pm

Re: Fast Sort BufferOverflowException

Postby tkramolis » Tue Nov 15, 2011 5:31 pm

Regarding the -config switch, it is not possible to use it for graphs running on the Server.

--Tom

pro7
Posts: 42
Joined: Fri Aug 05, 2011 6:54 pm

Re: Fast Sort BufferOverflowException

Postby pro7 » Mon Dec 19, 2011 5:54 pm

Tom,
Thanks. I have tried a number of things without much success. My apologies but this is going to be a long post.
The link you provided mentions :

[tomcat_home]/conf/Catalina/localhost/clover.xml
We are running on tomcat but I did not find the above file. I did find a file named ROOT.xml in path
E:\Apps\apache-tomcat-6.0.29\conf\Catalina\localhost\
I added :
<Parameter name="Record.MAX_RECORD_SIZE" value="24576" override="false" />
I restarted tomcat server and looked at the Clover Engine Properties tab on the server. It was still showing the old vale for Record.MAX_RECORD_SIZE (12288).
Next I searched for a clover.xml file and found one in path :
E:\Apps\apache-tomcat-6.0.29\webapps\ROOT\WEB-INF\classes\com\cloveretl\server\nodemanager\
So I tried making changes to this file and restarted server but again the Clover Engine Properties tab is showing the original value.
Next I created a custom config file with one line:
Record.MAX_RECORD_SIZE=24576
And in E:\Apps\apache-tomcat-6.0.29\webapps\ROOT\WEB-INF\config.properties
I changed
engine.config.file=
to
engine.config.file=E:\Apps\apache-tomcat-6.0.29\webapps\ROOT\WEB-INF\custconfig.properties
This change was reflected on the Clover Engine Properties tab but it was now the only property displayed on that tab. I suspect that all the default engine properties were still intact but I could not be sure. Maybe you could confirm?? Is it possible to have default, plus custom properties displayed on the tab?
Next I tried adding all the properties that had previously been displayed on the Clover Engine Properties tab to my custom config file. Like this:

ASSIGN_SIGN= :=
CLOVER_FIELD_INDICATOR= $
CLOVER_FIELD_REGEX= \$[\w]+
Component.KEY_FIELDS_DELIMITER= ;
Component.KEY_FIELDS_DELIMITER_REGEX= \s*([|;]|:(?!=))\s*
DEFAULT_BINARY_PATH= ./
DEFAULT_DATETIME_FORMAT= yyyy-MM-dd HH:mm:ss
DEFAULT_DATE_FORMAT= yyyy-MM-dd
DEFAULT_INTERNAL_IO_BUFFER_SIZE= 49152
DEFAULT_IOSTREAM_CHANNEL_BUFFER_SIZE= 2048
DEFAULT_JAVA_COMPILER= internal
DEFAULT_PATH_SEPARATOR_REGEX= \s*;\s*
DEFAULT_PLUGINS_DIRECTORY= ./plugins
DEFAULT_REGEXP_FALSE_STRING= false|F|FALSE|NO|N|f|0|no|n
DEFAULT_REGEXP_TRUE_STRING= true|T|TRUE|YES|Y|t|1|yes|y
DEFAULT_TIME_FORMAT= HH:mm:ss
Data.DATA_RECORDS_BUFFER_SIZE= 10485760
Data.MAX_BUFFERS_ALLOCATED= 99
DataFieldMetadata.DECIMAL_LENGTH= 8
DataFieldMetadata.DECIMAL_SCALE= 2
DataFormatter.DEFAULT_CHARSET_ENCODER= ISO-8859-1
DataFormatter.DELIMITER_DELIMITERS_REGEX= \\\|
DataFormatter.FIELD_BUFFER_LENGTH= 4096
DataParser.DEFAULT_CHARSET_DECODER= ISO-8859-1
DataParser.FIELD_BUFFER_LENGTH= 4096
Graph.DIRECT_EDGE_FAST_PROPAGATE_NUM_INTERNAL_BUFFERS= 4
GraphProperties.EXPRESSION_EVALUATION_ENABLED= true
GraphProperties.EXPRESSION_PLACEHOLDER_REGEX= (?<!\\)`(.*?)(?<!\\)`
GraphProperties.PROPERTY_PLACEHOLDER_REGEX= \$\{(\w+)\}
INCREMENTAL_STORE_KEY= incremental_store
InternalSortDataRecord.DEFAULT_INTERNAL_SORT_BUFFER_CAPACITY= 8000
Lookup.LOOKUP_INITIAL_CAPACITY= 512
OracleConnection.ROW_PREFETCH= 100
PACKAGES_EXCLUDED_FROM_GREEDY_CLASS_LOADING= java.;javax.;sun.misc.
PortReadingWriting.DATA_LENGTH= 2048
Record.DEFAULT_COMPRESSION_LEVEL= 5
Record.MAX_RECORD_SIZE= 12288
WatchDog.DEFAULT_WATCHDOG_TRACKING_INTERVAL= 5000
WatchDog.NUMBER_OF_TICKS_BETWEEN_STATUS_CHECKS= 1
WatchDog.WATCHDOG_SLEEP_INTERVAL= 1000

My graph started to get errors like :
Unclosed group near index 19 (?<!\)`(.*?)(?<!\)` ^
I suspect that some of the properties would need to be in double quotes? But I was not sure so I restored config.properties to look like : engine.config.file=
I also tried changing web.xml as mentioned in the linked page you provided but the Clover Engine properties tab again showed the original value.

pro7
Posts: 42
Joined: Fri Aug 05, 2011 6:54 pm

Re: Fast Sort BufferOverflowException

Postby pro7 » Tue Dec 27, 2011 2:30 am

Here is my question. Can I trust that my custom engine.config.file will only affect the properties defined in the custom file? All others will be unchanged defaults, correct? If I create a custom file is there some way to see all the settings since they are not being displayed on the Clover Engine Properties tab? thanks

jurban
Posts: 163
Joined: Fri Jul 20, 2007 9:25 am

Re: Fast Sort BufferOverflowException

Postby jurban » Fri Jan 06, 2012 4:46 pm

Hi,

yes, we can confirm that the custom engine.config.file will only affect the properties defined in itself, and the other properties will keep their defaults. The fact that you're seeing only your custom properties is a bug, I've reported it in our bugtracker (see https://bug.javlin.eu/browse/CLS-607).

Regards,
Jaro
Jaroslav Urban
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com


cron