# Fast Sort BufferOverflowException

**URL:** <https://forum.cloverdx.com/t/fast-sort-bufferoverflowexception/958>\
**Category:** CloverDX Platform\
**Created:** [November 2, 2011, 12:00am UTC](https://forum.cloverdx.com/t/fast-sort-bufferoverflowexception/958 "2011-11-02T00:00:00Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![pro7](https://avatars.discourse-cdn.com/v4/letter/p/13edae/32.png) [@pro7](https://forum.cloverdx.com/u/pro7)\
**Post date:** [November 2, 2011, 12:00am UTC](https://forum.cloverdx.com/t/fast-sort-bufferoverflowexception/958/1 "2011-11-02T00:00:00Z")

</div>

Hi,  
We have a graph that has been running along just fine. Today it started throwing BufferOverflowException. We bumped both  
Data.DATA\_RECORDS\_BUFFER\_SIZE and Data.MAX\_BUFFERS\_ALLOCATED but still getting the errors. The number of records has not changed significantly from when the graph ran fine. What are other possibilities? Could it be one bad data record that is causing this problem? Thanks

11-02 13:20:06,485[WatchDog] WatchDog ERROR 124553 Node SORT\_GOREMAL finished with status: ERROR caused by: null  
11-02 13:20:06,485[WatchDog] WatchDog ERROR 124553 Node SORT\_GOREMAL error details:  
java.nio.BufferOverflowException  
at java.nio.Buffer.nextPutIndex(Buffer.java:495)  
at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:250)  
at org.jetel.util.bytes.ByteBufferUtils.encodeLength(ByteBufferUtils.java:146)  
at org.jetel.graph.DirectEdge.eof(DirectEdge.java:343)  
at org.jetel.graph.Edge.eof(Edge.java:439)  
at org.jetel.graph.Node.closeAllOutputPorts(Node.java:854)  
at org.jetel.graph.Node.broadcastEOF(Node.java:865)  
at com.opensys.cloveretl.component.FastSort.b(Unknown Source)  
at com.opensys.cloveretl.component.FastSort.execute(Unknown Source)  
at org.jetel.graph.Node.run(Node.java:425)  
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
at java.lang.Thread.run(Thread.java:619)

---

<div class="post-metadata">

**Author:** ![pro7](https://avatars.discourse-cdn.com/v4/letter/p/13edae/32.png) [@pro7](https://forum.cloverdx.com/u/pro7)\
**Post date:** [November 3, 2011, 2:56pm UTC](https://forum.cloverdx.com/t/fast-sort-bufferoverflowexception/958/2 "2011-11-03T14:56:31Z")

</div>

I changed to an ExtSort but I still get the error.

---

<div class="post-metadata">

**Author:** ![pro7](https://avatars.discourse-cdn.com/v4/letter/p/13edae/32.png) [@pro7](https://forum.cloverdx.com/u/pro7)\
**Post date:** [November 3, 2011, 5:32pm UTC](https://forum.cloverdx.com/t/fast-sort-bufferoverflowexception/958/3 "2011-11-03T17:32:26Z")

</div>

Update… It turns out that the changes we made to defaultProperties were not picked up by the clover engine. I believe the file being used by clover engine is in a different path. I’ll try to edit there.

---

<div class="post-metadata">

**Author:** ![julrych](https://avatars.discourse-cdn.com/v4/letter/j/7ba0ec/32.png) [@julrych](https://forum.cloverdx.com/u/julrych)\
**Post date:** [November 3, 2011, 7:25pm UTC](https://forum.cloverdx.com/t/fast-sort-bufferoverflowexception/958/4 "2011-11-03T19:25:33Z")

</div>

CloverETL uses default engine properties file unless specified otherwise. In order to do so, in CloverETL designer go to menu Run | Run Configurations and on Arguments tab specify Program argument

```auto
-config engine_configuration_file

```

The engine\_configuration\_file is a full path or path relative to project.

---

<div class="post-metadata">

**Author:** ![pro7](https://avatars.discourse-cdn.com/v4/letter/p/13edae/32.png) [@pro7](https://forum.cloverdx.com/u/pro7)\
**Post date:** [November 4, 2011, 2:16pm UTC](https://forum.cloverdx.com/t/fast-sort-bufferoverflowexception/958/5 "2011-11-04T14:16:02Z")

</div>

Jan,  
Thanks. We would really like to make a change to the defaultProperties file used on the server but since that is proving difficult I will ask about using the –config switch on the Arguments tab of Run Configurations. I have done this successfully in Clover ETL Designer and tested on a graph I’m using to recreate our problem. However, the graph we are having problems with is running on the server. It is scheduled to run on a regular basis on the server. How can we set the –config switch such that it will persist on the server and be used for all scheduled runs of our graph??

Thanks

---

<div class="post-metadata">

**Author:** ![mzatopek](https://avatars.discourse-cdn.com/v4/letter/m/b5e925/32.png) [@mzatopek](https://forum.cloverdx.com/u/mzatopek)\
**Post date:** [November 7, 2011, 4:02pm UTC](https://forum.cloverdx.com/t/fast-sort-bufferoverflowexception/958/6 "2011-11-07T16:02:04Z")

</div>

The clover engine configuration file can be changed even on clover server, see ‘engine.config.file’ property at [http://server-demo-ec2.cloveretl.com/clover/docs/list-of-properties.html](http://server-demo-ec2.cloveretl.com/clover/docs/list-of-properties.html).

---

<div class="post-metadata">

**Author:** ![pro7](https://avatars.discourse-cdn.com/v4/letter/p/13edae/32.png) [@pro7](https://forum.cloverdx.com/u/pro7)\
**Post date:** [November 7, 2011, 4:27pm UTC](https://forum.cloverdx.com/t/fast-sort-bufferoverflowexception/958/7 "2011-11-07T16:27:03Z")

</div>

When I logon to the server there is a “Monitoring” link. Clicking on this link brings me to a page with a number of tabs. The “Clover Engine properties” tab shows all the values that are being used by the engine. The “Clover Server properties” tab has no data associated with engine.config.file. See attached. Also, for the –config switch question, is it only possible to use from within designer?

On the server where we are running clover I see three copies of defaultProperties in the following paths:  
E:\Archives\CloverETL\cloverETL.src.rel-3-0-0\cloverETL\cloveretl.engine\src\org\jetel\data  
E:\Archives\CloverETL\cloverETL.src.rel-2-9-4\cloverETL\cloveretl.engine\src\org\jetel\data  
E:\Apps

We are running 3.0.0 so I doubt the 2.9.4 file is used at all. I can change these files but any change is not being picked up by the server…I have tried.  
Thanks

- [clover\_server\_prop.JPG](https://cdn.cloverdx.com/forum/13662%5F96801bd0a598a06c4667b74444b8a9ba/clover%5Fserver%5Fprop.JPG)

---

<div class="post-metadata">

**Author:** ![pro7](https://avatars.discourse-cdn.com/v4/letter/p/13edae/32.png) [@pro7](https://forum.cloverdx.com/u/pro7)\
**Post date:** [November 14, 2011, 4:48pm UTC](https://forum.cloverdx.com/t/fast-sort-bufferoverflowexception/958/8 "2011-11-14T16:48:31Z")

</div>

Hello,  
Any thoughts on the above post? Where might we find the config file being used by our server?? Also, regarding the second part of the question. Is it possible to use the -config switch for a graph that is running scheduled on the server?

Thanks

---

<div class="post-metadata">

**Author:** ![tkramolis](https://avatars.discourse-cdn.com/v4/letter/t/eb8c5e/32.png) [@tkramolis](https://forum.cloverdx.com/u/tkramolis)\
**Post date:** [November 15, 2011, 4:22pm UTC](https://forum.cloverdx.com/t/fast-sort-bufferoverflowexception/958/9 "2011-11-15T16:22:56Z")

</div>

Hi,

You need to set path to your custom engine configuration file into the _engine.config.file_ server property. How to do that? I suggest 2 solutions. First is simpler, second is cleaner, I guess.

- Alter the _engine.config.file_ property of the default _config.properties_ server configuration file that comes with the Clover Server _clover.war_ distribution. It’s located inside _WEB-INF_ directory inside your _clover.war_ file (which is actually ordinary zip file). Then redeploy the modified _clover.war_ file on your application server. Check out the “Clover Server properties” tab then if the _engine.config.file_ property has desired value.

- In case you don’t want to change the _clover.war_ file, you need to somehow specify custom Server configuration file instead of the default _config.properties_ from your _clover.war_ Server distribution, see [http://server-demo-ec2.cloveretl.com/cl … urces.html](http://server-demo-ec2.cloveretl.com/clover/docs/config-sources.html) on possible ways how this is done.

Best regards,  
Tom

Javlin a.s.

---

<div class="post-metadata">

**Author:** ![tkramolis](https://avatars.discourse-cdn.com/v4/letter/t/eb8c5e/32.png) [@tkramolis](https://forum.cloverdx.com/u/tkramolis)\
**Post date:** [November 15, 2011, 4:31pm UTC](https://forum.cloverdx.com/t/fast-sort-bufferoverflowexception/958/10 "2011-11-15T16:31:16Z")

</div>

Regarding the _-config_ switch, it is not possible to use it for graphs running on the Server.

--Tom

---

<div class="post-metadata">

**Author:** ![pro7](https://avatars.discourse-cdn.com/v4/letter/p/13edae/32.png) [@pro7](https://forum.cloverdx.com/u/pro7)\
**Post date:** [December 19, 2011, 4:54pm UTC](https://forum.cloverdx.com/t/fast-sort-bufferoverflowexception/958/11 "2011-12-19T16:54:16Z")

</div>

Tom,  
Thanks. I have tried a number of things without much success. My apologies but this is going to be a long post.  
The link you provided mentions :

[tomcat\_home]/conf/Catalina/localhost/clover.xml  
We are running on tomcat but I did not find the above file. I did find a file named ROOT.xml in path  
E:\Apps\apache-tomcat-6.0.29\conf\Catalina\localhost\  
I added :  
  
I restarted tomcat server and looked at the Clover Engine Properties tab on the server. It was still showing the old vale for Record.MAX\_RECORD\_SIZE (12288).  
Next I searched for a clover.xml file and found one in path :  
E:\Apps\apache-tomcat-6.0.29\webapps\ROOT\WEB-INF\classes\com\cloveretl\server\nodemanager\  
So I tried making changes to this file and restarted server but again the Clover Engine Properties tab is showing the original value.  
Next I created a custom config file with one line:  
Record.MAX\_RECORD\_SIZE=24576  
And in E:\Apps\apache-tomcat-6.0.29\webapps\ROOT\WEB-INF\config.properties  
I changed  
engine.config.file=  
to  
engine.config.file=E:\Apps\apache-tomcat-6.0.29\webapps\ROOT\WEB-INF\custconfig.properties  
This change was reflected on the Clover Engine Properties tab but it was now the only property displayed on that tab. I suspect that all the default engine properties were still intact but I could not be sure. Maybe you could confirm?? Is it possible to have default, plus custom properties displayed on the tab?  
Next I tried adding all the properties that had previously been displayed on the Clover Engine Properties tab to my custom config file. Like this:

ASSIGN\_SIGN= :=  
CLOVER\_FIELD\_INDICATOR= $  
CLOVER\_FIELD\_REGEX= \$[\w]+  
Component.KEY\_FIELDS\_DELIMITER= ;  
Component.KEY\_FIELDS\_DELIMITER\_REGEX= \s\*([|;]|:(?!=))\s\*  
DEFAULT\_BINARY\_PATH= ./  
DEFAULT\_DATETIME\_FORMAT= yyyy-MM-dd HH:mm:ss  
DEFAULT\_DATE\_FORMAT= yyyy-MM-dd  
DEFAULT\_INTERNAL\_IO\_BUFFER\_SIZE= 49152  
DEFAULT\_IOSTREAM\_CHANNEL\_BUFFER\_SIZE= 2048  
DEFAULT\_JAVA\_COMPILER= internal  
DEFAULT\_PATH\_SEPARATOR\_REGEX= \s\*;\s\*  
DEFAULT\_PLUGINS\_DIRECTORY= ./plugins  
DEFAULT\_REGEXP\_FALSE\_STRING= false|F|FALSE|NO|N|f|0|no|n  
DEFAULT\_REGEXP\_TRUE\_STRING= true|T|TRUE|YES|Y|t|1|yes|y  
DEFAULT\_TIME\_FORMAT= HH:mm:ss  
Data.DATA\_RECORDS\_BUFFER\_SIZE= 10485760  
Data.MAX\_BUFFERS\_ALLOCATED= 99  
DataFieldMetadata.DECIMAL\_LENGTH= 8  
DataFieldMetadata.DECIMAL\_SCALE= 2  
DataFormatter.DEFAULT\_CHARSET\_ENCODER= ISO-8859-1  
DataFormatter.DELIMITER\_DELIMITERS\_REGEX= \\\|  
DataFormatter.FIELD\_BUFFER\_LENGTH= 4096  
DataParser.DEFAULT\_CHARSET\_DECODER= ISO-8859-1  
DataParser.FIELD\_BUFFER\_LENGTH= 4096  
Graph.DIRECT\_EDGE\_FAST\_PROPAGATE\_NUM\_INTERNAL\_BUFFERS= 4  
GraphProperties.EXPRESSION\_EVALUATION\_ENABLED= true  
GraphProperties.EXPRESSION\_PLACEHOLDER\_REGEX= (?\<!\\)`(.*?)(?\<!\\)`  
GraphProperties.PROPERTY\_PLACEHOLDER\_REGEX= \$\{(\w+)\}  
INCREMENTAL\_STORE\_KEY= incremental\_store  
InternalSortDataRecord.DEFAULT\_INTERNAL\_SORT\_BUFFER\_CAPACITY= 8000  
Lookup.LOOKUP\_INITIAL\_CAPACITY= 512  
OracleConnection.ROW\_PREFETCH= 100  
PACKAGES\_EXCLUDED\_FROM\_GREEDY\_CLASS\_LOADING= java.;javax.;sun.misc.  
PortReadingWriting.DATA\_LENGTH= 2048  
Record.DEFAULT\_COMPRESSION\_LEVEL= 5  
Record.MAX\_RECORD\_SIZE= 12288  
WatchDog.DEFAULT\_WATCHDOG\_TRACKING\_INTERVAL= 5000  
WatchDog.NUMBER\_OF\_TICKS\_BETWEEN\_STATUS\_CHECKS= 1  
WatchDog.WATCHDOG\_SLEEP\_INTERVAL= 1000

My graph started to get errors like :  
Unclosed group near index 19 (?\<!\)`(.*?)(?\<!\)` ^  
I suspect that some of the properties would need to be in double quotes? But I was not sure so I restored config.properties to look like : engine.config.file=  
I also tried changing web.xml as mentioned in the linked page you provided but the Clover Engine properties tab again showed the original value.

---

<div class="post-metadata">

**Author:** ![pro7](https://avatars.discourse-cdn.com/v4/letter/p/13edae/32.png) [@pro7](https://forum.cloverdx.com/u/pro7)\
**Post date:** [December 27, 2011, 1:30am UTC](https://forum.cloverdx.com/t/fast-sort-bufferoverflowexception/958/12 "2011-12-27T01:30:13Z")

</div>

Here is my question. Can I trust that my custom engine.config.file will only affect the properties defined in the custom file? All others will be unchanged defaults, correct? If I create a custom file is there some way to see all the settings since they are not being displayed on the Clover Engine Properties tab? thanks

---

<div class="post-metadata">

**Author:** ![jurban](https://avatars.discourse-cdn.com/v4/letter/j/cc9497/32.png) [@jurban](https://forum.cloverdx.com/u/jurban)\
**Post date:** [January 6, 2012, 3:46pm UTC](https://forum.cloverdx.com/t/fast-sort-bufferoverflowexception/958/13 "2012-01-06T15:46:08Z")

</div>

Hi,

yes, we can confirm that the custom engine.config.file will only affect the properties defined in itself, and the other properties will keep their defaults. The fact that you’re seeing only your custom properties is a bug, I’ve reported it in our bugtracker (see [https://bug.javlin.eu/browse/CLS-607](https://bug.javlin.eu/browse/CLS-607)).

Regards,  
Jaro