Hi
Problem: DelimitedDataReaderNIO errors out with java.nio.BufferOverflowException. Does any one know why? Has anyone faced the same problem?
Graph: I have 2 (DelimitedDataReaderNIO ) connected to a Merge with key. The Merge is followed by a Reformat and then a writer. Note there’s no sort between reader and merge (since the data is already sorted). Of the 2 (DelimitedDataReaderNIO) the first one is fed with a file with ~8K records (11 columns), the second is fed almost no records. When I run this graph I get java.nio.BufferOverfException on the first reader.
Observations:
1. This graph runs fine for smaller files.
2. The reformat transform has logger.debug statements, which when turned off, the problem goes away.
3. If I put an ext_sort between first reader and merge, the problem goes away. Since the data is already sorted I don’t want to incur this cost, could it be that the data if not sorted shall cause this problem?
I think there’s a problem of buffering, the reader is too fast and reformat transform too slow, and there’s not enough room to buffer in between. How do these components sync speed?
Also, if I replace DelimitedDataReaderNIO with DelimitedDataReader then the graph always fails, and none of the above work arounds work.
Any help is appreciated.
Thanks in advance.
Akhil