Here is my run down. Jforum broke. It’s not that big a problem except that the community is generally unhelpful. They tend to post once or twice then disapear. So i’m trying to convert my DB from JForum to PHPBB3.
The real work is converting the posts, topics and categories. It’s a personal small forum so there are few users.
JForum
Posts(DB Input1)
post_id
user_id
topic_id
forum_id
.bbcodes, smilies, time, ip,… the rest is not relative to orginization and data
Post_text(DB Input2)
post_id
post_text(the actual content of the post)
post_subject)
PHPBB3
Post(DB)
post_id
user_id
topic_id
forum_id
post_text(the actual content of the post)
post_subject)
basicly the phpbb just uses a single version of the post rather than splitting them.
So what i’ve done is have 2 DBInput tables. a ApproxMergerJoin and testing with a XLSDataWriter using a hand written meta data for output after joining. I’m just using the gui at the moment.
My current problem is with the size of data coming out of Database 2. posts_text. It seems to hit a memory limit. Anyhelp would be well extremly helpful.
INFO [main] - *** CloverETL framework/transformation graph runner ver 2.5, (c) 2002-06 D.Pavlis, released under GNU Lesser General Public License ***
INFO [main] - Running with framework version: 2.5 build#devel compiled 16/09/2008 16:58:06
INFO [main] - Running on 2 CPU(s) max available memory for JVM 65088 KB
INFO [main] - Loading default properties from: defaultProperties
DEBUG [main] - Plugin org.jetel.bulkloader loaded.
id - org.jetel.bulkloader
version - 2.5.2
provider-name -
component { className = org.jetel.component.OracleDataReader; type = ORACLE_DATA_READER; }
component { className = org.jetel.component.OracleDataWriter; type = ORACLE_DATA_WRITER; }
component { className = org.jetel.component.MysqlDataReader; type = MYSQL_DATA_READER; }
component { className = org.jetel.component.DB2DataWriter; type = DB2_DATA_WRITER; }
component { className = org.jetel.component.InformixDataWriter; type = INFORMIX_DATA_WRITER; }
component { className = org.jetel.component.MsSqlDataWriter; type = MS_SQL_DATA_WRITER; }
component { className = org.jetel.component.MysqlDataWriter; type = MYSQL_DATA_WRITER; }
component { className = org.jetel.component.PostgreSqlDataWriter; type = POSTGRESQL_DATA_WRITER; }
DEBUG [main] - Plugin org.jetel.component loaded.
id - org.jetel.component
version - 2.5.2
provider-name - null
component { className = org.jetel.component.DataReader; type = DATA_READER; }
component { className = org.jetel.component.DataWriter; type = DATA_WRITER; }
component { className = org.jetel.component.DelimitedDataReader; type = DELIMITED_DATA_READER; }
component { className = org.jetel.component.DelimitedDataWriter; type = DELIMITED_DATA_WRITER; }
component { className = org.jetel.component.SimpleCopy; type = SIMPLE_COPY; }
component { className = org.jetel.component.Concatenate; type = CONCATENATE; }
component { className = org.jetel.component.SimpleGather; type = SIMPLE_GATHER; }
component { className = org.jetel.component.Reformat; type = REFORMAT; }
component { className = org.jetel.component.DBInputTable; type = DB_INPUT_TABLE; }
component { className = org.jetel.component.Sort; type = SORT; }
component { className = org.jetel.component.DBOutputTable; type = DB_OUTPUT_TABLE; }
component { className = org.jetel.component.FixLenDataWriter; type = FIXLEN_DATA_WRITER; }
component { className = org.jetel.component.Dedup; type = DEDUP; }
component { className = org.jetel.component.FixLenDataReader; type = FIXLEN_DATA_READER; }
component { className = org.jetel.component.Merge; type = MERGE; }
component { className = org.jetel.component.MergeJoin; type = MERGE_JOIN; }
component { className = org.jetel.component.MergeJoin; type = EXT_MERGE_JOIN; }
component { className = org.jetel.component.MergeJoin; type = SORTED_JOIN; }
component { className = org.jetel.component.Trash; type = TRASH; }
component { className = org.jetel.component.DBExecute; type = DB_EXECUTE; }
component { className = org.jetel.component.HashJoin; type = HASH_JOIN; }
component { className = org.jetel.component.HashJoin; type = EXT_HASH_JOIN; }
component { className = org.jetel.component.CheckForeignKey; type = CHECK_FOREIGN_KEY; }
component { className = org.jetel.component.DBFDataReader; type = DBF_DATA_READER; }
component { className = org.jetel.component.ExtFilter; type = EXT_FILTER; }
component { className = org.jetel.component.ExtSort; type = EXT_SORT; }
component { className = org.jetel.component.Partition; type = PARTITION; }
component { className = org.jetel.component.DataIntersection; type = DATA_INTERSECTION; }
component { className = org.jetel.component.Aggregate; type = AGGREGATE; }
component { className = org.jetel.component.SystemExecute; type = SYS_EXECUTE; }
component { className = org.jetel.component.RunGraph; type = RUN_GRAPH; }
component { className = org.jetel.component.KeyGenerator; type = KEY_GEN; }
component { className = org.jetel.component.AproxMergeJoin; type = APROX_MERGE_JOIN; }
component { className = org.jetel.component.DBJoin; type = DBJOIN; }
component { className = org.jetel.component.XLSReader; type = XLS_READER; }
component { className = org.jetel.component.XLSWriter; type = XLS_WRITER; }
component { className = org.jetel.component.CloverDataWriter; type = CLOVER_WRITER; }
component { className = org.jetel.component.CloverDataReader; type = CLOVER_READER; }
component { className = org.jetel.component.StructureWriter; type = STRUCTURE_WRITER; }
component { className = org.jetel.component.Normalizer; type = NORMALIZER; }
component { className = org.jetel.component.Denormalizer; type = DENORMALIZER; }
component { className = org.jetel.component.JmsReader; type = JMS_READER; }
component { className = org.jetel.component.JmsWriter; type = JMS_WRITER; }
component { className = org.jetel.component.LookupJoin; type = LOOKUP_JOIN; }
component { className = org.jetel.component.LookupTableReaderWriter; type = LOOKUP_TABLE_READER_WRITER; }
component { className = org.jetel.component.DataGenerator; type = DATA_GENERATOR; }
component { className = org.jetel.component.MultiLevelReader; type = MULTI_LEVEL_READER; }
component { className = org.jetel.component.SequenceChecker; type = SEQUENCE_CHECKER; }
component { className = org.jetel.component.TextTableWriter; type = TEXT_TABLE_WRITER; }
component { className = org.jetel.component.XmlXPathReader; type = XML_XPATH_READER; }
component { className = org.jetel.component.Pacemaker; type = PACEMAKER; }
component { className = org.jetel.component.SpeedLimiter; type = SPEED_LIMITER; }
component { className = org.jetel.component.JavaExecute; type = JAVA_EXECUTE; }
component { className = org.jetel.component.XmlWriter; type = XML_WRITER; }
component { className = org.jetel.component.XSLDataTransformer; type = XSL_TRANSFORMER; }
DEBUG [main] - Plugin org.jetel.connection loaded.
id - org.jetel.connection
version - 2.5.2
provider-name - null
connection { className = org.jetel.connection.jdbc.DBConnection; type = JDBC; }
connection { className = org.jetel.connection.jms.JmsConnection; type = JMS; }
DEBUG [main] - Plugin org.jetel.engine loaded.
id - org.jetel.engine
version - 2.5.2
provider-name - null
dictionaryEntry { className = org.jetel.graph.dictionary.StringDictionaryEntryProvider; type = string; }
DEBUG [main] - Plugin org.jetel.jdbc loaded.
id - org.jetel.jdbc
version - 2.5.2
provider-name - null
jdbcSpecific { name = DB2; class = org.jetel.connection.jdbc.specific.impl.DB2Specific; database = DB2; }
jdbcSpecific { name = Informix; class = org.jetel.connection.jdbc.specific.impl.InformixSpecific; database = INFORMIX; }
jdbcSpecific { name = MS SQL; class = org.jetel.connection.jdbc.specific.impl.MSSQLSpecific; database = MSSQL; }
jdbcSpecific { name = MySQL; class = org.jetel.connection.jdbc.specific.impl.MySQLSpecific; database = MYSQL; }
jdbcSpecific { name = Oracle; class = org.jetel.connection.jdbc.specific.impl.OracleSpecific; database = ORACLE; }
jdbcSpecific { name = PostgreSQL; class = org.jetel.connection.jdbc.specific.impl.PostgreSpecific; database = POSTGRE; }
jdbcDriver { driverLibrary = lib/mysql/mysql-connector-java-5.1.5-bin.jar; jdbcSpecific = MYSQL; name = MySQL; dbDriver = org.gjt.mm.mysql.Driver; urlHint = jdbc:mysql://hostname:3306/database; database = MYSQL; }
jdbcDriver { driverLibrary = lib/postgre/postgresql-8.3-603.jdbc3.jar; jdbcSpecific = POSTGRE; name = PostgreSQL; dbDriver = org.postgresql.Driver; urlHint = jdbc:postgresql://hostname/database; database = POSTGRE; }
DEBUG [main] - Plugin org.jetel.lookup loaded.
id - org.jetel.lookup
version - 2.5.2
provider-name - null
lookup { className = org.jetel.lookup.SimpleLookupTable; type = simpleLookup; }
lookup { className = org.jetel.lookup.DBLookupTable; type = dbLookup; }
lookup { className = org.jetel.lookup.RangeLookupTable; type = rangeLookup; }
DEBUG [main] - Plugin org.jetel.sequence loaded.
id - org.jetel.sequence
version - 2.5.2
provider-name - null
sequence { className = org.jetel.sequence.SimpleSequence; type = SIMPLE_SEQUENCE; }
sequence { className = org.jetel.sequence.PrimitiveSequence; type = PRIMITIVE_SEQUENCE; }
DEBUG [main] - Plugin org.jetel.thirdparty loaded.
id - org.jetel.thirdparty
version - 2.5.2
provider-name - null
component { className = org.jetel.component.XMLExtract; type = XML_EXTRACT; }
component { className = org.jetel.component.Filter; type = FILTER; }
component { className = com.linagora.component.LdapReader; type = LDAP_READER; }
component { className = com.linagora.component.LdapWriter; type = LDAP_WRITER; }
DEBUG [main] - Plugin org.jetel.tlfunction loaded.
id - org.jetel.tlfunction
version - 2.5.2
provider-name - null
tlfunction { libraryName = math; className = org.jetel.interpreter.extensions.MathLib; function = sqrt,log,log10,exp,round,pow,pi,e,random,abs; }
tlfunction { libraryName = date; className = org.jetel.interpreter.extensions.DateLib; function = today,dateadd,datediff,trunc; }
tlfunction { libraryName = string; className = org.jetel.interpreter.extensions.StringLib; function = concat,uppercase,lowercase,substring,left,right,trim,length,soundex,replace,split,char_at,is_blank,is_ascii,is_number,is_integer,is_long,is_date,remove_diacritic,remove_blank_space,get_alphanumeric_chars,translate,join,index_of,count_char; }
tlfunction { libraryName = convert; className = org.jetel.interpreter.extensions.ConvertLib; function = num2str,date2str,str2date,date2num,str2num,try_convert,base64byte,byte2base64,bits2str,str2bits,hex2byte,byte2hex,num2num,num2bool,bool2num,str2bool,long2date,date2long,to_string; }
INFO [main] - Graph definition file: graph/posts.grf
INFO [main] - Checking graph configuration…
INFO [main] - Graph configuration is valid.
INFO [main] - Graph initialization (posts)
INFO [main] - Initializing connection:
INFO [main] - DBConnection driver[org.jetel.connection.jdbc.driver.JdbcDriver@12aea3e]:jndi[null]:url[jdbc:hsqldb:hsql://192.168.0.10/xdb]:user[sa] … OK
INFO [main] - Initializing connection:
INFO [main] - DBConnection driver[org.jetel.connection.jdbc.driver.JdbcDriver@97d026]:jndi[null]:url[jdbc:mysql://192.168.0.10/sreforum]:user[jasonjarvis] … OK
INFO [main] - register MBean with name:org.jetel.graph.runtime:type=CLOVERJMX_1223575998812_0
INFO [WatchDog] - [Clover] Initializing phase: 0
DEBUG [WatchDog] - initializing edges:
DEBUG [WatchDog] - all edges initialized successfully…
DEBUG [WatchDog] - initializing nodes:
DEBUG [WatchDog] - APROX_MERGE_JOIN0 …OK
DEBUG [WatchDog] - DB_INPUT_TABLE0 …OK
DEBUG [WatchDog] - DB_INPUT_TABLE2 …OK
DEBUG [WatchDog] - TRASH0 …OK
DEBUG [WatchDog] - XLS_WRITER0 …OK
INFO [WatchDog] - [Clover] phase: 0 initialized successfully.
INFO [WatchDog] - Starting up all nodes in phase [0]
DEBUG [WatchDog] - APROX_MERGE_JOIN0 … started
DEBUG [WatchDog] - DB_INPUT_TABLE0 … started
DEBUG [WatchDog] - DB_INPUT_TABLE2 … started
DEBUG [WatchDog] - TRASH0 … started
DEBUG [WatchDog] - XLS_WRITER0 … started
INFO [WatchDog] - Sucessfully started all nodes in phase!
DEBUG [DB_INPUT_TABLE0] - DB_INPUT_TABLE0: Sending query “select * from JFORUM_POSTS”
DEBUG [DB_INPUT_TABLE2] - DB_INPUT_TABLE2: Sending query “select * from JFORUM_POSTS_TEXT”
DEBUG [DB_INPUT_TABLE0] - DB_INPUT_TABLE0: Query execution time: 00:00:00.062
DEBUG [DB_INPUT_TABLE2] - DB_INPUT_TABLE2: Query execution time: 00:00:00.703
ERROR [WatchDog] - Graph execution finished with error
ERROR [WatchDog] - Node DB_INPUT_TABLE2 finished with status: ERROR caused by: The size of data buffer is only 12288. Set appropriate parameter in defautProperties file.
DEBUG [WatchDog] - Node DB_INPUT_TABLE2 error details:
java.lang.RuntimeException: The size of data buffer is only 12288. Set appropriate parameter in defautProperties file.
at org.jetel.data.StringDataField.serialize(StringDataField.java:380)
at org.jetel.data.DataRecord.serialize(DataRecord.java:453)
at org.jetel.graph.DirectEdge.writeRecord(DirectEdge.java:238)
at org.jetel.graph.Edge.writeRecord(Edge.java:342)
at org.jetel.graph.Node.writeRecordBroadcast(Node.java:754)
at org.jetel.component.DBInputTable.execute(DBInputTable.java:244)
at org.jetel.graph.Node.run(Node.java:379)
at java.lang.Thread.run(Unknown Source)
Caused by: java.nio.BufferOverflowException
at java.nio.Buffer.nextPutIndex(Unknown Source)
at java.nio.DirectByteBuffer.putChar(Unknown Source)
at org.jetel.data.StringDataField.serialize(StringDataField.java:377)
… 7 more
ERROR [WatchDog] - !!! Phase finished with error - stopping graph run !!!
INFO [WatchDog] - ----------------------** Final tracking Log for phase [0] **---------------------
INFO [WatchDog] - Time: 09/10/08 18:25:22
INFO [WatchDog] - Node Status Port #Records #KB Rec/s KB/s
INFO [WatchDog] - ----------------------------------------------------------------------------------
INFO [WatchDog] - APROX_MERGE_JOIN0 RUNNING
INFO [WatchDog] - %cpu:… In:0 240 21 0 0
INFO [WatchDog] - In:1 17 37 0 0
INFO [WatchDog] - Out:0 0 0 0 0
INFO [WatchDog] - Out:1 0 0 0 0
INFO [WatchDog] - DB_INPUT_TABLE0 FINISHED_OK
INFO [WatchDog] - %cpu:… Out:0 240 21 0 0
INFO [WatchDog] - DB_INPUT_TABLE2 ERROR
INFO [WatchDog] - %cpu:… Out:0 0 0 0 0
INFO [WatchDog] - TRASH0 RUNNING
INFO [WatchDog] - %cpu:… In:0 0 0 0 0
INFO [WatchDog] - XLS_WRITER0 RUNNING
INFO [WatchDog] - %cpu:0.01 In:0 0 0 0 0
INFO [WatchDog] - ---------------------------------** End of Log **--------------------------------
INFO [WatchDog] - Execution of phase [0] successfully finished - elapsed time(sec): 1
INFO [WatchDog] - -----------------------** Summary of Phases execution **---------------------
INFO [WatchDog] - Phase# Finished Status RunTime(sec) MemoryAllocation(KB)
INFO [WatchDog] - 0 ERROR 1 10846
INFO [WatchDog] - ------------------------------** End of Summary **---------------------------
INFO [WatchDog] - WatchDog thread finished - total execution time: 1 (sec)
INFO [main] - Freeing graph resources.
ERROR [main] - Execution of graph failed !