# Error while loading data in the Database
**URL:** https://forum.cloverdx.com/t/error-while-loading-data-in-the-database/393
**Category:** CloverDX Platform
**Created:** [September 19, 2008, 12:00am UTC](https://forum.cloverdx.com/t/error-while-loading-data-in-the-database/393 "2008-09-19T00:00:00Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@mishrahanuyahoocoin](https://forum.cloverdx.com/u/mishrahanuyahoocoin)
#### Post date: [September 19, 2008, 12:00am UTC](https://forum.cloverdx.com/t/error-while-loading-data-in-the-database/393/1 "2008-09-19T00:00:00Z")
Hi,
I am facing problem that , i have two million data in a csv file . when i start loading some where in the file data is disturbed and that is broke in two ldifferent line…so my first questin is that how can i check that which line has this type of problem.
Second thisng is that when i enter that data (Broken dat in two lines) it gives error at those lines because data disturbed in different column of the db …so it gives dat trucation error in the db…
So plese help how can i handle tis problem in Clover…it is very critical for me…
Apart of this…some time i get run time error becase of the wrong data in file…so this breaks my programe…what i want that if there is this type of data problm then it load that error line in the report.csv and the rest of the data should e loade in th database,
Means error should be logged in the file and the rest of the data should be loaded in the database.
Error coming in the file…
1. Line breaking
2. End of file is not currect
Please help me how to handle this…
Thansk,
Hauman Mishra
---
### Author:  [@avackova](https://forum.cloverdx.com/u/avackova)
#### Post date: [September 22, 2008, 8:21am UTC](https://forum.cloverdx.com/t/error-while-loading-data-in-the-database/393/2 "2008-09-22T08:21:14Z")
Hello,
if you set proper data policy on the Reader (see [dataPolicy](http://wiki.clovergui.net/doku.php?id=components:readers#readersdatapolicy)), data , which can’t be read are skipped. If you want to control db loading set on DBOUtputTable maxErrors attribute on some positive value and connect output port to it (see [DBOutputTable](http://wiki.clovergui.net/doku.php?id=components:writers#dboutputtable)).
---
### Author:  [@mishrahanuyahoocoin](https://forum.cloverdx.com/u/mishrahanuyahoocoin)
#### Post date: [September 22, 2008, 2:18pm UTC](https://forum.cloverdx.com/t/error-while-loading-data-in-the-database/393/3 "2008-09-22T14:18:05Z")
Can we log those records in a file or in DB(error table) , so that we can have report that these are the records having problem…otherwise it is not usefull…because we must have solid reason to show that this is the eroor.
Thaks,
Hanuman Mishra
---
### Author:  [@avackova](https://forum.cloverdx.com/u/avackova)
#### Post date: [September 23, 2008, 6:48am UTC](https://forum.cloverdx.com/t/error-while-loading-data-in-the-database/393/4 "2008-09-23T06:48:20Z")
Hi,
when you connect logging port to DataReader ([DataReader](http://wiki.clovergui.net/doku.php?id=components:readers#datareader)) you can do with wrong record what you want eg. write them to flat file (DataWriter) or do database (DBOutputTable); the same with the DBOutputTable - if db error occurs wrong records (with error message) can be processed by any way. See [Data policy example](http://wiki.clovergui.net/doku.php?id=simple%5Fexamples#data%5Fpolicy) and [DB Load example](http://wiki.clovergui.net/examples/graphDBLoad.grf)
---
### Author:  [@mishrahanuyahoocoin](https://forum.cloverdx.com/u/mishrahanuyahoocoin)
#### Post date: [September 24, 2008, 8:45am UTC](https://forum.cloverdx.com/t/error-while-loading-data-in-the-database/393/5 "2008-09-24T08:45:44Z")
I am getting error in the data that , is in the data…
dta is like this:-
“1473227”,“31580”,“16”,“”,“9”,“4”,“372-372”,“1962”,“Addendum to “Single-transit, large-radius E-type devicest””,“”,“Nunn, W.M., Jr.”,“IEEE”,“”,“”,“”
how to load in DB…
actually it is title is devided in two parts…
Addendum to “Single-transit
&
large-radius E-type devicest”
Now how can i handle thsi situation…please help me i am in trouble…i have relise of the project
---
### Author:  [@avackova](https://forum.cloverdx.com/u/avackova)
#### Post date: [September 24, 2008, 10:32am UTC](https://forum.cloverdx.com/t/error-while-loading-data-in-the-database/393/6 "2008-09-24T10:32:33Z")
Hello,
it is not possible to read such data directly. Work around can be Transformat component, which will prepare such data for data base: eg.
```auto
```
---
### Author:  [@mishrahanuyahoocoin](https://forum.cloverdx.com/u/mishrahanuyahoocoin)
#### Post date: [September 24, 2008, 10:46am UTC](https://forum.cloverdx.com/t/error-while-loading-data-in-the-database/393/7 "2008-09-24T10:46:17Z")
I am using java transform whic is using
public boolean transform(DataRecord source, DataRecord target){
method…
so please can you send me this transformer in Java.
How should i take list as you explained in your example…
Thanks,
hanuman
---
### Author:  [@avackova](https://forum.cloverdx.com/u/avackova)
#### Post date: [September 24, 2008, 10:53am UTC](https://forum.cloverdx.com/t/error-while-loading-data-in-the-database/393/8 "2008-09-24T10:53:38Z")
in java you can use [string split](http://java.sun.com/j2se/1.5.0/docs/api/java/lang/String.html#split%28java.lang.String%29) method or org.jetel.util.string.StringUtils.split method.
---
### Author:  [@mishrahanuyahoocoin](https://forum.cloverdx.com/u/mishrahanuyahoocoin)
#### Post date: [September 24, 2008, 10:58am UTC](https://forum.cloverdx.com/t/error-while-loading-data-in-the-database/393/9 "2008-09-24T10:58:44Z")
My question is not that how to split data///// but my question is that how to transform
DataRecord source
…
So please help me…just write initial code that how to use DataRecord source in your programe…
Thanks,
Hanuman
---
### Author:  [@avackova](https://forum.cloverdx.com/u/avackova)
#### Post date: [September 24, 2008, 11:12am UTC](https://forum.cloverdx.com/t/error-while-loading-data-in-the-database/393/10 "2008-09-24T11:12:12Z")
```auto
import org.jetel.component.DataRecordTransform;
import org.jetel.data.DataRecord;
import org.jetel.exception.TransformException;
import org.jetel.util.string.StringUtils;
public class Transform extends DataRecordTransform {
@Override
public int transform(DataRecord[] inputRecords, DataRecord[] outputRecords)
throws TransformException {
String[] list = StringUtils.split(inputRecords[0].getField("offending").getValue().toString());
for (int i = 0; i < outputRecords[0].getNumFields(); i++) {
outputRecords[0].getField(i).setValue(createString());
}
return 0;
}
private String createString() {
// TODO Auto-generated method stub
return null;
}
}
```
---
### Author:  [@mishrahanuyahoocoin](https://forum.cloverdx.com/u/mishrahanuyahoocoin)
#### Post date: [September 24, 2008, 12:17pm UTC](https://forum.cloverdx.com/t/error-while-loading-data-in-the-database/393/11 "2008-09-24T12:17:36Z")
i am not able to jhande because in metadat my seperator in ,
and when i read by
for(int i=0;i1473227
1-----\>31580
1-----\>16
1-----\>9
1-----\>4
1-----\>372-372
1-----\>1962
**1-----\>Addendum to ""Single-transi**
**1-----\> large-radius E-type"" devicest"**
**1-----\>Nunn, W.M., Jr.**
1-----\>IEEE
1-----\>“,”
How can i handle that please help…
Thanks,
Hanuman
---
### Author:  [@mishrahanuyahoocoin](https://forum.cloverdx.com/u/mishrahanuyahoocoin)
#### Post date: [September 26, 2008, 11:39am UTC](https://forum.cloverdx.com/t/error-while-loading-data-in-the-database/393/12 "2008-09-26T11:39:12Z")
Hi Please help me for the same which i discueed as above…
Can i use String str=“\”,\“” known as “,” as seperator…if i use simple java code like BufferReader and Writers the i use seperator as “,”, it works file…but my data is tool long 2 milllion so it is very slow while inserting in DB sand finding deduplicates…
So Please help me how can it be solved it, is road block for me , and it is going to screw my clover application…
i want to break my data at “,” sepeartor…
Please help help…
Thanks,
Hanuman mIshra
---
### Author:  [@avackova](https://forum.cloverdx.com/u/avackova)
#### Post date: [September 26, 2008, 11:48am UTC](https://forum.cloverdx.com/t/error-while-loading-data-in-the-database/393/13 "2008-09-26T11:48:31Z")
It should work. All you need then, is to strip the quote from the begin of first field and the end of last field.
---
### Author:  [@mishrahanuyahoocoin](https://forum.cloverdx.com/u/mishrahanuyahoocoin)
#### Post date: [September 26, 2008, 2:23pm UTC](https://forum.cloverdx.com/t/error-while-loading-data-in-the-database/393/14 "2008-09-26T14:23:04Z")
TransformationGraph institutionGraph = new TransformationGraph(“ArticlesLoadGraph”);
DataRecordMetadata inputMetaData= feedController.MetadataInput;
inputMetaData.setFieldDelimiter(“\”,\“”);
inputMetaData.setRecordDelimiters(“\n”);
i am using ","for seperator and "\n"record seperator…
But clover is not reading like that…
input:-
“371”,“18”,“9”,“”,“33”,“1”,“104-109”,“1988”,“Extension of the optimality of the threshold policy in heterogeneous multiserver queueing systems”,“10.1109/9.371”,“Viniotis, I.”,“IEEE”,“”,“”,“”
“417”,“21”,“35”,“”,“26”,“1”,“8-15”,“1988”,“G.722: a new CCITT coding standard for digital transmission of wideband audio signals”,“10.1109/35.417”,“Mermelstein, P.”,“IEEE”,“”,“”,“”
out put
:-
10
10
3
,"
2
143-147
1988
Performance characteristics of a 1.5 μm single-frequency semiconductor laser with an external waveguide Braff reflector
10.1109/3.107
Olsson, N.A.
IEEE
,“,”
“9
13
3
,“24”,“5”,“766-774”,“1988”,“The azimuthal effective-index method”,“10.1109/3.192”,“Marcatili, E.A.J.”,“IEEE”,”“,”“,”"
:::::::::::::::::::::::::::::::::::
124
83
9
,"
5
511-512
1988
Comments on ‘Dynamic path planning for a mobile automation with limited information on the environment’ [with reply]
10.1109/9.1243
Lucas, C.
IEEE
,“,”
“40
92
10
,“35”,“5”,“400-402”,“1988”,“An electronic device for triggering a stimulator during membrane responsiveness determinations in cardiac tissues”,“10.1109/10.1401”,“Pruett, J.K.”,“IEEE”,”“,”“,”"
:::::::::::::::::::::::::::::::::::
154
99
29
,"
3
417-420
1988
On estimating the instantaneous frequency of a Gaussian random signal by use of the Wigner-Ville distribution
10.1109/29.1543
White, L.V.
IEEE
,“,”
“54
100
29
,“36”,“4”,“433-439”,“1988”,“Cepstral domain talker stress compensation for robust speech recognition”,“10.1109/29.1547”,“Chen, Y.”,“IEEE”,”“,”“,”"
:::::::::::::::::::::::::::::::::::
19
13
3
,"
5
780-786
1988
Influence of Ag+-Na+ ion-exchange equilibrium on waveguide index profiles
10.1109/3.194
Ramaswamy, R.V.
IEEE
,“,”
“3
14
3
,“24”,“6”,“1114-1117”,“1988”,“Mn2+ as a potential solid-state laser ion”,“10.1109/3.234”,“Clausen, R.”,“IEEE”,”“,”“,”"
:::::::::::::::::::::::::::::::::::
23
14
3
,"
6
1141-1150
1988
CW arc-lamp-pumped alexandrite lasers
10.1109/3.237
Samelson, H.
IEEE
,“,”
“3
16
4
,“23”,“3”,“875-877”,“1988”,“A MOS implementation of totally self-checking checker for the 1-out-of-3 code”,“10.1109/4.334”,“Tao, D.L.”,“IEEE”,”“,”“,”"
:::::::::::::::::::::::::::::::::::
8
9
3
,"
1
13-21
1988
Analysis of lateral-mode behavior of “win-stripe"lasers related to the negative slope in their current-light characteristics
10.1109/3.88
Watanabe, M.
IEEE
,”,"
"
9
3
,“24”,“1”,“83-93”,“1988”,“Wavelength-tunable electrooptic polarization conversion in birefringent waveguides”,“10.1109/3.97”,“Heismann, F.”,“IEEE”,“”,“”,“”
So please llok into this…can i have your contact no or so that i can talk you…i ahve dead line…it is very critical for me…
Thanks,
Hnauman MIshra
---
### Author:  [@avackova](https://forum.cloverdx.com/u/avackova)
#### Post date: [September 29, 2008, 6:52am UTC](https://forum.cloverdx.com/t/error-while-loading-data-in-the-database/393/15 "2008-09-29T06:52:40Z")
Do you use last Clover.ETL version and DataReader component (not **DelimiterDataReader** )?
My output for your data is:
```auto
|Record |field1 |field2 |field3 |field4 |field5 |field6 |field7 |field8 |field9 |field10 |field11 |field12 |field13 |field14 |field15 |
+-------+-----------+---------+---------+---------+---------+---------+----------+---------+----------------------------------------------------------------------------------------------------+-----------------+------------------+----------+----------+----------+----------+
|# 1 |"1473227 |31580 |16 | |9 |4 |372-372 |1962 |Addendum to "Single-transit, large-radius E-type devicest" | |Nunn, W.M., Jr. |IEEE | | |" |
|# 2 |"371 |18 |9 | |33 |1 |104-109 |1988 |Extension of the optimality of the threshold policy in heterogeneous multiserver queueing systems |10.1109/9.371 |Viniotis, I. |IEEE | | |" |
|# 3 |"417 |21 |35 | |26 |1 |8-15 |1988 |G.722: a new CCITT coding standard for digital transmission of wideband audio signals |10.1109/35.417 |Mermelstein, P. |IEEE | | |" |
```
So all is needed, is to remove quote on the beginning of 1st field and on the end of last field.
---
### Author:  [@mishrahanuyahoocoin](https://forum.cloverdx.com/u/mishrahanuyahoocoin)
#### Post date: [October 1, 2008, 10:36am UTC](https://forum.cloverdx.com/t/error-while-loading-data-in-the-database/393/16 "2008-10-01T10:36:06Z")
Yes,
You are right but i found one big difference that DeLimiter handles automatically line breaks but…in the DataReared it consider that line as incorrect data and skip that line …f i put Controlled policy…
Is it works like that or i am wrong some where…so please suggest me that how should i handle line breaks…
Thanks,
Hauman