Email reader metadata configuration

CloverDX Platform
1

Hi,
I am new to Clover ETL. Have downloaded the ETL designer for fedora linux and installed it.
I need to setup a Email reader, read the attachments (CSV or XML) and write it to specified folder location.

I have setup the email properties for the reader (IMAP, server host, username and password) in the graph

Questions -
1. Help files do not mention any information on the field mappings to use. What should I be using in that field
2. How do I setup the metadata information for the edge output connection to view the data.

Also, I am seeing the following error in the log when Email reader was setup with JDK 1.6
Exception in thread “Thread-7” java.lang.NoClassDefFoundError: javax/mail/NoSuchProviderException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.jetel.component.ComponentFactory.getComponentClass(ComponentFactory.java:104)

Please help me out.

Thank you
Ram

2

Hello Ram,
mapping attribute defines a mapping between standard and user defined mail parts and clover fields.
Please see the attached graph. It illustrates 3 ways of getting the attachment files (each phase corresponds to one way):

  • Connect the attachment (1st) port but you don’t need to map anything to this output → all attachments are saved in tempURL directory (data-tmp of current project by default) with unique names

  • Map FileName and AttachmentRaw to attachment port and configure Writer to write the AttachmentRaw field to the file whose name depends on FileName field (see Partitioning Output into Different Output Files) → all attachments are saved in tempURL directory (data-tmp of current project by default) with unique names and in the files with the original name from the e-mail

  • Map FileName and AttachmentFile to attachment port and rename the files with SystemExecute - → attachments are stored only under the original names. Note that because of a minor bug (https://bug.javlin.eu/browse/CLD-2840), you need to add the user field AttachmentFile and map it into your clover AttachmentFile file.

The error suggests that you miss mail.jar in your CloverETL Designer. Please check the folder <path_to_designer_home_directory>/plugins/com.cloveretl.gui.commercial_3.X.X/lib/plugins/org.jetel.component.commercial/lib if it contains mail-1.4.2.jar file.

3

Thank you Agata Vackova.

Your example EmailReader graph helped me a lot.

I had to spend couple of hours before I could get it working.
Issues encountered -
1. The charset for the attachment. I left the default charset instead of UTF-16. Realized after a long time.
2. My email attachment was a single column csv file. The default delimiter ‘\n’ was not working. Always threw error that the SimpleDataParser could not read raw data. Had to change it to EOF delimiter.

Regarding the mail exception -
mail-1.4.2.jar is available in the <path_to_designer_home_directory>/plugins/com.cloveretl.gui.commercial_3.1.0/lib/plugins/org.jetel.component.commercial/lib/
I believe it has something to do with my jdk and the library path.
I am using Fedora 14 and it has inbuilt open-jdk jre 1.6 installed. Not sure if I have to configure anything different for that.
I also tried using JDK as

./clover-designer -vm /opt/jdk1.6.0_26/bin/java

but same issue.

4

Hello Ram,
I do not really see the relationship between the delimiter and reading the attachments with EmailReader. But if you want to treat the EOF as delimiter, you can set the attribute aeofAsDelimiter to true value in the last field of your metadata (see Details Pane for more information).
I’m also a bit confused with the error you get: Does the graph run successfully, but the log contains the error message? Could you send the full error stack trace?

5

Hi Agata,

I am sorry. Will be more specific.
The issue with email reader problem was the use of default charset instead of UTF-16. After I read the email attachment, I am feeding the attachment file (which is a single column csv) as input to a Universal Data reader.
File URL - port:$0.AttachmentRaw:discrete

From the Universal Data reader, I am reading the csv and writing it to a DBOutputTable. When I open the csv file using vi command, it opens up good with the data. The edge metadata between the universal data reader and dboutput table has one field with delimiter as ‘\r\n’. When I view the data in grid format or when I run the graph, I see the below parser error.

Node DATA_READER0 finished with status: ERROR caused by: Parsing error: Unexpected record delimiter, probably record has too few fields. in field # 1 of record # 1, value: '001070457
’
Attached the sample csv file used.

To fix it, I had to remove the delimiter and use EOF delim as true. This way I could read the fields in the file and insert into the db table. The field contained special characters at the end but I was able to trim them and insert into the column

Regarding the mail.jar error for class not found -
My classpath environment was incorrect. I fixed it and started to work. Thank you

6

Hello,
all you need is to set proper charset in DataReader - UTF-16LE. Please see the attached file: both DataReaders (when reading the data directly from file, as well as parsing it as attachment) have the exactly same configuration.

7

Thank you Agata