Loader reading from memory?

CloverDX Platform
1

I want to use “oracle data writer” node of clover that should read from java hash map objects in memory.

By default, It reads from input port or reads it from input file.

Kindly reply if you have any clue asap.

- Priyadarshini

2

You can use Simple lookup table for storing your data and then use LookupTableWriterWriter component.

3

I got LookupTableReaderWriter but not LookupTableWriterWriter. Can u give more details.

4

I’m sorry - I’ve thought LookupTableReaderWriter: If you are able to store the data in Map<HashKey, DataRecord>, then you can read them by LookupTableReaderWriter. If not you have to save data in file and then use OracleDataWriter component.

5

I am storing data in Map in Memory only, now giving name of LookupTable in property of LookupTableReaderWriter should not be sufficient ??

6

LookuTableReaderWriter can read data only from lookup table that implements org.jetel.data.lookup.LookupTable interface.

7

Also, I m finding that DataRecord object (that is value for MAP implementing LookupTable interface) should also belong to DataRecord class of clover. So, I need to extend my DataRecord class to same of clover. Again, I need to handle metadata part of it. It seems going in a very complex direction…is there any simple way.

8

The simplest way is to save data in flat file and them use OracleDataWriter. Or you can write a Reader, that will send data to OracleDataWriter.

9

Thats the problem…I do not want to use flat file as its extra overhead…

Again, If I use LookupReaderWriter, then the OracleDataWriter need to added to LookupReaderWriter…to finally load the thing to DB.

Is it a way, OracleDataWriter can directly read from memory and load to DB(using sqlldr)…pipe can help or not ?

10

All clover writers can read data from input port (edge). If the data are saved on disk or not depends on edge type and amount of data. See useFileForExchange attribute of OracleDataWriter component

11

On Unix, UseFileToExchange is set to false by default. OracleDataWriter reads from edge …write it to pipe and then load to sqlldr.

I was thinking if there is any way, I can write to pipe myself taking memory data…that will be consumed by sqlldr …

12

what I understood, loader can read from named pipe or can read from stdin as well. still exploring, how exactly can achieve it.

13

You can easily write your Reader, which will send data to Oracle Data Writer. Something like:

import java.util.Iterator;
import java.util.Map;
import java.util.Map.Entry;

import org.jetel.data.DataRecord;
import org.jetel.graph.Node;
import org.jetel.graph.Result;
import org.jetel.util.SynchronizeUtils;


/**
 * @author avackova
 *
 */
public class MapReader extends Node {


	public final static String COMPONENT_TYPE = "MAP_READER";

	private Map map;

	public MapReader(String id, Map data) {
		super(id);
		this.map = data;
	}

	/* (non-Javadoc)
	 * @see org.jetel.graph.Node#execute()
	 */
	@Override
	public Result execute() throws Exception {
		DataRecord record = new DataRecord(getOutputPort(OUTPUT_PORT).getMetadata());
		record.init();
		Iterator<Entry> data = map.entrySet().iterator();
		while (runIt && data.hasNext()) {
			getNextRecord(record, data.next());
			writeRecord(OUTPUT_PORT, record);
			SynchronizeUtils.cloverYield();
		}
		broadcastEOF();
        return runIt ? Result.FINISHED_OK : Result.ABORTED;
	}

	private void getNextRecord(DataRecord record, Entry data) {
		record.getField(0).setValue(data.getKey());
		record.getField(1).setValue(data.getValue());
	}

	/* (non-Javadoc)
	 * @see org.jetel.graph.Node#getType()
	 */
	@Override
	public String getType() {
		return COMPONENT_TYPE;
	}

}

For full instruction how to create the component see: Step by Step Component building

14

Thanks for reply…let me try this.