Clear Persistent Lookup

Support/help with CloverETL (4.9) and CloverDX (5.0 or newer) implementation problems

pfield
Posts: 7
Joined: Thu Mar 19, 2020 3:46 pm

Clear Persistent Lookup

Postby pfield » Sat Mar 21, 2020 3:34 pm

HI,

What is the correct way to clear and repopulate a persistent lookup?

I am using a shared persistent lookup as a method of locally storing a common set of mappings that need to be referenced by multiple graphs. However, when i get a new set of mappings that i need to upload it's very difficult to clear the existing values in the lookup and repopulate; i tried using the "Clear lookup table after finish" option in the LookupTableReaderWriter Component but this does nothing. The only method I can see is to physically delete the .db file which is stored - this often isn't possible as it seems they're held by some resources for a period of time so have to wait 5/10 minutes since the lookup was last used by a graph.

Am i using the lookup in the correct way? Is there a better way of approaching keeping local (not in a db) values for multiple graphs to reference?

Thanks,
Paul

dpavlis
Posts: 186
Joined: Sat Mar 10, 2007 8:12 pm

Re: Clear Persistent Lookup

Postby dpavlis » Thu Mar 26, 2020 6:18 pm

Hi Paul,

There is (unfortunately) no clear/direct way to remove all entries from Persistent Lookup Table. Your approach is what we would recommend - see this picture & notice the different phases of components:
image.png
image.png (171.34 KiB) Viewed 768 times


However there might be a different way to accomplish what you are trying to do using other means of CloverDX. Perhaps if you explain a bit more what kind of mapping logic you keep in the lookup and how that needs to be shared among multiple graphs, we could suggest a solution.

PS: we will create a change request for our R&D to potentially look at purging Persistent Lookup.
David Pavlis
CloverCARE Support
CloverDX | Rapid Data Integration

Visit us online at http://www.cloverdx.com

pfield
Posts: 7
Joined: Thu Mar 19, 2020 3:46 pm

Re: Clear Persistent Lookup

Postby pfield » Thu Mar 26, 2020 8:06 pm

Thanks David!

This workaround is acceptable for now but yes it would be good if you had the option to purge the lookup.

Context is I have number of graphs which are run in a sequence and many of them need this information (it mainly contains details on a data model being used). I'm not using server so jobflows aren't an option, this means i tend to kick off lots of graphs manually one after another and don't want to have to repopulate a lookup each time.

Interesting you were able to delete the .db file immediately though - i'm not sure what's holding onto my files for a period of time which stops me from deleting straight away. Not a big problem though.

Paul

bartonv
Posts: 143
Joined: Wed May 03, 2017 12:10 pm

Re: Clear Persistent Lookup

Postby bartonv » Tue May 19, 2020 3:03 pm

Hi Paul,
just a thought on the issue with being unable to delete the .db file, could you confirm that you intend to delete the file in a later phase than the lookup table is being written into (as the screenshot provided by David indicates)? Similarly, can you confirm that the file is not "in-use" by any of your other graphs while you are trying to delete it?
Kind regards,
---
Vladimir Barton
CloverCARE Support
CloverDX

Visit us online at http://www.cloverdx.com


cron