# Persisting a value from one graph execution to the next

**URL:** <https://forum.cloverdx.com/t/persisting-a-value-from-one-graph-execution-to-the-next/923>\
**Category:** CloverDX Platform\
**Created:** [September 2, 2011, 12:00am UTC](https://forum.cloverdx.com/t/persisting-a-value-from-one-graph-execution-to-the-next/923 "2011-09-02T00:00:00Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![rstark](https://avatars.discourse-cdn.com/v4/letter/r/b19c9b/32.png) [@rstark](https://forum.cloverdx.com/u/rstark)\
**Post date:** [September 2, 2011, 12:00am UTC](https://forum.cloverdx.com/t/persisting-a-value-from-one-graph-execution-to-the-next/923/1 "2011-09-02T00:00:00Z")

</div>

I have a requirement to capture the last run key from a graph execution. Store it and then re-inject this in the reader for the next run (in order to incrementally pull only new records). The challenge is the source is and HTTP connection and I do not have access to write to a RDBMS. Does Clover have a mechanism that will allow this( Without writing to file and joining this back in)?

---

<div class="post-metadata">

**Author:** ![avackova](https://avatars.discourse-cdn.com/v4/letter/a/65b543/32.png) [@avackova](https://forum.cloverdx.com/u/avackova)\
**Post date:** [September 5, 2011, 12:08pm UTC](https://forum.cloverdx.com/t/persisting-a-value-from-one-graph-execution-to-the-next/923/2 "2011-09-05T12:08:28Z")

</div>

Hello,  
when executing the graph through the HTTP Api (see [Operation graph\_run](http://server-demo-ec2.cloveretl.com/clover/docs/simple-http-api.html#operation-graph%5Frun)), eg. from HttpConnector component you get a _runId_ as output, so you can use it for configuring the next Reader.

---

<div class="post-metadata">

**Author:** ![rstark](https://avatars.discourse-cdn.com/v4/letter/r/b19c9b/32.png) [@rstark](https://forum.cloverdx.com/u/rstark)\
**Post date:** [September 12, 2011, 1:49pm UTC](https://forum.cloverdx.com/t/persisting-a-value-from-one-graph-execution-to-the-next/923/3 "2011-09-12T13:49:19Z")

</div>

Sorry for the confusion, but I am not executing a graph through the HTTP API I am utilizing a HTTP connector inside of the graph to source data from a REST API (e.g. Twitter search API ). To prevent me from pulling the same data again and again. Twitter provides a id which I can concatenate with the url to only pull records since the last time I ran. Currently I am storing this value in a flat file and joining this back in but it is sloppy.

---

<div class="post-metadata">

**Author:** ![avackova](https://avatars.discourse-cdn.com/v4/letter/a/65b543/32.png) [@avackova](https://forum.cloverdx.com/u/avackova)\
**Post date:** [September 13, 2011, 9:29am UTC](https://forum.cloverdx.com/t/persisting-a-value-from-one-graph-execution-to-the-next/923/4 "2011-09-13T09:29:30Z")

</div>

Hello,  
this is proper approach. Our incremental reading uses the external file for storing the incremental key also.

---

<div class="post-metadata">

**Author:** ![julrych](https://avatars.discourse-cdn.com/v4/letter/j/7ba0ec/32.png) [@julrych](https://forum.cloverdx.com/u/julrych)\
**Post date:** [September 13, 2011, 5:59pm UTC](https://forum.cloverdx.com/t/persisting-a-value-from-one-graph-execution-to-the-next/923/5 "2011-09-13T17:59:51Z")

</div>

You can also use parameters for this purpose:  
1. create a separate parameter file twitter.prm holding parameter TWITTER\_ID (do that in gui to have the syntax correct).  
2. in the graph, use syntax ${TWITTER\_ID} to access the actual parameter value.  
3. at the end of the graph run, overwrite twitter.prm file with updated value of TWITTER\_ID.

Note: When overwriting the twitter.prm file, you need to make sure that the resulting file has correct syntax.

I would also recommend to make the implementation fail-proof, i.e, handle case when the twitter.prm file gets corrupted or empty.

---

<div class="post-metadata">

**Author:** ![rstark](https://avatars.discourse-cdn.com/v4/letter/r/b19c9b/32.png) [@rstark](https://forum.cloverdx.com/u/rstark)\
**Post date:** [September 13, 2011, 7:00pm UTC](https://forum.cloverdx.com/t/persisting-a-value-from-one-graph-execution-to-the-next/923/6 "2011-09-13T19:00:19Z")

</div>

Jan,

I originally started with this approach; however I had issues with the parameter file being overwritten even if no records were processed. Without writing my own null handling logic is their a way to avoid this (eg using partitions on the writer)?

---

<div class="post-metadata">

**Author:** ![julrych](https://avatars.discourse-cdn.com/v4/letter/j/7ba0ec/32.png) [@julrych](https://forum.cloverdx.com/u/julrych)\
**Post date:** [September 14, 2011, 12:08pm UTC](https://forum.cloverdx.com/t/persisting-a-value-from-one-graph-execution-to-the-next/923/7 "2011-09-14T12:08:26Z")

</div>

Partitioning using lookup table which would put null values into different file should work. Sample implementation attached. The ${LOOKUP\_DIR}/OutputPartitioningTest.dat file contents should be

```auto

|_null

```

- [OutputPartitioningTest.grf](https://cdn.cloverdx.com/forum/11799%5F6716705c87825f83484668e88ab4a1ea/OutputPartitioningTest.grf)
