We have a flow that needs to poll an external API “list” endpoint for new entries in the list.
Over time the list will grow, so we don’t want to read the entire list every time. We’re only interested iin new items.
We can ask for entries only created after a given time (a `from_time` parameters is supported by the endpoint.)
Is there a way to set the `from_time` parameter based on the last time the REST Connector ran?
Or is there another approach?
I can see one approach to this, but I’m not sure if this is the ideal approach.
The Database connectors in Arc have a feature when querying a database that allows you to store in a local cache the last attempt that a query was made with, that will update the cache based on each successive request:
CData Arc - Database Advanced Configuration | Version 22.0.8473
What I’m about to suggest is based on this concept, and assumes that the way to progressively query your API is to use a simple timestamp argument, like a query string in the URL, or a header. Instead of putting the REST connector on a timer, precede it in your flow with a Script connector that runs on a timer, like so:
The purpose of DoNextCheck here is simple - this connector will figure out what the last query attempt was based on from some simple form of storage, like a table in a database, or in this case, a text file. This will pull that value, store the current time in it’s place, and generate a 0-byte signal message:
<arc:set attr="file.file" value="C:\\Program Files\\CData Arc 8560\\data\\LastAPICheck.txt" /><!-- read the last timestamp --><arc:call op="fileRead" item="file"> <arc:set attr="output.header:Check-Since" value="[file.file:data]" /> <arc:set attr="output.filename" value="Checking since [file.file:data]" /></arc:call><!-- set the current one --><arc:set attr="file.data" value="[_ | now]" /><arc:set attr="file.mode" value="truncate" /><arc:call op="fileWrite" item="file" /><arc:push item="output" />
This output message will contain a header that the REST connector can use when calling the next API method. If you enable the Advanced->Allow ArcScript in URL setting, for instance, you can go use this header as an argument in your query string:
Is that an option for you? The Script code here makes it clear what each query is being checked Since:
The precision of the timestamp and the storage medium are flexible, but in this example, I used a text file, so you would want to create that file and seed it with the first timestamp to start the process.
Thanks @James_B
I think that could work. the `from_date` parameters is indeed a query string parameter in this case. I’ll try it an then update the ‘best answer’ if it works.
One last thing to mention, is that the default format of that date is going to be in that yyyy-MM-ddTHH:mm:ss format, but you can use an argument to now in the code above to format your own timestamp, whatever is required:
<arc:set attr="file.data" value="[_ | now(yyyy-MM-dd-HH-mm-ss-ffff)]" />