Overview
Sometimes APIs return data to the connecting client in a page based format. This can happen when the requested dataset to be returned is too large for the API to return in one response, or it could be the way that the API administrator has built the API to return responses.
For example, if an API request is sent to GET the total number of items in an order, and the total number of items is 500, the API might return the results in a series of 10 pages where there are 50 items per page. The client requesting the information would then have to issue 10 separate API calls to get each page. This behavior is often coined “pagination”.
The REST Connector and the Flow
Pagination isn’t something that is natively supported within the REST connector however, it is typically possible for Arc users to develop a solution to accomplish this by use of some custom script within a script connector and the utilization of custom headers to hold the page numbers (or page URLs) and build the URL within the REST connector dynamically, based off of those headers.
At a high, flow-based level, a pagination flow in Arc might look something like this:
Where each connector is going to have a specific role, in order to accomplish this, specifically the script connectors that will need to contain the logic. A description of what each connector should do is below:
Script_FirstPage - This is a script connector that would basically be the connector that starts off at page 1. It would be responsible for adding a header onto the message that is passed to the REST connector that can be used dynamically in the URL to call the API for the first page.
REST_Pagination - The REST connector that will issue all of the requests to the API endpoint.
Copy_Output - This connector will take a copy of the output JSON data from the API response and send it down the rest of the flow, and another copy will flow into the Script_NextPage connector, described below.
Script_NextPage - This connector is where a majority of the logic will need to be written. This connector will need to be responsible for parsing out the page values from the JSON response - you can do this via the jsonDOMGet operation (https://cdn.arcesb.com/help/AZG/mft/op_jsonDOMGet.html). You will want to parse out the page data, whether it is a number or a part of the URL that needs to be used, or the whole URL and add it as a header on the message and then push that message as output.
The header here should be named the same as the header that is created by the Script_FirstPage connector and referenced in the URL field of the REST connector. This connector will also need logic written within it so that it knows when the last page of the API is.
This connector will then pass that message and header with the page/URL info to the REST connector where the REST connector will reference the header on the message within the URL field and issue a request to the API to get the next page available.
The way you go about creating the parsing logic is going to be determined based on what page data comes back in each response from the API - how does it tell you what the next page is? Is it a number that is used as a query string parameter? A entirely new URL?
For example, if the API response contains a full URL for the next page, you can set the custom header to have a value of "nextPageUrl" and then just reference that header within the URL field of the REST connector, like this:
NOTE: In order to evaluate arcscript/headers within the URL of the REST connector, "Allow ArcScript in URL" must be enabled within the Advanced tab:
The “boilerplate” arcflow that contains these connectors is attached below. Please note however due to the variability of the necessary scripts based on any given API response structure and your specific use case, the Script Connectors within the flow are empty and do not contain any code. You will need write the scripts in accordance with your requirements. Please feel free to take this flow and adapt it to your use case.
