Skip to main content

Pagination within the REST Connector

  • March 22, 2023
  • 1 reply
  • 544 views
C
Forum|alt.badge.img

Overview

Sometimes APIs return data to the connecting client in a page based format. This can happen when the requested dataset to be returned is too large for the API to return in one response, or it could be the way that the API administrator has built the API to return responses.

For example, if an API request is sent to GET the total number of items in an order, and the total number of items is 500, the API might return the results in a series of 10 pages where there are 50 items per page. The client requesting the information would then have to issue 10 separate API calls to get each page. This behavior is often coined “pagination”.

The REST Connector and the Flow

Pagination isn’t something that is natively supported within the REST connector however, it is typically possible for Arc users to develop a solution to accomplish this by use of some custom script within a script connector and the utilization of custom headers to hold the page numbers (or page URLs) and build the URL within the REST connector dynamically, based off of those headers.

At a high, flow-based level, a pagination flow in Arc might look something like this:

 

Where each connector is going to have a specific role, in order to accomplish this, specifically the script connectors that will need to contain the logic. A description of what each connector should do is below:

Script_FirstPage - This is a script connector that would basically be the connector that starts off at page 1. It would be responsible for adding a header onto the message that is passed to the REST connector that can be used dynamically in the URL to call the API for the first page.

REST_Pagination - The REST connector that will issue all of the requests to the API endpoint.

Copy_Output - This connector will take a copy of the output JSON data from the API response and send it down the rest of the flow, and another copy will flow into the Script_NextPage connector, described below.

Script_NextPage - This connector is where a majority of the logic will need to be written. This connector will need to be responsible for parsing out the page values from the JSON response - you can do this via the jsonDOMGet operation (https://cdn.arcesb.com/help/AZG/mft/op_jsonDOMGet.html). You will want to parse out the page data, whether it is a number or a part of the URL that needs to be used, or the whole URL and add it as a header on the message and then push that message as output.

The header here should be named the same as the header that is created by the Script_FirstPage connector and referenced in the URL field of the REST connector. This connector will also need logic written within it so that it knows when the last page of the API is.

This connector will then pass that message and header with the page/URL info to the REST connector where the REST connector will reference the header on the message within the URL field and issue a request to the API to get the next page available.

The way you go about creating the parsing logic is going to be determined based on what page data comes back in each response from the API - how does it tell you what the next page is? Is it a number that is used as a query string parameter? A entirely new URL?

For example, if the API response contains a full URL for the next page, you can set the custom header to have a value of "nextPageUrl" and then just reference that header within the URL field of the REST connector, like this:

NOTE: In order to evaluate arcscript/headers within the URL of the REST connector, "Allow ArcScript in URL" must be enabled within the Advanced tab:

The “boilerplate” arcflow that contains these connectors is attached below. Please note however due to the variability of the necessary scripts based on any given API response structure and your specific use case, the Script Connectors within the flow are empty and do not contain any code. You will need write the scripts in accordance with your requirements. Please feel free to take this flow and adapt it to your use case.

 

    1 reply

    James B
    Forum|alt.badge.img
    • James B
    • Employee
    • November 10, 2025

    I have a sample flow that makes use of a Webhook connector in the same workspace to mock up a custom endpoint that responds to different hardcoded page requests, so there is a simple result set to work with.

    Image_2025-11-07_14-55-56.png

    In the attached example (please rename PaginationSample.zip to a .arcflow file to import this into one of your workspaces), you'll notice that there is a floating Webhook (Letters) - this is meant to be a dummy API endpoint that looks for a Page querystring to implement 6 paged requests. In the response event of the Webhook, you'll see that the Webhook responds with a simple json response of 5 letters of the alphabet (5 pages of 5 and one page of 1) .

    The flow begins with a simple Script titled StartPage1 that simply creates a dummy message that will be trigger the first page in the REST connector - you can click on the Receive Messages button to trigger the start message that begins the flow. Consider this connector to be a placeholder for any steps in your flow that would precede the call to the REST endpoint that needs to be paged. 

    The PageSample REST connector, then, is set to call out to this API endpoint with a dynamic query string at the end:

    Image_2025-11-07_15-02-17.png


    The page query string is something that is added on subsequent requests to the API, but please note that to dynamically, evaluate an expression here, the Advanced setting Allow ArcScript in URL needs to be set:

    Image_2025-11-07_15-03-41.png

    If your page request involved a header, the Allow ArcScript in headers would be used instead. Finally, the REST connector has two headers that it looks for in the response to add to the returned message for the paging logic, in the Advanced->Response Headers property:

    Image_2025-11-07_15-05-21.png

    The custom paging API that is hardcoded into the Webhook uses the NextPage header to signal the next page in the API with responses, and the EndOfList header to signal the end of the pages. 

    NOTE: The POST keyword is used in this example, but that is because the Webhook that is used to return custom responses expects a POST - this concept will also work for API endpoints that need a GET.

    The rest of the flow is fairly simple - the Copy connector outputs 2 copies of the connector to two paths - one is a simple Script connector that represents the continuation of processing in your flow (right now, an empty Script):

    Image_2025-11-07_15-07-21.png

    And the other is a Script that the response passes through before going back to the Script connector. This is also a very light script:

    Image_2025-11-07_15-08-17.png

    The script checks to make sure that the header marking the end of the paging is not present, and if so, proceeds by building the querystring that will be appended to the next request.

    <!-- proceed only if endoflist header not there -->

    <arc:if exp="![_input.Header:endoflist | def | equals('true')]">
      <arc:set attr="output.header:pagequery" value="?page=[_input.header:nextpage]" />
      <arc:set attr="output.filepath" value="[_input.filepath]" />
      <arc:set attr="output.filename" value="Page[_input.header:nextpage].json" />
      <arc:push item="output" />
    </arc:if>


    Please give this a try and let me know if it helps as a more concrete example of the paging. Because paging is handled differently in every API, this may need to be modified to work against your example, but it should illustrate the concept cleanly. 

    NOTE: You'll find that the API endpoint targeted by the REST connector is  https://www.cdatacloud.net/arc1/connector/PaginationSample/Letters/webhook.rsb on our test Arc Cloud instance. I'd suggest that you update the REST connector to hit the Webhook Endpoint URL that you import the Webhook to, so you can control the API response if you wish to. This URL can be determined in the Settings tab of the Letters webhook connector after you import it. 

      State of AI Data Connectivity Report: 2026 Outlook
      Terms and ConditionsCookie settingsAccessibility statement