Introduction:
Working with large sets of data can be a daunting task in development. APIs are a powerful tool for retrieving and manipulating data, but when the data is too large to handle all at once, pagination comes to the rescue. Pagination is a technique used to retrieve large sets of data from an API in smaller, more manageable chunks. This is important because large amounts of data can be difficult to work with, especially when it comes to displaying it on a website or application. Pagination allows developers to break down the data into smaller pieces, making it easier to load, display, and navigate.
We will take a step-by-step approach to understand how pagination can be used to streamline data retrieval, improve performance and enhance user experience. Here, we shall take HubSpot API as reference to explain pagination.
Pagination with Step-by-Step Guide
Step 1:
We can use either HTTP module with authorization or "Make an API Call" for pagination. Once you have added the authorization parameters and URL, you can start implementing pagination.
Fig 1: Scenario of pagination with the HubSpot API
Step 2:
Start with the Repeater module in your scenario to iterate through all the contacts or other entities. To do this, input a higher value in the "Repeats" field of the Repeater module.
Fig 2: Repeater module
Connect the Get Variable module to the Repeater and provide any suitable name, say- "AfterKey Id"
Fig 3 : Connect the Get Variable module
Step 3 :
Run the Get Variable module and map the output variable in the HTTP module as shown below:
Fig 4: Map the output variable in the HTTP module
Step 4:
Connect the Set Variable module after the HTTP module and map the after variable as shown below. Ensure that the name of the Set Variable module should be the same as the Get Variable module.
Fig 5: Map the HTTP module's output in the Set variable
Now, as the Repeater runs, the Set Variable module will help to fetch the new after id and pass it to the HTTP module using the Get Variable module.
This will allow the module to repeat the action for a specified number of times, thus iterating through all the contacts/entities by pagination.
Now the biggest question is how to break the scenario after pagination?
Step 1:
To commit the scenario after paginating all records, firstly, apply appropriate filtering after the "Set Variable" module to check availability of result data.
Fig 6: Filtering to valid results
Step 2:
Connect an HTTP module (Error causing module) with the below settings to generate an error to break the scenario.
Fig 7: Scenario breaker
Step 4:
To prevent the scenario from getting affected due to error, we need to connect an error handler module Commit to the HTTP module. This module will stop the scenario without any failure.
Fig 8: Commit - Error handler
This is a basic example of how to paginate data with the HubSpot API. Depending on your specific use case, there may be additional steps or considerations to keep in mind. Feel free to ask your queries and share your opinions about the blog in the comment section.
New to Make?
Use following link to signup and get 10,000 operations (Pro plan) free on Make for first month. Later you can choose the plan suits your requirement.