I am writing an application to fetch all the data of an app from Podio. Code is making this request "/item/app/{appId}?offset={offset_val}&limit={limit_val}&sort_by=created_on" to get data from the app using these parameters "offset_val" and "limit_val" I am getting data for 500 (max supported by Podio) items in response. All my podio apps are items more than 20K.
Code is running and fetching data, for one single request it is getting 500 items and after getting records code filtering it, and adding in DB this operation takes 2-3 seconds. A case is failing when an item is deleted from the app and podio shifts next item position upwards, due to this shift, I am missing some items.
Scenario- Step 1: In any request I got data suppose the request offset is 1001 and the limit is 1500. Step 2: I got this data and my code works on filtering and DB operations. Step 3: In this interval period (filtering and DB operations, or can say before the next request to Podio), some user working on the same app deleted one item from the app and suppose it was the 1500th record in my request. (The hook for this item is received and code is handling this item) Step 4: Now Item at position 1500 is deleted from the podio app, now Podio will arrange records and item and 1501 position will take place of 1500 (the deleted item position). Step 5: Now after DB operation I will again trigger a request with an offset of 1501 and a limit is 2000. Now for this request, I will miss the data that was at position 1501 before this request (In step 4, the 1501st record is shifted to 1500). Step 6: If some items are deleted my code can get hook and delete those items.
But I am missing the items that shifted the position and after shifting went beyond offset and limit range. Tested this sifting behaviour by creating a test app.
Could you please help to fix this corner case?
Thanks a lot in advance.
I created a test app and confirms this behaviour.
The solution to address this issue depends on the frequency of item deletions from Podio.
If deletions are frequent:
In the event of an item deletion between these two points, the offset should be adjusted during step 2 to allow room for the potentially duplicated item. For instance, you may adjust the offset to cover the range from 500 to 1000. This adjustment ensures that you can properly check the 501st item in our database. If it already exists, you can proceed by skipping it. Otherwise, you'll need to adjust your approach accordingly.
Additionally, I recommend using offsets starting from 0 to 400, then 401 to 800, and so forth. This approach allows for adequate space and prevents potential issues that may arise from maximizing API limits.