How does Facebook Graph API Pagination works and how to iterate facebook user feed with it?

Question

I have a facebook Graph API call to get a facebook users feed:

dynamic myFeed = await fb.GetTaskAsync(
                    ("me/feed?fields=id,from {{id, name, picture{{url}} }},story,picture,link,name,description," +
                    "message,type,created_time,likes,comments")
                    .GraphAPICall(appsecret_proof));

The above returns a number of the latest user posts in a while say 21 or maybe 22 posts but not the complete list of user posts. I searched for a way to iterate through a users feed using facebook pagination and I ended up finding this solution which works with facebook Offset pagination.

dynamic myFeed = await fb.GetTaskAsync(
                    ("me/feed?fields=id,from {{id, name, picture{{url}} }},story,picture,link,name,description," +
                    "message,type,created_time,likes,comments")
                    .GraphAPICall(appsecret_proof), new {limit = "1000", offset = "21" });

This has taken me a step nearer to what I want to achieve but I think this is not the ideal way to do it and also it does not return all of users posts. Is there any workaround? Please help.

P.S: I am using Facebook C# SDK.

UPDATE1: As per Jeremy's answer. It seems that the facebook cursor pagination is the only right option for my requirements. I would like to know if C# facebook sdk provides any feature to iterate over the Next Edges so I can get all feed posts in one call, Is there any possible solution to this? P.S: I've been through facebook API docs many times and I know what exactly the Nodes, Edges and Fields are, the only unfortunate thing is that facebook doesn't support C# SDK as yet and I was unable to find a proper documentation on Facebook C# SDK too.

Answer 1

Finally after doing some researches and reading some blogs I found out that there is no direct API CAlls from facebook to fetch all user feeder posts in once. To achieve that functionality either one has to go for infinite scrolling as suggested by Jeremy Thomson or to iterate through different facebook data pages regardless of which facebook pagination type is supported by the edge. As far as I want a process without user interference/actions I would definitely go the second option which is iterating through facebook data pages with a while loop. To do That we first need our two most important parameters (facebook access_token + (facebook appsecret_proof) as described below:

var appsecret_proof = access_token.GenerateAppSecretProof();
var fb = new FacebookClient(access_token);

Point to be remembered: facebook access_token is generated by HttpContext class.

The facebook API call will get the users first 25 feeder post as below:

dynamic myFeed = await fb.GetTaskAsync(
                    ("me/feed?fields=id,from {{id, name, picture{{url}} }},story,picture,link,name,description," +
                    "message,type,created_time,likes,comments")
                    .GraphAPICall(appsecret_proof));

The API Call above return results in a Json array and that should be hydrated through the Model View properties as shown here:

var postList = new List<FacebookPostViewModel>();
    foreach (dynamic post in myFeed.data)
       {
         postList.Add(DynamicExtension.ToStatic<FacebookPostViewModel>(post));
       }

Until here everything was clear before, the most important part which is surely fetching all facebook user post is now in action. For that we need to set a string NextPageUri to empty as here:

string NextPageURI = string.Empty;

The final part of all is to check if there is another page for the data, If yes should iterate and add the data to the View Model until there is no page lift as shown here:

while (myFeed.paging != null && myFeed.paging.next != null)
                {
                    NextPageURI = myFeed.paging.next;
                    var nextURL = GetNextPageQuery(NextPageURI, access_token);
                    dynamic nextPagedResult = await fb.GetTaskAsync(nextURL.GraphAPICall(appsecret_proof));
                    foreach (dynamic post in nextPagedResult.data)
                    {
                        postList.Add(DynamicExtension.ToStatic<FacebookPostViewModel>(post));
                    }
                }

This helped me to get rid of the problem faced. But yet I have another task in hand to work on. It is the speed of fetching the posts that if posts are morethan 30k would take 10 minutes which is not ideal at least for me.

Answer 2

First up a bit of terminology:

nodes - basically "things" such as a User, a Photo, a Page, a Comment
edges - the connections between "things", such as a Page's Photos, or a Photo's Comments
fields - info about those "things", such as a person's birthday, or the name of a Page

When you make an API request to a node or edge, you usually don't receive all of the results of that request in a single response. This is because some responses could contain thousands of objects so most responses are paginated by default.

To get all posts by a user you have 3 options:

---

Cursor-based Pagination

Cursor-based pagination is the most efficient method of paging and should always be used where possible. A cursor refers to a random string of characters which marks a specific item in a list of data. Unless this item is deleted, the cursor will always point to the same part of the list, but will be invalidated if an item is removed. Therefore, your app shouldn't store any older cursors or assume that they will still be valid.

When reading an edge that supports cursor pagination, you will see the following JSON response:

{
  "data": [
     ... Endpoint data is here
  ],
  "paging": {
    "cursors": {
      "after": "MTAxNTExOTQ1MjAwNzI5NDE=",
      "before": "NDMyNzQyODI3OTQw"
    },
    "previous": "https://graph.facebook.com/me/albums?limit=25&before=NDMyNzQyODI3OTQw"
    "next": "https://graph.facebook.com/me/albums?limit=25&after=MTAxNTExOTQ1MjAwNzI5NDE="
  }
}

To get all posts by a user you keep surfing the "next" edges (upserting new items). This is how I do it when I dump entire groups into RDBMS's for statistical analysis. Often you will see edges with nodes you have already encountered, that's why I mention to UPSERT (update if it exists otherwise insert).

Time-based Pagination

Time pagination is used to navigate through results data using Unix timestamps which point to specific times in a list of data.

When using an endpoint that uses time-based pagination, you will see the following JSON response:

{
  "data": [
     ... Endpoint data is here
  ],
  "paging": {
    "previous": "https://graph.facebook.com/me/feed?limit=25&since=1364849754",
    "next": "https://graph.facebook.com/me/feed?limit=25&until=1364587774"
  }
}

To get all a users posts you keep iterating back in time. This method will get you the posts in order, although may want them returned in an order by FaceBooks edge algorithm.

Offset-based Pagination

Offset pagination can be used when you do not care about chronology (ie ordered by date) and just want a specific number of objects returned. This should only be used if the edge does not support cursor or time-based pagination.

So what you have found with Offset is the closest you will get to the stock-standard pagination you desire. However:

Offset based pagination is not supported for all API calls. To get consistent results, we recommend you to paginate using the previous/next links we return in the response.

You can read all this in the FB API docs.

https://developers.facebook.com/docs/graph-api/overview/
https://developers.facebook.com/docs/graph-api/using-graph-api/