Is there a way to store the value of all the GET requests my program does?

Question

So I have the following program:

client = Socrata("www.datos.gov.co", None)

# Example authenticated client (needed for non-public datasets):
# client = Socrata(www.datos.gov.co,
#                  MyAppToken,
#                  userame="[email protected]",
#                  password="AFakePassword")

# First results, returned as JSON from API / converted to Python list of
# dictionaries by sodapy.
  results = client.get("gt2j-8ykr", limit=800000)

# Convert to pandas DataFrame
  results_df = pd.DataFrame.from_records(results)

now, every time I run the code the variable 'results' has a new updated value as expectedly and so does the dataframe 'results_df'. What I want to do is to save all GET requests my program does (to be more precise I just want the len(results_df)). Some people have suggested me to make a list and append len(results_df). But, that obviously does not work as it just appends the current value of len(results_df), it does not save the previous value of len(results_df) so every time I run the code I end up with a list containing the current single value of len(results_df). But, what I want is the list to save the previous values of len(results_df) of previous program executions.

Im sorry if this a silly question but Im new to coding and I could not find any solution anywhere. Thanks

Answer 1

Use a persistent file storage and store the result length on disc:

def write_log_of_lengths(dataframelength):
    from datetime import datetime 
    import os.path

    log_name = "my_request_log"
    if not os.path.isfile(log_name):
        with open(log_name ,"w") as f:
            f.write( f"datetime,lenght_data\n" )   

    with open(log_name ,"a") as f:
        f.write( f"{datetime.now()},{dataframelength}\n" )

and then use

# your code

results_df = pd.DataFrame.from_records(results)
write_log_of_lengths(len(results_df))

---

Example:

write_log_of_lengths(5)
write_log_of_lengths(7)
write_log_of_lengths(22)

to get a file with

datetime,lenght_data
2020-09-07 07:37:17.889504,5
2020-09-07 07:37:17.892475,7
2020-09-07 07:37:17.895424,22