Selenium Stale elements in Python

Question

I am trying to complete a problem involved in Angela's '100 Days of Python' course.

It involves building a cookie clicker website that uses selenium to click the cookie from** http://orteil.dashnet.org/experiments/cookie/ **continually. Every 5 seconds, the program is supposed to check the upgrades (on the right of the cookie site) and click the most expensive one that I can afford to purchase it.

I have tried this challenge and my code generates the following error at random times (yoiu may need to run my code).

selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document

My code follows:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
import time

service = Service("~/Downloads/chromedriver_mac64/chromedriver")
driver = webdriver.Chrome(service=service)
driver.get("http://orteil.dashnet.org/experiments/cookie/")

def format_items():
    """Format the data in [{name, cost, element (to click)}]"""
    store_items = driver.find_elements("css selector", "#store div b")
    items_ = []
    for store_item in store_items:
        if store_item.text != "":
            description = store_item.text.split(" - ")
            name = description[0]
            cost = int(description[1].replace(",", ""))
            element = store_item

            items_.append({"name": name, "cost": cost, "element": element})
    return items_

cookie = driver.find_element("id", "cookie")


def find_most_expensive():
    affordable_upgrades = []
    for item in items:
        cost = item["cost"]
        if cost <= cookie_count:
            affordable_upgrades.append(item)

    expensive_affordable_cost = 0
    expensive_affordable_dict = {}
    for upgrade in affordable_upgrades:
        if upgrade["cost"] > expensive_affordable_cost:
            expensive_affordable_cost = upgrade["cost"]
            expensive_affordable_dict = upgrade

    return expensive_affordable_dict


base = time.time() + 5
while True:
    cookie.click()
    items = format_items()
    if time.time() >= base:
        cookie_count = int(driver.find_element("id", "money").text.replace(",", ""))
        most_expensive_item = find_most_expensive()
        if len(most_expensive_item) != 0:
            most_expensive_item["element"].click()
        base += 5

driver.quit()

I have no idea why this is because everything should work properly.

I have reviewed other similar articles on Stack overflow and none seem to solve my issue; many talk about the element being obsolete from the DOM, but I don't know how this can be because the items = format_items() happens every iteration of the While loop.

Would you mind taking a look at my code and telling me why this error is raised.

Answer 1

The is a common problem. After the first loop, the page either changes or updates invalidating the stored elements. Once you find where the invalidation is happening, you can just refetch the items inside the loop. For example,

store_items = driver.find_elements("css selector", "#store div b")
for store_item in store_items:

would become

for store_item in driver.find_elements("css selector", "#store div b"):

That way, all elements are refetched in each loop.

Answer 2

I solved my own problem using Angela's solution code. However I'm not sure why, so could you please let me know why you think the following change helped:

I changed the while loop so instead of being:

while True:
    cookie.click()
    **items = format_items()**
    if time.time() >= base:
        cookie_count = int(driver.find_element("id", "money").text.replace(",", ""))
        most_expensive_item = find_most_expensive()

It became... because I put the items = format_items() inside of the if statement.

while True:
    cookie.click()
    ####### items = get_items()
    if time.time() >= base:
        items = get_items()
        cookie_count = int(driver.find_element("id", "money").text.replace(",", ""))
        most_expensive_item = find_most_expensive()

I cannot figure out why this is working though because will the website really change in the amount of time between getting the current items (bold line) and checking if 5 seconds have passed since the last iteration, et cetera (condition in the if statement)? Anyways, thanks so much for your response