Python Selenium - StaleElementReferenceException: stale element not found

Question

I am trying to click on the next button at the bottom of the page to access the following page as in the image below:

However, when I reach page 3 I receive the following error:

selenium. common. exceptions. StaleElementReferenceException: Message: stale element reference: stale element not found

from the code line

next_button = wait.until(EC.element_to_be_clickable(span_element[1])).click()

Overall, the code I am using for this looks like

region_content = driver.find_element(By.CLASS_NAME, "pager__item--next")
span_element = region_content.find_elements(By.CSS_SELECTOR, "a > span")  
next_button = wait.until(EC.element_to_be_clickable(span_element[1])).click()
       

and the HTML code where the button can be found is the following:

<li class="pager__item pager__item--next">
          <a href="?field_headquarters_of_company_target_id=All&amp;ajax_page_state%5Btheme%5D=bootstrap&amp;ajax_page_state%5Blibraries%5D=addtoany/addtoany%2Casset_injector/css/css_injector%2Cbootstrap/popover%2Cbootstrap/tooltip%2Ccore/drupal.autocomplete%2Ccore/drupal.autocomplete%2Ccore/drupal.autocomplete%2Ccore/drupal.autocomplete%2Ccore/drupal.autocomplete%2Ccore/drupal.autocomplete%2Ccore/drupal.autocomplete%2Ccore/drupal.autocomplete%2Ccore/drupal.autocomplete%2Ccore/html5shiv%2Cextlink/drupal.extlink%2Cgoogle_analytics/google_analytics%2Csuperfish/superfish%2Csuperfish/superfish_hoverintent%2Csuperfish/superfish_smallscreen%2Csuperfish/superfish_style_white%2Csuperfish/superfish_supersubs%2Csuperfish/superfish_supposition%2Csuperfish/superfish_touchscreen%2Csystem/base%2Cviews/views.ajax%2Cviews/views.ajax%2Cviews/views.ajax%2Cviews/views.ajax%2Cviews/views.ajax%2Cviews/views.ajax%2Cviews/views.ajax%2Cviews/views.ajax%2Cviews/views.ajax%2Cviews/views.module%2Cviews/views.module%2Cviews/views.module%2Cviews/views.module%2Cviews/views.module%2Cviews/views.module%2Cviews/views.module%2Cviews/views.module%2Cviews/views.module%2Cwebform_bootstrap/webform_bootstrap&amp;ajax_page_state%5Btheme_token%5D=&amp;_wrapper_format=drupal_ajax&amp;title=&amp;field_company_category_primary_target_id&amp;field_market_cap_jul302023_value=&amp;field_company_website_uri=&amp;page=3" title="Go to next page" rel="next">
            <span class="visually-hidden">Next page</span>
            <span aria-hidden="true">Next ›</span>
          </a>
        </li>

I have tried with a try/except statement to refresh the page in case of stale element but still does not work. What is a bit puzzling to me is that to go from page 1 to 2 and from 2 to 3 there is no error.

Would anyone be willing to help out?
Thank you

Answer 1

What browser you use? From my experience, it's best to click elements by xpath.

Try something like:

while True:
    try:
        driver.find_element('xpath', 'some_xpath').click()
        break
    except:
        pass