How can I scrap a text included in a "_ngcontent" frontend?
Here is the code:
from bs4 import BeautifulSoup as bs
import requests
url = 'https://formosodoaraguaia.megasofttransparencia.com.br/receitas-e-despesas/empenho?faseDoEmpenho=4&etapaDaDespesa=4&dataInicial=01%2F01%2F2019'
page_to_scrap = requests.get(url)
soup = bs (page_to_scrap.text, 'html.parser')
data = soup.findAll("label _ngcontent-lpf-c7", attrs={"class":"valor"})
for i in data:
print (data.text)
The loop returns nothing, as if there is no content in the selector I choosed.
Does it have something to do with the content being in a pop-up page? How can I scrap something like this?
Thank you all!
Update -------------------------------------------------------------
The "_ngcontent" id changes when I reload the page. This is how it looks right now:
If you open the page source you will see only this:
No divs, no spans, no classes. That's all you get with
requests. That's because all the content is loaded dynamically via JavaScript modules afterwards.To get the dynamic content, you'll need a library that uses headless browser (a browser that has no GUI). The most popular ones for this purpose, which also support Python, are Selenium, Pyppeteer and Playwright.