BeautifulSoup gives an empty list

Question

I am trying to scrape website with Beautiful Soup. After printing the container, It gives me an empty list. How can I fix this?

import requests
from bs4 import BeautifulSoup
import lxml

URL = 'https://www.monster.com/jobs/search/?q=Software-Developer&where=Australia'
page = requests.get(URL)

soup = BeautifulSoup(page.content, 'lxml')
container = soup.find_all('div', class_="results-list")
print(container)

Answer 1

The data is loaded from external URL via Ajax call. You can use requests module to simulate it and load the data:

import json
import requests
from textwrap import wrap
from bs4 import BeautifulSoup


api_url = "https://services.monster.io/jobs-svx-service/v2/monster/search-jobs/samsearch/en-us"

payload = {
    "fingerprintId": "",
    "jobAdsRequest": {
        "placement": {"appName": "monster", "component": "JSR_SPLIT_VIEW"},
        "position": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    },
    "jobQuery": {
        "companyDisplayNames": [],
        "excludeJobs": [],
        "locations": [{"address": "Australia", "country": "us"}],
        "query": "Software-Developer",
    },
    "offset": 0,
    "pageSize": 20,
    "searchId": "",
}


data = requests.post(api_url, json=payload).json()

# uncomment to print all data:
# print(json.dumps(data, indent=4))

for r in data["jobResults"]:
    s = BeautifulSoup(r["jobPosting"]["description"], "html.parser")
    print("\n".join(wrap(s.get_text(strip=True, separator=" "))))
    print("-" * 80)

Prints:

Description: The role of the Mobile Developer is to analyze business
requirements, develop a design plan, translate plan into program
specifications (or a manual process), code, test, and coordinate
implementation. They will be focusing on enhancements within existing
applications (not building out anything new). Working with multiple
applications- all very client specific. This developer could also be
tasked with some frontend work for a specific application. Need to be
able to utilize Angular skills for frontend development. -Must be
highly proficient with mobile platform Application Programming
Interfaces (API) such as Apple iOS and Android Mobile -Hands on
experience with latest iOS and Android tech -Deep knowledge of Angular


...and so on