Scraping Baidu Search Results with Python: A Comprehensive Guide

Baidu is the dominant search engine in China, with over 500 million active users. As the "Chinese Google", Baidu indexes a vast amount of Chinese web pages and provides keyword search services similar to Google.

For many businesses, researchers, and individuals, scraping Baidu search results provides valuable insights into Chinese language content and what people in China are searching for. By extracting and analyzing Baidu search data, you can uncover Chinese consumer trends, monitor brand mentions, conduct market research, and more.

However, scraping Baidu can be challenging for several reasons:

• Baidu employs strict anti-scraping measures like IP blocks, CAPTCHAs, and frequent DOM changes.

• Baidu search results are displayed dynamically via JavaScript, making extraction difficult.

• The large volume of Chinese text requires encoding and language expertise.

• Proxies or residential IPs located in China are often needed to access Baidu freely.

In this comprehensive guide, I‘ll walk through how to scrape Baidu effectively step-by-step using Python. We‘ll cover:

• Baidu search engine basics
• The legality of scraping Baidu
• Required tools & packages
• Query parameter customization
• Extracting data via the Baidu Scraper API
• Storing and analyzing results

Let‘s get started!

Overview of Baidu Search Engine

Baidu provides keyword search capabilities for the Chinese language web. Some key facts about Baidu:

• It has over 1 billion monthly active users globally.

• Baidu accounts for over 70% of internet searches in China.

• In addition to web search, Baidu offers additional services like news, images, video, shopping, travel booking, and more.

• Baidu is accessible globally but also has country-specific domains like Baidu.com (international) and Baidu.jp (Japan).

When you search on Baidu, the search engine results page (SERP) displays various types of results:

Organic Search Results

These are the main web page results that Baidu‘s algorithm determines are most relevant for the search query. Organic results typically make up the majority of the SERP and are ranked by relevance.

Paid/Sponsored Results

Advertisers can pay Baidu to show ads above or within the organic results. Paid results are marked "广告" and tend to show at the top and bottom of the SERP.

Special Results

Depending on the search, Baidu may display unique content blocks above organic results like news tabs, video carousels, shopping modules, and knowledge panels.

Related Searches

At the bottom of the results, Baidu suggests alternative searches related to the original query to help users refine their search.

Understanding these key elements of Baidu‘s SERP will help guide your scraping approach. Next, we‘ll look at the legality of scraping Baidu specifically.

Is Scraping Baidu Legal?

The legality of web scraping is nuanced and varies across jurisdictions. However, in most cases, it is legal to scrape publicly available data from sites like Baidu if done reasonably.

Here are some best practices to follow when scraping Baidu to stay on the right side of the law:

• Only extract data that Baidu makes publicly visible to anyone without needing to log in. Scraping non-public user info or content behind a login would be unethical and illegal.

• Check Baidu‘s Terms of Service and respect any usage limits or restrictions they place on automated scraping. Generally, individual non-commercial scraping is allowed.

• Make sure not to download or redistribute content that may be copyrighted, like news articles or images. Only collect factual data like keywords, URLs and metadata.

• Use throttling, random delays, and other courteous scraping tactics to minimize load on Baidu‘s servers. Don‘t overload their systems with an excessive number of rapid requests.

• Cache previously scraped data when possible rather than hitting Baidu repeatedly for the same queries.

As long as you follow reasonable practices like above and aren‘t reselling Baidu‘s data, scraping modest amounts of organic search data for internal analysis purposes appears to be legal. Of course, always consult a lawyer for legal advice tailored to your specific situation.

Now let‘s look at the tools and methodology for scraping Baidu SERPs in Python.

Tools & Packages for Scraping Baidu in Python

To extract data from Baidu in Python, we‘ll need:

Python 3 – The programming language we‘ll write our scraper in. I recommend the latest version.

Requests – A Python package for sending HTTP requests to web pages. We‘ll use it to fetch Baidu‘s results pages.

BeautifulSoup – A handy Python library for parsing and extracting information from HTML and XML documents through DOM traversal.

JSON – For encoding scraped search data into JSON format. Part of Python‘s standard library.

Oxylabs Baidu Scraper API – A commercial API service that provides proxy rotation and handles Baidu‘s anti-scraping measures, which can be integrated into a Python scraper.

We‘ll install Requests and BeautifulSoup via pip, Python‘s package manager. Let‘s create a virtual environment first to avoid cluttering the global Python install:

# Create virtual env 
python3 -m venv scraper-env 

# Activate virtual env
source scraper-env/bin/activate

# Install requests and BeautifulSoup
pip install requests beautifulsoup4

Now we‘re ready to start coding the scraper in Python!

Customizing Query Parameters

Before we write the scraper, let‘s discuss how to customize the search results through query parameters.

Query parameters are options we can append to the Baidu search URL to configure our scraped data, like:

• The keywords/phrases to search for
• Number of results pages to extract
• Results per page
• Search domain (Baidu.com, Baidu.jp, etc)
• Language, region, device type, etc.

Here are some common Baidu query parameters:

wd – The search query keywords, equivalent to Google‘s q parameter.

pn – Page number of results to return. Baidu displays 10 results per page by default.

rn – Number of results per page. Maximum is 100.

ie – Input encoding, usually utf-8.

oe – Output encoding, usually utf-8.

domain – Domain to search, like com or jp.

gpc – Region setting, can be ip for geo-located results.

So an example search URL with parameters could be:

https://www.baidu.com/s?wd=python&pn=2&rn=20&ie=utf-8&domain=com

This would search for "python" on Baidu.com, display the 2nd page of results, with 20 results per page, and use UTF-8 encoding.

When you scrape Baidu programmatically, you can generate these parameterized URLs to customize the search data.

Now let‘s see how we can extract Baidu results in Python.

Scraping Baidu Search in Python

We‘ll break the scraping process down into several steps:

1. Make a request to Baidu for the SERP of our keyword.

2. Parse the HTML with BeautifulSoup to extract the organic results.

3. Clean and structure the scraped data.

4. Handle pagination by incrementing page parameters.

5. Manage proxies and blocks using the Baidu Scraper API.

Let‘s go through each step.

1. Fetching Baidu‘s SERP Page

We‘ll use the Requests library to send a GET request to Baidu for our chosen keyword.

Let‘s search for "python" on Baidu.com:

import requests

search_term = "python"
page = 1 

# Set Baidu search URL with parameters   
url = f"https://www.baidu.com/s?wd={search_term}&pn={page}"  

# Fetch page with Requests
response = requests.get(url)

This fetches the HTML content of the first page of results. Next, we‘ll parse the HTML.

2. Extracting Results with BeautifulSoup

Now we can parse the scraped HTML using BeautifulSoup to extract the data we want – the organic search results.

from bs4 import BeautifulSoup

# Create BeautifulSoup object from response content 
soup = BeautifulSoup(response.content, ‘html.parser‘)

# Extract organic results
results = soup.select(‘div.result‘) 

Here BeautifulSoup parses the HTML into a navigable DOM we can search through using CSS selectors. We grab all <div class="result"> elements, which contain the individual organic results.

3. Cleaning & Structuring Data

The raw organic result HTML needs some cleaning to extract the most useful fields:

cleaned_results = []

for result in results:

    title = result.h3.text
    url = result.h3.a[‘href‘] 
    snippet = result.find(‘div‘, class_=‘c-abstract‘).text

    cleaned_result = {
        ‘title‘: title,
        ‘url‘: url,
        ‘snippet‘: snippet
    }

    cleaned_results.append(cleaned_result)

This parses out the key fields – title, URL, and snippet – into a cleaner dictionary for each result. We append each dictionary to a list to collate the cleaned results.

4. Paginating Results

To scrape multiple pages of results, we can increment the pn page parameter:

# Number of pages to scrape
pages = 10 

results = []

for i in range(pages):

    page = i + 1

    # Fetch and parse page
    # ...

    results.extend(cleaned_results)

This iterates through the first 10 pages, extracts the cleaned results per page, and combines all results into a single list.

5. Managing Blocks with Baidu Scraper API

The code above covers the basics of scraping Baidu SERPs in Python. However, in reality Baidu employs advanced anti-scraping systems to detect and block bots.

To scrape Baidu reliably at scale, it‘s best to use a commercial web scraping API like Oxylabs.

The Oxylabs Baidu Scraper API handles necessary scraping infrastructure like:

• Global residential IPs and datacenters to prevent blocks.
• CAPTCHA solving services.
• Automatic proxy rotation following best practices.

It also provides out-of-the-box support for Baidu scraping in Python.

To integrate the API, you first sign up for an Oxylabs account to get an API key.

Then you can install the oxylabs Python package:

pip install oxylabs

And modify the request logic to use the API:

from oxylabs import BaiduScraper

client = BaiduScraper(api_key=‘YOUR_API_KEY‘)

proxy = client.get_proxy() 

response = requests.get(url, proxies=proxy)

This fetches a rotating proxy from Oxylabs for each request, preventing Baidu from seeing a consistent scraping IP.

The API also takes care of CAPTCHAs, blocks, and other anti-scraping barriers automatically behind the scenes.

So by leveraging a commercial web scraping API, you can build a robust Baidu scraper in Python without dealing with the underlying proxy management and antic-scraping logic yourself.

Storing & Analyzing Scraped Baidu Data

Once you‘ve scraped your desired amount of pages from Baidu, you‘ll likely want to store the data for further analysis.

Here are some options for storing the scraped search results:

• JSON – Simple and useful for prototyping. You can export results to a JSON file.

• CSV – For analysis in Excel or Tableau. CSV is a compact tabular format.

• MySQL, PostgreSQL – Structured relational databases for more complex analysis with SQL queries. Requires database setup.

• MongoDB – A popular document-based database for unstructured or semi-structured data like JSON.

• Google BigQuery – A serverless data warehouse suitable for large scraping projects. Can directly ingest JSON.

For ad-hoc analysis and visualization, I recommend JSON or CSV files, which are easy to export using Python‘s built-in JSON and CSV modules.

Once your Baidu data is collected in an analysis-friendly format, there are countless directions for mining insights, including:

• Tracking search volume and trends over time for keywords related to your business, brand, or industry.

• Monitoring new sites/domains ranking for key terms.

• Analyzing search market share between you and competitors.

• Optimizing SEM campaigns and SEO keywords for the Chinese market.

• Informing product, marketing, and expansion plans with Chinese consumer demand data.

Scraping Baidu opens up a goldmine of search data for tapping into market trends and consumer behavior unique to China and Chinese language users worldwide.

Conclusion

Scraping data from Baidu can provide invaluable market insights, but requires technical expertise to overcome its anti-scraping systems.

In this guide, we covered:

• How Baidu search results work
• The legality of Baidu scraping
• Configuring search via query parameters
• Extracting SERP data in Python with Requests and BeautifulSoup
• Pagination, proxy rotation, and the Oxylabs API for robust scraping
• Storing and analyzing results

The techniques outlined here should equip you to build an effective Baidu scraper in Python tailored to your needs, whether for market research, brand monitoring, SEO, or other use cases requiring Chinese search data.

As your needs grow, leveraging a commercial web scraping solution like Oxylabs removes the headaches of handling proxies, blocks, captchas, and precision page handling at scale.

Scraping a niche search engine like Baidu requires nuance – but done properly can provide invaluable visibility into the Chinese market.

I hope this guide serves as a solid blueprint for unlocking Baidu‘s data potential with Python. Let me know if you have any other questions!