| CARVIEW |
Select Language
HTTP/2 200
date: Mon, 28 Jul 2025 22:09:09 GMT
content-type: text/html; charset=utf-8
cf-ray: 9667b2fd5a35c1cc-BLR
cf-cache-status: DYNAMIC
cache-control: private
set-cookie: prov=f39a1f59-4faf-4bf6-86f8-eb061cf082e1; expires=Tue, 28 Jul 2026 22:09:09 GMT; domain=.stackexchange.com; path=/; secure; httponly
strict-transport-security: max-age=31536000; includeSubDomains
vary: Accept-Encoding
content-security-policy: upgrade-insecure-requests; frame-ancestors 'self' https://stackexchange.com
x-clacks-overhead: GNU Terry Pratchett
x-frame-options: SAMEORIGIN
x-request-guid: c56ee567-6c3f-486c-8e8a-c56b44074083
x-worker-origin-response-time: 947000000
x-dns-prefetch-control: off
set-cookie: prov=f39a1f59-4faf-4bf6-86f8-eb061cf082e1; Path=/; HttpOnly; Domain=stackexchange.com
set-cookie: __cf_bm=MhhNF3XcfN_fzae0e5528HM7b90rx5f1eKhrp1gvma8-1753740549-1.0.1.1-VWpWyO7ZJutoCiea1OvbSdvJEQp.Qw5OnvOaSIc_BJRwiZbv3DgU0msyhuE5zT1II_AylfVRMnc4HNHFrzxUzK4z9P_XwUlXI_MX4YMmQWQ; path=/; expires=Mon, 28-Jul-25 22:39:09 GMT; domain=.stackexchange.com; HttpOnly; Secure; SameSite=None
set-cookie: _cfuvid=3_UGfbAyQPZSl7CBbke5oxfongppdIIeX3GFqu9_zyA-1753740549659-0.0.1.1-604800000; path=/; domain=.stackexchange.com; HttpOnly; Secure; SameSite=None
server: cloudflare
content-encoding: gzip
Newest 'web-scraping' Questions - Code Review Stack Exchange
Skip to main content
Stack Exchange Network
Stack Exchange network consists of 183 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.
Visit Stack Exchange
Teams
Q&A for work
Connect and share knowledge within a single location that is structured and easy to search.
Learn more about TeamsQuestions tagged [web-scraping]
Web scraping is the use of a program to simulate human interaction with a web server or to extract specific information from a web page.
611 questions
- Bountied 0
- Unanswered
- Frequent
- Score
- Trending
- Week
- Month
- Unanswered (my tags)
3
votes
1
answer
89
views
Multi-Page Web Scraping Code Using Selenium with Multithreading
I have written a web scraping script using Selenium to crawl blog content from multiple URLs. The script processes URLs in batches of 1000 and uses multithreading with the ThreadPoolExecutor to ...
5
votes
2
answers
681
views
Readability and error handling improvements for Python web scraping class
Description
I recently wrote a Python script to download files from the Library of Congress (LOC) based on a search query. The code fetches metadata, extracts file ...
4
votes
1
answer
86
views
Scraping the calendar of some public libraries from their websites
I've been learning some Haskell as an amateur (to be precise: I started programming with this language, and it has been a year or less since I started seriously). So far, I have realised only small ...
0
votes
0
answers
66
views
Scrapy Spider - Code Audit and Optimization Suggestions
I've written a Scrapy spider to scrape product details from a specific website. The spider is designed to navigate through various pages, locate specific products, and gather detailed information ...
2
votes
1
answer
73
views
Scrapy Spider for fetching product data from multiple pages of a website
I have written a Scrapy spider to scrape product data from a website. The spider navigates through multiple pages to reach a specific product and extracts details such as the product name, price, ...
3
votes
2
answers
79
views
Validating a web crawlers page visits with a decorator
I am writing a crawler that is going to end up in production and I was trying to come up with a way to validate its page visits. It scrapes asp.net pages so each scraping process involves a few ...
5
votes
3
answers
834
views
code format and steps web scraping using beautiful soup
I've done simple web scraping and want to make sure all my steps are correct? Is it considered clean code? Is there a better way to use the multi-page scraping feature?
...
3
votes
1
answer
90
views
Scraping website with Python and Selenium to collect data from dynamic website
Summary:
The code scrapes the website and collects the data to store it in CSV. It also downloads selected information that is available for download in PDF format. The details and the entire code are ...
0
votes
2
answers
143
views
Drayage Webscraper: Limited to table structure
This is my first working scraper. I'm sure a lot can be improved. My biggest question is how can I better specify what data to pull? All the data I'm currently grabbing is needed, but I couldn't ...
2
votes
1
answer
69
views
A selenium web scraper to package NBA data
I'm building a selenium web scraper for basketball-reference.com that takes a player name and returns data in either a JSON format or Pandas DataFrame object. The class in question is one of many that ...
4
votes
1
answer
114
views
Java classes for downloading all in-coming/out-going links of an article in the Wikipedia article graph
(The entire project is in GitHub.)
Introduction
This project provides facilities for generating in-coming or out-going links in a given Wikipedia page.
Code
...
5
votes
1
answer
189
views
Scraping the Divar.ir
I've wrote a code to scrape the Divar, which is an equivalent of Ebay in Iran. I have a few questions:
Am I doing the error handling and logging ok?
Is there a better way to optimize this code? (note ...
1
vote
2
answers
161
views
Web scraping spider
I'm currently working on my first web scraping project and I need to scrape a lot of websites. With my current code it takes more than a day but for my project I need to scan the same websites every 5 ...
3
votes
2
answers
195
views
Enum to deserialize HTML sizes from JSON with serde
I added an enum for my webscraper to deserialize data from a JSON field that represents an HTML image size, which can either be an unsigned int like 1080 or a ...
2
votes
1
answer
80
views
Automatically extract useful cars from car site
I am using puppeteer to extract data and see when a car that meets my requirements shows up, this is what I did so far. I would like some basic syntax advice, or more advanced tips as well.
I tried to ...
- The Overflow Blog
-
- Featured on Meta
-
-
Related Tags
python × 430
python-3.x × 205
beautifulsoup × 140
selenium × 71
performance × 58
beginner × 47
vba × 35
object-oriented × 32
javascript × 24
java × 23
scrapy × 22
excel × 19
python-2.x × 19
parsing × 18
python-requests × 18
c# × 17
php × 17
html × 16
multithreading × 16
node.js × 16
pandas × 16
regex × 16
ruby × 16
api × 15
http × 15
more related tags
Hot Network Questions
- Is the six million dollar hut challenge real?
- Who has faced the country of winning and losing?
- The puzzle of chess pieces beating one another in “rock—paper—scissors” way
- Self explanatory
- What is the proper course of action when a published paper utilizes my previously published methodology without providing citation?
- Should I regularly reapply thermal paste?
- Will this work as an XOR gate?
- Dependent type theory and free variables
- Sci-fi novel human-centered, with alien empire using stargates, humans try to avoid overruling
- How can my dwarves keep humans from reverse engineering their technology?
- How to speed up this NIntegrate which is calculated many times?
- If a mathematical theorem is true, what it is true of?
- "I met a monk and philosopher." How many people did I meet?
- Does group action on a ring induce an action on a Morita equivalent ring
- How to scale down a benzene inside a larger molecule
- What kind of 7-pin connector is this from a 1997 Compaq laptop?
- How does one justify the rationality of buying insurance?
- Should I simply accept the terms without reviewing them for legal appropriateness?
- How to find all PDAs before closing a program
- Kepler's first law proof error - can anyone help me see what the error is?
- forcing line break in url while preserving link url, without using \href
- How to analytically sample from the conditional distribution of a t-statistic under normal data-generating process?
- Why does power drop occur when the buzzer is activated?
- Civilian airplanes with inside fuselage jet engines