| CARVIEW |
Select Language
HTTP/2 200
date: Sun, 27 Jul 2025 09:09:04 GMT
content-type: text/html; charset=utf-8
cf-ray: 965afee8dd5ff424-BLR
cf-cache-status: DYNAMIC
cache-control: private
set-cookie: prov=f2955b91-5cb9-4b97-af40-d3da200e7e10; expires=Mon, 27 Jul 2026 09:09:03 GMT; domain=.stackexchange.com; path=/; secure; httponly
strict-transport-security: max-age=31536000; includeSubDomains
vary: Accept-Encoding
content-security-policy: upgrade-insecure-requests; frame-ancestors 'self' https://stackexchange.com
x-clacks-overhead: GNU Terry Pratchett
x-frame-options: SAMEORIGIN
x-request-guid: 5bd089a8-6bfb-425e-900b-1cef1ea2a68d
x-worker-origin-response-time: 533000000
x-dns-prefetch-control: off
set-cookie: prov=f2955b91-5cb9-4b97-af40-d3da200e7e10; Path=/; HttpOnly; Domain=stackexchange.com
set-cookie: __cf_bm=divdmOsNwZLIS794bwFztydOrD7sH9Q9E_5QLLV9.dE-1753607344-1.0.1.1-8HKuRsnHw0oW20qikAFAuGqmD08JL5aVBKwSN86zOn5xqZK_fPnGjfH15ser8rG8ghSY3lQNIN1UTnd9J6YvOkiNAbVU7_PerWAjFlEna2s; path=/; expires=Sun, 27-Jul-25 09:39:04 GMT; domain=.stackexchange.com; HttpOnly; Secure; SameSite=None
set-cookie: _cfuvid=O6ggf4b6ksCBP90KapNEYeL9lZSSYfBStxuoULlta14-1753607344049-0.0.1.1-604800000; path=/; domain=.stackexchange.com; HttpOnly; Secure; SameSite=None
server: cloudflare
content-encoding: gzip
Newest 'web-scraping' Questions - Code Review Stack Exchange
Skip to main content
Stack Exchange Network
Stack Exchange network consists of 183 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.
Visit Stack Exchange
Teams
Q&A for work
Connect and share knowledge within a single location that is structured and easy to search.
Learn more about TeamsQuestions tagged [web-scraping]
Web scraping is the use of a program to simulate human interaction with a web server or to extract specific information from a web page.
611 questions
- Bountied 0
- Unanswered
- Frequent
- Score
- Trending
- Week
- Month
- Unanswered (my tags)
3
votes
1
answer
89
views
Multi-Page Web Scraping Code Using Selenium with Multithreading
I have written a web scraping script using Selenium to crawl blog content from multiple URLs. The script processes URLs in batches of 1000 and uses multithreading with the ThreadPoolExecutor to ...
5
votes
2
answers
681
views
Readability and error handling improvements for Python web scraping class
Description
I recently wrote a Python script to download files from the Library of Congress (LOC) based on a search query. The code fetches metadata, extracts file ...
4
votes
1
answer
86
views
Scraping the calendar of some public libraries from their websites
I've been learning some Haskell as an amateur (to be precise: I started programming with this language, and it has been a year or less since I started seriously). So far, I have realised only small ...
0
votes
0
answers
66
views
Scrapy Spider - Code Audit and Optimization Suggestions
I've written a Scrapy spider to scrape product details from a specific website. The spider is designed to navigate through various pages, locate specific products, and gather detailed information ...
2
votes
1
answer
73
views
Scrapy Spider for fetching product data from multiple pages of a website
I have written a Scrapy spider to scrape product data from a website. The spider navigates through multiple pages to reach a specific product and extracts details such as the product name, price, ...
3
votes
2
answers
79
views
Validating a web crawlers page visits with a decorator
I am writing a crawler that is going to end up in production and I was trying to come up with a way to validate its page visits. It scrapes asp.net pages so each scraping process involves a few ...
5
votes
3
answers
834
views
code format and steps web scraping using beautiful soup
I've done simple web scraping and want to make sure all my steps are correct? Is it considered clean code? Is there a better way to use the multi-page scraping feature?
...
3
votes
1
answer
90
views
Scraping website with Python and Selenium to collect data from dynamic website
Summary:
The code scrapes the website and collects the data to store it in CSV. It also downloads selected information that is available for download in PDF format. The details and the entire code are ...
0
votes
2
answers
143
views
Drayage Webscraper: Limited to table structure
This is my first working scraper. I'm sure a lot can be improved. My biggest question is how can I better specify what data to pull? All the data I'm currently grabbing is needed, but I couldn't ...
2
votes
1
answer
69
views
A selenium web scraper to package NBA data
I'm building a selenium web scraper for basketball-reference.com that takes a player name and returns data in either a JSON format or Pandas DataFrame object. The class in question is one of many that ...
4
votes
1
answer
114
views
Java classes for downloading all in-coming/out-going links of an article in the Wikipedia article graph
(The entire project is in GitHub.)
Introduction
This project provides facilities for generating in-coming or out-going links in a given Wikipedia page.
Code
...
5
votes
1
answer
189
views
Scraping the Divar.ir
I've wrote a code to scrape the Divar, which is an equivalent of Ebay in Iran. I have a few questions:
Am I doing the error handling and logging ok?
Is there a better way to optimize this code? (note ...
1
vote
2
answers
161
views
Web scraping spider
I'm currently working on my first web scraping project and I need to scrape a lot of websites. With my current code it takes more than a day but for my project I need to scan the same websites every 5 ...
3
votes
2
answers
195
views
Enum to deserialize HTML sizes from JSON with serde
I added an enum for my webscraper to deserialize data from a JSON field that represents an HTML image size, which can either be an unsigned int like 1080 or a ...
2
votes
1
answer
80
views
Automatically extract useful cars from car site
I am using puppeteer to extract data and see when a car that meets my requirements shows up, this is what I did so far. I would like some basic syntax advice, or more advanced tips as well.
I tried to ...
- The Overflow Blog
-
-
- Featured on Meta
-
-
Related Tags
python × 430
python-3.x × 205
beautifulsoup × 140
selenium × 71
performance × 58
beginner × 47
vba × 35
object-oriented × 32
javascript × 24
java × 23
scrapy × 22
excel × 19
python-2.x × 19
parsing × 18
python-requests × 18
c# × 17
php × 17
html × 16
multithreading × 16
node.js × 16
pandas × 16
regex × 16
ruby × 16
api × 15
http × 15
more related tags
Hot Network Questions
- What Isp can be gotten from a napthalene (mothball) hot gas thruster? Did it work in space?
- Do TeX engines support Shavian OpenType fonts?
- rock paper scissors game with randomized system choices. NOT using rnd()
- Can expanding foam provide interior strength for a large model?
- Automatic drawing of polygon around selected polygons in QGIS
- I am looking for a French mathematician who contributed to complex analysis
- Can I switch the stand on my Apple Studio Display?
- In Jurassic Park, is the lysine dependency intentionally fake (in-universe)?
- leftmargin in enumerate
- Question about relativity of simultaneity vs signal delay
- Why is time reversal not a natural part of the Galilean group?
- What is the best way to bevel these corners without these deformations showing?
- Regenerative cooling of rocket engine turbine's blades
- I don't have enough alive employees to start the day. What now?
- Is it possible to use the PC power button to pre-select the OS in rEFInd UEFI dual boot setup?
- Why are police personnel of all ranks called officers?
- How do I set up a recursive algorithm to repeatedly square root a positive real number?
- Tracking stored procedure performance metrics
- Blinded despite Blindsight?
- How much of a really big a spaceship could a nuke take out?
- What does this word "ahn-shdah-heh" mean?
- Source for Adam naming the angels
- Has the Silver Surfer's clothing (or lack thereof) ever been addressed in the comics?
- Why is 0.0 printed as 0.00001 when rounding upward?