| CARVIEW |
Select Language
HTTP/2 200
date: Sun, 27 Jul 2025 23:07:05 GMT
content-type: text/html; charset=utf-8
vary: X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, X-Requested-With,Accept-Encoding, Accept, X-Requested-With
x-robots-tag: none
etag: W/"f7bf6d2f9e5e58cd3ea9a3d348571191"
cache-control: max-age=0, private, must-revalidate
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 0
referrer-policy: no-referrer-when-downgrade
content-security-policy: default-src 'none'; base-uri 'self'; child-src github.githubassets.com github.com/assets-cdn/worker/ github.com/assets/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com www.githubstatus.com collector.github.com raw.githubusercontent.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com *.rel.tunnels.api.visualstudio.com wss://*.rel.tunnels.api.visualstudio.com objects-origin.githubusercontent.com copilot-proxy.githubusercontent.com proxy.individual.githubcopilot.com proxy.business.githubcopilot.com proxy.enterprise.githubcopilot.com *.actions.githubusercontent.com wss://*.actions.githubusercontent.com productionresultssa0.blob.core.windows.net/ productionresultssa1.blob.core.windows.net/ productionresultssa2.blob.core.windows.net/ productionresultssa3.blob.core.windows.net/ productionresultssa4.blob.core.windows.net/ productionresultssa5.blob.core.windows.net/ productionresultssa6.blob.core.windows.net/ productionresultssa7.blob.core.windows.net/ productionresultssa8.blob.core.windows.net/ productionresultssa9.blob.core.windows.net/ productionresultssa10.blob.core.windows.net/ productionresultssa11.blob.core.windows.net/ productionresultssa12.blob.core.windows.net/ productionresultssa13.blob.core.windows.net/ productionresultssa14.blob.core.windows.net/ productionresultssa15.blob.core.windows.net/ productionresultssa16.blob.core.windows.net/ productionresultssa17.blob.core.windows.net/ productionresultssa18.blob.core.windows.net/ productionresultssa19.blob.core.windows.net/ github-production-repository-image-32fea6.s3.amazonaws.com github-production-release-asset-2e65be.s3.amazonaws.com insights.github.com wss://alive.github.com api.githubcopilot.com api.individual.githubcopilot.com api.business.githubcopilot.com api.enterprise.githubcopilot.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com copilot-workspace.githubnext.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src viewscreen.githubusercontent.com notebooks.githubusercontent.com; img-src 'self' data: blob: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com avatars.githubusercontent.com private-avatars.githubusercontent.com github-cloud.s3.amazonaws.com objects.githubusercontent.com release-assets.githubusercontent.com secured-user-images.githubusercontent.com/ user-images.githubusercontent.com/ private-user-images.githubusercontent.com opengraph.githubassets.com copilotprodattachments.blob.core.windows.net/github-production-copilot-attachments/ github-production-user-asset-6210df.s3.amazonaws.com customer-stories-feed.github.com spotlights-feed.github.com objects-origin.githubusercontent.com *.githubusercontent.com; manifest-src 'self'; media-src github.com user-images.githubusercontent.com/ secured-user-images.githubusercontent.com/ private-user-images.githubusercontent.com github-production-user-asset-6210df.s3.amazonaws.com gist.github.com; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; upgrade-insecure-requests; worker-src github.githubassets.com github.com/assets-cdn/worker/ github.com/assets/ gist.github.com/assets-cdn/worker/
server: github.com
content-encoding: gzip
accept-ranges: bytes
set-cookie: _gh_sess=dh13Xm4ZOe2nrLPJYfO3rcn6RcqyFNIvbRnWK43FvlxXjkP3rduyHwuVkaztCSbgewuLyyjkZOeGqan62qj6I5%2FpwruQvll8Va3qrVHqG6woiz%2BJjnyMunL6OO%2BS%2BR%2FW4RsiQKJnKzaZ8mcerb4gDZ6PylGdBYwgtJlj7kUuP6AkApPmD5JjN6kh0vg%2B0pFzwWpiAK6YggTZqoAsLmjQ2UA2zZmpb65QH3hLeFuWeXt9U6CC%2BNWdcyWTkhuL10gjSkuhoXTruVPBhmuEg8qQSA%3D%3D--hROhI67u2NUtJV%2Fy--Gm7kl9NzHx9kkoL276zszQ%3D%3D; Path=/; HttpOnly; Secure; SameSite=Lax
set-cookie: _octo=GH1.1.595232000.1753657625; Path=/; Domain=github.com; Expires=Mon, 27 Jul 2026 23:07:05 GMT; Secure; SameSite=Lax
set-cookie: logged_in=no; Path=/; Domain=github.com; Expires=Mon, 27 Jul 2026 23:07:05 GMT; HttpOnly; Secure; SameSite=Lax
x-github-request-id: 9AD8:363053:1213A04:17D5788:6886B119
History · rmax/scrapy-redis Wiki · GitHub
Skip to content
Navigation Menu
{{ message }}
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
History
Jeremy Chou edited this page May 30, 2023
·
4 revisions
- Move docs to GitHub Wiki
- Update tox and support dynamic tests
- Update support for json data
- Refactor max idle time
- Add support for python3.7~python3.10
- Deprecate python2.x support
- Fix RedisStatsCollector._get_key()
- Fix redis-py dependency version
- Added maximum idle waiting time MAX_IDLE_TIME_BEFORE_CLOSE
- Fixes datetime parse error for redis-py 3.x.
- Add support for stats extensions.
- Fixes datetime parse error for redis-py 3.x.
- Add support for stats extensions.
- Unreleased.
- Fixed automated release due to not matching registered email.
- Fixes bad formatting in logging message.
- Fixes wrong message on dupefilter duplicates.
- Fixed typo in default settings.
- Fixed data decoding in Python 3.x.
- Added
REDIS_ENCODINGsetting (defaultutf-8). - Default to
CONCURRENT_REQUESTSvalue forREDIS_START_URLS_BATCH_SIZE. - Renamed queue classes to a proper naming conventiong (backwards compatible).
- Added
REDIS_START_URLS_KEYsetting. - Fixed spider method
from_crawlersignature.
- Support
redis_clsparameter inREDIS_PARAMSsetting. - Python 3.x compatibility fixed.
- Added
SCHEDULER_SERIALIZERsetting.
-
Backwards incompatible change: Require explicit
DUPEFILTER_CLASSsetting. - Added
SCHEDULER_FLUSH_ON_STARTsetting. - Added
REDIS_START_URLS_AS_SETsetting. - Added
REDIS_ITEMS_KEYsetting. - Added
REDIS_ITEMS_SERIALIZERsetting. - Added
REDIS_PARAMSsetting. - Added
REDIS_START_URLS_BATCH_SIZEspider attribute to read start urls in batches. - Added
RedisCrawlSpider.
- Updated code to be compatible with Scrapy 1.0.
- Added
-a domain=...option for example spiders.
- Added
REDIS_URLsetting to support Redis connection string. - Added
SCHEDULER_IDLE_BEFORE_CLOSEsetting to prevent the spider closing too quickly when the queue is empty. Default value is zero keeping the previous behavior. - Schedule preemptively requests on item scraped.
- This version is the latest release compatible with Scrapy 0.24.x.
- Added
RedisSpiderandRedisMixinclasses as building blocks for spiders to be fed through a redis queue. - Added redis queue stats.
- Let the encoder handle the item as it comes instead converting it to a dict.
- Added support for different queue classes.
- Changed requests serialization from
marshaltocPickle.
- Improved backward compatibility.
- Added example project.
- First release on PyPI.
You can’t perform that action at this time.