| CARVIEW |
Select Language
HTTP/2 200
date: Fri, 25 Jul 2025 08:40:19 GMT
content-type: text/html; charset=utf-8
cache-control: max-age=0, private, must-revalidate
content-security-policy: default-src 'none'; base-uri 'self'; child-src github.githubassets.com github.com/assets-cdn/worker/ github.com/assets/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com www.githubstatus.com collector.github.com raw.githubusercontent.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com *.rel.tunnels.api.visualstudio.com wss://*.rel.tunnels.api.visualstudio.com objects-origin.githubusercontent.com copilot-proxy.githubusercontent.com proxy.individual.githubcopilot.com proxy.business.githubcopilot.com proxy.enterprise.githubcopilot.com *.actions.githubusercontent.com wss://*.actions.githubusercontent.com productionresultssa0.blob.core.windows.net/ productionresultssa1.blob.core.windows.net/ productionresultssa2.blob.core.windows.net/ productionresultssa3.blob.core.windows.net/ productionresultssa4.blob.core.windows.net/ productionresultssa5.blob.core.windows.net/ productionresultssa6.blob.core.windows.net/ productionresultssa7.blob.core.windows.net/ productionresultssa8.blob.core.windows.net/ productionresultssa9.blob.core.windows.net/ productionresultssa10.blob.core.windows.net/ productionresultssa11.blob.core.windows.net/ productionresultssa12.blob.core.windows.net/ productionresultssa13.blob.core.windows.net/ productionresultssa14.blob.core.windows.net/ productionresultssa15.blob.core.windows.net/ productionresultssa16.blob.core.windows.net/ productionresultssa17.blob.core.windows.net/ productionresultssa18.blob.core.windows.net/ productionresultssa19.blob.core.windows.net/ github-production-repository-image-32fea6.s3.amazonaws.com github-production-release-asset-2e65be.s3.amazonaws.com insights.github.com wss://alive.github.com api.githubcopilot.com api.individual.githubcopilot.com api.business.githubcopilot.com api.enterprise.githubcopilot.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com copilot-workspace.githubnext.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src viewscreen.githubusercontent.com notebooks.githubusercontent.com; img-src 'self' data: blob: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com avatars.githubusercontent.com private-avatars.githubusercontent.com github-cloud.s3.amazonaws.com objects.githubusercontent.com release-assets.githubusercontent.com secured-user-images.githubusercontent.com/ user-images.githubusercontent.com/ private-user-images.githubusercontent.com opengraph.githubassets.com copilotprodattachments.blob.core.windows.net/github-production-copilot-attachments/ github-production-user-asset-6210df.s3.amazonaws.com customer-stories-feed.github.com spotlights-feed.github.com objects-origin.githubusercontent.com *.githubusercontent.com; manifest-src 'self'; media-src github.com user-images.githubusercontent.com/ secured-user-images.githubusercontent.com/ private-user-images.githubusercontent.com github-production-user-asset-6210df.s3.amazonaws.com gist.github.com; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; upgrade-insecure-requests; worker-src github.githubassets.com github.com/assets-cdn/worker/ github.com/assets/ gist.github.com/assets-cdn/worker/
link: ; rel=preload; as=fetch; crossorigin=use-credentials
referrer-policy: no-referrer-when-downgrade
server-timing: issue_layout-fragment;desc="issue_layout fragment";dur=231.675833,issue_conversation_content-fragment;desc="issue_conversation_content fragment";dur=536.599734,issue_conversation_sidebar-fragment;desc="issue_conversation_sidebar fragment";dur=48.850407,nginx;desc="NGINX";dur=0.475904,glb;desc="GLB";dur=99.656884
strict-transport-security: max-age=31536000; includeSubdomains; preload
vary: X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, X-Requested-With, Accept,Accept-Encoding, Accept, X-Requested-With
x-content-type-options: nosniff
x-frame-options: deny
x-voltron-version: a2eb102
x-xss-protection: 0
server: github.com
content-encoding: gzip
accept-ranges: bytes
set-cookie: _gh_sess=o8d18c5eACYw0IPRrYr4y2MfK%2Bf0d1ydR%2BVYCJTQ8rO6UTXfOYq0g7W396tzITDnAzE4JKDATBCU7dv0L8ul0ISIX%2FRGY%2FyE4xc3jeusz7Pmvq49%2B4wFR3VDwda9Fwn2v%2FV%2Fl55uG9VD1Rz8lbw%2BXUN%2BLBxgR5eARZFFH6YKkYdEqo8PAfNE2XzMhZPHShRhshNcbivxzzQgvgMPzS59HygSbNKVSDn8um9fOOQDFCIvK9wJEcTlVeMKEGpJEL%2BaipB8d%2FV0gSwBzOjO%2BDp6Hg%3D%3D--RSp6IQnpu9tnoYet--B879ztsID%2FWOnujC5w2DCA%3D%3D; Path=/; HttpOnly; Secure; SameSite=Lax
set-cookie: _octo=GH1.1.1447683539.1753432818; Path=/; Domain=github.com; Expires=Sat, 25 Jul 2026 08:40:18 GMT; Secure; SameSite=Lax
set-cookie: logged_in=no; Path=/; Domain=github.com; Expires=Sat, 25 Jul 2026 08:40:18 GMT; HttpOnly; Secure; SameSite=Lax
x-github-request-id: A2B4:1D2B8F:57FC1A:69EA52:688342F2
ArchiveBox index corruption when running multiple import processes on v0.5.0 · Issue #454 · ArchiveBox/ArchiveBox · GitHub
No one assignedNo projectsNo milestoneNone yetNo branches or pull requests
Skip to content
Navigation Menu
{{ message }}
-
-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Closed
Labels
status: doneWork is completed and released (or scheduled to be released in the next version)Work is completed and released (or scheduled to be released in the next version)touches: data/schema/architecture
Description
I am trying to import a lot of URLs from a text file. While it was importing already, I tried to import a specific URL parallel. It spat out an error about a mismatched SQL / JSON index and I didn't think anything of it. Now, almost every archive that had already been archived now has no content.
https://tinf.io/admin/core/snapshot/?p=0
All of those archive.whatever URLs were successfully captured. After I tried to download a second page at the same time, it broke hundreds of them.
archivebox init yields:
[*] Updating existing ArchiveBox collection in this folder...
/opt/archive
------------------------------------------------------------------
[*] Verifying archive folder structure...
√ /opt/archive/sources
√ /opt/archive/archive
√ /opt/archive/logs
√ /opt/archive/ArchiveBox.conf
[*] Verifying main SQL index and running migrations...
√ /opt/archive/index.sqlite3
Operations to perform:
Apply all migrations: admin, auth, contenttypes, core, sessions
Running migrations:
No migrations to apply.
[*] Collecting links from any existing indexes and archive folders...
√ Loaded 574 links from existing main index.
! Skipped adding 540 invalid link data directories.
X /opt/archive/archive/1597919542.044602 [1597919542.044602] https://archive.md/o/abcd/https://twitter.com/... "Some title"
X ...(redacted)...
Hint: For more information about the link data directories that were skipped, run:
archivebox status
archivebox list --status=invalid
[*] [2020-08-20 15:46:43] Writing 574 links to main index...
√ /opt/archive/index.sqlite3
√ /opt/archive/index.json
√ /opt/archive/index.html
------------------------------------------------------------------
[√] Done. Verified and updated the existing ArchiveBox collection.
Hint: To view your archive index, run:
archivebox server # then visit https://127.0.0.1:8000
To add new links, you can run:
archivebox add ~/some/path/or/url/to/list_of_links.txt
For more usage and examples, run:
archivebox help
I can verify these folders exist and have content. They are owned by archive:archive, the user that archivebox runs on.
/opt/archive# ls -l /opt/archive/archive/1597919542.046462
total 11108
drwxr-xr-x 4 archive archive 4096 Aug 20 12:25 archive.fo
-rwxr-xr-x 1 archive archive 66 Aug 20 12:26 archive.org.txt
-rwxr-xr-x 1 archive archive 686 Aug 20 12:25 favicon.ico
-rwxr-xr-x 1 archive archive 21021 Aug 20 12:26 index.html
-rwxr-xr-x 1 archive archive 11631 Aug 20 12:26 index.json
drwxr-xr-x 2 archive archive 4096 Aug 20 12:26 media
-rwxr-xr-x 1 archive archive 437299 Aug 20 12:26 output.html
-rwxr-xr-x 1 archive archive 6008835 Aug 20 12:25 output.pdf
drwxr-xr-x 2 archive archive 4096 Aug 20 12:26 readability
-rwxr-xr-x 1 archive archive 382290 Aug 20 12:25 screenshot.png
-rwxr-xr-x 1 archive archive 4473399 Aug 20 12:25 singlefile.html
drwxr-xr-x 2 archive archive 4096 Aug 20 12:25 warc
Metadata
Metadata
Assignees
Labels
status: doneWork is completed and released (or scheduled to be released in the next version)Work is completed and released (or scheduled to be released in the next version)touches: data/schema/architecture
Type
Projects
Milestone
Relationships
Development
Issue actions
You can’t perform that action at this time.