CARVIEW |
Select Language
HTTP/2 200
date: Wed, 23 Jul 2025 17:38:14 GMT
content-type: text/html; charset=utf-8
vary: X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, X-Requested-With,Accept-Encoding, Accept, X-Requested-With
x-robots-tag: none
etag: W/"9816bd900d19fb327babd6743fdaceac"
cache-control: max-age=0, private, must-revalidate
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 0
referrer-policy: no-referrer-when-downgrade
content-security-policy: default-src 'none'; base-uri 'self'; child-src github.githubassets.com github.com/assets-cdn/worker/ github.com/assets/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com www.githubstatus.com collector.github.com raw.githubusercontent.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com *.rel.tunnels.api.visualstudio.com wss://*.rel.tunnels.api.visualstudio.com objects-origin.githubusercontent.com copilot-proxy.githubusercontent.com proxy.individual.githubcopilot.com proxy.business.githubcopilot.com proxy.enterprise.githubcopilot.com *.actions.githubusercontent.com wss://*.actions.githubusercontent.com productionresultssa0.blob.core.windows.net/ productionresultssa1.blob.core.windows.net/ productionresultssa2.blob.core.windows.net/ productionresultssa3.blob.core.windows.net/ productionresultssa4.blob.core.windows.net/ productionresultssa5.blob.core.windows.net/ productionresultssa6.blob.core.windows.net/ productionresultssa7.blob.core.windows.net/ productionresultssa8.blob.core.windows.net/ productionresultssa9.blob.core.windows.net/ productionresultssa10.blob.core.windows.net/ productionresultssa11.blob.core.windows.net/ productionresultssa12.blob.core.windows.net/ productionresultssa13.blob.core.windows.net/ productionresultssa14.blob.core.windows.net/ productionresultssa15.blob.core.windows.net/ productionresultssa16.blob.core.windows.net/ productionresultssa17.blob.core.windows.net/ productionresultssa18.blob.core.windows.net/ productionresultssa19.blob.core.windows.net/ github-production-repository-image-32fea6.s3.amazonaws.com github-production-release-asset-2e65be.s3.amazonaws.com insights.github.com wss://alive.github.com api.githubcopilot.com api.individual.githubcopilot.com api.business.githubcopilot.com api.enterprise.githubcopilot.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com copilot-workspace.githubnext.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src viewscreen.githubusercontent.com notebooks.githubusercontent.com; img-src 'self' data: blob: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com avatars.githubusercontent.com private-avatars.githubusercontent.com github-cloud.s3.amazonaws.com objects.githubusercontent.com release-assets.githubusercontent.com secured-user-images.githubusercontent.com/ user-images.githubusercontent.com/ private-user-images.githubusercontent.com opengraph.githubassets.com copilotprodattachments.blob.core.windows.net/github-production-copilot-attachments/ github-production-user-asset-6210df.s3.amazonaws.com customer-stories-feed.github.com spotlights-feed.github.com objects-origin.githubusercontent.com *.githubusercontent.com; manifest-src 'self'; media-src github.com user-images.githubusercontent.com/ secured-user-images.githubusercontent.com/ private-user-images.githubusercontent.com github-production-user-asset-6210df.s3.amazonaws.com gist.github.com; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; upgrade-insecure-requests; worker-src github.githubassets.com github.com/assets-cdn/worker/ github.com/assets/ gist.github.com/assets-cdn/worker/
server: github.com
content-encoding: gzip
accept-ranges: bytes
set-cookie: _gh_sess=%2By2PLsj1ep0OOOGNwXtpSn0GmoJHSTEw%2BPuYdlyVxdJ6roSCR7B9pkHM%2B5MxZ3GYvaWjFvuiN5MU48nPA3iJJqshZTN%2Fph20W4ZdIpFGBXk8bOW1UQ9DAVhpAMMEfDXtXYoV%2FdKg3LXafrFajuBXsA2%2FTMMb6Kf4h9916O2rRSmYhV9ZlyQ0q9jvHPnpGJOLwzantP2Vp2DGgSN1X%2BMNXX1rQHK2nUC4RPfMUUBJw5vSVTpVzE1De%2BUbh8e285POB5ixDSmlAeUId2AmadhSxA%3D%3D--mf7u1%2FBa7kZqABP3--AFUWsqtrrB4%2B9OSlaW6qhg%3D%3D; Path=/; HttpOnly; Secure; SameSite=Lax
set-cookie: _octo=GH1.1.614859600.1753292293; Path=/; Domain=github.com; Expires=Thu, 23 Jul 2026 17:38:13 GMT; Secure; SameSite=Lax
set-cookie: logged_in=no; Path=/; Domain=github.com; Expires=Thu, 23 Jul 2026 17:38:13 GMT; HttpOnly; Secure; SameSite=Lax
x-github-request-id: C63A:167F3A:F4B14F:12168C8:68811E05
Erasure coding · gilbertchen/duplicacy Wiki · GitHub
Skip to content
Navigation Menu
{{ message }}
-
Notifications
You must be signed in to change notification settings - Fork 345
Erasure coding
gilbertchen edited this page Sep 27, 2020
·
1 revision
This feature is available since CLI version 2.7.0.
To initialize a storage with erasure coding enabled, run this command (assuming 5 data shards and 2 parity shards):
duplicacy init -erasure-coding 5:2 repository_id storage_url
Then you can run backup, check, prune, etc as usual.
When a bad chunk is detected, you'll see log messages like this:
Restoring /private/tmp/duplicacy_test/repository to revision 1
Recovering a 1824550 byte chunk from 364910 byte shards: ***--**
Downloaded chunk 1 size 1817347, 1.73MB/s 00:00:11 9.0%
Recovering a 6617382 byte chunk from 1323477 byte shards: **--***
Downloaded chunk 2 size 6591322, 8.02MB/s 00:00:02 42.0%
Recovering a 5136934 byte chunk from 1027387 byte shards: --*****
Downloaded chunk 3 size 5116593, 12.90MB/s 00:00:01 67.6%
Recovering a 2515494 byte chunk from 503099 byte shards: -*****-
Downloaded chunk 4 size 2505558, 15.29MB/s 00:00:01 80.1%
Recovering a 3984934 byte chunk from 796987 byte shards: --*****
Downloaded chunk 5 size 3969180, 19.07MB/s 00:00:01 100.0%
Downloaded file1 (20000000)
To check if a storage is configured with erasure coding, run duplicacy -d list
and it should report the numbers of data and parity shards:
Data shards: 5, parity shards: 2
The encoded chunk file starts with a 10 byte unique banner, then a 14 byte header containing the chunk size and parity parameters, followed by hashes of each shard, then the contents of shards, and finally the 14 byte header again for redundancy:
----------------------------
| duplicacy\0003 (10 bytes) |
-------------------------------------------------------------------------------------------------
| chunk size (8 bytes) | #data shards (2 bytes) | #parity shards (2 bytes) | checksum (2 bytes) |
-------------------------------------------------------------------------------------------------
| hash of data shard #1 (32 bytes) |
------------------------------------
...
| hash of parity shard #1 (32 bytes) |
------------------------------------
...
| data shard #1 |
-----------------
...
| parity shard #1 |
-----------------
...
-------------------------------------------------------------------------------------------------
| chunk size (8 bytes) | #data shards (2 bytes) | #parity shards (2 bytes) | checksum (2 bytes) |
You can’t perform that action at this time.