ratarmount

Fixes

• Fix possible issue when deleting from write overlay.
• Do not filter chained fsspec protocols as unknown protocol.
• Return the correct number of blocks for files. This fixes results from du.
• Improve automatic mount point inference.

ratarmountcore

Fixes

• Python file object implementations did not return True for closed after closing them.
• Patch broken TAR parsing for >8 GiB file with sparsity.
• Patch wrong checksum for >8 GiB file with sparsity.
• Support files larger than 8 GiB for GNU sparse detection.
• Fix very long parsing time during GNU sparse detection with --ignore-zeros.
• Make @overrides a test-time error not a runtime-error.
• Make fsspec tar:// protocol work. For most cases, this should be avoided, though, because it does not
  use the performance benefits of ratarmount.
• Use exceptions over asserts for functional SQLAR magic bytes checks so that it works with python3 -O
  optimization mode.
• Detect Zstandard files created by pzstd, which start with a skippable frame.
• Show the correct permissions for the archives opened via libarchive.

AppImages

The AppImages can be downloaded, made executable with chmod u+x *.AppImage, and executed directly to start ratarmount.

There are also usability helper for AppImages such as AppImageLauncher and AM.

If the AppImage is too slow to start up or to reduce memory overhead for the AppImage itself, the Appimage can also be manually unpacked and installed:

• Extract with ratarmount-*.AppImage --appimage-extract
• The extracted AppImage can be started by executing squashfs-root/AppRun.
• Rename and move the resulting generic squashfs-root folder wherever you want, e.g., /opt/ratarmount-<version>.
• Add a link to /opt/ratarmount-<version>/AppRun into some folder that is in your PATH variable, e.g., with:
  ln -s /opt/ratarmount-<version>/AppRun ~/.local/bin/ratarmount

Slim Version

Supports: 7z, ASAR, bzip2, EXT4, FAT, gzip, RAR, SQLAR, TAR, XZ, ZIP, zlib, ZStandard

Normal / Full Version

Supports:

• all slim formats
• libarchive (ar, CAB, cpio, grzip, ISO9660, lrzip, LZ4, lzip, LZMA, lzop, RPM, UU, WARC, XAR, Z), SquashFS
• all remote protocols (ftp://, git://, https://, ssh://, ...)

ratarmount

Features

• Add support for argument completion in your shell with argcomplete if it is installed.
• Make --unmount accept multiple mount points to unmount.
• Add support for extended file attributes.
• Extract libfuse3 support into mfusepy, a fork of fusepy, and depend on it.
• Add --log-file option to write output to file when daemonized.

Performance

• Try backends in smarter order based on file suffix.

ratarmountcore

Feature

• Add support for SQLAR SQLite archives.
• Add support for mounting .index.sqlite ratarmount indexes directly without the associated archive-
• Add support for encrypted 7z files via py7zr.
• Add support for EXT4 images.
• Add support for SAR archives.
• Add new method to specify fine-granular parallelization for each backend.
• Make --recursion-depth 0 only undo the compression on TAR files.
• Show better error message when trying to open supported file formats with missing Python modules.
• Libarchive: Add warning when file contents are encrypted and no password specified.

Performance

• Do not import everything in main module to aid some delayed import workflows for smaller latencies.

API

• Rename AutoMountLayer.recursionDepth to maxRecursionDepth.
• Restructure ratarmountcore file hierarchy. Move MountSource implementations into subfolders.
• Use snake_case for functions and class methods, one step further towards PEP 8.
• Accept os.PathLike in open_mount_source.

Fixes

• The progress bar was wrong for parallelized XZ decompression.
• Collect correct file permissions for SquashFS, RAR, ZIP, and Libarchive.
• FileVersionLayer: Do not dispatch requests on .versions folder to underlying mount source.
• AutoMountLayer: Account for recursions depths introduced by SQLiteIndexedTar.
• Mounting github:// without a prefix did not work correctly.
• Avoid misdetection of images as TAR by libarchive.
• SubvolumeMountSource: Return clone of root file info to avoid duplicate userdata elements.

AppImages

The AppImages can be downloaded, made executable with chmod u+x *.AppImage, and executed directly to start ratarmount.

There are also usability helper for AppImages such as AppImageLauncher and AM.

If the AppImage is too slow to start up or to reduce memory overhead for the AppImage itself, the Appimage can also be manually unpacked and installed:

• Extract with ratarmount-*.AppImage --appimage-extract
• The extracted AppImage can be started by executing squashfs-root/AppRun.
• Rename and move the resulting generic squashfs-root folder wherever you want, e.g., /opt/ratarmount-<version>.
• Add a link to /opt/ratarmount-<version>/AppRun into some folder that is in your PATH variable, e.g., with:
  ln -s /opt/ratarmount-<version>/AppRun ~/.local/bin/ratarmount

Slim version

Supports: 7z, ASAR, bzip2, EXT4, FAT, gzip, RAR, SQLAR, TAR, XZ, ZIP, zlib, ZStandard

Normal / Full Version

Supports:

• all slim formats
• libarchive (ar, CAB, cpio, grzip, ISO9660, lrzip, LZ4, lzip, LZMA, lzop, RPM, UU, WARC, XAR, Z), SquashFS
• all remote protocols (ftp://, git://, https://, ssh://, ...)

This is the newest as of yet unreleased but tested state.
Try this if you have trouble with other versions or you want cutting edge features.

ratarmount

Semantic versioning of GUIs and CLIs is not easy to define. The simplest GUI usage has not changed since the first version and even the index file format has been mostly compatible since version 0.2.0 and there is a lot of code for version checking. Based on the semver FAQ, the 1.0.0. release should probably have been a long time ago. Here it is now!

Features

• Bundle modified fusepy in order to add libfuse3 support in case only that one is installed.
  Contributions to the mfusepy fork would be welcome!
• Add message for first time users to show the result mount point.

Performance

• Disable Python-side buffering when opening files via FUSE.
• Forward underlying archive block sizes to statfs and stat implementations.

Fixes

• statfs did not work when using a write overlay and calling it on a file not in the overlay folder.
• Union mounting of inputs with the same name, even if in different folders, ignored all but the first.
• Suppress teardown warning in case the mount source was not even created yet.
• Make mounting work with only a write overlay.
• Avoid hangs and errors caused by non-joined threads before forking into the background by checking for running threads.
• Set locale to C when calling GNU tar to get more reproducible output on --commit-overlay.

ratarmountcore

Features

• Add fsspec implementation and register it as ratar://.
• Add support for new formats: SquashFS, Git, FAT12, FAT16, FAT32.
• Add support for fsspec backends. Archives and even index files can now be specified via URIs:
  dropbox://, ftp://, git://, github://, https://, https://, ipfs://, ipns://, s3://, ssh://, sftp://, smb://, webdav://.
• Add support for remote and compressed index files. Ratarmount will automatically look for
  index files with .gz and other common extensions and extracts these into /tmp/ or RATARMOUNT_INDEX_TMPDIR
  before using them.
• MountSource.open: Add buffering argument to enable/disable buffering or set the buffer size.

Fixes

• Argument to --gzip-seek-point-spacing was ignored when using the rapidgzip backend.
• Index creation did not work with default arguments with an archive in a read-only location.
• Close sqlite3 dummy connection after querying the SQLite version.
• Avoid resource leaks in case a MountSource constructor throws.
• SQLiteIndex: Do note store checkMetadata callback as a member to avoid dependency cycles.
• SQLiteIndex: Ignore errors when the connection is already closed.
• SQLiteIndexedTar: Avoid resource leak when constructor fails.

Performance

• Import compiled zip decrypter for 100x speedup for Standard ZIP 2.0 encrypted ZIP files
• Speed up readdir and therefore simple use cases such as find to iterate all files and folders by 3x.
• Avoid reading the whole appended TAR parts into memory for the check has-been-appended-to check.
• Fix block size being ignored when reading everything via io.BufferedReader.
• Do not use parallelization with possibly huge prefetches for simple file type checks.

ratarmount

The AppImage now uses manylinux_2_28 instead of manylinux2014 because CentOS 7 was EOL 2024-06-30, which broke the Github Actions pipeline for building the AppImage because some of the "actions" such as "checkout" broken because they are using npm, which requires a newer glibc version and does not work anymore inside the manylinux2014 container. I tried to manually check out the repo, which worked, but then I lost my patience when the upload-artifact action also failed.

Fixes

• Setting either the owner or group for a file with --write-overlay would reset the group or user respectively.
• Setting owner or group to 0 (root) with --write-overlay was not working.

ratarmountcore

Fixes

• Fix error when combining --recursive or --recursion-depth with --write-overlay.

ratarmount

Fixes

• Show install suggestions when FUSE is missing.
• File deletion with --commit-overlay did not work.

ratarmountcore

Fixes

• Fix the missing indentation for the index version check.
• Make --help and --version work even if libarchive.so is not installed.
• Fix LibarchiveMountSource compatibility issues with older libarchive versions.
• Fix the exception in StenciledFile when seeking before the file start.

ratarmount

Added

• Print indicators for versions of loaded shared libraries.

ratarmountcore

Added

• Add libarchive backend and detection support for:
  grzip, lrzip, lz4, lzip, lzma, lzop, rpm, uuencode, compress, 7zip, ar, cab, deb, xar, cpio, iso, war, cxar.
• Add --transform option to map each archive entry path via a regex to some user-specified one.
• Add zlib support (RFC1950).
• Store backend name into the index and check that the index fits to the current backend / MountSource.
• Store isGnuIncremental flag in the index.
• Apply specified priorities for opening all archives not just gzip.

Performance

• Determine incremental archives from index rows to avoid seeks.
• Remove indexed_bzip2 dependency in favor of rapidgzip, which in the future should support even more formats.

Fixes

• utils.findModuleVersion: Return version not name if __version__ does not exist.

rapidgzip

Added

• The rapidgzip Python library now also bundles IndexedBzip2File from indexed_bzip2.
• Enable checksum verification by default.
• Support for decompression from non-seekable inputs such as stdin.

Performance

• Windows / seek points are compressed in memory for reduced memory usage.
• Avoid doubling memory usage during index import and export by streaming the data directly to the output file.

ratarmount

Fixes

• Do not check mount point because of faulty os.path.ismount, simply try fusermount.
• Avoid total I/O hang when lazy-mounting a folder with archives onto itself.

ratarmountcore

Fixes

• Return a valid file info and file version count for /.
• Make the original archive viewable as an older file version when using AutoMountLayer.
• Resolve symbolic links pointing outside the source folder given to FolderMountSource to not break them.
• Do not return a valid FileInfo for invalid paths such as ../...
• Make --index-minimum-file-count work for the TAR backend.
• The index should not be created for very small archives.
• Root file info userdata was not initialized correctly.
• Index validation did fail for TAR entries with more than 2 metadata blocks.
• Do not check for folder consistency because parent folders get automatically added to the index.
• Move _createFileInfo out of MountSource class to fix "protected-access" warning.
• Joined files (a.001, a.002) did not work because of an accidentally shared list.
• Do not check file header for zip, only for the footer, to detect self-extracting archives.

ratarmount

Fixes

• Fix AttributeError: module 'fuse' has no attribute 'errno'.
• Fix --commit-overlay, which did not add newly created empty folders to TARs.
• Do not ask for confirmation if there is nothing to commit when using --commit-overlay.
• Improve unmounting with ratarmount -u, especially with the AppImage.

ratarmountcore

Fixes

• Properly close opened file objects in mount sources.
• Fix --disable-union-mount, which returned an I/O error.
• Fix that --use-backend=indexed_gzip still did use rapidgzip. It only worked when both were specified.
• Fix detection of self-extracting RAR files.
• Improve the error message when a file cannot be read because of a missing dependency.
• Improve debug message when the index does not yet contain a gzip index.
• Fix faulty seek forward for files opened via RarMountSource when whence is not io.SEEK_SET.

Performance

• Skip data offset collection for zip files because it takes too long and is unused.

ratarmount

Added

• Add --disable-union-mount option to present multiple archives in subfolders under the mount point.
• Add fine-grained control over the union mount cache:
  • --union-mount-cache-max-depth
  • --union-mount-cache-max-entries
  • --union-mount-cache-timeout
• Only use the number of cores the ratarmount process has affinity for by default.
• Add --index-minimum-file-count with sane default (1000) to avoid creating index files for
  rather small ZIP archives.

Fixes

• Disable auto-detection of GNU incremental TAR archives by default.
  Use --detect-gnu-incremental or better --gnu-incremental/--no-gnu-incremental
  instead.
• Apply name change: pragzip -> rapigzip.
• Fix ambiguous FUSE behavior that resulted in find not descending into recursively
  mounted subarchives.

ratarmountcore

Added

• Use rapidgzip instead of indexed_gzip by default.
• Do not parallelize gzip on slow drives because pread is slower than
  sequential read on those.
• Enable profiling output for rapidgzip with -d 2 or greater.

Fixes

• Do not check for GNU incremental TAR archive after an index has been loaded because
  it is only necessary for index creation. Note that this means thatSQLiteIndexedTar.isGnuIncremental
  member will remain False when an index has been loaded.
• Test for incremental archive after loading the compression index to avoid having
  to recreate it.
• Fix missing warning for multi-frame requirement of Zstandard archives.
• Fix unnecessary warning about mismatching gzip-seek-point-spacing
  when loading an index for
  archives without gzip compression.
• Change the default value of the SQLiteIndexedTar constructor argument
  gzipSeekPointSpacing from 4 MiB tp 16 MiB to make it consistent with the ratarmount CLI option.