TrueZIP Virtual File System API, Version 6.6

"Archive files are virtual directories!" (the TrueZIP mantra)

Contents

1. Overview
2. News
3. Introduction
4. Downloading and getting started
5. Documentation
6. Support
7. Unit Tests
8. Building from Source

Overview

TrueZIP is a Java based Virtual File System (VFS) which enables client applications to access ZIP, TAR and derivative archive types transparently as if they were just directories in a file's path name. Following are TrueZIP's primary features:

Easy to use:

TrueZIP features drop-in replacements for the java.io.File/FileInputStream/FileOutputStream classes. If you know how to deal with these classes, you can instantly use TrueZIP - though you should really, really read the manual to bypass some common pitfalls and power up your client applications with some advanced I/O tricks.

Thread-safe:

Multiple threads can access the same archive file at the same time. Where restrictions apply, TrueZIP enforces them to prevent client applications from corrupting archive files.

Unlimited nesting:

Client applications can create and access nested archives up to a virtually unlimited nesting level: outer.zip/inner.tar.gz/nuts.jar/META-INF/LICENSE.TXT is a perfectly valid path name.

Strong cryptography:

Forget WinZIP's contended encryption, abstain from PKZIP's patent bombed encryption! TrueZIP's Random Access Encryption Specification (RAES) is open source and features continuous 256 bit encryption and authentication with SHA-256/AES-256/CTR. Furthermore, RAES can be applied to any data payload, not just ZIP files.

Easily extensible:

TrueZIP can support virtually any archive type via its pluggable archive driver architecture. TrueZIP ships with archive drivers for ZIP, TAR and all derivatives (JAR, TAR.GZ, TAR.BZ2, TZP, ...).

Relentlessly robust:

TrueZIP ships with numerous assertions and comprehensive unit tests to ensure maximum reliability. Companies like JBoss, Vignette and many more rely on TrueZIP to deploy applications and content.

Fu*ing fast:

Despite the startup time of the JVM and despite the overhead required to implement a VFS, TrueZIP's nzip utility main class unzips the sources for JDK1.5.0_09 even slightly faster than 7-Zip (details to follow in a future article on this web site).

100% open source:

Covered by the Apache Software License, Version 2.0.

100% pure Java:

No native code incorporated.

News

• February 26^th, 2007 - TrueZIP 6.5.2 has been released
• December 21^st, 2006 - TrueZIP 6.4 has been released
• October 10^th, 2006 - TrueZIP 6.3.1 has been released
• October 6^th, 2006 - TrueZIP 6.3 has been released
• September 7^th, 2006 - TrueZIP 6.2.1 has been released
• August 7^th, 2006 - TrueZIP 6.2 and 5.1.8 have been released
• July 10^th, 2006 - TrueZIP 6.1 and 5.1.7 have been released
• April 5^th, 2006 - TrueZIP 6.0 and 5.1.6 have been released
• Older news...

Introduction

Although the ZIP file format it is a proprietary, de-facto standard by PKWARE Inc. (see https://www.pkware.com/company/standards/appnote/appnote.txt), ZIP files are ubiquitous on the Internet and thus on many platforms today. The J2SE API provides the well known package java.util.zip with classes like ZipInputStream, ZipOutputStream and ZipFile for easy access to ZIP compatible files. However, this package has some serious limitations/disadvantages:

• The J2SE API always uses UTF-8 (eight bit Unicode character set) for entry names and comments instead of IBM437 (a.k.a. IBM437, the genuine IBM-PC character set) as used by the de-facto standard PKZIP from PKWARE. As a result, you cannot read or write ZIP files with international entry file names such as "täscht.txt" in a ZIP file created by a (southern) German.
• You need to use an additional API (the package java.util.zip) if your application needs to support ZIP compatible files in addition to ordinary files and directories, thereby adding unnecessary dependencies and complexity (which implies lots of bugs and work).
• You can either read or completely write ZIP compatible files, but you cannot update individual entries.
• The classes do not support the concept of a directory, which you may want to create, modify, list, rename, recursively copy or delete.
• You cannot browse a ZIP compatible file with a JFileChooser or FileSystemView.
• There is no GUI tree class to browse and edit file systems.

The TrueZIP Library has been developed to overcome these limitations/disadvantages.

The TrueZIP API provides drop-in replacements for the well-known classes File, FileInputStream and FileOutputStream. This design makes TrueZIP very easy to use: All that is required to archive-enable a legacy application is to add a few import statements for the package de.schlichtherle.io, add some type casts where required, and add a final call to File.update() once before the client application's main method terminates.

Now you can simply address archive files like directories in a path name. For example, the path name "archive.zip/readme" addresses the archive entry "readme" within the ZIP file "archive.zip". Note that file name suffixes are fully configurable and TrueZIP automatically detects false positives and reverts back to treat them like ordinary files or directories. This works unlimitedly recursive, so an archive file may even be enclosed in another archive file, like in "outer.zip/inner.zip/readme".

Throughout an application's process life cycle, TrueZIP updates archive files automatically whenever it's required and there is a JVM shutdown hook which updates all archive files which need an update. However, the JVM shutdown hook is just a last resort (like a parachute) and a client application should always call File.update() once before its main method terminates - see the example in the tutorial for version 6 for a more detailed explanation.

TrueZIP features an extensible archive driver architecture (actually a Service Provider Interface) in order to support virtually any archive type. TrueZIP 6 ships with archive drivers for ZIP, TAR and all derivatives (JAR, TAR.GZ, TAR.BZ2, TZP) and support for other archive types is expected to be added over time (contributors wanted: ARJ, RAR anyone?).

Even if a client application never uses an archive file, using TrueZIP may still be beneficial because the enhanced methods in the File class such as recursive renaming, deleting or (archive) copying with limited transaction support. These operations use asynchronous I/O and hence deliver a performance which is comparable to the Channel.transfer(...) method in the java.nio package, although they operate on plain InputStream and OutputStream instances.

Or maybe a client application is just requiring an easy to use, yet secure file format for AES encrypted data which supports transparent random read access just as if it were reading plain data from a RandomAccessFile? Then the classes in the package de.schlichtherle.crypto.io might be a perfect fit.

If you would like to see TrueZIP in action, please visit https://truemirror.schlichtherle.de/en/ and web start the TrueMirror application. TrueMirror is a tool for easy and reliable file synchronization. TrueMirror 3.0 uses TrueZIP 6.2.1 to do all its I/O, not just when synchronizing a folder with a ZIP file, which is supported in order to save storage space on removable media.

Features

• Random access to archive files using drop-in replacements for the classes File, FileInputStream, FileOutputStream.
• Browse archive files using the drop-in replacements for JFileChooser and FileSystemView.
• Browse and edit the file system with a custom JTree, including archive file up to any nesting level.
• Fully transparent access to RAES encrypted ZIP files using AES encryption with up to 256 bit key length.
• Properly supports nested archive files, i.e. new FileInputStream("dir/outer.zip/anotherdir/inner.zip/readme.txt") is perfectly OK.
• Asynchronous I/O for ordinary copying of InputStreams to OutputStreams, delivering performance comparable to the java.nio package.
• Adds a lot of reliable utility methods for handling File objects (cat(), catFrom(), catTo(), copyFrom(), copyTo(), archiveCopyFrom(), archiveCopyTo() etc.).
• Supports IBM437 (and any other) character set encoding for ZIP compatible files with international entry names and comments.
• Configurable suffixes and encodings for ZIP compatible files (.zip, .jar, .tzp, .tar, .tbz2, whatever).
• Provides an enhanced low level ZIP API which is backwards compatible to Sun's java.util.zip package - although you will hardly need that anymore.
• Most classes are thread safe with documented exceptions.
• Runs on J2RE 1.4.2 or higher, reflectively uses new features in JRE 5.0 and JRE 6.0.
• Optimized for maximum performance and minimum memory footprint (in order of priority).
• Ships with comprehensive unit tests for maximum reliability.

Benefits

• Treats ZIP entries like files or directories, i.e. you can create, modify or delete each of them individually.
• One API less to get a headache from: java.util.zip.
• Minimal integration: Some import statements are usually all that is required to make your applications ZIP-enabled.
• Provides fully transparent access to RAES encrypted ZIP files for maximum security.
• Eases dealing with nested ZIP compatible files by treating them as subdirectories of another ZIP compatible file.
• Offers reliable enhanced file operations like cat(), copy*(), archiveCopy*() and many more.

Current Limitations / Caveats

• TrueZIP currently implements the ZIP32 specification only. This limits the maximum ZIP file length to 4GB and the compound size of the name and comment to 64KB per entry. Please note that there are plans to support the ZIP64 format. It is currently evaluated how backwards compatibility could be provided and transparency for applications be retained.
• The RAES encryption is not compatible to WinZip's encryption scheme. This is simply because of security issues with WinZip's encryption scheme. For more information, please refer to the news section on RAES.
• TrueZIP may seem to behave erratically if it is loaded and used by multiple class loaders. For each archive file which has been presented to TrueZIP (via the de.schlichtherle.io.File* classes), TrueZIP associates some internal state with it. This data is mostly held in some static maps. Either make sure that multiple class loader instances never use the same archive file or prevent the loading of multiple instances by assigning a shared parent class loader which loads all TrueZIP classes first.

Downloading and getting started

Important: For new client applications, you should always use TrueZIP 6 rather than TrueZIP 5!

To compile against TrueZIP and run your application, you will need the following:

In your compile time class path:

• The JAR for TrueZIP "truezip.jar". You can download it here.

In your run time class path:

• In case you want to access TAR, TAR.GZ and TAR.BZIP2 file formats you need ant.jar from Apache Ant Version 1.7 or higher. You can download Ant at https://ant.apache.orgg. Note that you need to configure the class de.schlichtherle.io.DefaultArchiveDetector too in order to recognize the associated file suffixes.
• In case your application needs to access or create RAES encrypted ZIP files, you need Bouncy Castle's Lightweight Crypto API for JDK 1.4 Version 1.30 or higher to process these RAES encrypted ZIP files. Check for lcrypto-jdk14-130.jar at https://www.bouncycastle.org/latest_releases.html. Note that you need to configure the class de.schlichtherle.io.DefaultArchiveDetector too in order to recognize the associated file suffixes.

Additional resources:

• For Javadoc: The source distribution of TrueZIP truezip-src.zip. You can download it here.
• For unit testing: JUnit 3.8.1 or later and Jemmy 2.2.6 (since TrueZIP 6.1) or later. You can download JUnit at https://junit.org and Jemmy at https://jemmy.netbeans.org.

To play around with TrueZIP, I recommend to start running the nzip utility main class first. For a tutorial on this utility, read here.

To start using the API, I recommend to read the tutorial matching your version and then read the source code of the nzip main class.

Documentation

Documentation for TrueZIP 6:

• Manual for TrueZIP 6
• Javadoc for the TrueZIP API: As a starting point, please refer to the package de.schlichtherle.io.
• Release Notes for TrueZIP 6.6
• Release Notes for TrueZIP 6.5.4
• Release Notes for TrueZIP 6.4
• Release Notes for TrueZIP 6.3
• Release Notes for TrueZIP 6.2.1
• Release Notes for TrueZIP 6.2
• Release Notes for TrueZIP 6.1
• Release Notes for TrueZIP 6.0
• The nzip main class - a comprehensive ZIP file utility

Documentation for TrueZIP 5:

• Tutorial for TrueZIP versions up to 5.X
• Release Notes for TrueZIP 5.1.8
• Release Notes for TrueZIP 5.1.7

Support

Free support is available via the mailing list users@truezip.dev.java.net (you need to subscribe to the mailing list first).

Please do not send emails directly! Answers to any direct emails regarding this project may get relentlessly copied to the appropriate mailing list, so that everybody can benefit from the communication. Thank you very much for your understanding.

Unit Tests

Exhaustive JUnit tests have been added over time to guarantee stability of this API. The tests may serve as an example on how to deal with some advanced methods of the class File and the side effects which are introduced by the static File.update() and File.umount() methods. Please note that although this code may help to understand some of TrueZIP's advanced features, it does not serve as a clean coding example (the objective of unit tests is more to try to break things and see them fail gracefully rather than to make them work).

Building from Source

The base directory for TrueZIP is a regular project directory for NetBeans Version 5.0. However, there is a central build script build.xml in the base directory for Apache ant, so you don't need to use NetBeans at all.

1. Prerequisites

This is what you need to build TrueZIP from its source:

1. Sun's Java Development Kit (JDK) Version 1.4.2 or higher.
2. Bouncy Castle's Lightweight Crypto API for JDK 1.4 Version 1.30 or higher (check for "lcrypto-jdk14-130" at https://www.bouncycastle.org/latest_releases.html).
3. Apache Ant Version 1.7 or higher. You can download it at https://ant.apache.org.
4. Optional: Any recent CVS client to check out the code from cvs.dev.java.net.

2. Downloading or checking out the Source Code

Once you have all the tools handy for the build process, you first need to either download or check out the project's source code from the CVS repository at cvs.dev.java.net.

You can download the TrueZIP source code now: Download truezip-src.zip. Once you have downloaded the source code, please unzip it to your favorite directory and go on with the next section.

If you would like to check out the source code from the CVS repository instead, consider the following first:

• The latest stable release is marked by a CVS tag of the form "TrueZIP*". The tag indicating the latest stable release can either be found in the following example or (if in doubt) in the history of the file build.xml in the base directory. This is the recommended procedure.
• The latest developer release including all intermediate commits is simply the HEAD revision of CVS and does not have a special tag. Note that this release may be only partially completed and thus is not necessarily stable because files are usually committed one-by-one (CVS does not support transactions).

Using a command line CVS tool, you can initiate the check out of the source by executing:

cd [Any empty directory will do]
cvs -d :pserver:guest@cvs.dev.java.net:/cvs login

Then execute either the following command to checkout the latest stable release (preferred):

cvs -d :pserver:guest@cvs.dev.java.net:/cvs checkout -r TrueZIP-6_6 truezip

... or execute the following command to check out the latest developer version (if you like living on the edge):

cvs -d :pserver:guest@cvs.dev.java.net:/cvs checkout truezip

Now you should have at least the subdirectories nbproject, src and the file build.xml in the formerly empty directory.

If you are using NetBeans Version 5 or higher, I recommend to use the built-in CVS client to checkout the truezip CVS module and then to open the complete module as an existing project. This provides more comfort throughout your development process.

For more information about CVS, please consult Google.

3. Configuring the Build Process

TrueZIP uses the regular ant script build.xml in its base directory to execute the build process. However, its configuration slightly depends on your IDE environment:

Environment	Configuration Steps
NetBeans 5.5	Right click on the project node TrueZIP and select Properties for editing.
Other	Please edit the file custom.properties in the base directory.

In either case, please check that your platform is correctly identified and that the debugging code generation option is set to your preferences.

4. Building

The following ant targets are provided to perform the build process:

• jar: Compiles all Java sources and creates the JAR file in the dist directory. This is the default.
• compile: Compiles all Java sources only.
• clean: Deletes all build Java classes and the JAR.
• javadoc: Creates Javadoc in the dist directory.

Again, if you are using NetBeans Version 5.5 or higher you can simply right click on the project node to execute any of the ant targets from the context menu.