DJL v0.13.0 brings the new TensorRT and Python engines, and updates the engines PyTorch to 1.9.0, ONNXRuntime to 1.9.0, PaddlePaddle to 2.0.2, and introduces several new features:

Key Features

• Introduces TensorRT engine
• Introduces Python engine which allows you to run Python scripts with DJL
• Upgrades PyTorch engine to 1.9.0 with CUDA 11.1 support
• Upgrades ONNXRuntime engine to 1.9.0 with UINT8 support
• Upgrades PaddlePaddle engine to 2.0.2
• Introduces the djl-bench snap package: sudo snap install djlbench --classic
• Introduces dynamic batch feature for djl-serving (#1154)
• DJL serving becomes a standalone repository: https://github.com/deepjavalibrary/djl-serving (#1170)
• Allows load ModelZoo model using url (#1120)
• Support .npy and .npz file format (#1131)
• djl-serving is available in dockerhub
• Publishes d2l book Chinese translation preview (chapter 1-5) : https://d2l-zh.djl.ai

Enhancement

• Introduces several new features in djl-serving:
  • Improves djl-serving API to make it easy to get HTTP headers (#1134)
  • Loads models on all GPUs at startup for djl-serving (#1132)
  • Enables asynchronous logging for djl-serving
  • Makes djl-serving access log in separate log file (#1150)
  • Adds configuration to support number worker threads for GPU inference (#1153)
  • Improves auto-scale algorithm for djl-serving (#1149)
• Introduces several new features in djl-bench:
  • Adds a command line option to djl-bench to generate NDList file (#1155)
  • Adds warmup to benchmark (#1152)
  • Improves djll-bench to support djl:// urls (#1146)
  • Adds support to benchmark on multiple GPUs (#1144)
  • Adds support to benchmark onnx on GPU machines (#1148)
  • Adds support to benchmark TensorRT models (#1257)
  • Adds support to benchmark Python models (#1267)
• Introduces several new features in PyTorch engine:
  • Supports PyTorch custom input data type with IValue (#1208)
• Introduces several new features in OnnxRuntime:
  • Adds UINT8 support for OnnxRuntime (#1271)
• Introduces several new features in PaddlePaddle:
  • Adds more model loading options for PaddlePaddle (#1173)
  • Adds load functionalities to PaddlePaddle (#1140)
  • Adds remove pass option to PaddlePaddle (#1141)
• Introduces several API improvements:
  • Adds missing NDList.get(String) API (#1194)
  • Adds support to directly load models from a TFHub url (#1231)
  • Improves repository API to support passing argument in the URL query string (#1139)
  • Avoids loading the default engine if it is not being used (#1136)
  • Improves IO by adding a buffer to read/write (#1135)
  • Improves NDArray.toString() debug mode performance (#1142)
  • Makes GPU device detection engine specific to avoid confusion when using multiple engines (#1138)

Documentation and examples

• Adds Style Transfer example with CycleGAN (#1180)

Breaking change

• Removes support for Apache MXNet 1.6.0
• Deprecates Device.getDevices() API - Use Engine.getDevices() instead
• Renames SimpleVocabulary to DefaultVocabulary

Bug Fixes

• Fixes broken link in documents
• Fixes TensorFlow NDArray was created on CPU instead of GPU bug (#1279)
• Fixes default image processing pipeline (#1268)
• Fixed XGBoost NDArray multiple read bug (#1239)
• Fixes platform matching bug (#1167)
• Fixes NullPointerException in NDArray.toString() (#1157)
• Fixes PaddlePaddle crash due to GC (#1162)
• Fixes NDArrayAdapter.getSparseFormat() unsupported bug (#1151)
• Fixes mixed device issue in multiple engine use case (#1123)
• Fixes handle duplicate plugin issue for djl-serving (#1108)
• Fixes XGBoost NDArray creation bug (#1109)
• Fixes runtime exception running benchmark arm machine(#1107)
• Fixes unregister model regression (#1101)

Contributors

This release is thanks to the following contributors:

• Akshay Rajvanshi (@aksrajvanshi)
• Aziz Zayed (@AzizZayed)
• Elchanan Haas (@ElchananHaas)
• Erik Bamberg (@ebamberg)
• Frank Liu (@frankfliu)
• Jake Lee (@stu1130)
• Kimi MA (@kimim)
• Paul Greyson
• Qing Lan (@lanking520)
• Raymond Liu (@raymondkhliu)
• Sindhu Somasundaram (@sindhuvahinis)
• Zach Kimberg (@zachgk)