• Add DDQN
• Add prioritized sampling
• Improve explore strategy
• Fix minor issues

• Fix DiscreteRLPolicy set_state bug (a9a6381, b3c6a58)

• Refine RL workflow
  • Add **kwargs to support more problem setting (e.g., Graph based ones) (#589)
    • add **kwargs to RL models' forward funcs and _shape_check()
    • add **kwargs to RL policies' get_action related funcs and _post_check()
    • add **kwargs to choose_actions of AbsEnvSampler; remain it None in current sample() and eval()
  • Add detached loss to the return value of update_critic() and update_actor() of current TrainOps; add default False early_stop to update_actor() of current TrainOps (#589)
  • Refine random seed setting logic in RL workflow (#584)
  • Refine rollout workflow (#577) to support:
    • Run a specific number of steps in rollout
    • Run a specific number of episodes during evaluation with num_eval_episodes
    • Flexible metrics management during rollout with AbsEnvSampler.metrics
  • Add AbsEnvSampler.metrics to support flexible metrics management during roullout (#577)
  • Add Callback as a general interface to support customized operations in each phase of the workflow.
    • Two instances Checkpoint and MetricsRecorder are added. (#577)
    • Add customized_callbacks to RLComponentBundle. (#589)
  • Re-organize RL job's output paths. (#577)
  • Fix several RL algorithm bugs. (#577, #589)
• Replace the numpy data type with python common data type in whole project (#571)
• Add RL benchmark on Mujoco as a module to tests/, compared with spinning up benchmark, performance results can be found in tests/rl/performance.md (#575, #577, #583, #584)
• Other minor code refinements

Update build workflow to fix pymaro package failed in manylinux issue

• Refine decision event logic (#559)
  • Add DecisionEventPayload and ActionPayload as the base class for payloads of decision events & action events.
  • Add related type check during running.
  • Rename related variables under simulator/.
  • Refine the action handling logic. Modify Env and related BEs.
• Refine RL component bundle logic (#549)
  • Refine rl_component_bundle. Use a more straightforward & easier way to organize. In this new version, we do not need easyrl anymore.
  • Update related examples.
  • Fix bugs in distributed training. Add config YAML file for distributed training in CIM scenario.
  • Add rl_formulation.ipynb as an example.
• Fix CITI BIKE hello world dashboard (#555)
• Update data model doc (#554)
• Update requirements (#552, #553)

• RL Toolkit: ✨ A whole new design of RL Toolkit (#539)
• CLI: refactorization (#539)
• Backend: data frame/snapshot precision issue fix (#544)
• MISC: Whole repository code formating (#538, #547)
• Citi bike visualization tool hot fix (#543)

• Add higher Python version 3.8, 3.9 (#398)
• Refinement for some core modules, including more type hints, better coding style, small interface adjustment, potential bug fix
  • Core & Abstract Business Engine refinement (#392)
  • Event Buffer refinement (#389)
  • Interface updates for SimRandom to better support scenario buildings (#401, #400)
• CIM scenario refinement (#400)
  • Disable auto action type detection, i.e., requires for explicit ActionType (#399)
  • Data processing updates, including data generation part and real data loader part (#395)
  • Some testing program updates, and enlarge the test coverage (#395)
• Examples update
  • OnlineLP for Citi Bike: change the default solver from GLPK to CBC (#391)

Scenario/CIM:

• Fix the potential bug of vessel plan for some specific topologies after calling Env.reset() (Related issue: #385, Related PR: #387, #388)

Simulator/CIM Scenario:

• Fix the environment reset issue #385, and update the simulator random seed interfaces (#387)
• Fix the "Wrong future stop tick predictions" issue #384 (#386)

RL Toolkit:

• Add the arguments used by Actor to the Trajectory.on_env_feedback() interface (#373, #374)

Visualization Tool/CIM:

• Fix Geo vis IP address & SQL logic bugs (#352, #383)

Others:

• update dataclasses requirement to setup
• fix some typo in codes comments and online document

• CIM data container/data loader interface for real data mode updated (#372)
• README for pymaro description updated