ESIM Web App Playground

A Web App for Generating Events from Upsampled Video Frames & Visualization of Events from Live Video of a Webcam!

Siddhant Sahu | Daniel Gehrig | Nico Messikommer

Table of Contents

About The Project
Getting Started
Roadmap

About The Project

The goal of this project is to make ESIM and VID2E available to researchers who do not possess a real event camera. For this we want to deploy VID2E and ESIM as an interactive web app. The app should be easy to use and have the following functional requirements:

Generation of events from video through dragging and dropping a video into the browser. The resulting events should be downloadable as raw events and rendered video.
Incorporate video interpolation via an existing video interpolation method (e.g. Super SloMo) before event generation with ESIM.
Visualization and inspection of the event stream in the browser.
The event generation should be configurable by changing ESIM parameters.

(back to top)

Getting Started

Follow these steps in order to make this app run in your local system.

This code base assumes you have access to a NVIDIA GPU in your system with proper drivers installed along with the cuda-toolkit version 10.1.

Prerequisites & Installation

Make sure that you have followed the Instalation with Anaconda Instruction and have created the vid2e Conda environment with the esim_torch package installed in this environment.

Activate the Conda environment

conda activate vid2e
conda list | grep 'esim'

The above command should output the following

esim-cuda                 0.0.0                    pypi_0    pypi
esim-py                   0.0.1                    pypi_0    pypi

Install the streamlit package in this Conda environment, along with a couple of additional packages.

pip install streamlit stqdm numba h5py

Running the Web App Locally

If you want to run this app locally on your system then follow these steps.

Run the App

streamlit run web_app.py

Then you can access the app in Local URL: https://localhost:8501

When using the 'video upload' feature, the uploaded file is saved in the 'data/original/video_upload' directory. Subsequently, upon clicking the 'Generate Events' button, two new directory, namely, 'data/upsampled/video_upload' and 'data/events/video_upload' are created. Additionally, after the process has finished, you will see a 'Download Events' button, you can click this button to download the events. Events can be downloaded in three different formats: as .h5 file, as .npz or as a rendered video.

(back to top)

Running the Web App on Google Colab

To run this web app on Google Colab follow these steps in order.

Make sure you select the Run time type as GPU in Google Colab

Register and create an account in the remote.it.
Install remote.it service in your Google Colab instance.

!curl -LkO https://raw.githubusercontent.com/remoteit/installer/master/scripts/auto-install.sh
! chmod +x ./auto-install.sh
! sudo ./auto-install.sh

Install Miniconda in your Google Colab Instance

! wget https://repo.anaconda.com/miniconda/Miniconda3-py37_4.8.2-Linux-x86_64.sh
! chmod +x Miniconda3-py37_4.8.2-Linux-x86_64.sh
! bash ./Miniconda3-py37_4.8.2-Linux-x86_64.sh -b -f -p /usr/local

Update Python Path

import sys
sys.path.insert(0,'/usr/local/lib/python3.7/site-packages/')

Install Streamlit package

!pip install streamlit

Run the remote.it service

!sudo connectd_installer

In the menu selection choose 1
Enter your remote.it email address and password
Enter the name of your device esim
Choose 1 Attach/reinstall a remote.it Service to an application
Protocol Selection Menu, Choose 2, Web (HTTP) on port 80
Enter a name for this remote.it service, esim
Main menu, choose, 5 Exit

Mount your Google Drive to this Google Colab Instance
Change directory to where to want this project to reside.

Clone the repo

git clone https://github.com/uzh-rpg/rpg_vid2e.git

Check if the esim_torch package is installed

import esim
print(esim_torch.__path__)

Built the esim_torch package with pybind11

pip install esim_torch/

Change directory inside web_app and run the app on port number 80 as a webservice which can then be accessed for the remote.it service.

!streamlit run --server.port 80 web_app.py&>/dev/null&

Finaly, you can navigate your remote.it dashboard and open these service and use the app!

Note: Remeber when running this app on Google Colab you need to change the paths for input and output directories for the upsampling to the specific directory in Google Drive and also other paths and esim import.

(back to top)

Roadmap

Add video upload dragging and dropping a video into the browser.
Add video interpolation via an existing video interpolation method (e.g. Super SloMo) before event generation with ESIM.
Add events download button (output format)
- HDF5
- NPZ
- Rendered Video
Add webcam functionality
Run this app on Google Colab

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!