notes/real_time_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Real-time demo\n",
    "\n",
    "One of the strengths of DSINE is that it runs in real-time. \n",
    "\n",
    "Here we explain how you can build your own real-time demo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2\n",
    "import torch \n",
    "\n",
    "import sys\n",
    "sys.path.append('../')\n",
    "\n",
    "from utils.demo_data import define_input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`InputStream` will load frames from some source. You can specify the source by setting `input` to the following:\n",
    "\n",
    "* `screen`: screenshot of your screen\n",
    "* `webcam`: webcam\n",
    "* `rs`: a realsense camera\n",
    "* `youtube`: extract frames from a Youtube video\n",
    "\n",
    "Besides `input`, you should also provide `device` where the loaded image would be sent to.\n",
    "\n",
    "Each option requires additional keyword arguments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `input = \"screen\"`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* For this option, you need to provide:\n",
    "  * `intrins`: \n",
    "    * Camera intrinsics as a torch tensor of shape (3, 3)\n",
    "    * If `None`, it will be assumed that the principal point is at the center and that the field-of-view is 60 degrees\n",
    "  * `top`, `left`, `width`, `height`:\n",
    "    * Section of the screen to capture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "input = 'screen'\n",
    "device = 0\n",
    "\n",
    "kwargs = dict(\n",
    "    intrins = None,\n",
    "    top = (1080-480) // 2,\n",
    "    left = (1920-640) // 2,\n",
    "    height = 480,\n",
    "    width = 640,\n",
    ")\n",
    "\n",
    "InputStream = define_input(input=input, device=device, **kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `input = \"webcam\"`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* For this option, you need to provide:\n",
    "  * `intrins`: \n",
    "    * Camera intrinsics as a torch tensor of shape (3, 3)\n",
    "    * If `None`, it will be assumed that the principal point is at the center and that the field-of-view is 60 degrees\n",
    "  * `new_width`:\n",
    "    * If the webcam image resolution is too low/high, you can set `new_width` to resize it\n",
    "    * By default, the image will not be resized\n",
    "  * `webcam_index`:\n",
    "    * OpenCV will find a video source based on the provided index\n",
    "    * Normally, `0` or `1` would work"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "input = 'webcam'\n",
    "device = 0\n",
    "\n",
    "kwargs = dict(\n",
    "    intrins = None,\n",
    "    new_width = -1,\n",
    "    webcam_index = 1,\n",
    ")\n",
    "\n",
    "InputStream = define_input(input=input, device=device, **kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `input = \"rs\"` (realsense)\n",
    "\n",
    "NOTE: install `pyrealsense2` by \n",
    "\n",
    "```\n",
    "python -m pip install pyrealsense2\n",
    "```\n",
    "\n",
    "* For this option, you need to provide:\n",
    "  * `enable_auto_exposure`: turn on/off auto exposure\n",
    "  * `enable_auto_white_balance`: turn on/off auto WB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "input = 'rs'\n",
    "device = 0\n",
    "\n",
    "kwargs = dict(\n",
    "    enable_auto_exposure = True,\n",
    "    enable_auto_white_balance = True,\n",
    ")\n",
    "\n",
    "InputStream = define_input(input=input, device=device, **kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `input = \"youtube\"`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE: Install `vidgear` and `yt_dlp` by\n",
    "\n",
    "```\n",
    "python -m pip install vidgear\n",
    "python -m pip install yt_dlp\n",
    "```\n",
    "\n",
    "* For this option, you need to provide:\n",
    "  * `intrins`: \n",
    "    * Camera intrinsics as a torch tensor of shape (3, 3)\n",
    "    * If `None`, it will be assumed that the principal point is at the center and that the field-of-view is 60 degrees\n",
    "  * `new_width`:\n",
    "    * The video will be resized to have this width\n",
    "  * `video_id`:\n",
    "    * Youtube video id (e.g. `https://www.youtube.com/watch?v=dQw4w9WgXcQ` $\\rightarrow$ `video_id=\"dQw4w9WgXcQ\"`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m17:33:06\u001b[0m :: \u001b[1;35m   Helper    \u001b[0m :: \u001b[1;36m  INFO  \u001b[0m :: \u001b[1;37mRunning VidGear Version: 0.3.2\u001b[0m\n",
      "\u001b[32m17:33:06\u001b[0m :: \u001b[1;35m   Helper    \u001b[0m :: \u001b[1;31m\u001b[2;33mWARNING \u001b[0m :: \u001b[1;37mGStreamer not found!\u001b[0m\n",
      "\u001b[32m17:33:06\u001b[0m :: \u001b[1;35m   Helper    \u001b[0m :: \u001b[1;33m DEBUG  \u001b[0m :: \u001b[1;37mSelecting `best` resolution for streams.\u001b[0m\n",
      "\u001b[32m17:33:06\u001b[0m :: \u001b[1;35m   CamGear   \u001b[0m :: \u001b[1;36m  INFO  \u001b[0m :: \u001b[1;37mVerifying Streaming URL using yt-dlp backend. Please wait...\u001b[0m\n",
      "\u001b[32m17:33:09\u001b[0m :: \u001b[1;35m   CamGear   \u001b[0m :: \u001b[1;36m  INFO  \u001b[0m :: \u001b[1;37m[Backend] :: Streaming URL is fully supported. Available Streams are: [144p, 240p, 360p, 480p, 720p, 1080p, best, worst]\u001b[0m\n",
      "\u001b[32m17:33:09\u001b[0m :: \u001b[1;35m   CamGear   \u001b[0m :: \u001b[1;33m DEBUG  \u001b[0m :: \u001b[1;37mUsing `best` resolution for streaming.\u001b[0m\n",
      "\u001b[32m17:33:09\u001b[0m :: \u001b[1;35m   CamGear   \u001b[0m :: \u001b[1;33m DEBUG  \u001b[0m :: \u001b[1;37mYouTube source ID: `dQw4w9WgXcQ`, Title: `Rick Astley - Never Gonna Give You Up (Official Music Video)`, Quality: `best`\u001b[0m\n",
      "\u001b[32m17:33:09\u001b[0m :: \u001b[1;35m   CamGear   \u001b[0m :: \u001b[1;33m DEBUG  \u001b[0m :: \u001b[1;37mEnabling Threaded Queue Mode for the current video source!\u001b[0m\n",
      "\u001b[32m17:33:10\u001b[0m :: \u001b[1;35m   Helper    \u001b[0m :: \u001b[1;31m\u001b[2;33mWARNING \u001b[0m :: \u001b[1;37mGStreamer not found!\u001b[0m\n",
      "\u001b[32m17:33:10\u001b[0m :: \u001b[1;35m   Helper    \u001b[0m :: \u001b[1;33m DEBUG  \u001b[0m :: \u001b[1;37mSelecting `best` resolution for streams.\u001b[0m\n",
      "\u001b[32m17:33:10\u001b[0m :: \u001b[1;35m   CamGear   \u001b[0m :: \u001b[1;36m  INFO  \u001b[0m :: \u001b[1;37mVerifying Streaming URL using yt-dlp backend. Please wait...\u001b[0m\n",
      "\u001b[32m17:33:12\u001b[0m :: \u001b[1;35m   CamGear   \u001b[0m :: \u001b[1;36m  INFO  \u001b[0m :: \u001b[1;37m[Backend] :: Streaming URL is fully supported. Available Streams are: [144p, 240p, 360p, 480p, 720p, 1080p, best, worst]\u001b[0m\n",
      "\u001b[32m17:33:12\u001b[0m :: \u001b[1;35m   CamGear   \u001b[0m :: \u001b[1;33m DEBUG  \u001b[0m :: \u001b[1;37mUsing `best` resolution for streaming.\u001b[0m\n",
      "\u001b[32m17:33:12\u001b[0m :: \u001b[1;35m   CamGear   \u001b[0m :: \u001b[1;33m DEBUG  \u001b[0m :: \u001b[1;37mYouTube source ID: `dQw4w9WgXcQ`, Title: `Rick Astley - Never Gonna Give You Up (Official Music Video)`, Quality: `best`\u001b[0m\n",
      "\u001b[32m17:33:12\u001b[0m :: \u001b[1;35m   CamGear   \u001b[0m :: \u001b[1;33m DEBUG  \u001b[0m :: \u001b[1;37mEnabling Threaded Queue Mode for the current video source!\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "input = 'youtube'\n",
    "device = 0\n",
    "\n",
    "kwargs = dict(\n",
    "    intrins = None,\n",
    "    new_width = 1640,\n",
    "    video_id = \"dQw4w9WgXcQ\",\n",
    ")\n",
    "\n",
    "InputStream = define_input(input=input, device=device, **kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building a demo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you have all the ingredients. The code below will load the next frame and display it in some window. Replace `out` with something that you want to visualize (e.g. a network output). You can get keyboard input using `cv2.waitKey` to do some action (e.g. pause/quit the demo)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "frame_name = 'insert your frame name here'\n",
    "cv2.namedWindow(frame_name, cv2.WINDOW_NORMAL)\n",
    "# uncomment below to get full-screen\n",
    "# cv2.setWindowProperty(frame_name, cv2.WND_PROP_FULLSCREEN, cv2.WINDOW_FULLSCREEN)\n",
    "\n",
    "pause = False\n",
    "\n",
    "with torch.no_grad():\n",
    "    while True:\n",
    "        if pause:\n",
    "                pass\n",
    "        else:\n",
    "            data_dict = InputStream.get_sample()\n",
    "            color_image = data_dict['color_image']\n",
    "\n",
    "            #↓↓↓↓\n",
    "            #NOTE: do some operation to the img and get the output\n",
    "            img = data_dict['img']\n",
    "            intrins = data_dict['intrins']\n",
    "\n",
    "            out = color_image\n",
    "            #↑↑↑↑\n",
    "\n",
    "            cv2.imshow(frame_name, out)\n",
    "\n",
    "        # keyboard input\n",
    "        k = cv2.waitKey(1)\n",
    "        if k == ord(' '):\n",
    "            pause = not pause\n",
    "        elif k == ord('q'):\n",
    "            break\n",
    "\n",
    "    cv2.destroyWindow(frame_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mono3D",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}