Advanced camera and sensor control

The Khronos OpenKCam working group was founded to create an open, royalty-free standard for advanced, low-level control of mobile and embedded cameras and sensors:

• Mobile and embedded devices are increasingly being equipped with the sensors and processing power for advanced camera-based applications such as computational photography, face and gesture processing, augmented reality and 3D object and scene reconstruction;
• While Khronos is defining APIs for vision and image processing the industry still lacks a camera API with low-level control of the camera sensor, lens and flash to generate the input image stream needed by cutting-edge computational photography and computer vision;
• The OpenKCam working group sought to drive industry consensus to create a cross-platform API that provides functionality such as: burst control over sensor, flash and lens, system wide time stamping of sensor samples, multiple sensor control, output format and resolution selection, region of interest extraction, and consistent device and frame metadata.

The OpenKCam working group collaborated as appropriate with other Khronos working groups, including:

• Potential use of EGL for data streaming and accessing cross API sensor synchronization time stamps;
• Synchronized operation with the StreamInput sensor fusion API for motion aware vision apps such as augmented reality;
• Efficient interoperation with, and fulfillment of key use cases for, OpenVX, OpenGL, OpenGL ES and OpenCL when used for image and vision processing.

Advanced Camera Control Use Cases

• High-dynamic range (HDR) and computational flash photography
  • High-speed burst with individual frame control over exposure and flash
• Rolling shutter elimination
  • High-precision intra-frame synchronization between camera and motion sensor
• HDR Panorama, photo-spheres
  • Continuous frame capture with constant exposure and white balance
• Subject isolation and depth detection
  • High-speed burst with individual frame control over focus
• Time-of-flight or structured light depth camera processing
  • Aligned stacking of data from multiple sensors
• Augmented Reality
  • 60Hz, low-latency capture with motion sensor synchronization
  • Multiple Region of Interest (ROI) capture
  • Multiple sensors for scene scaling
  • Detailed feedback on camera operation per frame

Precursor APIs for Camera Control Initiative

• FCAM – Open source project
  • Capture of stream of camera images with precision control
    • A pipeline that converts requests into image stream
    • All parameters packed into the requests - no global state
    • Programmer has full control over sensor settings for each frame in stream
  • Control over focus and flash
    • No hidden daemon running
  • Control ISP
    • Can access supplemental statistics from ISP if available

• Android New Camera HAL (2013)
  • Uses some of these concepts

Potential OpenKCam API Functionality

• Burst Sensor Control
  • Exposure, time, gain, CFA pattern …
• Burst Lens control
  • Target focus distance, aperture, focal length, position state …
• Burst Flash Control
  • Brightness, duration, burst, activity state …
• ISP Control
  • Demosaic-ing quality, denoising quality, 3A, CCM, Gamma, color space …
• Output(s) Control
  • Resolution, ROI extraction, quality, format (including Bayer, YUV) …
• Frame timestamp
  • To be used to synch with motion sensors
• Parameter access
  • Standard-defined parameters AND Vendor-specific extensions registry
• Feedback Statistics
  • Histogram, sharpness map …
• Multi-sensor control
  • Synchronization, master sensor controlling other sensors, image stacking
• Metadata
  • Per frame: Focal length (fx, fy), principal point (cx, cy), skew (s), image resolution (h, w), exposure
  • Per device: Cameras and sensors physical layout, calibration and lens distortion

OpenKCam API Usage