Framework for digital camera in Linux ( )
Framework for digital camera in Linux Dongsoo Kim ([email protected]) Heungjun Kim([email protected]) Samsung Electronics Contents Introducing mobile camera devices - SoC camera subsystem - Mobile camera module devices Framework for digital camera - Video4Linux2 and camera device - New APIs for high-end mobile camera devices - Dual camera framework for 3G handset Porting issues in SoC camera subsystem 2 Terms AP : Application Processor Camera interface : interface device which is syncing image data from external camera device and gives some commands to them Camera module : external camera device packaged with lens, sensor and ISP on it ISP : Image Signal Processor. Also supports lots of additional functions V4L2 : Video for Linux 2 3 This document contains some animations. Slideshow mode is recommended 4 What makes camera “a camera”? Exposure control - Aperture (Iris) - Shutter - ISO Strobe support - Various program mode : slow sync, red-eye reduction and on…. Lens control - Focus - Zoom 5 Introducing mobile camera devices 6 History of camera phone 1.3MP in 2003 First Camera Phone 5MP in2004 3MP 7MP with in 2004 2005 2MP in 12MP in 2009 in 10MP 2000 3.5MP 2006 7 Mobile camera module devices Old and low-end mobile camera modules Feature Exposure Lens Strobe Low end camera Aperture Not supported Shutter Just change frame rate ISO Low sensitivity & High noise level Focus Pan focus Zoom Digital zoom Just LED light most of time 8 Mobile camera module devices Brand new and high-end mobile camera modules Feature Exposure Lens Strobe High end camera Aperture Supported with lens packaging Shutter Supported with lens packaging ISO High sensitivity & Low noise level Focus Movable focus point (Auto/manual) Zoom Optical zoom & Digital zoom XENON flash is also supported 9 SoC camera subsystem What is SoC camera subsystem? ARM SoC solution Peripherals Camera interface Peripheral3 (USB) ARM Core External Camera module Peripheral2 (MMC) Peripheral1(Display) 10 SoC camera subsystem Request for data Image data in expected format Camera interface Command interface Resolution handling Data format handling (YUV/RAW/JPEG…) Buffer handling I2C RAW DATA Camera module Exposure control Lens control White balance control Effect control Face detect Zoom control Strobe control . . . 11 SoC camera subsystem Various ways of preview (Live view) in SoC camera subsystems DATA ARM core Camera Module DATA Camera Interface User Application M e m o r y COPY! Frame buffer OK Cancel 12 SoC camera subsystem Various ways of preview (Live view) in SoC camera subsystems User Application DATA ARM core Camera Module DATA Camera Interface M e m o r y D M A COPY! Save as file Through DMA Frame buffer OK Cancel 13 SoC camera subsystem How to capture still shots in SoC camera subsystem - Your camera module has a dedicated JPEG encoder on it Camera module ARM core DATA JPEG PROCESS MEMORY DATA RAW DATA PROCESS DATA Camera Interface User Application M e m o r y COPY! Save as JPEG file 14 SoC camera subsystem How to capture still shots in SoC camera subsystem - No JPEG encoder in your camera module User Application DATA ARM core Camera Module DATA Camera Interface M e m o r y COPY! RAW DATA Codec S/W Save as JPEG file 15 SoC camera subsystem How to take motion pictures in SoC camera subsystem COPY! DATA ARM core Camera Module DATA Camera Interface M e m o r y Frame buffer User Application OK Codec S/W Cancel Save as… COPY! 16 What we’ve got in Linux to control camera In Kernel V4L2 (AKA Video For Linux 2) - The second version of the Video For Linux API - Kernel interface for analog radio and video capture and output drivers In user space Multimedia middleware Gstreamer - Pipeline based multimedia framework written in the C programming language with the type system based on GObject OpenMAX - Cross-platform set of C-language programming interfaces that provides abstractions for routines especially useful for audio, video, and still images 17 Mobile camera module devices Is V4L2 enough for mobile digital camera modules? No way 18 Framework for digital camera 19 Video4Linux2 and camera device Camera subsystem in Linux kernel aspect V4L2 videobuf videobuf -core V4l2dev videobufsg-dma V4l2ioctl videodev V4l2-device V4l2-intdevice (old) V4l2subdev (new) Camera interface V4L2 driver Command interface Resolution handling Data format handling (YUV/RAW/JPEG…) Buffer handling Camera module V4L2 driver Exposure control Lens control White balance control Effect control Face detect Zoom control Strobe control . . . 20 Abstract view of V4L2 working with video capture device How does it work with V4L2? Application or Middleware Copy memory to userspace Open device VIDIOC_QUERYCAP VIDIOC_S_INPUT VIDIOC_QBUF VIDIOC_REQBUF VIDIOC_QBUF VIDIOC_DQBUF VIDIOC_QUERYBUF VIDIOC_STREAMON 21 New V4L2 APIs for digital camera Exposure control 22 New V4L2 APIs for exposure control Need for detailed control in exposure - We’ve got camera devices with mechanical shutter - Iris could be handled also - ISO controlled output is somehow useful Why not using enhanced exposure control like a regular digital camera? 23 New V4L2 APIs for exposure control What we’ve got in mainline V4L2 right now? #define V4L2_CID_EXPOSURE_AUTO (V4L2_CID_CAMERA_CLASS_BASE+1) enum v4l2_exposure_auto_type { V4L2_EXPOSURE_AUTO = 0, V4L2_EXPOSURE_MANUAL = 1, V4L2_EXPOSURE_SHUTTER_PRIORITY = 2, V4L2_EXPOSURE_APERTURE_PRIORITY = 3 }; #define V4L2_CID_EXPOSURE_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+2) #define V4L2_CID_EXPOSURE_AUTO_PRIORITY (V4L2_CID_CAMERA_CLASS_BASE+3) No way to control iris No way to control exposure metering method 24 New V4L2 APIs for exposure control Additional exposure control enum v4l2_exposure_auto_type { V4L2_EXPOSURE_AUTO = 0, V4L2_EXPOSURE_MANUAL = 1, V4L2_EXPOSURE_SHUTTER_PRIORITY = 2, V4L2_EXPOSURE_APERTURE_PRIORITY = 3, /* Additional features for digital camera */ V4L2_EXPOSURE_ISO_PRIORITY =4, + + }; #define V4L2_CID_EXPOSURE_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+2) #define V4L2_CID_EXPOSURE_AUTO_PRIORITY (V4L2_CID_CAMERA_CLASS_BASE+3) . . . +#define V4L2_CID_EXPOSURE_LOCK (V4L2_CID_CAMERA_CLASS_BASE+18) 25 New V4L2 APIs for exposure control +#define V4L2_CID_CAM_APERTURE(V4L2_CID_CAMERA_CLASS_BASE+19) +#define V4L2_CID_CAM_SHUTTER (V4L2_CID_CAMERA_CLASS_BASE+20) +#define V4L2_CID_CAM_ISO (V4L2_CID_CAMERA_CLASS_BASE+21) Supported aperture stages, shutter speed, and ISO speed could be different between every different camera module products Supported values for aperture, shutter and ISO should be made in V4L2_CTRL_TYPE_MENU { .id = V4L2_CID_CAM_APERTURE, .type = V4L2_CTRL_TYPE_MENU, .name = "Aperture", .minimum = 0, .maximum = 4, .step = 1, .default_value = 0, }, static const char *camera_iris_stages[] = { /* This module supports 5 Iris stages * on it's own but F number depends * how you package Lens module */ "2.8“, "4“, "5.6“, "8“, "11“, NULL }; 26 New V4L2 APIs for exposure control New exposure metering control +/* Exposure Methods */ +#define V4L2_CID_PHOTOMETRY (V4L2_CID_CAMERA_CLASS_BASE+17) +enum v4l2_photometry_mode { + V4L2_PHOTOMETRY_MULTISEG = 0, + V4L2_PHOTOMETRY_CWA = 1, + V4L2_PHOTOMETRY_SPOT = 2, + V4L2_PHOTOMETRY_AFSPOT = 3, +}; 27 New V4L2 APIs for digital camera Lens control 28 New V4L2 APIs for lens control Need for enhanced Focus mode API - Current V4L2 framework just slightly covers MANUAL & AUTO FOCUS mode for usb webcam - High-end camera modules support for various focus mode presets like • AF Macro • AF-S / AF-C For compatibility Focus control API should be defined 29 New V4L2 APIs for lens control Focus Mode & Enumeration values #define V4L2_CID_FOCUS_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+10) #define V4L2_CID_FOCUS_RELATIVE (V4L2_CID_CAMERA_CLASS_BASE+11) +#define V4L2_CID_FOCUS_MODE (V4L2_CID_CAMERA_CLASS_BASE+13) +/* Focus Methods */ +enum v4l2_focus_mode { + V4L2_FOCUS_AUTO + V4L2_FOCUS_MANUAL + V4L2_FOCUS_MACRO + V4L2_FOCUS_CONTINUOUS +}; = = = = 0, 1, 2, 3, 30 New V4L2 APIs for digital camera Object recognition 31 What is object detection? Object detection Gee! It recognized my face! Hello everyone 32 What should we consider? What do we set? - Detect mode - Type of object - How many objects to detect - Triggered action What do we get? - How many objects are detected - Detected object’s type - Detected object’s coordinates 33 New V4L2 APIs for object recognition New API for object recognition - New capability #define V4L2_CAP_ASYNCIO #define V4L2_CAP_STREAMING 0x02000000 /* async I/O */ 0x04000000 /* streaming I/O ioctls */ +#define V4L2_CAP_OBJ_RECOGNITION 0x10000000 - New IOCTL +#define VIDIOC_S_RECOGNITION _IOWR ('V', 85, struct v4l2_detect) +#define VIDIOC_G_RECOGNITION _IOR ('V', 86, struct v4l2_detect) 34 New V4L2 APIs for object recognition New API for object detection +/* Object detection and triggered actions */ +enum v4l2_recog_mode { + V4L2_RECOGNITION_MODE_OFF = 0, + V4L2_RECOGNITION_MODE_ON = 1, + V4L2_RECOGNITION_MODE_LOCK = 2, +}; + +enum v4l2_recog_action { + V4L2_RECOG_ACTION_NONE = 0,/* only detection */ + V4L2_RECOG_ACTION_BLINK = 1,/* Capture on blinking */ + V4L2_RECOG_ACTION_SMILE = 2,/* Capture on smiling */ +}; + +enum v4l2_recog_pattern { + V4L2_RECOG_PATTERN_FACE = 0; /* Face */ + V4L2_RECOG_PATTERN_HUMAN = 1; /* Human */ + V4L2_RECOG_PATTERN_CHAR = 2; /* Character */ +} 35 New V4L2 APIs for object recognition New API for object detection +struct v4l2_recog_rect { + enum v4l2_recog_pattern p; /* What pattern detected */ + struct v4l2_rect o; /* detected area */ + __u32 reserved[4]; +} + +struct v4l2_recognition_data { + __u32 detect_cnt; /* detected object counter */ + struct v4l2_recog_rect obj; /* detected area */ + __u32 reserved[4]; +}; + +struct v4l2_recognition { + enum v4l2_recog_mode mode; + enum v4l2_recog_pattern pattern; /* What pattern to detect */ + __u32 obj_num; /* How many object to detect */ + __u32 detect_idx; /* select detected object */ + struct v4l2_recog_data data; /* read only */ + enum v4l2_recognition_action action; + __u32 reserved[4]; +}; 36 What should we consider? What do we set? - Detect mode - How many objects to detect - Triggered action What do we get? - How many objects are detected - Detected object’s coordinates 37 New V4L2 APIs for object recognition New API for object recognition - New capability #define V4L2_CAP_ASYNCIO #define V4L2_CAP_STREAMING 0x02000000 /* async I/O */ 0x04000000 /* streaming I/O ioctls */ +#define V4L2_CAP_OBJ_RECOGNITION 0x10000000 - New IOCTL +#define VIDIOC_S_RECOGNITION _IOWR ('V', 85, struct v4l2_detect) +#define VIDIOC_G_RECOGNITION _IOR ('V', 86, struct v4l2_detect) 38 New V4L2 APIs for object recognition New API for object detection +/* Object detection and triggered actions */ +enum v4l2_recognition_mode { + V4L2_RECOGNITION_MODE_OFF = 0, + V4L2_RECOGNITION_MODE_ON = 1, + V4L2_RECOGNITION_MODE_LOCK = 2, +}; + +enum v4l2_recognition_action { + V4L2_RECOG_ACTION_NONE = 0,/* only detection */ + V4L2_RECOG_ACTION_BLINK = 1,/* Capture on blinking */ + V4L2_RECOG_ACTION_SMILE = 2,/* Capture on smiling */ +}; + +struct v4l2_recognition_data { + __u8 detect_cnt; /* detected object counter */ + struct v4l2_rect o; /* detected area */ +}; + +struct v4l2_recognition { + enum v4l2_recognition_mode mode; + __u8 obj_num; /* How many object to detect */ + __u8 detect_idx; /* select detected object */ + struct v4l2_recognition_data data; /* read only */ + enum v4l2_recognition_action action; +}; 39 New V4L2 APIs for digital camera Dual camera 40 Dual camera for 3G handset What is dual camera? Flip side camera : for still shots Front side camera : for special purpose Dual camera applications - Video telephony - Camera applications • Taking self portrait shots 41 Dual camera for 3G handset H/W restrictions in Application Processor’s camera peripheral interface Mega camera VGA camera Data path 42 Why should we make single device node? H/W restrictions in Application Processor’s camera peripheral interface “SINGLE” camera interface on Application Processor ! Only one camera could be handled at a time Only one camera could be using data pins at a time Each camera is different in Working MCLK Working resolution (SYNC) Working PCLK 43 What do we have in V4L2? We already have API for dual camera - Enumerate video inputs VIDIOC_ENUMINPUT - Query or select the current video input VIDIOC_G_INPUT VIDIOC_S_INPUT 44 What do we need to consider for dual camera? What do we need to consider? •Let user space application get noticed about camera input devices •Don’t forget VIDIOC_ENUMINPUT •Check out supported pixel format using VIDIOC_ENUM_FMT and VIDIOC_ENUM_FRAMESIZES •Context of each camera module •Capability : Supported color spaces, resolutions, additional functions… •Latest working status : resolution, color space, effect and so on •Context of camera interface •Working information of the camera module getting switched : Clock information and so on 45 Example of switching camera How to switch active camera Condition 1 How to enable camera module: Give power source Enable expecting camera Give MCLK Condition 2 Shared things Camera power enable pin Camera data pins MCLK Mega camera VGA camera Mega Enable VGA Enable Mega Reset VGA Reset CAM POWER MCLK DATA PINS CPU Condition 3 Dedicated things Camera enable pins Camera reset pins 46 Abstract work flow of switching camera Cam power off STREAMON Camera (A) initialize Start preview Camera (A) enable pin off Camera (B) Enable pin on Cam power on Switch to camera STREAMOFF (B) Stop preview G_INPUT/ S_INPUT Camera (B) initialize Start preview Camera interface reset & setting Stop preview Stop DMA Reset camera interface’s current setting Make a proper MCLK for camera (B) Setup resolution, color-space, DMA 47 Porting issues in SoC camera subsystem 48 I2C issue I2C protocol has start and stop condition Some camera modules generate unexpected noise until their power up sequence - Makes I2C bus in bus busy state 49 I2C issue How to prevent this malfunctioning issue Use a “noise free on power up” camera module OR Use a level shifter to block and ignore noise from camera module OR Change functionality of I2C pins while turning on camera module 50 Resolution issue All - about between camera interface and camera module Expected resolution from camera interface Served resolution from camera module Symptoms depend on camera interface Expecting resolution is bigger than served resolution - A “select time out” error in user space application or - Preview image is floating in some direction Serving resolution is bigger than expected resolution - A “select time out” error in user space application or - Preview angle looks like to be zoomed in 51 Color space issue Color order mismatched between camera interface and camera module 52 Lens control issue Preview freezing while checking lens response - Using while() loop while it checks response, preview will freeze until focusing job gets finished - If the ISP driver use the kernel thread when it check response from current lens status could make smooth preview on the LCD while lens is on focusing. 53 Lens control issue Solving preview freezing while checking lens status struct platform_isp { const struct platform_data *pdata; struct v4l2_int_device *v4l2_int_device; struct i2c_client *i2c_client; int reschk; . . . . . . . . . } static int ioctl_s_ctrl(struct v4l2_int_device *s, struct v4l2_control *vc) { struct platform_isp *isp = s->priv; struct i2c_client *client = isp->i2c_client; switch (vc->id) { case V4L2_CID_FOCUS_AUTO: isp->vfm = vc->value; if (vc->value == V4L2_CID_FOCUS_MACRO) { err = i2c_command_for_isp( client, focus_macro ); isp->reschk = 1; } If isp->reschk is Set, break; The kernel thread is active. } } 54 Lens control issues Solving preview freezing while checking lens status int pseudo_reschk_thread (void *data) Platform Data Vairable for isp, or static variable. { int err = 0; This function return when the ISP is ……… matched the focus(+ return) or do { timeout(- return). switch (isp->reschk) { case 1: err = i2c_command_for_isp(client, focus_check); if (err < 0) { isp->reschk = 1; /* i2c command function failed. */ } else { isp->reschk = 0; /* if succeed, reschk ended. */ } break; …… This is able to re-check } to the next check time. } while(1); return 0; } 55 JPEG data synchronization issues JPEG has no synchronization protocol when it gets transferred All about camera interface and camera module’s data synchronization - When camera interface supports JPEG sync • Follow user manual and make camera module to follow that protocol - When camera interface does not support JPEG sync • Make synchronization protocol: make JPEG data to be synchronized with VSYNC 56 Q&A 57 Thank you 58
© Copyright 2025