Recent Changes - Search:

edit SideBar

Intel RealSense Facial Recognition

Previous versions of the Intel RealSense had facial recognition capabilities, but those capabilities were https://software.intel.com/en-us/realsense-sdk-windows-eol removed and are not available for the D400 series.

How Apple stomped on Intel's plans to make RealSense cameras detect emotions (PC World, March 2016)

librealsense OpenCV Wrapper

librealsense OpenCV Wrapper (GitHub)

"RealSense examples have been designed and tested with OpenCV 3.4, Working with latest OpenCV 4 requires minor code changes"

  1. "ImShow - Minimal OpenCV application for visualizing depth data"
  2. "GrabCuts - Simple background removal using the GrabCut algorithm"
  3. "Latency-Tool - Basic latency estimation using computer vision"
  4. "DNN - Intel RealSense camera used for real-time object-detection"

RealSense, OpenCV and Python

OpenCV and Python under Mac OS

bash-3.2$ python
Python 2.7.16 (default, Apr 22 2019, 18:46:22)
[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cv2.__version__
'4.1.0'
>>> quit
Use quit() or Ctrl-D (i.e. EOF) to exit
>>> quit();
bash-3.2$ which python

bash-3.2$ ls -l /usr/local/bin/python
lrwxr-xr-x 1 cxh admin 36 Apr 22 18:49 /usr/local/bin/python -> ../Cellar/python@2/2.7.16/bin/python
bash-3.2$

Get the Haar files:

wget https://raw.githubusercontent.com/opencv/opencv/master/data/haarcascades/haarcascade_eye.xml
wget https://raw.githubusercontent.com/opencv/opencv/master/data/haarcascades/haarcascade_frontalface_default.xml

Get the image:

wget https://upload.wikimedia.org/wikipedia/commons/9/96/Anjali-Sachin.jpg

Create faceDetector.py:

import numpy as np
import cv2 as cv
face_cascade = cv.CascadeClassifier('haarcascade_frontalface_default.xml')
eye_cascade = cv.CascadeClassifier('haarcascade_eye.xml')
img = cv.imread('Anjali-Sachin.jpg')
gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY)
faces = face_cascade.detectMultiScale(gray, 1.3, 5)
for (x,y,w,h) in faces:
    cv.rectangle(img,(x,y),(x+w,y+h),(255,0,0),2)
    roi_gray = gray[y:y+h, x:x+w]
    roi_color = img[y:y+h, x:x+w]
    eyes = eye_cascade.detectMultiScale(roi_gray)
    for (ex,ey,ew,eh) in eyes:
        cv.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(0,255,0),2)
cv.imshow('img',img)
cv.waitKey(0)
cv.destroyAllWindows()

Run it:

python faceDetector.py

Below is the result:


OpenCV Facial Recognition

Reading RealSense Data via .bag files

"Replace pipe.start() with:"

rs2::config cfg;
cfg.enable_device_from_file(<filename>);
pipe.start(cfg); // Load from file

read_bag_example.py has

    # Create pipeline
    pipeline = rs.pipeline()

    # Create a config object
    config = rs.config()
    # Tell config that we will use a recorded device from filem to be used by the pipeline through playback.
    rs.config.enable_device_from_file(config, args.input)
    # Configure the pipeline to stream the depth stream
    config.enable_stream(rs.stream.depth, 1280, 720, rs.format.z16, 30)

    # Start streaming from file
    pipeline.start(config)

If we copy an example by going to https://github.com/IntelRealSense/librealsense/blob/master/doc/sample-data.md and then clicking on the "Outdoors scene captured with D415 pre-production sample (Depth from Stereo)" example and downloading the the 403Mb (!) file.

Unfortunately, outdoors.bag does not contain any face data

Other file names:

  • single_depth_color_640x480.bag
  • D435i_Depth_and_IMU_Stands_still.bag

RealSense, OpenCV and Python under MacOS

Under MacOS, build librealsense with:

   cmake .. -DBUILD_EXAMPLES=true -DBUILD_WITH_OPENMP=false -DHWM_OVER_XU=false       -DBUILD_PYTHON_BINDINGS=true -DBUILD_PYTHON_DOCS=true -DPYTHON_EXECUTABLE=/opt/local/bin/python3.7 -G "Unix Makefiles"

What worked for me was

    bash-3.2$ cp build/wrappers/python/*.so /usr/local/lib
    bash-3.2$ cp build/*.dylib /usr/local/lib
    bash-3.2$ export PYTHONPATH=/usr/local/lib
    bash-3.2$ python3.7
    Python 3.7.5 (default, Oct 19 2019, 01:20:12)
    [Clang 10.0.1 (clang-1001.0.46.4)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import pyrealsense2
    >>> 
# First import library
import pyrealsense2 as rs
# Import Numpy for easy array manipulation
import numpy as np
# Import OpenCV for easy image rendering
import cv2 as cv
# Import os.path for file path manipulation
import os.path


face_cascade = cv.CascadeClassifier('haarcascade_frontalface_default.xml')
eye_cascade = cv.CascadeClassifier('haarcascade_eye.xml')

# img = cv.imread('Anjali-Sachin.jpg')

# Create pipeline
pipeline = rs.pipeline()

# Create a config object
config = rs.config()

# Tell config that we will use a recorded device from filem to be used by the pipeline through playback.
# FIXME: Unfortunately, @@outdoors.bag@@ does not contain any face data'.  So, we should create a bag file with some faces.
rs.config.enable_device_from_file(config, 'outdoors.bag')

# Configure the pipeline to stream the depth stream
#    config.enable_stream(rs.stream.depth, 1280, 720, rs.format.z16, 30)

# Start streaming from file
pipeline.start(config)

window_name = "Display Image";
cv.namedWindow(window_name, cv.WINDOW_AUTOSIZE);

# Create colorizer object
colorizer = rs.colorizer();

try:
    while True:
        # Get frameset of depth
        frames = pipeline.wait_for_frames()

        # Get depth frame
        # frame = frames.get_depth_frame()
        # Colorize depth frame to jet colormap
        # color_frame = colorizer.colorize(frame)

        color_frame = frames.get_color_frame()

        # Convert depth_frame to numpy array to render image in opencv
        img = np.asanyarray(color_frame.get_data())
        gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY)
        faces = face_cascade.detectMultiScale(gray, 1.3, 5)
        for (x,y,w,h) in faces:
            print(x, y, w, h)
            cv.rectangle(img,(x,y),(x+w,y+h),(255,0,0),2)
            roi_gray = gray[y:y+h, x:x+w]
            roi_color = img[y:y+h, x:x+w]
            eyes = eye_cascade.detectMultiScale(roi_gray)
            for (ex,ey,ew,eh) in eyes:
                cv.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(0,255,0),2)
        # Query frame size (width and height)
        # const int w = depth.as<rs2::video_frame>().get_width();
        # const int h = depth.as<rs2::video_frame>().get_height();

        # Create OpenCV matrix of size (w,h) from the colorized depth data
        # Mat image(Size(w, h), CV_8UC3, (void*)depth.get_data(), Mat::AUTO_STEP);                                                                                                                      
        # Update the window with new data
        cv.imshow(window_name, img);
        key = cv.waitKey(1)
        # if pressed escape exit program
        if key == 27:
            cv.destroyAllWindows()
            break
        # cv.imshow('img',img)                                                                                                                                                                          
        # cv.waitKey(0)
        # cv.destroyAllWindows()
finally:
    pipeline.stop()

Problems

ImportError: No module named pyrealsense2

bash-3.2$ python faceDetectionRealSense.py
Traceback (most recent call last):
File "faceDetectionRealSense.py", line 2, in <module>
import pyrealsense2 as rs
ImportError: No module named pyrealsense2

Solution: Copy the shared library, which unfortunately is .so instead of .dylibv

cp /usr/local/lib/pyrealsense2.2.23.cpython-37m-darwin.so pyrealsense2.so

Symbol not found: _PyInstanceMethod_Type

ealmac23:opencv root# python faceDetectionRealSense.py
Traceback (most recent call last):
File "faceDetectionRealSense.py", line 2, in <module>
import pyrealsense2 as rs
ImportError: dlopen(/Users/cxh/src/opencv/pyrealsense2.so, 2): Symbol not found: _PyInstanceMethod_Type
Referenced from: /Users/cxh/src/opencv/pyrealsense2.so
Expected in: flat namespace
in /Users/cxh/src/opepythonncv/pyrealsense2.so

PyBind FAQ says:

“ImportError: dynamic module does not define init function”
You are likely using an incompatible version of Python (for instance, the extension library was compiled against Python 2, while the interpreter is running on top of some version of Python 3, or vice versa).
“Symbol not found: __Py_ZeroStruct / _PyInstanceMethod_Type”
See the first answer.

Solution: Use Python3:

/usr/local/Cellar/python/3.7.3/bin/python3.7 faceDetectionRealSense.py

Couldn't resolve requests

There is no local camera, so I've been trying to read from .bag files.

The RealSense OpenCV C++ example (BGR_example.cpp) reads using color. If I change it to read from a file:

      //Add desired streams to configuration
      cfg.enable_stream(RS2_STREAM_COLOR, 640, 480, RS2_FORMAT_BGR8, 30);

+     cfg.enable_device_from_file("/Users/cxh/src/opencv-bak/outdoors.bag");
      //Instruct pipeline to start streaming with the requested configuration
      pipe.start(cfg);

and compile with

g++ -I/opt/local/include -std=c++11 BGR_sample.cpp -L/opt/local/lib -lrealsense2 -lopencv_core -lopencv_highgui -o BGR

Then I get an error:

bash-3.2$ ./BGR
libc++abi.dylib: terminating with uncaught exception of type rs2::error: Couldn't resolve requests
Abort trap: 6

The same happens with my three files:

  /Users/cxh/src/librealsense/unit-tests/resources/single_depth_color_640x480.bag
  /Users/cxh/src/opencv-bak/D435i_Depth_and_IMU_Stands_still.bag
  /Users/cxh/src/opencv-bak/outdoors.bag

However, the https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/read_bag_example.py read_bag_example.py works with outdoors.bag:

  python3.7 ./read_bag_example.py -i /Users/cxh/src/opencv-bak/outdoors.bag

and, read_bag_example.py works with d435i_walk_around.bag and d435i_walking.bag, which were downloaded from http://realsense-hw-public.s3.amazonaws.com/rs-tests/TestData/d435i_sample_data.zip

However, it fails with the other two .bag files:

bash-3.2$ python3.7 ./read_bag_example.py -i /Users/cxh/src/librealsense/unit-tests/resources/single_depth_color_640x480.bag
python3.7 ./read_bag_example.py -i /Users/cxh/src/librealsense/unit-tests/resources/single_depth_color_640x480.bag
Traceback (most recent call last):
File "./read_bag_example.py", line 46, in <module>
pipeline.start(config)
RuntimeError: Couldn't resolve requests
bash-3.2$ python3.7 ./read_bag_example.py -i /Users/cxh/src/opencv-bak/D435i_Depth_and_IMU_Stands_still.bag
python3.7 ./read_bag_example.py -i /Users/cxh/src/opencv-bak/D435i_Depth_and_IMU_Stands_still.bag
Traceback (most recent call last):
File "./read_bag_example.py", line 46, in <module>
pipeline.start(config)
RuntimeError: Couldn't resolve requests
bash-3.2$

Solution? Looking at librealsense docs for rs2::config Class Reference:

  • "void enable_stream (rs2_stream stream_type, int width, int height, rs2_format format=RS2_FORMAT_ANY, int framerate=0)"

So, editing BGR_Sample.cpp:

       cfg.enable_stream(RS2_STREAM_COLOR, 640, 480, RS2_FORMAT_ANY, 30);

seems to work with outdoors.bag!

.bag fileBGR_Sample.cppread_bag_example.cppopencv_viewer_example.cpp!BGR_Sample.cpp with enable_stream(... RS2_FORMAT_ANY)
D435i_Depth_and_IMU_Stands_still.bagnononono
d435i_walk_around.bagnoyesnoyes
d435i_walking.bagnoyesnoyes
object_detection.bagnoyesnono
outdoors.bagnoyesnoyes
single_depth_color_640x480.bagnononono

It seems like the problem has to do with combining enabling streams for depth and color and reading from a bag file. Interestingly, these worked:

   config.enable_stream(rs.stream.depth, 1280, 720, rs.format.z16, 30)
   config.enable_stream(rs.stream.color, 640, 480, rs.format.any, 30)

Also, using this for color works:

   config.enable_stream(rs.stream.color, 640, 480, rs.format.rgb8, 30)

TypeError: Expected Ptr<cv::UMat> for argument '%s'

Now, if I edit realsend_face_detect.py and remove the calls concerning depth and add

   config.enable_stream(rs.stream.color, 640, 480, rs.format.any, 30)
   ...   
        print(color_frame)
	print(type(color_frame))
        width = color_frame.get_width()
        height = color_frame.get_height()
        stride = color_frame.get_stride_in_bytes()
        bits_per_pixel = color_frame.get_bits_per_pixel()
        bytes_per_pixel = color_frame.get_bytes_per_pixel()
        print("w ", width, " h ", height, " stride: ", stride, " bits/pixel: ", bits_per_pixel, " bytes_per_pixel: ", bytes_per_pixel)

python3.7 realsense_face_detect.py ../webcam_face_detect/haarcascade_frontalface_default.xml

I get:

    <pyrealsense2.video_frame object at 0x10f35c270>
    <class 'pyrealsense2.video_frame'>
    w  640  h  480  stride:  1920  bits/pixel:  24  bytes_per_pixel:  3
    Traceback (most recent call last):
      File "realsense_face_detect.py", line 68, in <module>
    gray_frame = cv2.cvtColor(color_frame, cv2.COLOR_BGR2GRAY)
    TypeError: Expected Ptr<cv::UMat> for argument '%s'

Solution: I was using video frames instead of arrays.

## License: Apache 2.0. See LICENSE.
## Copyright(c) 2015-2017 Intel Corporation. All Rights Reserved.

## Based on https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/opencv_viewer_example.py

###############################################
##      Open CV and Numpy integration        ##
###############################################

import pyrealsense2 as rs
import numpy as np
import cv2
import sys, traceback

cascPath = "../webcam_face_detect/haarcascade_frontalface_default.xml"
faceCascade = cv2.CascadeClassifier(cascPath)

# Configure depth and color streams
pipeline = rs.pipeline()
config = rs.config()
config.enable_device_from_file("/Users/cxh/src/bags/outdoors.bag");

config.enable_stream(rs.stream.color, 640, 480, rs.format.any, 30)

pipeline.start(config);

try:
    while True:

        # Wait for a coherent pair of frames: depth and color
        frames = pipeline.wait_for_frames()
        color_frame = frames.get_color_frame()
        if not color_frame:
            continue

        # print(color_frame)
        # print(type(color_frame))

        # width = color_frame.get_width()
        # height = color_frame.get_height()
        # stride = color_frame.get_stride_in_bytes()
        # bits_per_pixel = color_frame.get_bits_per_pixel()
        # bytes_per_pixel = color_frame.get_bytes_per_pixel()

        # print("w ", width, " h ", height, " stride: ", stride, " bits/pixel: ", bits_per_pixel, " bytes_per_pixel: ", bytes_per_pixel)

        color_image = np.asanyarray(color_frame.get_data())
        gray_frame = cv2.cvtColor(color_image, cv2.COLOR_BGR2GRAY)
        faces = faceCascade.detectMultiScale(
            gray_frame,
            scaleFactor=1.1,
            minNeighbors=5,
            minSize=(30, 30),
            flags = cv2.CASCADE_SCALE_IMAGE
        )
        # Convert images to numpy arrays
        color_image = np.asanyarray(color_frame.get_data())
        # Draw a rectangle around the faces
        for (x, y, w, h) in faces:
            print("face: ", x, y, w, h)
            cv2.rectangle(color_image, (x, y), (x+w, y+h), (0, 255, 0), 2)

        # Show images
        cv2.namedWindow('RealSense', cv2.WINDOW_AUTOSIZE)
        cv2.imshow('RealSense', color_image)
        cv2.waitKey(1)

finally:

    # Stop streaming
    pipeline.stop()

OpenCV, Python and Intel RealSense under Windows

Under Windows, the Intel RealSense pyrealsense2 package seems to require Python 2 and will not work with Python 3, see Python RealSense

However, Support of pyrealsense2 package on Python 3.7 (Windows) states

  • "Pip support for Python 3.7 was apparently added in the recent 2.28.0 SDK."

So, Intel RealSense pyrealsense2 will work with both Python2.7 and Python 3.x

In addition, LabJackPython does install under Python3, despite Why does labjackpython package still enforce python=2.7.

So, Python 3.x can work. The 64-bit version of Python amd64 is preferred because that is what works with tensorflow. The 32-bit version of Python (x86) also works.

Installing the Intel RealSense Python pyrealsense2 module requires the pip command-line tool.

These instructions are for the 32-bit version of Python 2.7, which will not work with tensorflow

  1. https://pip.pypa.io/en/stable/installing/ says that pip is included in Python downloads from https://python.org.
    1. I selected the 32-bit "Windows x86 MSI installer" at https://www.python.org/ftp/python/2.7.17/python-2.7.17.msi
  2. Install Python in the default location (C:/Python27)
  3. The version of OpenCV that is installed requires the 0xa version of numpy, so I used pip:
    1. Start up Cygwin (a regular DOS shell would also work, just use \ instead of /)
    2. cd to the scripts directory and run pip:
      cd c:/Python27/Scripts
      ./pip.exe install numpy
  4. Install the latest OpenCV 2.x for Windows: https://sourceforge.net/projects/opencvlibrary/files/opencv-win/
    1. I installed https://sourceforge.net/projects/opencvlibrary/files/opencv-win/2.4.13/opencv-2.4.13.6-vc14.exe/download
    2. The default install location of OpenCV is on the Desktop.
    3. Go to the opencv/build/python/2.7/x86/ folder
    4. Copy cv2.pyd to C:/Python27/Lib/site-packages/
  5. Start up Python Ease (look at the Windows Start Menu)
    1. Try the import:
      >>> import cv2
      >>> cv2.__version__
      '2.4'13.6'
  6. Install the Intel RealSense Python interface by invoking pip.exe while in c:/Python27/Scripts:
    ./pip.exe install pyrealsense2
  7. To install LabJack, either use pip or install from the repo
    1. pip install labjackpython
    2. OR
    3. Install from the repo:
      1. Find the Cygwin setup binary, typically setup-x86_64.exe in the Desktop directory
      2. Use git to download from https://github.com/labjack:
        git clone https://github.com/labjack/LabJackPython.git
      3. Do the installation
        cd LabJackPython/; C:/Python27/python.exe setup.py install

Robot Operation System (ROS)

Using Robot Operating System (ROS) could be interesting

  • 3D Face Detection with RealSense D400 series and ROS (qiita.com)
  • RealSense (ros wiki)
  • realsense-ros - "These are packages for using Intel RealSense cameras (D400 series SR300 camera and T265 Tracking Module) with ROS." "ROS Indigo, on Ubuntu 14.04, and ROS Kinetic, on Ubuntu 16.04."
    • librealsense2 (ROS package) "SDK for communicating with Intel® RealSense™ SR300 and D400 cameras."
      • Installs only under Ubuntu 16 or 18 and ROS Indigo or Kinetic.
    • realsense2_camera (ROS package) "RealSense Camera package allowing access to Intel T265 Tracking module and SR300 and D400 3D cameras"
  • ROS Face Detection Tutorial (ROS package) - Uses OpenCV
  • face_detector (ROS package) - Uses stereo cameras to improve OpenCV face detection.
  • face_detection_tracking (ROS package) - OpenCV for face detection combined with tracking (last update in 2016)
  • opencv_apps (ROS package) - nodes that run various OpenCV systems including face detection. Seems active.
Edit - History - Print - Recent Changes - Search
Page last modified on December 16, 2019, at 03:26 AM