Hello_Depth_Perception 任务二：Project Tango采集深度感知数据

Java API Depth Perception Tutorial深度感知教程

Configuration 配置信息

In order to use depth perception, your TangoConfig must have KEY_BOOLEAN_DEPTH set to true. In the default TangoConfig, KEY_BOOLEAN_DEPTH is set to false.

为了使用深度感知，你的TangoConfig必须将KEY_BOOLEAN_DEPTH设为真。

 

try {
    mConfig = new TangoConfig();
    mConfig = mTango.getConfig(TangoConfig.CONFIG_TYPE_CURRENT);
    mConfig.putBoolean(TangoConfig.KEY_BOOLEAN_DEPTH, true);
} catch (TangoErrorException e) {
    // handle exception
}

Define the callback定义回调

The caller is responsible for allocating memory, which will be released after the callback function has finished.

调用方负责分配内存空间，该空间将在该回调函数结束之后释放。

 

private void setTangoListeners() {
    final ArrayList<TangoCoordinateFramePair> framePairs = new ArrayList<TangoCoordinateFramePair>();
    framePairs.add(new TangoCoordinateFramePair(
        TangoPoseData.COORDINATE_FRAME_START_OF_SERVICE,
        TangoPoseData.COORDINATE_FRAME_DEVICE));

    // Listen for new Tango data
    mTango.connectListener(framePairs, new OnTangoUpdateListener() {

        @Override
        public void onXyzIjAvailable(TangoXyzIjData arg0) {
            byte[] buffer = new byte[xyzIj.xyzCount * 3 * 4]; //文件流输入缓冲区
            FileInputStream fileStream = new FileInputStream(
                     xyzIj.xyzParcelFileDescriptor.getFileDescriptor()); //xyzIj的定义在哪里？输入xyzIj的文件路径
            try {
                fileStream.read(buffer,
                        xyzIj.xyzParcelFileDescriptorOffset, buffer.length);
                fileStream.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
            // Do not process the buffer inside the callback because不要处理该回调中的缓冲，因为
            // you will not receive any new data while it processes在处理中你接收不到任何新数据
        }

        @Override
        public void onPoseAvailable(final TangoPoseData pose) {
            // Process pose data from device with respect to start of service
        }

        @Override
        public void onTangoEvent(final TangoEvent event) {
            // This callback also has to be here
        }
    });
}

Define the onXYZijAvailable() callback. Do not do any expensive processing on the data within the callback; you will not receive new data until the callback returns.

定义onXYZijAvailable()回调。不要在回调中做任何高消耗的数据处理。你只有在回调返回时才能接收新数据。

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 3.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

上次更新日期：六月 9, 2016

Depth Perception

How it works

Depth Perception gives an application the ability to understand the distance to objects in the real world. To implement Depth Perception, manufacturers of Tango devices can choose among common depth technologies, including Structured Light, Time of Flight, and Stereo. Structured Light and Time of Flight require the use of an infrared (IR) projector and IR sensor; Stereo does not.

Depth Perception can be useful in a number of ways:

• A game might want to detect when the user is approaching a wall or other object in the area and have that be part of the gameplay.

• By combining Depth Perception with Motion Tracking, the device can measure distances between points in an area that aren't in the same frame.

• Depth data can be associated with the color image data to look up the color for a point cloud for texturing or meshing.

Usability tips

Current devices are designed to work best indoors at moderate distances (0.5 to 4 meters). This configuration gives good depth at a distance while balancing power requirements for IR illumination and depth processing. It may not be ideal for close-range object scanning or gesture detection.

For a device that relies on viewing IR light using its camera, there are some situations where accurate Depth Perception is difficult. Areas lit with light sources high in IR like sunlight or incandescent bulbs, or objects that do not reflect IR light, cannot be scanned well.

Point clouds

The Tango APIs provide a function to get depth data in the form of a point cloud. This format gives (x, y, z) coordinates for as many points in the scene as are possible to calculate. Each dimension is a floating point value recording the position of each point in meters in the coordinate frame of the depth-sensing camera.

For more information, see the TangoPointCloud struct API reference.

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 3.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

上次更新日期：二月 14, 2017

tangoxyzijdata-structure

Description:
This data structure holds information about a Tango Point cloud, including the timestamp and how many points are contained within the current Point cloud.

Note that Blueprint users are restricted in how they may use data- we suggest either using the "Get Single Point" function to access individual array points, or if iteration over all Depth points is required, using C++. For more information read here.

Fields:

• Timestamp [Float]: The seconds since tango service was started, when this structure was generated.
• Count [Integer]: The number of Points (FVectors) contained within the latest point cloud.

How do I match xyzIj sets to pose data?

I am trying to work with the Google Project Tango tablet, I currently am sending data from my tablet to Epic's Unreal Engine 4 and trying to visualize it properly. The orientation of the cloud data however is having consistent issues lining up with the orientation of the Tango's pose data.

Currently I am taking the pose from the same frame as the xyzIj data, but the resulting point clouds never line up quite right. I tried doing manual rotation and correction of the cloud data so that it matched up with the pose data, however the solution to rotating one set of xyzIj data does not correct the remaining sets either.

I am currently under the assumption that the poses I am using to match these sets are in error, but I am unsure as to how to find the correct one. I also find that using getPoseAtTime tends to produce similarly flawed results, but that may have something to do with the fact that I don't quite understand it's usage.

Is there a solution to this issue outside of hand correcting each data set that I have failed to realize, or does everyone deal with similar issues?

Which part of the xyzij data is the point cloud?

I want to graphically show the point cloud data to the user, and I'm wondering what part of the xyzIj data is the actual coordinates of the point cloud data. I assume it's XyzIj.xyz, but on the developers website, this call is described as "packed coordinate triplet". When I print it out it's just separate point numbers; does that mean it prints out an x point, y point, z point, x point, y point, etc... or am I understanding it wrong?

mPointCount = xyzIj.xyzCount;
    //Is this the coordinate points of the point cloud?
    FloatBuffer newPointCloudPoints = xyzIj.xyz;
    int i = 0;
    //Printing out all the points
    while(newPointCloudPoints.get(i) != 0) {
         Log.i(TAG, "point cloud " + i + " " + newPointCloudPoints.get(i));
         i++;
    }

Answering:

Each "point" has three entries in the buffer.

The first index of the float buffer is x, second is y, third is z. The sequence repeats for every point.

for (int i = 0; i < xyzIjData.xyzCount; i += 3) {
            float x = xyz.get(i);
            float y = xyz.get(i + 1);
            float z = xyz.get(i + 2);
        }

Project Tango: Depthmap Transformation from XYZij data

I'm currently trying to filter the depth information using OpenCV. For that reason I need to transform Project Tango's depth information XYZij into a image like depthmap. (Like the output of Microsoft Kinect) Unfortunately the official APIs lacking the ij part of XYZij. That's why I'm trying to project the XYZ part using the camera intrinsics projection, wich is explained in the official C API Dokumentation. My current approach looks like this:

    float fx = static_cast<float>(ccIntrinsics.fx);
    float fy = static_cast<float>(ccIntrinsics.fy);
    float cx = static_cast<float>(ccIntrinsics.cx);
    float cy = static_cast<float>(ccIntrinsics.cy);

    float k1 = static_cast<float>(ccIntrinsics.distortion[0]);
    float k2 = static_cast<float>(ccIntrinsics.distortion[1]);
    float k3 = static_cast<float>(ccIntrinsics.distortion[2]);

    for (int k = 0; k < xyz_ij->xyz_count; ++k) {

        float X = xyz_ij->xyz[k][0];
        float Y = xyz_ij->xyz[k][1];
        float Z = xyz_ij->xyz[k][2];

        float ru = sqrt((pow(X, 2) + pow(Y, 2)) / pow(Z, 2));
        float rd = ru + k1 * pow(ru, 3) + k2 * pow(ru, 5) + k3 * pow(ru, 7);

        int x = X / Z * fx * rd / ru + cx;
        int y = X / Z * fy * rd / ru + cy;

        // drawing into OpenCV Mat in red
        depth.at<cv::Vec3b>(x, y)[0] = 240;

    }

The resulting depthmap can be seen in the lower right corner. But it seems that this calculation result in a linear representation ... Does anyone has already done something similar? Are the XYZ points already correct positioned for this projection?

answering:

up vote0down voteaccepted

I have actually found a solution ... Just skipped the distortion calculation like they do in the rgb-depth-sync-example. My code now looks like this: float fx = static_cast<float>(ccIntrinsics.fx); float fy = static_cast<float>(ccIntrinsics.fy); float cx = static_cast<float>(ccIntrinsics.cx); float cy = static_cast<float>(ccIntrinsics.cy); int width = static_cast<int>(ccIntrinsics.width); int height = static_cast<int>(ccIntrinsics.height); for (int k = 0; k < xyz_ij->xyz_count; ++k) { float X = xyz_ij->xyz[k * 3][0]; float Y = xyz_ij->xyz[k * 3][1]; float Z = xyz_ij->xyz[k * 3][2]; int x = static_cast<int>(fx * (X / Z) + cx); int y = static_cast<int>(fy * (Y / Z) + cy); uint8_t depth_value = UCHAR_MAX - ((Z * 1000) * UCHAR_MAX / 4500); cv::Point point(y % height, x % width); line(depth, point, point, cv::Scalar(depth_value, depth_value, depth_value), 4.5); } And the working OpenCV result looks like this:

>>Google Tango初学者教程

>>Hello_Motion_Tracking 任务一：Project Tango采集运动追踪数据