Converting a pointcloud to a depth/multi channel image

Question

I have a pointcloud generated by scanning a planar surface using stereo cameras. I have generated features such as normals, fpfh etc and using this information I want to classify areas in the pointcloud. To enable the use of more traditional CNN approaches I want to convert this pointcloud to a multi-channel image in opencv. I have the pointcloud collapsed to the XY plane, and aligned to the X and Y axes so that I can create a bounding box for the image.

I am looking for ideas on how to proceed further with the mapping from points to pixels. Specifically, I am confused about the image size, and how to go about filling in each pixel with the appropriate data. (Overlapping points would be averaged out, empty ones will be labelled accordingly). Since this is an unorganized pointcloud, I do not have camera parameters to use, and I guess PCL's RangImage class would not work in my case.

Any help is appreciated!

Answer 1

Try creating an empty cv::Mat of predetermined size first. Then iterate through every pixel of that Mat to determine what value it should take.

Here is some code which does something similar to what you were describing:

cv::Mat makeImageFromPointCloud(pcl::PointCloud<pcl::PointXYZI>::Ptr cloud, std::string dimensionToRemove, float stepSize1, float stepSize2)
{
    pcl::PointXYZI cloudMin, cloudMax;
    pcl::getMinMax3D(*cloud, cloudMin, cloudMax);

    std::string dimen1, dimen2;
    float dimen1Max, dimen1Min, dimen2Min, dimen2Max;
    if (dimensionToRemove == "x")
    {
        dimen1 = "y";
        dimen2 = "z";
        dimen1Min = cloudMin.y;
        dimen1Max = cloudMax.y;
        dimen2Min = cloudMin.z;
        dimen2Max = cloudMax.z;
    }
    else if (dimensionToRemove == "y")
    {
        dimen1 = "x";
        dimen2 = "z";
        dimen1Min = cloudMin.x;
        dimen1Max = cloudMax.x;
        dimen2Min = cloudMin.z;
        dimen2Max = cloudMax.z;
    }
    else if (dimensionToRemove == "z")
    {
        dimen1 = "x";
        dimen2 = "y";
        dimen1Min = cloudMin.x;
        dimen1Max = cloudMax.x;
        dimen2Min = cloudMin.y;
        dimen2Max = cloudMax.y;
    }

    std::vector<std::vector<int>> pointCountGrid;
    int maxPoints = 0;

    std::vector<pcl::PointCloud<pcl::PointXYZI>::Ptr> grid;

    for (float i = dimen1Min; i < dimen1Max; i += stepSize1)
    {
        pcl::PointCloud<pcl::PointXYZI>::Ptr slice = passThroughFilter1D(cloud, dimen1, i, i + stepSize1);
        grid.push_back(slice);

        std::vector<int> slicePointCount;

        for (float j = dimen2Min; j < dimen2Max; j += stepSize2)
        {
            pcl::PointCloud<pcl::PointXYZI>::Ptr grid_cell = passThroughFilter1D(slice, dimen2, j, j + stepSize2);

            int gridSize = grid_cell->size();
            slicePointCount.push_back(gridSize);

            if (gridSize > maxPoints)
            {
                maxPoints = gridSize;
            }
        }
        pointCountGrid.push_back(slicePointCount);
    }

    cv::Mat mat(static_cast<int>(pointCountGrid.size()), static_cast<int>(pointCountGrid.at(0).size()), CV_8UC1);
    mat = cv::Scalar(0);

    for (int i = 0; i < mat.rows; ++i)
    {
        for (int j = 0; j < mat.cols; ++j)
        {
            int pointCount = pointCountGrid.at(i).at(j);
            float percentOfMax = (pointCount + 0.0) / (maxPoints + 0.0);
            int intensity = percentOfMax * 255;

            mat.at<uchar>(i, j) = intensity;
        }
    }

    return mat;
}