Convert an image to a SceneKit Node

Question

I have a bit-map image:

( However this should work with any arbitrary image )

I want to take my image and make it a 3D SCNNode. I've accomplished that much with this code. That takes each pixel in the image and creates a SCNNode with a SCNBox geometry.

static inline SCNNode* NodeFromSprite(const UIImage* image) {
  SCNNode *node = [SCNNode node];
  CFDataRef pixelData = CGDataProviderCopyData(CGImageGetDataProvider(image.CGImage));
  const UInt8* data = CFDataGetBytePtr(pixelData);
  for (int x = 0; x < image.size.width; x++)
  {
    for (int y = 0; y < image.size.height; y++)
    {
      int pixelInfo = ((image.size.width * y) + x) * 4;
      UInt8 alpha = data[pixelInfo + 3];
      if (alpha > 3)
      {
        UInt8 red   = data[pixelInfo];
        UInt8 green = data[pixelInfo + 1];
        UInt8 blue  = data[pixelInfo + 2];
        UIColor *color = [UIColor colorWithRed:red/255.0f green:green/255.0f blue:blue/255.0f alpha:alpha/255.0f];
        SCNNode *pixel = [SCNNode node];
        pixel.geometry = [SCNBox boxWithWidth:1.001 height:1.001 length:1.001 chamferRadius:0];
        pixel.geometry.firstMaterial.diffuse.contents = color;
        pixel.position = SCNVector3Make(x - image.size.width / 2.0,
                                        y - image.size.height / 2.0,
                                        0);
        [node addChildNode:pixel];
      }
    }
  }
  CFRelease(pixelData);
  node = [node flattenedClone];
  //The image is upside down and I have no idea why.
  node.rotation = SCNVector4Make(1, 0, 0, M_PI);
  return node;
}

But the problem is that what I'm doing takes up way to much memory!

I'm trying to find a way to do this with less memory.

All Code and resources can be found at: https://github.com/KonradWright/KNodeFromSprite

Answer 1

Now you drawing each pixel as SCNBox of certain color, that means:

• one GL draw per box
• drawing of unnecessary two invisible faces between adjancent boxes
• drawing N of same 1x1x1 boxes in a row when one box of 1x1xN can be drawn

Seems like common Minecraft-like optimization problem:

1. Treat your image is 3-dimensional array (where depth is wanted image extrusion depth), each element representing cube voxel of certain color.
2. Use greedy meshing algorithm (demo) and custom SCNGeometry to create mesh for SceneKit node.

Pseudo-code for meshing algorithm that skips faces of adjancent cubes (simplier, but less effective than greedy meshing):

#define SIZE_X = 16; // image width
#define SIZE_Y = 16; // image height

// pixel data, 0 = transparent pixel
int data[SIZE_X][SIZE_Y];

// check if there is non-transparent neighbour at x, y
BOOL has_neighbour(x, y) {
    if (x < 0 || x >= SIZE_X || y < 0 || y >= SIZE_Y || data[x][y] == 0)
        return NO; // out of dimensions or transparent
    else
        return YES; 
}

void add_face(x, y orientation, color) {
    // add face at (x, y) with specified color and orientation = TOP, BOTTOM, LEFT, RIGHT, FRONT, BACK
    // can be (easier and slower) implemented with SCNPlane's: https://developer.apple.com/library/mac/documentation/SceneKit/Reference/SCNPlane_Class/index.html#//apple_ref/doc/uid/TP40012010-CLSCHSCNPlane-SW8
    // or (harder and faster) using Custom Geometry: https://github.com/d-ronnqvist/blogpost-codesample-CustomGeometry/blob/master/CustomGeometry/CustomGeometryView.m#L84 
}

for (x = 0; x < SIZE_X; x++) {
    for (y = 0; y < SIZE_Y; y++) {

        int color = data[x][y];
        // skip current pixel is transparent
        if (color == 0)
            continue;

        // check neighbour at top
        if (! has_neighbour(x, y + 1))
            add_face(x,y, TOP, );

        // check neighbour at bottom
        if (! has_neighbour(x, y - 1))
            add_face(x,y, BOTTOM);

        // check neighbour at bottom
        if (! has_neighbour(x - 1, y))
            add_face(x,y, LEFT);

        // check neighbour at bottom
        if (! has_neighbour(x, y - 1))
            add_face(x,y, RIGHT);

        // since array is 2D, front and back faces is always visible for non-transparent pixels
        add_face(x,y, FRONT);
        add_face(x,y, BACK);

    }
}

A lot of depends on input image. If it is not big and without wide variety of colors, it I would go with SCNNode adding SCNPlane's for visible faces and then flattenedClone()ing result.

Answer 2

An approach similar to the one proposed by Ef Dot:

1. To keep the number of draw calls as small as possible you want to keep the number of materials as small as possible. Here you will want one SCNMaterial per color.
2. To keep the number of draw calls as small as possible make sure that no two geometry elements (SCNGeometryElement) use the same material. In other words, use one geometry element per material (color).

So you will have to build a SCNGeometry that has N geometry elements and N materials where N is the number of distinct colors in your image.

1. For each color in you image build a polygon (or group of disjoint polygons) from all the pixels of that color
2. Triangulate each polygon (or group of polygons) and build a geometry element with that triangulation.
3. Build the geometry from the geometry elements.

If you don't feel comfortable with triangulating the polygons yourself your can leverage SCNShape.

1. For each polygon (or group of polygons) create a single UIBezierPath and a build a SCNShape with that.
2. Merge all the geometry sources of your shapes in a single source, and reuse the geometry elements to create a custom SCNGeometry

Note that some vertices will be duplicated if you use a collection of SCNShapes to build the geometry. With little effort you can make sure that no two vertices in your final source have the same position. Update the indexes in the geometry elements accordingly.

Answer 3

I can also direct you to this excellent GitHub repo by Nick Lockwood:

https://github.com/nicklockwood/FPSControls

It will show you how to generate the meshes as planes (instead of cubes) which is a fast way to achieve what you need for simple scenes using a "neighboring" check.

If you need large complex scenes, then I suggest you go for the solution proposed by Ef Dot using a greedy meshing algorithm.