Stenciling using OpenGl ES 2.0

Question

I am trying to figure out a way to cut out a certain region of a background texture such that a certain custom pattern is not rendered on the screen for that background. For example:

This square can be any pattern. I am using Frame Buffer Object and Stencil Buffer to achieve this kind of effect. Here is the code:

fbo.begin();
//Disables ColorMask and DepthMask so that all the rendering is done on the Stencil Buffer
Gdx.gl20.glColorMask(false, false, false, false);
Gdx.gl20.glDepthMask(false);
Gdx.gl20.glEnable(GL20.GL_STENCIL_TEST);
Gdx.gl20.glStencilFunc(GL20.GL_ALWAYS, 1, 0xFFFFFFFF);
Gdx.gl20.glStencilOp(GL20.GL_REPLACE, GL20.GL_REPLACE, GL20.GL_REPLACE);

stage.getSpriteBatch().begin();
rHeart.draw(stage.getSpriteBatch(), 1); //Draws the required pattern on the stencil buffer

//Enables the ColorMask and DepthMask to resume normal rendering
Gdx.gl20.glColorMask(true, true, true, true);
Gdx.gl20.glDepthMask(true);

Gdx.gl20.glStencilFunc(GL20.GL_EQUAL, 1, 0xFFFFFFFF);
Gdx.gl20.glStencilOp(GL20.GL_KEEP, GL20.GL_KEEP, GL20.GL_KEEP);

background.draw(stage.getSpriteBatch(), 1); //Draws the background such that the background is not rendered on the required pattern, leaving that area black.

stage.getSpriteBatch().end();
Gdx.gl20.glDisable(GL20.GL_STENCIL_TEST);
fbo.end();

However this is not working at all. How am I supposed to do this using Stencil Buffers? I am also facing some difficulty understanding glStencilFunc and glStencilOp. It would be very helpful if anyone can shed some light on these two.

UPDATE: I have also tried producing something of the same kind using glColorMask. Here is the code:

Gdx.gl20.glClearColor(0, 0, 0, 0);
stage.draw();
FrameBuffer.clearAllFrameBuffers(Gdx.app);
fbo1.begin();
Gdx.gl20.glClearColor(0, 0, 0, 0);
batch.begin();
rubber.draw(batch, 1);
Gdx.gl20.glColorMask(false, false, false, true);
coverHeart.draw(batch, 1);
Gdx.gl20.glColorMask(true, true, true, false);
batch.end();        
fbo1.end();

toDrawHeart = new Image(new TextureRegion(fbo1.getColorBufferTexture()));
batch.begin();
toDrawHeart.draw(batch, 1);
batch.end();

This code is producing this:

Instead of something like this: (Ignore the windows sizes and colour tones)

Note: I am using the libgdx library.

Answer 1

While drawing to a SpriteBatch, state changes are ignored, until end() is called. If you want to use stenciling with SpriteBatch, you'll need to break up the batch drawing. One thing, I've left out FBOs, but that shouldn't make a difference.

@Override
public void create() {      
    camera = new OrthographicCamera(1, 1);
    batch = new SpriteBatch();

    texture = new Texture(Gdx.files.internal("data/badlogic.jpg"));
    texture.setFilter(TextureFilter.Linear, TextureFilter.Linear);

    TextureRegion region = new TextureRegion(texture, 0, 0, 256, 256);

    sprite = new Sprite(region);
    sprite.setSize(1f, 1f);
    sprite.setPosition(-0.5f, -0.5f);

    spriteUpsideDown = new Sprite(new TextureRegion(texture, 1f, 1f, 0f, 0f));
    spriteUpsideDown.setSize(1f, 1f);
    spriteUpsideDown.setPosition(-0.5f, -0.5f);

    pattern = new Sprite(region);
    pattern.setSize(0.5f, 0.5f);
    pattern.setPosition(-0.25f, -0.25f);

    << Set Input Processor >>
}

The input processor allows to set two boolean flags breakBatch1 and breakBatch2 via keyboard (libgdx on desktop), which are used to break the SpriteBatch drawing.

@Override
public void render() {      
    Gdx.gl.glClearColor(1, 1, 1, 1);
    Gdx.gl.glClear(GL10.GL_COLOR_BUFFER_BIT | GL10.GL_STENCIL_BUFFER_BIT);
    batch.setProjectionMatrix(camera.combined);

    // setup drawing to stencil buffer
    Gdx.gl20.glEnable(GL20.GL_STENCIL_TEST);
    Gdx.gl20.glStencilFunc(GL20.GL_ALWAYS, 0x1, 0xffffffff);
    Gdx.gl20.glStencilOp(GL20.GL_REPLACE, GL20.GL_REPLACE, GL20.GL_REPLACE);
    Gdx.gl20.glColorMask(false, false, false, false);

    // draw base pattern
    batch.begin();
    pattern.draw(batch);

    if(breakBatch1) { batch.end(); batch.begin(); }

    // fix stencil buffer, enable color buffer
    Gdx.gl20.glColorMask(true, true, true, true);
    Gdx.gl20.glStencilOp(GL20.GL_KEEP, GL20.GL_KEEP, GL20.GL_KEEP);

    // draw where pattern has NOT been drawn
    Gdx.gl20.glStencilFunc(GL20.GL_NOTEQUAL, 0x1, 0xff);
    sprite.draw(batch);

    if(breakBatch2) { batch.end(); batch.begin(); }

    // draw where pattern HAS been drawn.
    Gdx.gl20.glStencilFunc(GL20.GL_EQUAL, 0x1, 0xff);
    spriteUpsideDown.draw(batch);
    batch.end();
}

Answer 2

Gdx.gl20.glStencilFunc(GL20.GL_REPLACE, GL20.GL_REPLACE, GL20.GL_REPLACE);

These are not the right arguments to glStencilFunc. I think you mean glStencilOp here.

You need to use glGetError in your code, it will alert you to these kinds of errors.

Answer 3

I believe your problem is that your initial GL_REPLACE stencil operation is applied to all the drawn pixels by your rHeart.draw regardless of the shape of any texture applied on the quad.

Thus, the stencil value is applied to every pixel of your quads which gives your problem.

If the texture applied on your quad has an alpha channel, as GL_ALPHA_TEST is not supported, you could setup your shader to discard the totally transparent pixels, preventting them from being drawn to the stencil buffer.