multisampling and fragment shader

Question

Multisampling does not seem to work for fragments generated by a fragment shader. In the example below, the fragment shader is used to produce a check-board procedural texture. The outer edges of the square are properly antialiased, but the inner edges of the procedural texture are not.

Is the fragment shader evaluated only per pixel? Or are the texture coordinates the same for each fragment of a given pixel?

Below is the code and the image shows its output (notice that the procedural edges —between white and gray square— are not antialiased, whereas geometry edges —between black and white/gray— are):

#!/usr/bin/env python3
# -*- coding: utf-8 -*-


# imports ####################################################################

import sys

from OpenGL.GLUT import *
from OpenGL.GL import *


# display ####################################################################

def reshape(width, height):
    """window reshape callback."""
    glViewport(0, 0, width, height)

    glMatrixMode(GL_PROJECTION)
    glLoadIdentity()
    r = float(min(width, height))
    w, h = 2*width/r, 2*height/r
    glOrtho(-w, w, -h, h, -1, 1)

    glMatrixMode(GL_MODELVIEW)
    glLoadIdentity()
    glRotate(45, 0, 0, 1)

def display():
    """window redisplay callback."""
    glClear(GL_COLOR_BUFFER_BIT)
    glBegin(GL_TRIANGLE_STRIP)
    for x in [-1, 1]:
        for y in [-1, 1]:
            glTexCoord(x, y)
            glVertex(x, y)
    glEnd()
    glutSwapBuffers()


# setup ######################################################################

glutInit(sys.argv)
glutInitDisplayString(b"rgba double samples=4")
glutInitWindowSize(100, 100)
glutCreateWindow(sys.argv[0].encode())

glutReshapeFunc(reshape)
glutDisplayFunc(display)

glEnable(GL_BLEND)
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA)

shader = glCreateShader(GL_FRAGMENT_SHADER)
glShaderSource(shader, """
    void main() {
        vec2 c = gl_TexCoord[0].xy;
        vec4 color = gl_Color;
        if(c.x*c.y < 0.) color.a *= .5;
        gl_FragColor = color;
    }
""")
glCompileShader(shader)
program = glCreateProgram()
glAttachShader(program, shader)
glLinkProgram(program)
glUseProgram(program)

glutMainLoop()  

Answer 1

This is the basic idea of MSAA (multisample anti-aliasing). The fragment shader is only executed once per fragment. The sample mask is then used to control which samples the resulting fragment is written to.

Say you use 4x MSAA, where each fragment consists of 2x2 samples. If the edge of a triangle passes through a fragment, only the samples on the inside of the edge are updated with a new color. So if the fragment is only partly inside the triangle, 1, 2, or 3 of the 4 samples may be updated. But the samples that are updated are all updated with the same color. Therefore, the fragment shader only needs to be evaluated once.

The big advantage of this approach is that it is very efficient. It will often add only a very modest overhead compared to rendering without MSAA. As already established, the number of shader executions is unchanged. The render target theoretically gets 4 times larger, which could increase memory usage substantially. But that memory can often be compressed effectively, so it is not as bad as it sounds. There obviously is a downsampling step at the end of rendering the frame, which does add overhead.

The disadvantage is that MSAA only helps to smooth triangle boundaries and intersections. If you have sharp transitions within a triangle, e.g. because of texture content, or due to procedural textures, it will not help at all. Mipmapping can reduce jagged edges from texturing, and procedural textures can reduce sharp transitions by taking the gradient into account. But MSAA does not help.

If you want anti-aliasing that addresses all sources of aliasing from sharp transitions, you can go all out and use supersampling. This means that you render the whole scene at a higher resolution. For example, if the size of your final render target is w times h, you render to a surface with size 2 * w times 2 * h, and downsample in the end. This is substantially more expensive, since you now execute the fragment shader 4 times as often, and compression on the larger surface will also not be nearly as effective as on the MSAA surface.

I haven't used the ARB_sample_shading extension you found. But from what I understand about it, it tries to approach the visual quality of supersampling while maintaining at least some of the performance benefits of MSAA.

Answer 2

The sample_shading extension is needed: "In standard multisample rendering, an implementation is allowed to assign the same color and texture coordinate values to each sample, which then allows the optimization where the shader is only evaluated once and then distributed to the samples that have been determined to be covered by the primitive currently being rasterized."