Stencil Tests are a process used in rendering engines like OpenGL to determine if the pixels of an object should be rendered
Stencil tests start by rendering objects in order that they should appear on in the GPU. This is influenced by many factors such as,
- The
OffsetFactorandOffsetUnitsof a material. - The
SortingOrderof either a Component:MeshRenderer or Component:SkinnedMeshRenderer. - The
RenderQueueof a material.
Depending on these factors, polygons are rendered in order on top of one another, generating the scene.
Now what Stencil Tests do is during the rendering process, is every pixel in the in-progress-frame (frame buffer) has a stencil buffer number associated with it (0->255). When the first polygon is rendered, if the material that polygon is using allows for interacting with the stencil buffer, it will read from the stencil buffer of the pixels it is rendering on top of.
When reading, the StencilReadMask is bitwise ANDed with the stencil numbers in the pixels in the current frame buffer before the test compares them. So if the buffer holds the value "12" (base 10) in the buffer and the StencilReadMask is "4" (base 10) then the result would be four (in binary (base 2): 0100 & 1100 == 0100).
The previous paragraph can be hard to follow for someone who doesn't know binary. To explain it in an example, you can use a binary calculator site to make a set of 1 and 0s to make a number. The number created can act as a set of switches. If the switches (binary data) in the pixels rendered before don't match the mask's switches (binary data), then they aren't read by the material for comparing numbers (the copy of the switches that got read will turn off). This can allow for having a material with the switches 00010010 (18) and a mask of 00000010 (2) then you end up with 00000010 (2) being read to compare. Using a mask of 255 or 11111111 allows all stencil numbers in previously rendered polygons to be read regardless of what number they hold for comparison.
After reading the stencil buffer numbers from all the pixels it is reading from, it will perform a Type:StencilComparison against the polygon material's StencilID. If the comparison fails for a pixel, the polygon at that pixel is not rendered at all. If the comparison passes, the polygon is rendered for that pixel. For example, if your StencilComparison is Equals and your StencilID is 3, and it reads 2 from the buffer after being filtered by the StencilReadMask it won't render for that pixel. but if your StencilID equals to 2, it will render.
Lastly the Stencil buffer writes the StencilID to the stencil buffer, then does a Bitwise AND for every pixel's stencil mask it is rendering on with StencilReadMask before finally completeing
Examples
Below are examples of Stencil masks being used to create many objects, like portable holes, objects only visible through portals, 4th dimensional objects, and even cubes that show different shapes inside depending on which direction they are looked at from.
This is all done by rendering pixels of polygons that appear after another object only if the numbers that they read from the stencil buffer before match the needed criteria (like drawing a cone with "4" and then rendering polygons on top that only render their pixels if they read "4" from the objects drawn before)
For the page that this info was gathered from, see learnopengl Stencil testing