MRT Support

[takahirox]

Fixes #5179.

Kudos to @mattdesl and @edankwan

MRT PR was opened #9358 #8439. It needed to be cleaned up to get merged but that PR hasn't been updated for years. I still want to see MRT in Three.js, so I cleaned up and created a new PR.

API

In the current implementation, users need to use RawShaderMaterial (or onBeforeCompile() for built-in material) to render to WebGLMultiRenderTarget.

const attachmentNums = 2;
const renderTarget = new THREE.WebGLMultiRenderTarget( width, height, attachmentNums );

// rendering to MultiRenderTarget
renderer.setRenderTarget( renderTarget );
renderer.render( scene, camera );

// rendering to screen with MultiRenderTarget as input
material1.map = renderTarget.textures[ 0 ];
material2.map = renderTarget.textures[ 1 ];
renderer.setRenderTarget( null );
renderer.render( scene2, camera2 );

// WebGL1 fragment shader

#extension GL_EXT_draw_buffers : require
...
void main() {
	gl_FragData[ 0 ] = ...
	gl_FragData[ 1 ] = ...
}

// WebGL2 fragment shader

#version 300 es
layout(location = 0) out vec4 gColor;
layout(location = 1) out vec4 gNormal;
...
void main() {
	gColor = ...
	gNormal = ...
}

[WestLangley]

Nice! I did not review the renderer code, though. The hard part. :-)

[fernandojsg]

Looking great! It could be nice to add also a WebGL1 MRT example to see the differences and test between both implementations

[takahirox]

Thanks for the review. I updated.

One concern with WebGL 1. No proper way to inject #extension GL_EXT_draw_buffers : require to the top of fragment shader code with ShaderMaterial because it adds common code to the top. So users need to use RawShaderMaterial instead of ShaderMaterial with WebGL 1. I'd like to think of the solution and WebGL 1 example later, after this PR is merged.

[fernandojsg]

@takahirox thanks, sounds good, we could get webgl2 support landed and iterate on the webgl1 example in another PR.

[nayyden]

Hi, any plans to release this in r106?

[valenn]

Thanks for the review. I updated.

One concern with WebGL 1. No proper way to inject #extension GL_EXT_draw_buffers : require to the top of fragment shader code with ShaderMaterial because it adds common code to the top. So users need to use RawShaderMaterial instead of ShaderMaterial with WebGL 1. I'd like to think of the solution and WebGL 1 example later, after this PR is merged.

There is already ShaderMaterial.extensions.drawBuffers that does this, or does that cause any problems? (also if there is this extension already supported, is it possible to use it already?)

[nayyden]

There is already ShaderMaterial.extensions.drawBuffers that does this, or does that cause any problems? (also if there is this extension already supported, is it possible to use it already?)

For MRT we need a way to bind multiple color attachments to the framebuffer and I don't think this is possible with the current API.

[oskardahlberg]

This is great. Hope this can land soon. Two notes,

Firstly, built in materials currently write out highp vec4 pc_fragColor; into their frag shaders and this prohibits them from being used with this target. Changing this (in WebGLProgram) to layout(location = 0) out highp vec4 pc_fragColor; solves this and I can't see any potential issues with doing so.

Secondly, in the example, the texture options is set like this

renderTarget = new THREE.WebGLMultiRenderTarget( window.innerWidth * window.devicePixelRatio, window.innerHeight * window.devicePixelRatio, 2 );
renderTarget.texture.format = THREE.RGBAFormat;
renderTarget.texture.minFilter = THREE.NearestFilter;
renderTarget.texture.magFilter = THREE.NearestFilter;
renderTarget.texture.type = THREE.FloatType;
renderTarget.texture.generateMipmaps = false;

but to my understanding, this wont actually do anything. There is already a way to set it that does seem to work though, so changing this might be good before releasing.

renderTarget = new THREE.WebGLMultiRenderTarget(
  window.innerWidth * window.devicePixelRatio,
  window.innerHeight * window.devicePixelRatio,
  2,
  {
    format: THREE.RGBAFormat,
    minFilter: THREE.NearestFilter,
    magFilter: THREE.NearestFilter,
    type: THREE.FloatType,
    generateMipmaps: false
  }
);

Do verify this before changing it, but it seems highly odd the way it is.

[gkjohnson]

This can probably wait but is it possible for the different color attachments to have different sizes?

[nayyden]

I am not sure if different image sizes have any real-world usage (if supported at all). Although the fragment stage outputs several colors, the rasterization stage has been executed only once, so separate target resolutions are not possible. To write to the smaller/bigger images the driver would need to down/up sample which might be possible but the gain of having some of them smaller is probably none. The gain of MRT is to exactly execute everything prior the fragment shader once only.

According to this post it might work but will be rendered with the size of the smallest image https://community.khronos.org/t/fbo-with-different-size-attachments/62925/2

P.S. Different formats and types for the images are possible and have practical usage. I read that some combination could possibly fail depending on the implementation. Not sure if this applies to WebGL or OpenGL only.

[takahirox]

I'd like to go as is so far, and consider different size if users actually demand it.

[takahirox]

@oskardahlberg

Thanks for the comments.

Firstly, built in materials currently write out highp vec4 pc_fragColor; into their frag shaders and this prohibits them from being used with this target. Changing this (in WebGLProgram) to layout(location = 0) out highp vec4 pc_fragColor; solves this and I can't see any potential issues with doing so.

Built-in material as is seems working fine with WebGLMultiRenderTarget where numAttachments = 1. I guess if there is only one shader buffer output without location specification it seems being recognized as location = 0 (I couldn't find that definition in the specs tho).

Secondly, in the example, the texture options is set like this

The last argument of RenderTarget constructor is for its texture parameters so I think renderTarget.texture.xxx = yyy works but yeah defining via the constructor looks properer way. Otherwise, texture parameters won't set to .textures. Updated Example.

[takahirox]

Thanks for the comments. Updated. And using module in example now. I think ready to go.

[mrdoob]

I think this is almost there! We should be able to land this at the beginning of the next dev cycle 🤞

A few more changes...

I don't think we should add setTexture(), setSize() or even copy(). For example, WebGLCubeRenderTarget doesn't have them and no one has requested them yet.

As per the class name, I think WebGLMultipleRenderTargets would be more clear. Otherwise it can be confused with WebGLMultisampleRenderTarget. And making the class name plural helps too.

I'm not sure it's a good idea to extend from WebGLRenderTarget either...

[takahirox]

We should be able to land this at the beginning of the next dev cycle 🤞

That sounds good to me.

I don't think we should add setTexture(), setSize() or even copy(). For example, WebGLCubeRenderTarget doesn't have them and no one has requested them yet.

I have removed setTexture(). But I think setSize() is necessary for example it's called on window resize.

And copy() would be necessary, too. Because it's used by clone() defined in the parent WebGLRenderTarget.

As per the class name, I think WebGLMultipleRenderTargets would be more clear. Otherwise it can be confused with WebGLMultisampleRenderTarget. And making the class name plural helps too.

Sounds good to me. I renamed it.

I'm not sure it's a good idea to extend from WebGLRenderTarget either...

Personally I prefer extending WebGLRenderTarget for the consistency with other render targets.

[takahirox]

So, r128 has been released. It's time to go now?

[marcofugaro]

Happy 2nd birthday to this PR!! 🥳🥳🥳🥳

[mrdoob]

What better way to celebrate than...

[mrdoob]

Thanks!

[takahirox]

Finally the time has come, thanks!

The PR was originally made two years ago. There are a lot of updates which may affect the PR in the core during that time. I think I carefully kept updating the PR but let me know if you folks find anything which doesn't match the newest core API/design/behavior/etc.

@Mugen87 As I wrote in the PR comment, I took over @mattdesl and @edankwan works. Would you please add their names in the change log next to me?

[takahirox]

And I think there is a big room for API improvement. For example currently WebGLRenderMutlipleRendertargets is expected to be used with RawShaderMaterial. But we may enable built-in materials MRT support.

When I made this PR, there were a lot of difficulties and limitations to improve the API. But during the two years we have had tons of new features and simplifications in Three.js (mainly done by @Mugen87 thanks). Such difficulties and limitations might have been removed and we might be able to improve the API now. Let's cooperate and work on it all together.