This is obviously a call to WEBGL-Gurus out there:
I'm rendering to a 2x2 pixel resolution with a second depth camera, but I'm reading only a single pixel.
I have tried:
renderer.readRenderTargetPixels()in Three.js (which I assume it will be the same asgl.readPixelsin pureWEBGL).getImageData()after rendering toCanvas.
Both seem to be extremely slow: they add up to 25ms in my case, even for a 2x2-pixel render target. The unjustified low-efficiency especially of the gl.readpixels is also mentioned here: what is the correct way to use gl.readPixels?
So, I'm looking for a solution -any solution, that will read that single-pixel efficiently, and put it into a JS array, or object. Thanks.
UPDATE:
-I updated the title and the main text to be more on point.
-I created a JSFiddle to demonstrate the readRenderTargetPixels() (equiv. to gl.readpixels()) latency. I tried to make it as simple as possible but not... simpler :)
Notes / how to use:
This demo runs both in normal mode, and in WebXR mode. You don't have to enter VR to see the huge delay. JSFiddle has a BUG and it doesn't show the VRButton on mobile. In order to enter VR on mobile, you either need to copy the code to an index.html and run a secure local server (HTTPS) via WIFI or use a JSFiddle-alternative that doesn't make the VRButton ...invisible -I have tested many, and I couldn't find one that worked(!)
This demo shows rendering statistics (min and max milliseconds rendering time) for the last 10 frames, both in normal and WebXR mode on the screen for your convenience, so it's easy to see it on mobile too.
There is a main camera and a depth camera. The depth camera has a resolution of 2x2 pixels and a FOV of 0.015 degrees. It points on a rotating cube, it measures depth and draws a ray and a point on the cubes' surface. I have even optimized the code to eliminate the costly decoding math in JS.
I'm interested mostly on mobile, where the latency is much higher and I have provided QR codes for your convenience. So to test it on mobile please scan the #1 QR code bellow, wait for a minute to stabilize, then scan the second QR code to compare. To see/edit the code in JSFiddle, remove "show" from the url and reload the page.
Please test the demo on Chrome browser. If you run the demo on PC, it is also important to wait a minute before judging the latency. I've noticed that Firefox on PC has far lower latency than Chrome on PC and is far more stable, but the features I'm interested in, are not supported on FF, so I'm only interested in Chrome. On my PC, Chrome starts with around 5ms rendering time (which is still huge vs the 1-2ms without that function) then after a while it doubles and triples. On mobile it's almost always high, between 15 and 30ms (powerful mobile).
readRenderTargetPixels() ON:
https://jsfiddle.net/dlllb/h3ywjeud/show
readRenderTargetPixels() OFF:
https://jsfiddle.net/dlllb/mbk4Lfy1/show
<!DOCTYPE html>
<html lang="en" >
<head>
<title>READPIXELS LATENCY</title>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no, viewport-fit=cover" />
<meta name="mobile-web-app-capable" content="yes">
<meta name="HandheldFriendly" content="true" />
<style>
html{height: 100%}
body {text-align:center; background-color: #000000; height: 100%; margin: 0px; overflow: hidden}
#info { position: absolute; top: 0px; width: 800px; margin: 10px; text-align: left; z-index: 90; display:none}
</style>
</head>
<body>
<div id="container"></div>
<script type="module" defer>
import * as THREE from "https://threejs.org/build/three.module.js";
import { VRButton } from "https://threejs.org/examples/jsm/webxr/VRButton.js";
var win = {
x: window.innerWidth,
y: window.innerHeight
}
// multiply deg with deg2rad to find radians (rad = deg * Pi/180)
const deg2rad = 0.017453292519943295769236907685;
var tmp;
var startms = 0;
var endms = 0;
var deltams = 0;
var fcounter = 0;
var fbuffer = [0,0,0,0,0,0,0,0,0,0];
var maxms = 0;
var minms = 1000;
let avframes = 10;
//_________________________SCENE___________________________________________
var scene = new THREE.Scene();
scene.background = new THREE.Color( "#346189" );
//___________________xrRig __xrCam________________________________________
var xrRig = new THREE.Object3D();
scene.add(xrRig);
var fov = 50;
var aspect = win.x/win.y;
var cnear = 10;
var cfar = 4000;
var xrCam = new THREE.PerspectiveCamera( fov, aspect, cnear, cfar );
xrRig.add(xrCam);
xrRig.position.set(0, 20, 125);
//___________________ depthRig ____ depthCam ____________________________
var depthRig = new THREE.Object3D();
scene.add(depthRig);
var dres = 2;
var lfov = 0.015625;
var laspect = 1;
var lnear = 1;
var lfar = 2000;
var depthCam = new THREE.PerspectiveCamera( lfov, laspect, lnear, lfar );
depthRig.add(depthCam);
depthRig.position.set(40, 0, 50);
depthRig.rotateOnAxis(new THREE.Vector3(0,1,0), 40 * deg2rad)
// show camera cone (depth won't work)
// const helper = new THREE.CameraHelper( depthCam );
// scene.add( helper );
//_________________________________________________________________
var depthTarget = new THREE.WebGLRenderTarget( dres, dres );
depthTarget.texture.format = THREE.RGBAFormat;
// depthTarget.texture.minFilter = THREE.NearestFilter;
// depthTarget.texture.magFilter = THREE.NearestFilter;
// depthTarget.texture.generateMipmaps = false;
// depthTarget.stencilBuffer = false;
// depthTarget.depthBuffer = true;
// depthTarget.depthTexture = new THREE.DepthTexture();
// depthTarget.depthTexture.format = THREE.DepthFormat;
// depthTarget.depthTexture.type = THREE.UnsignedShortType;
var depthMaterial = new THREE.MeshDepthMaterial({depthPacking: THREE.RGBADepthPacking});
var pb = new Uint8Array(4);
var onpos = new THREE.Vector3();
//_________________________________________________________________
const Dlight = new THREE.DirectionalLight( 0xffffff, 1);
Dlight.position.set( 0, 1000, 1000 );
scene.add( Dlight );
//_________________________________________________________________
// *** WebGLRenderer XR ***
var renderer = new THREE.WebGLRenderer({ antialias: true, precision:'highp'});
renderer.setPixelRatio( window.devicePixelRatio );
renderer.setSize( win.x, win.y );
renderer.autoClear = false;
renderer.xr.enabled = true;
var cont = document.getElementById( 'container' );
cont.appendChild( renderer.domElement );
document.body.appendChild( VRButton.createButton( renderer ) );
//____________ LASER RAY VARS _________________________________
var startray = new THREE.Vector3();
var endray = new THREE.Vector3();
var raygeom = new THREE.BufferGeometry();
var points = [startray, endray];
raygeom.setFromPoints( points );
var rayline = new THREE.Line( raygeom, new THREE.MeshBasicMaterial({color: 0xff0000}) );
scene.add(rayline);
var marker = new THREE.Mesh(new THREE.SphereGeometry(0.8), new THREE.MeshBasicMaterial({color: 0xff0000}));
scene.add(marker);
//____________ CUBE _________________________________
var cubeGeometry = new THREE.BoxGeometry(40,40,40);
var material = new THREE.MeshStandardMaterial({color: "#eabf11"});
var cube = new THREE.Mesh(cubeGeometry, material);
scene.add(cube);
cube.position.set(0,0,0);
// ______________________VERTICAL_PLANE____________________________
var ccw = 500;
var cch = 150;
var vplane = new THREE.PlaneBufferGeometry( ccw, cch );
var vmap = new THREE.MeshBasicMaterial();
var m = new THREE.Mesh( vplane, vmap );
m.visible = false;
m.position.set(0, 150, -1000);
scene.add( m );
//_________ CANVAS _______________
var canvas = document.createElement("canvas");
var ctx = canvas.getContext("2d");
ctx.canvas.width = ccw;
ctx.canvas.height = cch;
ctx.font = "60px Arial";
ctx.textBaseline = "top";
var img, tex;
function drawtext(t1, t2){
ctx.clearRect(0, 0, ccw, cch);
ctx.fillStyle = "#346189";
ctx.fillRect(0, 0, ccw, cch);
ctx.fillStyle = "#ffffff";
ctx.fillText(t1, 100, 10, ccw);
ctx.fillText(t2, 100, 80, ccw);
img = ctx.getImageData(0, 0, ccw, cch);
tex = new THREE.Texture(img);
tex.needsUpdate = true;
m.material.map = tex;
m.material.needsUpdate = true;
tex.dispose();
m.visible = true;
}
//_________________________________
window.addEventListener('resize', onResize, false);
renderer.setAnimationLoop( render );
//_________handle_window_resizing___________________________
function onResize(){
if (renderer.xr.isPresenting) return;
win.x = window.innerWidth;
win.y = window.innerHeight;
xrCam.aspect = win.x / win.y;
xrCam.updateProjectionMatrix();
renderer.setSize( win.x, win.y );
}
// ____________________render_frame______________________________
function render() {
startms = Date.now();
renderer.clear();
cube.rotation.y += 0.01;
renderer.xr.enabled = false;
//---------------------- RENDER RGBA-Depth to depthTarget--------------------------
renderer.setClearColor("#ffffff", 1);
rayline.visible = false;
marker.visible = false;
renderer.setRenderTarget(depthTarget);
scene.overrideMaterial = depthMaterial;
renderer.render(scene, depthCam);
// ******* COMMENT-OUT THE FOLLOWING LINE TO COMPARE ******
renderer.readRenderTargetPixels(depthTarget, dres/2, dres/2, 1, 1, pb);
var dp = pb[0]*0.0000000002328306436538696
+ pb[1]*0.00000005960464477539063
+ pb[2]*0.0000152587890625
+ pb[3]*0.00390625;
var viewZ = (lnear * lfar) / ((lfar - lnear) * dp - lfar);
var midZ = viewZ;
if (viewZ < -lfar) {
midZ = -lfar;
}
onpos.set(0, 0, 0.5).applyMatrix4(depthCam.projectionMatrixInverse);
onpos.multiplyScalar(midZ / onpos.z);
onpos.applyMatrix4(depthCam.matrixWorld);
startray = new THREE.Vector3();
depthCam.getWorldPosition(startray);
raygeom.attributes.position.setXYZ(0, startray.x, startray.y, startray.z);
raygeom.attributes.position.setXYZ(1, onpos.x, onpos.y, onpos.z);
raygeom.attributes.position.needsUpdate = true;
//-------------------- RENDER NORMAL SCENE ------------------------
renderer.setClearColor("#346189", 1);
renderer.xr.enabled = true;
rayline.visible = true;
marker.visible = true;
marker.position.copy(onpos);
scene.overrideMaterial = null;
renderer.setRenderTarget(null);
renderer.render( scene, xrCam );
//------- delta time statistics for the last 10 frames -----------
endms = Date.now();
deltams = endms - startms;
for (let f=avframes; f>0; f--){
tmp = fbuffer[f];
minms = Math.min(tmp, minms);
maxms = Math.max(tmp, maxms);
fbuffer[f+1]=tmp;
}
fbuffer[1] = deltams;
fcounter++;
if (fcounter === avframes){
fcounter = 0;
drawtext("max-ms:"+maxms, "min-ms:"+minms);
minms = 1000;
maxms = 0;
}
}//end render() _______________________________________________________________________________
</script>
</body>
</html>
UPDATE #2
Andre van Kammen's mod: still getting very high delays in mobile (Xiaomi Redmi Note 9S).
Video captured with a webcam: https://www.bitchute.com/video/5WJxdo649KiF/
There is an extensive article on Pixel Buffer Objects (PBO): http://www.songho.ca/opengl/gl_pbo.html It looked very promising, it was about asynchronous read of GPU data that would not stall the GPU ...until I tried the demo that switches OFF and ON when you press space, and got ZERO (0) difference! on my PC: http://www.songho.ca/opengl/files/pboUnpack.zip
So, apparently, after 30 years since gl.readpixels was introduced, technology has failed to provide a reliable and efficient way to read the damn pixels... which is extremely shameful, I was designing hardware, and one thing I've learned through the years is that every problem has a solution in electronics, and the same applies to software, and all other sectors. Apparently it's sloppiness-first, instead of performance-first for some part of the industry. Please, prove me wrong.
If you look at the code of readRenderTargetPixels in:
https://github.com/mrdoob/three.js/blob/65e6b4835ae52ea6136392b12ee7114bccefc35a/src/renderers/WebGLRenderer.js
You can see it's using readPixels to do the reading and has some checks around it. ReadPixels is always slow because it forces to sync, meaning the GPU has to finish all the work before reading the pixels and giving them back, the amount of pixels doesn't matter a lot until you reach megabytes.
25ms is a very long time though, what are you doing before reading the pixel?
Update: I've run your fiddle, it runs considerably slower with readPixels (on 1-7ms vs 0-2ms), I checked it in chrome's performance tab and 35.2% of the time goes to the readPixels and 23.4% to the checkFrameBufferStatus. This is a lot indeed, but it's also about half the time, which probably comes from waiting on the scene to render.
If you remove the line everything run's async and you won't measure the waiting on the graphics card. I moved the read to a settimeout to show the difference:
https://jsfiddle.net/rdvz15fx/4/
It runs a bit faster because it can do the read async so it doesn't have to wait. But it also means the data is a frame behind.
Since you use it to do a change on the camera position it would be better to handle it in a vertex shader that reads a texture attached to the framebuffer you render to. That way you wouldn't have to use readPixels and keep it in the videocard. I wouldn't know how to do that in threejs.
Here a piece of code for checking if the GPU is ready before doing readpixels, place this after the draw
Then you can check if the samples are ready with
To bad you still have to poll, but there isn't a callback or event for this unfortunately.