Trouble with Maplibre-GL's projection matrix

[unphased]

I'm doing some three.js rendering, and now want to employ a three raycaster. It's very simple code at the end of the day but many things aren't making much sense.

I have to get my hands slightly dirty as the typical Three.js Raycaster codepath will call ray.origin.setFromMatrixPosition with camera.matrixWorld but the canonical way to "bind" three's camera to maplibre's camera is via assignment of projectionMatrix. When that happens the matrixWorld is left in a stale and invalid state, it remains the identity matrix.

Even though I figured the above out, I am testing the code ray.origin.setFromMatrixPosition( {elements: mercatorMatrix} ) with the raw mercator matrix from maplibre.

Now, this matrix is clearly valid and correct in some sense because it clearly controls three.js properly and all the 3d objects I insert (granted I need to fiddle with some inversion to deal with the left handed/right handed coordinate system situation) are correctly moving with respect to the map.

My roadblock now is that raycasting simply doesn't work. I'm looking at the ray origin position, which is (per setFromMatrixPosition) simply grabbing the mercatorMatrix's [12], [13], and [14] values, which are huge, with values such as {x: 244512.32882916788, y: -934734.2579967699, z: -1089157.7139949538}.

Clearly there is some division or some other unknown operation I need to take into account for the mat4's coming from maplibre, that is not accounted for by setFromMatrixPosition simply taking the usual position vector out of the matrix.

I know that when my camera is still positioned somewhere near earth, in terms of maplibre's coordinate system that means my coordinates should never be larger than 1 or 2 or so. Not in the millions.

[unphased]

I found

maplibre-gl-js/src/geo/transform.ts

Lines 939 to 941 in 868680e

	// The mercatorMatrix can be used to transform points from mercator coordinates
	// ([0, 0] nw, [1, 1] se) to clip space.
	this.mercatorMatrix = mat4.scale([] as any, m, [this.worldSize, this.worldSize, this.worldSize]);

But I am still confused. Why must we scale the matrix with worldsize in order to work with [0, 0] nw, [1, 1] se coords? Wouldn't the scaling by worldsize mean this matrix is suitable for working with a meter based mercator space, which is not what i'm doing?

[unphased]

since this [this.worldSize, this.worldSize, this.worldSize] was very easy to find in my maplibre-gl minified js source, I hacked it to [1,1,1], and this changed the mercatorMatrix values, but the positional values [12] thru [14] are still huge.

[unphased]

Good news. I tried a few things and one that worked for me was:

• Compute inverse of maplibre's mercatorMatrix. This seems to be a view-projection matrix.

• Multiply <0,0,0,1> with this inverse, divide by result's w. This yields camera position!

• Use a pointer vec2 calculated from the mouse position as NDC with

  raycaster.ray.direction.set(pointer.x, pointer.y,  1).normalize().applyMatrix4(invMat).sub(camPos).normalize();
  

  Now this isn't 100% matching the raycast position, when the aspect ratio is not 1:1 it's shooting a ray a little bit closer toward the center of the display (NDC <0,0>). I need to fiddle with it more but at least I found that with aspect at 1:1 it shoots a perfect ray from the mouse. This is showing signs of being workable.