Lighting system is corrupted

[bil-bas]

Mostly doesn't work :(

[ragekit]

hey, i don't know if you've found the solution, i'm trying to implement the Catalinzima shader baser light engine in libgdx with Opengl myself too. I think the error is when you make the texture reduction to 2pixel wide shadowmap. But i didn't find how to correct it

[bil-bas]

No, I haven't resolved this issue, but I haven't done any work on the library for some months. I made significant alterations to the original code I adapted it from to both optimise it and convert it to Gosu's coordinates system and seem to have messed up somewhere. I'd certainly appreciate any insights you might have!

[ragekit]

well i'm comparing my 2 texture, one with the distorted rays and the one with the reduction and things don't seem to match, so i'm guessing the reduction isn't correct
(right : distorted shadowcaster map, left, 2px shadowmap stretched)

[ragekit]

You're writing the reduction directly in a 2 pixel wide texture right ?

[bil-bas]

I wish I could remember how I did it in the end ;) But yes, I think I removed steps to be able to do that.

[ragekit]

ahah, i'm begining to go insane on this issue :)

[bil-bas]

It would be great if I could get it fixed though. It looks quite amazing, even if it does cast shadows in the wrong directions!

[ragekit]

ho, i think i'm onto something

[ragekit]

okaaaaaay, found it, you have to modify the main of distort.frag like this :

float pixel_width = 1.0 / float(u_textureWidth);
                int half_width = u_textureWidth / 2;

                // Get the left-most pixel.
                gl_FragColor = distort(v_texCoords);

                // Grab each pixel to the right in turn and min it with the previous one.
                float xOffset = (v_texCoords.x <0.5 ? 0.0 : 0.5);
                for(int i = 1; i < half_width; i++)
                {
                    vec2 color = distort(vec2(xOffset + pixel_width * float(i), v_texCoords.y)).rg;
                    gl_FragColor.rg = min(gl_FragColor.rg, color.rg);
                }

(i changed the varying name to match libgdx ones, but you get the idea)
I'm fairly new to GLSL, but i guess the algorithm need an v_texCoords that was 0 for the first horizontal pixel and 0.5 for the second horizontal pixel, but GLSL was giving him something else, so i forced the value.

[ragekit]

i hope this was your issue too :D

[bil-bas]

I changed the current code to mirror what you suggested, but it doesn't fix it. Maybe looks a bit different though. Maybe the issue is in the vertex shader used so that it sometimes reads from the wrong side of the 2-pixel wide texture? I would expect, however that var_TexCoord.x would only be 0 or 0.5 to begin with.

float pixel_width = 1.0 / float(in_TextureWidth);
int half_width = in_TextureWidth / 2;

// Get the left-most pixel.
gl_FragColor = distort(var_TexCoord);

// Grab each pixel to the right in turn and min it with the previous one.
float xOffset = (var_TexCoord.x < 0.5 ? 0.0 : 0.5);
for(int i = 1; i < half_width; i++)
{
    vec2 color = distort(vec2(xOffset + pixel_width * float(i), var_TexCoord.y)).rg;

    gl_FragColor.rg = min(gl_FragColor.rg, color.rg);
}

By the way, are you running this in Ruby or running my shader code in your OpenGL environment?

Ultimately, if my code has given you any of the insight you needed to fix yours, then we've not lost entirely today ;)

[ragekit]

I'm running it in java, throught LWJGL, my vertex shader is ultra simple, basically an identity shader

attribute vec4 a_position;
attribute vec4 a_color;
attribute vec2 a_texCoord0;

uniform mat4 u_projTrans;

varying vec4 v_color;
varying vec2 v_texCoords;

void main()
{
    v_color = a_color;
    v_texCoords = a_texCoord0;

    gl_Position = u_projTrans * a_position;
}

[ragekit]

(btw projTrans is an identity Matrix4, i don't know why its still here )

[bil-bas]

I use https://github.com/Spooner/ashton/blob/master/lib/ashton/shaders/default.vert which is even simpler, I think. Problem is that I banged my head against these formulae for quite a few hours and ultimately I realised I didn't understand them quite well enough to fix them (I roughly know what is happening, of course).

[ragekit]

hum, maybe its there, which version of Opengl do you use ? libgdx is with ES2.0 so i can't use ftransform(), maybe use a vertex attribute instead as stated here : http://en.wikipedia.org/wiki/GLSL#A_sample_trivial_GLSL_vertex_shader

[bil-bas]

I'm using glsl 1.1, since I wanted maximum compatibility for the default shaders used by the library. As that example says, ftransform() is just the same as what you have to do manually in later versions.

[ragekit]

hu, yeah, i mismatched GLSL and OpenGL, :s do you have a screenshot of how it behave ?
EDIT : not mismatch, mix up, it's 2 A.M here and i lost my English

[bil-bas]

GLSL 1.1 is what we got in OpenGL2.0, so it is pretty prehistoric compared to any version any sane person would use. I still have had people complain it is too new and doesn't work on their hardware though (really!).

The way the shadows are messed up isn't terribly clear unless you move the light source around interactively.e.g. As you move the light UP towards an object, the shadow is correct for a bit, then is shown mirrored BELOW the light as the light gets close to the object. Other shadows are mirrored or flipped depending on both the orientation and distance of the light with respect to the solid solid object.

I'm sure the problem is even as simple as an inverted sign somewhere, but I haven't a clue where to look for it 😆

It is 1am here and I'm past being able to think about this either ;) Thanks for your input. If you have any more ideas, I'd be very interested to hear about them. Could you give me a link to your working code; perhaps it could help me work out my problem more easily than comparing my code to the original code we both adapted?

[ragekit]

yeah, here are the 2 shaders https://www.dropbox.com/s/n27piaq6do9mssb/shaders.zip?m
the java code is a bit of a mess, buts i don't think the issue is from it
(sorry, i can't give you access off the full repo, its for a game, and its private for now)