Mip Map support for High resolution texture packs.

[blackhand1001]

It would be very helpful to be able to have mip maps for texture packs. They could either be generated at runtime or as part of the pack itself. This would be separate from the original built in n64 style mip maps. Most texture packs want those disabled anyway. This would help with the severe shimmering issues that high resolution textures have when they are larger than the sample size on screen. Dolphin generates mip maps from its png textures if I recall correctly and also has a format for handling mip map effects like the princess/bowser portrait as well. It may be useful to generate mip maps for the stock textures as well as an optional image quality boost as many games don't have them.

[gonetz]

As I remember, the idea to generate mip-map tiles from the most detailed tile was already discussed.
It should be doable. I may try to implement that if I find time.

[blackhand1001]

That would be great. Right now the only way to get sort of acceptable quality is to run crazy amounts of super sampling and that really kills the performance especially when ever smoke effects appear on screen. It also kills the battery life on devices like the steam deck.

[gonetz]

Ok, draft implementation of mip-map support for HD textures is ready: #2826
It works, but not optimal and bugs are possible. So far I implemented it only for accurate texture mapping.

GLideN64 and RiceVideo dump textures for T0 and T1 textures. In case of mip-map textures these are the first two tiles,
or detailed texture and the first tile if detailed mode is on. Thus, texture packs have HD versions for the first two tiles at most.
HD mip-mapping generates mip maps from the provided HD tile using image resize.

After I got that working, I started to analyze, what is changed.
I selected Zelda MM Termina Field to test. Bushes use a mip-map texture there. The texture has 4 tiles of sizes 32x32, 16x16, 8x8, 8x8. The last two tiles have the same size, N64 mip-mapping supports that. Here how it looks with original textures:

I extracted corresponding HD textures from Nerrel's pack. The first tile has size 2048x2048, the second tile is 512x512.

The master build of GLideN64 has partial support for HD textures mip-mapping: it calculates lod factor, but ignores mip-map level, using only the most detailed tiles, because there are no other tiles. It looks like that:

New HD mip-mapping generates missing tiles of size 256x256 from the original 512x512 tile. So, HD texture has tiles 2048x2048, 512x512, 256x256, 256x256. The result:

May be I'm blind, but I don't see any difference with the master build.

You may suspect that the algorithm does not work. It does. To prove that I hacked the mip-map rendering, making each next mip-map level look darker.
You may clearly see where the new level starts on the result:

May be the HD texture is too large and the difference in details level is not noticeable in that resolution?
I checked that. I reduced the HD texture to 128x128 and 64x64 tiles.
Master build looks like that with these textures:

New HD mip-mapping result:

Again, I don't see any difference with the master.

So, I'm in doubt now. Was it worth the efforts? May be Zelda MM is a bad example? It seldom uses mip-map textures. I found another one only for a floor in Clock-Town.
May be Nerrel's texture is too detailed and does not work well as mip-mapped one?
I need a feedback from users and texture artists: is there a case where the new HD mip-mapping works better then the master one?

[blackhand1001]

Is the lod Bias set properly? Also you may need more mip map levels to really get a satisfactory result with such a large texture. For a texture that size you really need like 12 mip map levels. 4 is not nearly enough as a 256x256 texture is still gonna be way larger than the sample size on the screen. Luckily the additional mip levels don't consume much vram as each level only takes up 1/4 the amount as the last.

Is this piggy backing off the n64 style mip mapping code or is it using the native opengl mip map code where the video driver handles the lod levels, anisotropic filtering etc. Ideally this should really be handled completely outside of the n64 mip map code and let the opengl sampler do its thing.

Also does this mip mapping work for textures that aren't mip mapped in the original game?

[gonetz]

It is the n64 style mip-mapping. Number if mip-map levels is the same, as in the original texture. Lod bias is calculated the same way as for N64 textures. The only difference is that HD tiles are used instead of the original ones.

"Ideally this should really be handled completely outside of the n64 mip map code and let the opengl sampler do its thing."
Well, probably, for HD textures, it would be better. It won't work for every case though. There are special cases, like like the princess/bowser portraits in SM64. Zelda MM have a special case in swamps. Also, using OpenGL mip-map mechanism would be very hackish, because I need to bypass N64 color combiners to use the OpenGL lod bias.

[weinerschnitzel]

I recall the mip mapping effect was more obvious with Perfect Dark on the floors inside the first level Carrington Institute.

Is it possible to interpolate number of mip map levels, based on the scale of the internal resolution? It might not work for special cases like bowser/peach portraits, but maybe it will, and maybe it can show a smoother transition in I.E. Perfect Dark.

[GhostlyDark]

Does it really work as intended? What I'm seeing is some "warping" at the spot where I think the first mip-map starts, followed by the full resolution texture again.

[gonetz]

Is it possible to interpolate number of mip map levels, based on the scale of the internal resolution?

Suppose, I use 8x scale of the internal resolution. Main tile of the original texture has size 32x32. HD tile has size 1024x1024. that is 32x. How many tiles are necessary to get positive effect from HD mip-mapping?

[gonetz]

Does it really work as intended?

Yes. To test that, I set different colors for each tile. We have 4 tiles there. Color of the most detailed tile is unchanged, next is blue, then green and finally red. The result:

[gonetz]

I also tried to force set tile in the mip-map shader to the least detailed one, and I got the low-detailed texture. It works.
I suppose @blackhand1001 is right: "256x256 texture is still gonna be way larger than the sample size on the screen."
Even when I use 128x128 main tile, the less detailed tile is 32x32, which is probably too large.

[gonetz]

Here are the tiles I used to test:
Zelda-mipmap.zip

[weinerschnitzel]

@gonetz 1024x1024 HD texture can have x and y divided by 2, 5 times to match 32x32 original texture size.

32x32 original texture size gets split to 16x16, 8x8, 8x8. So 2 additional levels of fidelity.

I.E.

A = Number of times HD texture x and y can be halved until it is at original resolution

B = Number of times original resolution texture x and y gets halved to reach lowest fidelity.

A + B = max number of tiles supported by texture pack.

5 + 2 = 7 max tiles for majoras mask example.

We could call it a day with an option that does the above "Interpolate additional HD mip map levels by texture pack"

Interpolation by internal resolution seems more complicated. I dont see it working without generating odd texture sizes.

Most basic, we would want number of lod changes times the resolution factor. 24 tiles for 8x... seems like overkill.

Or number of lod changes times internal resolution's power of 2 plus 1.

2x would be original resolution x2^1. 1+1=2. So 2 (power of 2 + 1) x 3 (number of times lod changes) would be 6 levels.

4x would be original resolution x2^2. 2+1=3. So 3 x 3 = 9.

8x would be x2^3. So 4x3 = 12.

There are some doubts:

Doesn't the game determine where lod levels are set? I.E. if we interpolate to 7 tiles, would we end up with compressed bands of different fidelities where we see blue and green levels? The whole red section may end up being lowest fidelity and maybe not what we want.

That said, people could gameshark hack the lod distances if they want lowest fidelity to be farther away.

I think that would be the best we could do to mathematically show the greatest fidelity and least fidelity on screen at once for smooth transitions.

[weinerschnitzel]

Another idea:

Larger tile could be blurred before being reduced in size to produce smaller tile. That should improve the jagginess that is seen.

However if we are manipulating the texture, best to just let the graphic artist modify all tiles.

Afterthought: Blurrier mip maps would not be as good as working anisotropic filtering.

[gonetz]

Doesn't the game determine where lod levels are set?

Mip-map tile is selected depending on polygon's size.
For example, if max tile is 32x32, polygons with size above 32x32 will use that tile.
If polygon size is between 32x32 and 16x16, its color is blend of tiles 32x32 and 16x16.
For HD resolutions polygon size is scaled, so tiles selection works the same regardless of the screen resolution.

I suppose, the problem with Zelda MM example is that the difference between hi-res and low-res tiles is not that drastic, as with the original texture. Look:

We have nothing like this with Nerrel's HD texture. This HD texture is very detailed. Automatic resize does not remove the details, it just makes them less distinct. It probably is not what we need there. I'm not a texture artist, I can't create a less detailed texture to see if it works better than automatically generated one.

However if we are manipulating the texture, best to just let the graphic artist modify all tiles.

Yes, it may work. However, it is currently unsupported: texture dumping dumps only 2 most detailed tiles.
We need to dump all tiles, to get texture names to texture artists. I think I can do it, but I need to know that it can help the situation. Is there anybody, who can create a full HD tile set for a mip-map texture, to see if it can improve the situation with mip-mapping? @Nerrel @GhostlyDark ?

[GhostlyDark]

I still can't get it to work. The plugin seems to pick the auto-generated textures from the dump folder, so I can't fully control how the mip-maps look.

hires_texture

texture_dump

In case this has something to do with us having different settings:
settings.txt

[gonetz]

I still can't get it to work. The plugin seems to pick the auto-generated textures from the dump folder, so I can't fully control how the mip-maps look.

Yes. Since there are no texture pack with full mip-map texture, the plugin looks only for the first two tiles in the HD pack. The texture names in my dump folder are fake ones. I just increased texture CRC by 1 to get different texture name for a next tile. I did not implement proper mip-map texture dump with all tiles yet. Can you give me the full tile set to test? I'll manually rename the tiles.

[GhostlyDark]

My plan was to just resize the textures to lower sizes and see how low they need to be to get a visual improvement.
mm_grass.zip

[gonetz]

I implemented texture dump for all mip-map tiles, #2826
It was quite a work.
Now mip-map tiles dumped into a sub-folder with name <title>#<first_tile_crc>_mipmap, so you don't need to search for tiles among all dumped textures.

The situation is a bit different for mip-maps with detail tile.
Detail tile usually has different format and image pattern than other tiles.
The same detail tile can be used for different mip-map textures.
If detail tile is used, it is stored in <title>#<detail_tile_crc>_detail sub-folder. Other tiles are stored in sub-sub-folder with name <title>#<second_tile_crc>_mipmap. If several textures use the same detail tile, the <title>#<detail_tile_crc> folder will have several <title>#<second_tile_crc>_mipmap sub-folders.

Note: N64 mip-maps can have tiles with size 2x2 and even 1x1. Such tiles hardly will get unique checksum within the whole texture pack. When texture pack has textures with non-unique checksum, only the first texture is loaded and used for all original textures with that checksum. To calculate unique checksum for small tiles, I use checksum of the largest tile as a seed (that is initial value of checksum).

Now, how HD mip-map texture load works.
Previous version loaded only the two most detailed tiles and used image resize to get the rest.
Moreover, due to the problem described in #1711, in many cases only the first tile could be dumped, so HD mip-mapping was impossible to support at all, as only the first HD tile could be found.
This problem is hopefully fixed, so now all tiles can be dumped and their HD counterparts can be loaded.
The new version looks for HD tile for each original one. If some HD tile is not found in the texture pack, it will be generated by resizing the previous tile.

I hope, this is enough to finally implement proper HD packs for games like GE and PD.
Feedback is welcome.

[gonetz]

I fixed load of HD tiles. Test texture pack:

mm_mipmap_colored.zip

Result:

[GhostlyDark]

I noticed that if I reduce the size of the lower mipmaps too much, it'll cause graphical issues.
size-difference-too-large.zip

It appears that the mipmaps need to be offset to fix some weird misalignments going on, which makes me believe that dynamically resizing higher res textures is not a good idea - at least not for that specific grass.

I did create a set offsetting the mipmap textures (16x16 and 8x8 dumps), but it's still not quite an improvement. The lowest size texture (8x8) may still need a lower resolution, but I can't freely adjust it without introducing the aforementioned graphical issue.
new_hd_set.zip

[gonetz]

I used that texture

to test mip-map misalignments. Yes, the problem exists:

[gonetz]

I found a solution for the mip-map misalignment problem, but I don't understand, why it works.
Probably, there is another bug somewhere, so now we have even number of bugs, which eliminate each other.
So far, I consider that fix as a hack.
I also added hotkey to toggle mip-map emulation. It is useful for testing.
BTW, this checkered texture is convenient to create test mip-map levels.
You may cut a quarter and get less detailed next mip-map level.

[GhostlyDark]

I did stumble upon a crash when resizing the window during the intro of Majora's Mask (when the mask flies in, with the motion blur being active). I am not 100% sure this is related, but I do believe this only happens with the HD mipmap branch. I can't reproduce this on current master.

[GhostlyDark]

I found a solution for the mip-map misalignment problem, but I don't understand, why it works.

Are you referring to the textureEngine1 hack commit? It seems to affect all textures, not just mipmap ones. For recent texture pack releases, some textures are adjusted for the accurate texture coordinates to avoid something like this floor looking misaligned:

With this hack, while still using accurate texture coordinates, the HD texture behavior seems to be the same as the one with the current Inaccurate texture coordinates option. Discussion about the differences:
#2528 (comment)

To put it in other words: This hack unifies the appearance of HD textures for both inaccurate and accurate texture coordinates, seemingly without affecting SD textures - eliminating any need to offset textures. It's pretty neat to be honest.

[GhostlyDark]

To clarify: The above is only true for "combined" textures, such as the East Clock Town floor or mipmap textures. From my understanding, it's anything that combines two textures that have different resolutions (e.g. 64x64 together with 32x32).

[weinerschnitzel]

Interesting, and convenient.

Probably worth keeping to have HD texture creation sane with Majoras Mask.

Are there any games that suffer the same HD misalignments mentioned here? I thought Marjoras Mask was the one off and it was a game side bug.

If Majoras Mask is the only game with HD misalignments, and the mipmap misalignment issue is the same: Maybe make the hack for Majoras Mask only.

I. E. Does the mipmapping offset issue show up in Perfect Dark, too?

@GhostlyDark instead of simply reducing the size of lower mipmaps, what if you create each mipmap as a blurred version of the last? Each time tile size decreases, the smaller tile can have another blur applied, so the effect should be more noticeable. We might have something interesting there.

[GhostlyDark]

The misalignment is something that was introduced with the accurate coordinates and has nothing to do with mipmaps. Dolphin and the Ship PC ports also have the same misalignments on all textures, except that no one is able to check mipmap textures as they simply don't work there. I haven't checked other games, but it should affect pretty much all of them, at least in theory.

Regarding the idea to blur lower mipmaps:
The game doesn't replace a texture with a lower sized mipmap, it is layered on top. Unfortunately, this doesn't look very good and sometimes (stairs in Clock Town) the mipmap doesn't even overlay the full sized texture completely. You also need to be very close to see the textures swap. Some mipmaps in the game aren't blurrier versions like some leafs in Woodfall (a white texture to make them partially disappear in the distance).

I generally don't see the benefit of reducing moire via mipmaps for MM as there are (if I haven't missed any) exactly 19 instances. If we want to fix HD shimmering, DDS support (files can have mipmaps integrated) is the way to go.