OVL - Support Disneyland Adventures

[dabbers]

I've pulled out the OVL files from the game Disneyland Adventures. Trying to open the files with the tool results in a ton of

list index out of range
and
Error -3 while decompressing data: invalid code lengths set

Errors. No files extract successfully. It's likely that the files are slightly different format.

I see several references to octl files in some debug files that were added in the Steam version. Is there any documentation on how to read these OVL files so I can try and help reverse engineer the special formatting of these files?

[HENDRIX-ZT2]

File format specification is done here: https://github.com/OpenNaja/pyffi_ext
per file type, an XML defines the basic data structures and a python script stores the actual logic to read those blocks from a file.

eg. the OVL header data layout:
https://github.com/OpenNaja/pyffi_ext/blob/master/formats/ovl/ovl.xml#L112

eg. the main OVL reading function:
https://github.com/OpenNaja/pyffi_ext/blob/master/formats/ovl/__init__.py#L177

Good luck and keep us posted!

[dabbers]

Something is going on with the file structure. Not sure what's going on. Based on the number of mimes (and their length), the file structures begin about here, and partly looks like the highlighted:

The code parses it out like below. And reads an invalid extension. Notice the offset becomes repeated which seems an unlikely circumstance. I'm unsure what the problem is here. I'll keep poking around but if you have any ideas I'd be open to hear them.

* files :
    <class 'pyffi.object_models.xml.array.Array'> instance at 0x2A67FC30778
    0: <struct 'FileEntry'> instance at 0x2A67FDD2FC8
    * offset : 0
    * file_hash : 59323
    * unkn_0 : -55
    * unkn_1 : -97
    * extension : 24097
    1: <struct 'FileEntry'> instance at 0x2A685F92388
    * offset : 4
    * file_hash : 66535
    * unkn_0 : 104
    * unkn_1 : -111
    * extension : 2658
    2: <struct 'FileEntry'> instance at 0x2A685F92688
    * offset : 65540
    * file_hash : 38567
    * unkn_0 : 24
    * unkn_1 : -77
    * extension : 5492
    3: <struct 'FileEntry'> instance at 0x2A685F928C8
    * offset : 65540
    * file_hash : 83033
    * unkn_0 : 31
    * unkn_1 : -102
    * extension : 11376
    4: <struct 'FileEntry'> instance at 0x2A685F92B08
    * offset : 65540
    * file_hash : 9481
    * unkn_0 : 79
    * unkn_1 : 98
    * extension : 19137

[HENDRIX-ZT2]

There is probably a different number of Bytes per file entry. You'll want to figure out what was added or removed, and edit the File entry struct accordingly.

[HENDRIX-ZT2]

You can find what the correct offsets are by looking at the name block.

[dabbers]

So turns out there was an extra buffer of 4 bytes (uint) of 0s between the names and the Mime. After correcting for that, I can parse up until the archive now.

I've also learned that this archive data isn't compressed. The uncompressed_size is 0 for every archive entry (I had 64 archives that each made up the rest of the file, instead of just the first one). So that was simple to fix to not zlib decompress. I'm now having issues with header_type mismatch Mismatch between total amount of headers. I don't quite know what's going on here, but I'm going to keep looking into this. The header_type.num_headers are 34088 and 51531, and we're expecting 2. So Something's definitely off with the uncompressed stream (what used to be the zlib stream but I'm avoiding uncompressing).

Any thoughts at this point? I don't think I'm blocked as of yet but would appreciate any insights.

[HENDRIX-ZT2]

You can have it dump the decompressed data as DAT. Make sure that's "readable" and not zlib gibberish.
If you've done that, the byte raster is probably thrown off again by a missing or additional structs before the count.

[dabbers]

Are there any zlib specific markers I could look for? I'm not familiar with the underlying workings of zlib so I wasn't sure if there was a header I could look for.

[HENDRIX-ZT2]

Well data that is still zlib-compressed will be cryptic gibberish, while correctly decompressed data will have some regularity due to the byte raster and does often contain readable plain text in some spots. Compressed zlib data does have two header bytes that you can look for to make sure it's trying to decompress at the right spot. These are usually but not always one of the following:
78 01 - No Compression/low
78 9C - Default Compression
78 DA - Best Compression

[HENDRIX-ZT2]

Any luck with these? We've recently improved the code base to be more flexible, hoping to expand it to more known game versions.

[dabbers]

I got pretty stalled and ran out of time. I hope to get back to working on it when some of my other projects wrap up.

[Zhilian1881]

Hello,
is possible to export end import text files from Disneyland Adventures game by this tool? I tried it but have some error. Is there a chance to support this game?
Or if this game is supported, what did i do wrong?

Here is error mesage and *.ovl files from the game:
https://ibb.co/qFzs3rZ
https://www88.zippyshare.com/v/EmwCciSF/file.html

[HENDRIX-ZT2]

Yeah, probably an easy fix.
Zippyshare seems to be down right now. Can you upload the ovl elsewhere? Like wetransfer.com
Scratch that, I got the files with a VPN.

[HENDRIX-ZT2]

Disneyland adventure txts can now be extracted!
Be ware, extraction is sloooow. Need to refactor.

[Zhilian1881]

Disneyland adventure txts can now be extracted!
Be ware, extraction is sloooow. Need to refactor.

Great work, buddy.
It works perfectly fine for *.ovl files. By the way, is there any chance this tool will works with *.ovs txt files too?

[HENDRIX-ZT2]

OVS files are just additional data for the OVL and are opened automatically if the corresponding OVL is opened.

[Zhilian1881]

OVS files are just additional data for the OVL and are opened automatically if the corresponding OVL is opened.

Sorry, man. I do not notice that. Thank you very much.

[HENDRIX-ZT2]

Pushed a speedup for txt extraction.

[dabbers]

Those changes help a lot for getting more extraction working. I'm tempted to pick this back up now...

[Tandashi]

Version I used: Latest Master (Also tried the latest release, same result)
Files I used: Files.zip
Log: ovl_tool_gui.log

Is Lua file extraction / injection supported for Disneyland Adventures yet?

I always get some sort of binary file as output when trying to extract Lua files from the Game OVL's.

Example:

I also sometimes get Errors like this:

Am I doing something wrong / using the tool in a wrong way?
If it's not supported yet, would gladly help to implement, but very new to this sort of topic. If there are some resources that might make it easier to get into the topic / documentations would appreciate that.

[HENDRIX-ZT2]

No, not supported.
Getting LUA files from OVLs that do load should be possible.
The Main.ovl archive you added does not seem to follow the usual rules. The data entries vary in size and I have no way to tell why they do.

[Tandashi]

Alright thanks :)

Tried my best to find some sort of indicator over the last 2-3 days as well for a bit, but no luck. If I should find something, will keep you posted.

What I have so far (might or might not be common knowledge already)

# Unknown. (Database.MainAvatar)
44 61 74 61 62 61 73 65 2E 4D 61 69 6E 41 76 61 74 61 72 

# Probably as terminator for string
00 

# Some path maybe where the file was located? (D:\QA\KDA_HD\Project_KDA_HD\Game\Res\GameMain\Lua\Database\MainAvatar.lua)
44 3A 5C 51 41 5C 4B 44 41 5F 48 44 5C 50 72 6F 6A 65 63 74 5F 4B 44
41 5F 48 44 5C 47 61 6D 65 5C 52 65 73 5C 47 61 6D 65 4D 61 69 6E 5C 
C 75 61 5C 44 61 74 61 62 61 73 65 5C 4D 61 69 6E 41 76 61 74 61 72 2E
6C 75 61 

# Probably as terminator for string
00

# Unknown
01 E4 29
# Unknown Maybe terminator or data
00 00 
# Unknown
56 4C 69 07



# File seems to start here

# Lua Compiler Header (https://stackoverflow.com/questions/32968051/determine-compiler-version-of-compiled-lua)
1B 4C 75 61 51 01 04 04 04 00 00 00

[HENDRIX-ZT2]

Alright thanks :)

Tried my best to find some sort of indicator over the last 2-3 days as well for a bit, but no luck. If I should find something, will keep you posted.

What I have so far (might or might not be common knowledge already)

# Unknown. (Database.MainAvatar)
44 61 74 61 62 61 73 65 2E 4D 61 69 6E 41 76 61 74 61 72 

# Probably as terminator for string
00 

# Some path maybe where the file was located? (D:\QA\KDA_HD\Project_KDA_HD\Game\Res\GameMain\Lua\Database\MainAvatar.lua)
44 3A 5C 51 41 5C 4B 44 41 5F 48 44 5C 50 72 6F 6A 65 63 74 5F 4B 44
41 5F 48 44 5C 47 61 6D 65 5C 52 65 73 5C 47 61 6D 65 4D 61 69 6E 5C 
C 75 61 5C 44 61 74 61 62 61 73 65 5C 4D 61 69 6E 41 76 61 74 61 72 2E
6C 75 61 

# Probably as terminator for string
00

# Unknown
01 E4 29
# Unknown Maybe terminator or data
00 00 
# Unknown
56 4C 69 07



# File seems to start here

# Lua Compiler Header (https://stackoverflow.com/questions/32968051/determine-compiler-version-of-compiled-lua)
1B 4C 75 61 51 01 04 04 04 00 00 00

Can you provide an ovl that loads and includes LUAs? If the lua version matches, our decompiler should handle them correctly if the meta data is removed

[Tandashi]

This OVL has a .lua and I am not getting any errors while loading it. The lua still doesn't get decompiled correctly.
Couldn't find any OVL from the game that had a correct lua export from 20-ish I've tested.

File: SpatialDatabaseTest.zip

Hopes this helps somehow.

[HENDRIX-ZT2]

The code now extracts the correct binary lua data, however it is a different, older LUA version (LuaQ = Lua 5.1). The fork of luadec used for the newer games only works with Lua 5.3. So if you can find a working luadec, we can implement that.

[HENDRIX-ZT2]

I've also fixed FGM materials.

[HENDRIX-ZT2]

TEX files also extract properly now and decode to png!

[Tandashi]

So I tried to build luadec from sources myself for version 5.1 - 5.1.5 but no matter which version I tried, trying to decompile the extracted lua gives the following error: bad header in precompiled chunk.

Not sure if I am missing something or not. Will look a bit deeper into this tomorrow though.

[HENDRIX-ZT2]

I have no clue on luadec. It's a total PITA.
What I do know is the Lua should start with �LuaQ in hex 1B 4C 75 61 51, make sure you run it with files like that. Previously, 8 extra bytes were included in the output.

[Tandashi]

Yeah, I used the latest version of master to extract. Also checked with a Hex Editor so the filed looked good extraction wise.

After some trial and error and looking at Lua bytecode a bit closer, I noticed that the header in the Lua files seems a bit off...

For a normal Lua file (64-bit, double number type), it would be:

0000  1B4C7561           header signature: "\27Lua"
0004  51                 version (major:minor hex digits)
0005  00                 format (0=official)
0006  01                 endianness (1=little endian)
0007  04                 size of int (bytes)
0008  08                 size of size_t (bytes)
0009  04                 size of Instruction (bytes)
000A  08                 size of number (bytes)
000B  00                 integral (1=integral)

But the header for the DLA Lua's is:

0000  1B4C7561           header signature: "\27Lua"
0004  51                 version (major:minor hex digits)
0005  01                 format (0=official)
0006  04                 endianness (1=little endian)
0007  04                 size of int (bytes)
0008  04                 size of size_t (bytes)
0009  00                 size of Instruction (bytes)
000A  00                 size of number (bytes)
000B  00                 integral (1=integral)

Which is super off. The endianness value for example could never be 4 since it should be either 0 (big endian) or 1 (little endian).
Also, the size of number in bytes should definitely not be 0.

Side note: Currently assuming that the Lua's were compiled for 32-bit with float as number type.
(Important since tweaks to the decompiler need to be made, but also with those it doesn't work due to the weird header format)

Also validated the extraction because my first thought was it might have missed a byte or two for whatever reason, but this is not the case. The OVL contains the same header. Did check some other OVL's as well, and the Lua files inside had the same header.

Will take a closer look at it when I got some spare time in the next couple of days, but thought I'd update with my findings.

[Tandashi]

Some more information I found out today, thought I'd document:

Lua Version: 5.1.3

Custom Header (16 bytes instead of the normal 12 bytes) :

0000  1B4C7561           header signature: "\27Lua"
0004  51                 version (major:minor hex digits)
0005  01                 endianness (1=little endian)
0006  04                 size of  either int, insturction or lua_number (bytes)
0007  04                 size of  either int, insturction or lua_number (bytes)
0008  04                 size of  either int, insturction or lua_number (bytes)
0009  00                 Unkown
000A  00                 Unkown
000B  00                 Unkown
000C  00                 Unkown
000D  00                 Unkown
000E  00                 Unkown
000F  00                 Unkown

So from that Header that means we have:
Integer Width: 4 bytes
Number Type: 4 bytes (float)
VM Instruction Width: 4 bytes per op-code

Values not in the Header because they are fixed:
size_t: 8 Bytes (64-bit compile)
format: Not specified since it's custom (doesn't really matter though)

Those are my current findings. With those values, one can at least load the VM Instructions. Modified Luadec crashes at the constants though... Will take a look at that in the next couple of days.

[Tandashi]

Loading constants is fixed now. For documentation purposes, the constants "Block" in the DLA Lua's is constructed as follows:

# Number of Constants
19 00 00 00 

# String
06 
# Padding for 8 byte blocks
00 00 00 
# size_t Length
03 00 00 00 00 00 00 00 
# Value -> _G
5F 47 00 

# String
06 
# Padding for 8 byte blocks
00 00 00 00 
# size_t Length
04 00 00 00 00 00 00 00 
# Value -> api
61 70 69 00 

# String
06 
# Padding for 8 byte blocks
00 00 00 
# size_t Length
06 00 00 00 00 00 00 00 
# Value -> table
74 61 62 6C 65 00 

...

Note: They use a custom Constant Type (0x06) which is just a regular string,

They also add padding for 8 bytes blocks after each instruction type. Meaning for e.g. type 0x6 between the length information of the string and the type of the constant 0's will be used to pad to get 8 byte blocks.

[SoraTrash15]

Is it possible yet to rip the meshes from the whole park itself, or its areas?

[HENDRIX-ZT2]

Is it possible yet to rip the meshes from the whole park itself, or its areas?

Nope. Some ovl files from DLA can not be loaded. We have not investigated models yet. If you provide an OVL with models, we may.

[HartLeeKonnor]

Is it possible yet to rip the meshes from the whole park itself, or its areas?

Nope. Some ovl files from DLA can not be loaded. We have not investigated models yet. If you provide an OVL with models, we may.

Hopefully I'm not wrong which file includes models but .ms2 files are listed in this ovl though greyed out and not importing to blender giving an error stating "padding_too_long"
SplashMountainOvlData

[HENDRIX-ZT2]

Could anyone who owns the game download current master, run cobra tools > Generate Hash Table and post the resulting files that are created in cobra_tools/hashes?
@ilodev has provided the new hashes

[HENDRIX-ZT2]

It appears that some (but not all) OVS files are somehow compressed as a whole, and have to be decompressed before the data for each archive can be read. QuickBMS bulk decompression test was not successful.

[ilodev]

On Wed, 15 Mar 2023 at 16:28, HENDRIX-ZT2 ***@***.***> wrote: It appears that some (but not all) OVS files are somehow compressed as a whole, and have to be decompressed before the data for each archive can be read. QuickBMS bulk decompression test was not successful. — Reply to this email directly, view it on GitHub <#55 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG42R75ES2AOBG73MFMF753W4HN3XANCNFSM4ON76X4Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

I’ll have a look, there might be a flag in the header indicating when it is.

[spiralingbrother]

Is it possible yet to rip the meshes from the whole park itself, or its areas?

Nope. Some ovl files from DLA can not be loaded. We have not investigated models yet. If you provide an OVL with models, we may.

I've got this ovl of the small world park area that I've been trying to get decompiled but it keeps giving list index out of range errors for some reason
the ovl and ovs stuff
aside from that weird error, there's also times where ovl files do get through but some to most items aren't properly extracted and are pretty much inaccessible

[huckleberrypie]

I have a bunch of OVL files from a legit copy of the game if anyone's up to reversing them. Replacing Black Barty with Johnny Depp would be a priority, for one.

[ilodev]

We have a copy of the game, don't worry about it!. Priority is extremely low tho, older versions of the engine usually fall in the back burner.

…

On Sat, 2 Sep 2023, 06:05 Huck Jones, ***@***.***> wrote: I have a bunch of OVL files from a legit copy of the game if anyone's up to reversing them. Replacing Black Barty with Johnny Depp would be a priority, for one. — Reply to this email directly, view it on GitHub <#55 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG42R72CIO4FTISBGGIVHALXYKWBDANCNFSM4ON76X4Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[huckleberrypie]

We have a copy of the game, don't worry about it!. Priority is extremely low tho, older versions of the engine usually fall in the back burner.
…
On Sat, 2 Sep 2023, 06:05 Huck Jones, @.> wrote: I have a bunch of OVL files from a legit copy of the game if anyone's up to reversing them. Replacing Black Barty with Johnny Depp would be a priority, for one. — Reply to this email directly, view it on GitHub <#55 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG42R72CIO4FTISBGGIVHALXYKWBDANCNFSM4ON76X4Q . You are receiving this because you were mentioned.Message ID: @.>

I see, I just wish I could mod the game somehow tho.

[AaronTheSailor]

We have a copy of the game, don't worry about it!. Priority is extremely low tho, older versions of the engine usually fall in the back burner.
…
On Sat, 2 Sep 2023, 06:05 Huck Jones, @.> wrote: I have a bunch of OVL files from a legit copy of the game if anyone's up to reversing them. Replacing Black Barty with Johnny Depp would be a priority, for one. — Reply to this email directly, view it on GitHub <#55 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG42R72CIO4FTISBGGIVHALXYKWBDANCNFSM4ON76X4Q . You are receiving this because you were mentioned.Message ID: @.>

We have a copy of the game, don't worry about it!. Priority is extremely low tho, older versions of the engine usually fall in the back burner.
…
On Sat, 2 Sep 2023, 06:05 Huck Jones, @.> wrote: I have a bunch of OVL files from a legit copy of the game if anyone's up to reversing them. Replacing Black Barty with Johnny Depp would be a priority, for one. — Reply to this email directly, view it on GitHub <#55 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG42R72CIO4FTISBGGIVHALXYKWBDANCNFSM4ON76X4Q . You are receiving this because you were mentioned.Message ID: @.>

Do you possibly have the files for the SmallWorldLand area that are extractable? The files in SpiralBrother's post a few messages (like he stated) are innaccessable. So I was wondering if you have those files that CAN be extracted properly.

[HENDRIX-ZT2]

Do you possibly have the files for the SmallWorldLand area that are extractable? The files in SpiralBrother's post a few messages (like he stated) are innaccessable. So I was wondering if you have those files that CAN be extracted properly.

It appears that there is no 'working' version of those files, they are just different from the usual format for unknown reasons.

[AaronTheSailor]

Do you possibly have the files for the SmallWorldLand area that are extractable? The files in SpiralBrother's post a few messages (like he stated) are innaccessable. So I was wondering if you have those files that CAN be extracted properly.

It appears that there is no 'working' version of those files, they are just different from the usual format for unknown reasons.

that's what I've been hearing about some other files, perhaps there is some way the format can be identified (and put to use for the program)

[HENDRIX-ZT2]

It appears that there is no 'working' version of those files, they are just different from the usual format for unknown reasons.

that's what I've been hearing about some other files, perhaps there is some way the format can be identified (and put to use for the program)

Sure, with enough dedication. We looked for a while but didn't find it. Feel free to investigate and report back!

[Gilimaster28]

Is it possible yet to rip the meshes from the whole park itself, or its areas?

Nope. Some ovl files from DLA can not be loaded. We have not investigated models yet. If you provide an OVL with models, we may.

Hopefully I'm not wrong which file includes models but .ms2 files are listed in this ovl though greyed out and not importing to blender giving an error stating "padding_too_long" SplashMountainOvlData

Can you put the splash mountain files in a google drive please?