Converting GH3 tracks to FOF - New format used in GH3 ?

[YMS]

D'oh!
I've just read at Scorehero Board that someone has successfully extracted MIDI files from the PAK files (though they are unreadable for him yet). I guess I was wrong with that, though I don't understand how this can be .

I've contacted dgendreau, and hopefully we'll know more soon.

EDIT: Yeah, just what FTGF quoted in his post .

[FTGF]

D'oh!
I've just read at Scorehero Board that someone has successfully extracted MIDI files from the PAK files (though they are unreadable for him yet). I guess I was wrong with that, though I don't understand how this can be .

I've contacted dgendreau, and hopefully we'll know more soon.

Haha we saw the same thing at the same time, keep us updated. I might start working on putting the songs together then when we get the MIDI's we can combine and publish

Edit: Maybe not, the torrent is not seeded. If anyone has the ps2 version of the game, if you could post some songs (as may as your willing to do) to mediafire so we can get started combining!

[adthrawn]

I'm uploading them all right now to rapidshare....I'll make a torrent to. Hold on a few.

EDIT
Here is the torrent of ALL of the music files. I'll seed this for a day or two.
Download

EDIT 2
Let me know if it doesn't work becasue when I hit the address above it shows no seeds, but that can be deceiving.

EDIT 3
Here a 3 or 4 songs.
Download

[Def_Monk]

The torrent works, I figured I'd test for you. I'm not gonna download the thing though, I already have the full DVD. I tried emulating it with Pcsx2 (Pc run PS2 emulator that runs iso's) but it didnt work. It crashed from incompatibility and whatnot. That suks. Oh well, hopefully we can find out how to rip them. lol :p.

[adthrawn]

All you have to do is moun the iso using MagicISO or something adn you can see the files.

[Lange]

All you have to do is moun the iso using MagicISO or something adn you can see the files.

Or open it with winrar.

[YMS]

There has been a problem with short stream names in MSV2VAG 1.0. So here comes version 1.1, in which I also removed "rest data" extraction. As rest data doesn't contain sound information nor the MIDI, we won't need it.


As those PAK files posted here have not been extracted correctly, can someone post the WAD / HED file they were contained in?

[adthrawn]

Yeah. I'll also see about making a separate torrent for the pak files if you want it.

EDIT

You'll have to wait till around 2:30, I can't upload from school.

[cama]

YMS this file have 400mb maybe faster will be if you tell me how extract this correctly ? I used HED Extractor suggested in scorehero and im not had any errors durring extraction. But maybe you have right maybe ned any other HED extractor hmm

I checked from this hed extracted package .ini files should be txt files i think but here emplty bytes , i think really extractor not workin properly

[YMS]

Oh, well, okay.

The following is what dgendreau from Scorehero wrote me about that topic. I don't know if that helps you / how much you can follow, but if you are able to use a hex editor, you could at least follow this special instructions:

"I extracted the file "anarchyintheuk.pak.ps2" from datap.wad using a hex editor.
Starting at offset 0x0015098000, I extracted 0x2FBC0 bytes to a separate file."

If you could manage to send me that file (it will be < 200 kB this time ), that would be great!


So here's dgendreau text:


"Notes on disc:

Code: Select all

SECTOR_SIZE = 0x800

// we find the WAD sub-file descriptor for "\songs\anarchyintheuk.pak.ps2"
in datap.hed:
0000061AC0 30 A1 02 00 // STARTING_SECTOR in wad file: 0x02a130 * SECTOR_SIZE == 0x15098000
0000061AC4 C0 FB 02 00 // SIZE(bytes) (end offset=0x150C7BC0
0000061AC8 "\songs\anarchyintheuk.pak.ps2" // pad out to nearest DWORD with 0s

in datap.wad
// anarchyintheuk.pak.ps2 file starts here
0015098000 5E E8 C1 4B B0 01 00 00 A4 F9 02 00 12 E3 66 F5 00 00 00 00 FF 21 6A A8 00 00 00 00 20 00 00 00
0015098020 "data/songs/anarchyintheuk.mid.qb.ps2"
...
00150980C0 5E E8 C1 4B A0 FA 02 00 48 00 00 00 D4 1B 18 A2 00 00 00 00 92 3D D4 33 00 00 00 00 20 00 00 00
00150980E0 "data/songs/anarchyintheuk_song_scripts.qb.ps2"

To make things simpler, I extracted the file "anarchyintheuk.pak.ps2" from datap.wad using a hex editor.
Starting at offset 0x0015098000, I extracted 0x2FBC0 bytes to a separate file:

The sub-file descriptor table at the start looks like this:

Code: Select all

anarchyintheuk.pak
 Ã‚   SUBFILE_DESCRIPTOR
0000000000 5E E8 C1 4B // ???? weird magic number
0000000004 B0 01 00 00 // SUBFILE_START_OFFSET,
0000000008 A4 F9 02 00 // SUBFILE_SIZE (190kb sounds ok for a midi)
000000000C 12 E3 66 F5 // unknown
0000000010 00 00 00 00
0000000014 FF 21 6A A8 // filename for "anarchyintheuk" song file is "A86A21FF.IMF"!!!
0000000018 00 00 00 00
000000001C 20 00 00 00
0000000020 "data/songs/anarchyintheuk.mid.qb.ps2"

 Ã‚   SUBFILE_DESCRIPTOR
00000000C0 5E E8 C1 4B // ???? weird magic number
00000000C4 A0 FA 02 00 // SUBFILE_START_OFFSET, (0x2FAA0+0xC0 = 0x2FB60)
00000000C8 48 00 00 00 // SUBFILE_SIZE (72 bytes)
00000000CC D4 1B 18 A2 // unknown
00000000D0 00 00 00 00
00000000D4 92 3D D4 33 // unknown
00000000D8 00 00 00 00
00000000DC 20 00 00 00
00000000E0 "data/songs/anarchyintheuk_song_scripts.qb.ps2"

 Ã‚   SUBFILE_DESCRIPTOR
0000000180 5F 56 24 B5 // ????end of sub-file descriptor table magic number
0000000184 30 FA 02 00 // seems to offset to an empty sub-file at the end of the pak file
0000000188 04 00 00 00
000000018C 00 00 00 00
0000000190 00 00 00 00
0000000194 00 00 00 00
0000000198 00 00 00 00
000000019C 00 00 00 00
00000001A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

I havent nailed down how the SUBFILE_START_OFFSET works exactly, but I believe you have to add the start of each sub-file descriptor with its SUBFILE_START_OFFSET.

This yields the following partial dump of each sub-file

Code: Select all

   somewhere near sub-file "anarchyintheuk.mid.qb.ps2"
00000001B0 00 00 00 00
00000001B4 A4 F9 02 00 // same value as SUBFILE_SIZE in table above
00000001B8 1C 08 02 04 10 04 08 0C
00000001C0 0C 08 02 04 14 02 04 0C 10 10 0C 00 00 04 0C 00
00000001D0 DF BA 39 D4 80 30 6B 69 30 00 00 00 00 00 00 00
00000001E0 00 01 00 00 00 00 00 00 00 00 00 00 00 04 0C 00
00000001F0 87 E5 45 D5 80 30 6B 69 50 00 00 00 00 00 00 00
...

 Ã‚   somewhere near sub-file "anarchyintheuk_song_scripts.qb.ps2"
000002FB60 00 00 00 00
000002FB64 48 00 00 00 // same value as SUBFILE_SIZE in table above
000002FB68 1C 08 02 04 10 04 08 0C
000002FB70 0C 08 02 04 14 02 04 0C 10 10 0C 00 00 04 07 00
000002FB80 5A FA 44 0A D4 1B 18 A2 30 00 00 00 00 00 00 00
000002FB90 0B F7 19 01 09 00 00 00 09 00 00 00 01 16 3B 4D
000002FBA0 0B AD 2C 01 24 00 00 00 00 00 00 00 00 00 00 00
000002FBB0 AB AB AB AB 00 00 00 00 00 00 00 00 00 00 00 00

Notice how each sub file seems to start with the same header of some sort:
00 00 00 00 (DWORD)sf_size 1C 08 02 04 10 04 08 0C 0C 08 02 04 14 02 04 0C 10 10 0C 00 00 04
Then they start to deviate.


...


The reason you are seeing repeating useless junk in the pak files is because they were not extracted correctly.
The sizes seem all wrong and the contents are not the same as the pak files that I have extracted by hand with a hex editor.

Most likely, the extractor utility is not interpreting the hed file format correctly.
Each record in the hed file is formatted as follows:
NOTE: integer format is little-endian

Code: Select all

DWORD file_start_sector // sub-file start offset into WAD file, in SECTORS
DWORD file_size // size of sub-file in BYTES
char[] pathname // null terminated string for pathname
BYTE[] padding // pad record out to next DWORD boundary with 0s

Most likely, the extraction utility is interpreting the first field as an offset in bytes instead of Sectors (0x800 bytes per sector). This would explain the repetition and overlaps.

Another gotcha could be the padding after each string in the hed file.

Finally, when I wrote VGSPlayer, I ran into several DVDROM drives that did not read UDF format discs larger than 2GB in size correctly.
I fixed this by reading the raw disc sectors and parsing the UDF disc format directly."

[cama]

wow big instruction thanks but allready 3 mins ago i extracted i think succesfully this HED file using other extractor program found by google i see extracted ini looks like ini file :>

now songs .pak file looks like pak have inside data/songs/closer.mid.qb.ps2 etc

Here 2 .pak songs extracted: http://uploaded.to/?id=gp9o7t

[YMS]

Ah, now that's better! Big Thanks! I'll work with it.

[DesktopMan]

The .pak files has a list of headers first, 192 bytes for each file it contains. As they contain midi files but no midi headers are visible the .pak files are apparently packed and/or scrambled.

It seems a full 160 bytes are set to file name, which greatly reduces the work required to understand the header.

5EE8C14B seems to be the header magic word. Something like:

struct header
{
char magic[4];
int offset;
int size;
char unknown[20];
char name[160];
};

Of course, this is just from 2 minutes of looking, I might look more at it later.

The midi file might not be in the pak file, there's also the possibility that .mid.qb files are *only* script files without note charts. .qb is script files used on TonyHawk.

[YMS]

The .pak files has a list of headers first, 192 bytes for each file it contains. As they contain midi files but no midi headers are visible the .pak files are apparently packed and/or scrambled.

It seems a full 160 bytes are set to file name, which greatly reduces the work required to understand the header.

5EE8C14B seems to be the header magic word. Something like:

struct header
{
char magic[4];
int offset;
int size;
char unknown[20];
char name[160];
};

Of course, this is just from 2 minutes of looking, I might look more at it later.

Yes, I can verify what you said (except the packed thing - I don't believe that). Or, in more detail, I can verify what dgendreau said, who also found out how to calculate Offsets, and such (see one of my last posts).

So here's the MID.QB files from the two PAKs cama has uploaded, for anybody who's interessted in looking at them. Now it's time to decode them!):

http://rapidshare.com/files/65082657/gh3-mid.qb.rar.html


Those MID.QB files are organized in 4-Byte-blocks, where first byte always is != 0, second byte most times is != 0, third byte most times is == 0, fourth byte almost always == 0.
So I don't believe this is a packed format, rather one with control characters which might be converted to MIDI comparatively easy.

[DesktopMan]

.qb is used in Tony Hawk, so if they kept the scripting system from there I don't know why there's .mid added to the extension... hmm