Converting GH3 tracks to FOF - New format used in GH3 ?

[tma]

I'm doing final testing of the script now. One thing of interest is that there is a pak file called dbg.pak.ps2. Inside this is a bunch of files with the following sort of data:

Code: Select all

00000000  5b 4c 69 6e 65 4e 75 6d  62 65 72 73 5d 0d 0a 44  |[LineNumbers]..D|
00000010  75 6d 6d 79 20 31 20 28  30 2c 31 29 0d 0a 0d 0a  |ummy 1 (0,1)....|
00000020  5b 43 68 65 63 6b 73 75  6d 73 5d 0d 0a 30 78 66  |[Checksums]..0xf|
00000030  35 36 36 65 33 31 32 20  64 61 74 61 2f 73 6f 6e  |566e312 data/son|
00000040  67 73 2f 61 6e 61 72 63  68 79 69 6e 74 68 65 75  |gs/anarchyintheu|
00000050  6b 2e 6d 69 64 2e 71 62  0d 0a 30 78 61 32 31 38  |k.mid.qb..0xa218|
00000060  31 62 64 34 20 64 61 74  61 2f 73 6f 6e 67 73 2f  |1bd4 data/songs/|
00000070  61 6e 61 72 63 68 79 69  6e 74 68 65 75 6b 5f 73  |anarchyintheuk_s|
00000080  6f 6e 67 5f 73 63 72 69  70 74 73 2e 71 62 0d 0a  |ong_scripts.qb..|
00000090  0d 0a                                             |..|


My guess is that your newly created HED/WAD files aren't working because these .dbg files need to be updated with new checksums.

[tma]

Ok, my unpak perl script is finished:

http://www.moto-coda.org/gh/unpak

Run it without options to see the usage. It will handle regular pak files and the split pak/pab file archives. Let me know if you spot any bugs.

[GameZelda2]

I'm doing final testing of the script now. One thing of interest is that there is a pak file called dbg.pak.ps2. Inside this is a bunch of files with the following sort of data:

Code: Select all

00000000  5b 4c 69 6e 65 4e 75 6d  62 65 72 73 5d 0d 0a 44  |[LineNumbers]..D|
00000010  75 6d 6d 79 20 31 20 28  30 2c 31 29 0d 0a 0d 0a  |ummy 1 (0,1)....|
00000020  5b 43 68 65 63 6b 73 75  6d 73 5d 0d 0a 30 78 66  |[Checksums]..0xf|
00000030  35 36 36 65 33 31 32 20  64 61 74 61 2f 73 6f 6e  |566e312 data/son|
00000040  67 73 2f 61 6e 61 72 63  68 79 69 6e 74 68 65 75  |gs/anarchyintheu|
00000050  6b 2e 6d 69 64 2e 71 62  0d 0a 30 78 61 32 31 38  |k.mid.qb..0xa218|
00000060  31 62 64 34 20 64 61 74  61 2f 73 6f 6e 67 73 2f  |1bd4 data/songs/|
00000070  61 6e 61 72 63 68 79 69  6e 74 68 65 75 6b 5f 73  |anarchyintheuk_s|
00000080  6f 6e 67 5f 73 63 72 69  70 74 73 2e 71 62 0d 0a  |ong_scripts.qb..|
00000090  0d 0a                                             |..|


My guess is that your newly created HED/WAD files aren't working because these .dbg files need to be updated with new checksums.

Mmm, what do you mean? I'm building the HED/WAD again with the same files, so there shouldn't be any checksum change.

I've also tried to match those checksums with the CRC32 of the files and they doesn't coincid ???

EDIT: Also, I don't know Perl (I can understand some things, but not too much). But what does mean the "0x559566CC" magic? I see that the 2 magics on the top of it are the File entry/End of table DWORDS, but I don't know what this magic means

[GameZelda2]

I think that I've found a new type of track (I don't remember any reference to it before).

First, check my MIDI-QB inspection tool (finds all tracks by magic number and detects type, for now suports "Nothing", "SingleList" and "MultiList", and unknown types.

http://www.savefile.com/files/1172001

After running it with "The Metal", it reported 2 unknown tracks at 51936 and 145968 offsets (decimal).

The "Type" variable for those is 0x00010A00 (655616). I haven't too much idea about what they contain.

Also, there's anything about the HO-POs? Could they be defined in another file or automatically using a algorithm?

[tma]

My guess is that your newly created HED/WAD files aren't working because these .dbg files need to be updated with new checksums.

Mmm, what do you mean? I'm building the HED/WAD again with the same files, so there shouldn't be any checksum change.

I've also tried to match those checksums with the CRC32 of the files and they doesn't coincid ???

EDIT: Also, I don't know Perl (I can understand some things, but not too much). But what does mean the "0x559566CC" magic? I see that the 2 magics on the top of it are the File entry/End of table DWORDS, but I don't know what this magic means

I thought you said you were modifying some files in the WAD earlier, before you said you were going to retry with the original files? If you're not changing files then I'm not really sure why it wouldn't work. I just spotted the checksum parts and thought that might be related. I've no idea how they're calculated, I haven't looked into it at all.

I found the 0x559566CC magic number in the file "dbg.pak.ps2" inside the "pak" sub-directory of the WAD.

One oddity I have found is that some of the .pak file headers are missing the first letter of the filename. The "dbg.pak.ps2" is a good example, the filenames look like they should be things like "zones", "songs", "scripts" etc, but they are in fact "ones", "ongs", "cripts". It's not a bug in my script, the filenames are actually stored that way and I couldn't see where the first character might be (I'm open to any suggestions).

For example:

Code: Select all

00000000  cc 66 95 55 b0 26 02 00  7d 00 00 00 be 86 bf dc  |?f.U?&..}...?.??|
00000010  00 00 00 00 f8 17 f6 d6  00 00 00 00 20 00 00 00  |....?.??.... ...|
00000020  6f 6e 65 73 5c 7a 5f 77  6f 72 6c 64 5c 7a 5f 77  |ones\z_world\z_w|
00000030  6f 72 6c 64 5f 73 63 72  69 70 74 73 5f 70 61 6b  |orld_scripts_pak|
00000040  2e 64 62 67 00 00 00 00  00 00 00 00 00 00 00 00  |.dbg............|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000080  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000090  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|


There you can see the magic number I referred to (little endian of course), then the filename starts at 0x20.

As I was looking further through that file just now - perhaps the "Checksums" are actually magic numbers? In fact:

dbg/cripts/guitar/songlist.dbg:

[LineNumbers]
get_song_original_artist 6 (28,1859) (55,1860) (84,1861) (86,1862) (87,1864) (136,1865)
get_song_title 6 (28,1868) (55,1869) (84,1870) (86,1871) (87,1873) (136,1874)
get_song_prefix 6 (28,1877) (55,1878) (84,1879) (86,1880) (87,1882) (136,1883)
get_song_artist 10 (44,1886) (71,1887) (83,1888) (136,1889) (140,1890) (169,1891) (171,1892) (173,1893) (174,1895) (224,1896)
get_song_artist_text 6 (28,1899) (55,1900) (84,1901) (86,1902) (87,1904) (136,1905)
get_song_struct 6 (28,1908) (55,1909) (78,1910) (80,1911) (81,1913) (132,1914)
get_songlist_size 8 (0,1917) (36,1918) (51,1919) (104,1920) (140,1921) (162,1922) (164,1923) (178,1925)
get_songlist_checksum 7 (0,1928) (36,1929) (55,1930) (79,1931) (83,1932) (116,1933) (118,1934)
is_song_downloaded 12 (28,1937) (57,1939) (120,1940) (132,1941) (150,1942) (196,1943) (200,1944) (248,1945) (250,1946) (254,1947) (300,1948) (302,1949)

[Checksums]
0x54e72043 themetal
0x21902065 checksum
0x700c4085 lagrange
0x523e20a4 download_songlist
0x614910ae paintitblack
0x46a640ce startup_Script
0xddebc0ce mynameisjonas
0x6a9010dd imintheband
[..]

That checksum number corresponds with the filenames of the .imf!

[GameZelda2]

My guess is that your newly created HED/WAD files aren't working because these .dbg files need to be updated with new checksums.

Mmm, what do you mean? I'm building the HED/WAD again with the same files, so there shouldn't be any checksum change.

I've also tried to match those checksums with the CRC32 of the files and they doesn't coincid ???

EDIT: Also, I don't know Perl (I can understand some things, but not too much). But what does mean the "0x559566CC" magic? I see that the 2 magics on the top of it are the File entry/End of table DWORDS, but I don't know what this magic means

I thought you said you were modifying some files in the WAD earlier, before you said you were going to retry with the original files? If you're not changing files then I'm not really sure why it wouldn't work. I just spotted the checksum parts and thought that might be related. I've no idea how they're calculated, I haven't looked into it at all.

I found the 0x559566CC magic number in the file "dbg.pak.ps2" inside the "pak" sub-directory of the WAD.

One oddity I have found is that some of the .pak file headers are missing the first letter of the filename. The "dbg.pak.ps2" is a good example, the filenames look like they should be things like "zones", "songs", "scripts" etc, but they are in fact "ones", "ongs", "cripts". It's not a bug in my script, the filenames are actually stored that way and I couldn't see where the first character might be (I'm open to any suggestions).

For example:

Code: Select all

00000000  cc 66 95 55 b0 26 02 00  7d 00 00 00 be 86 bf dc  |?f.U?&..}...?.??|
00000010  00 00 00 00 f8 17 f6 d6  00 00 00 00 20 00 00 00  |....?.??.... ...|
00000020  6f 6e 65 73 5c 7a 5f 77  6f 72 6c 64 5c 7a 5f 77  |ones\z_world\z_w|
00000030  6f 72 6c 64 5f 73 63 72  69 70 74 73 5f 70 61 6b  |orld_scripts_pak|
00000040  2e 64 62 67 00 00 00 00  00 00 00 00 00 00 00 00  |.dbg............|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000080  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000090  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|


There you can see the magic number I referred to (little endian of course), then the filename starts at 0x20.

As I was looking further through that file just now - perhaps the "Checksums" are actually magic numbers? In fact:

dbg/cripts/guitar/songlist.dbg:

[LineNumbers]
get_song_original_artist 6 (28,1859) (55,1860) (84,1861) (86,1862) (87,1864) (136,1865)
get_song_title 6 (28,1868) (55,1869) (84,1870) (86,1871) (87,1873) (136,1874)
get_song_prefix 6 (28,1877) (55,1878) (84,1879) (86,1880) (87,1882) (136,1883)
get_song_artist 10 (44,1886) (71,1887) (83,1888) (136,1889) (140,1890) (169,1891) (171,1892) (173,1893) (174,1895) (224,1896)
get_song_artist_text 6 (28,1899) (55,1900) (84,1901) (86,1902) (87,1904) (136,1905)
get_song_struct 6 (28,1908) (55,1909) (78,1910) (80,1911) (81,1913) (132,1914)
get_songlist_size 8 (0,1917) (36,1918) (51,1919) (104,1920) (140,1921) (162,1922) (164,1923) (178,1925)
get_songlist_checksum 7 (0,1928) (36,1929) (55,1930) (79,1931) (83,1932) (116,1933) (118,1934)
is_song_downloaded 12 (28,1937) (57,1939) (120,1940) (132,1941) (150,1942) (196,1943) (200,1944) (248,1945) (250,1946) (254,1947) (300,1948) (302,1949)

[Checksums]
0x54e72043 themetal
0x21902065 checksum
0x700c4085 lagrange
0x523e20a4 download_songlist
0x614910ae paintitblack
0x46a640ce startup_Script
0xddebc0ce mynameisjonas
0x6a9010dd imintheband
[..]

That checksum number corresponds with the filenames of the .imf!

Oh, thanks for that info about the checksums! I was thinking that they were CRC32 (and they could be it, anyway ??? ).

And I was trying to rebuild an HED/WAD file with the original files, but I had that problem with the game adding more padding than the necessary.

Also, i'm thinking, could those "Checksums" be used for the game as reference for the files on the PAK? Like create an array:

files[checksum][offset] = offset
files[checksum][size] = size

and when the game wants any file, it looks at the array reference for the checksum, and gets the file with the offset/size.

[tma]

I think that I've found a new type of track (I don't remember any reference to it before).

First, check my MIDI-QB inspection tool (finds all tracks by magic number and detects type, for now suports "Nothing", "SingleList" and "MultiList", and unknown types.

http://www.savefile.com/files/1172001

After running it with "The Metal", it reported 2 unknown tracks at 51936 and 145968 offsets (decimal).

The "Type" variable for those is 0x00010A00 (655616). I haven't too much idea about what they contain.

Also, there's anything about the HO-POs? Could they be defined in another file or automatically using a algorithm?

Debr worked out those 0x000A0100 records (I think you got the endian wrong), they are another type of multilist...

There are also 0x00010000 (65536) entries, which I'm guessing contain event data for character movement and stage effects. They have a text sub-type (0x00000700) in some songs which contains data such as "guitarist_walk01" and "guitarist_start".

[tma]

I haven't got to HO/POs yet, (it's next on my list after extracting the song section names) but Debr mentioned an extra bit used on the note field that is likely to be it.

[GameZelda2]

I haven't got to HO/POs yet, (it's next on my list after extracting the song section names) but Debr mentioned an extra bit used on the note field that is likely to be it.

Yeah, I've checked it now. Bit 6 is used to define a HO-PO (and maybe a manually strum too, or another it is in another bit) of a note that shouldn't be hopoable with the normal algorithm (or at least it seems this). I didn't notice because The Metal hasn't any custom ho-po.

Black Magic Woman, entries 242 and 243 only easy level (notes 243 and 244, you can check this on the path of debr site, http://www.bradleyzoo.com/GuitarHero/gh3-ps2.easy.html, they're the R and G notes at the start of measure 106).

EDIT: Notes 3 and 4 of BMW on MEDIUM are custom hopoable too. To waste less time :;):

[tma]

I'm thinking of moving this thread back to ScoreHero as it's starting to delve into the innards of the qb files and doesn't directly relate to any FoF requirements, and any discussions about custom GH3 songs should probably be on ScoreHero anyway.

A call to FoF people: what other info do you need? I think most of the requirements (note charts, timing, beats) has been worked out now and this is all you really need to get the tracks working in FoF, right?

[Desertman123]

This is a nooby question, but how do you use python scripts?

[tma]

This is a nooby question, but how do you use python scripts?

I've no idea how you would run it on Windows (assuming you're on Windows). Most Linux and all Mac OS X installations have python installed.

http://www.python.org/ would be a place to start.

[Pudding]

I'm thinking of moving this thread back to ScoreHero as it's starting to delve into the innards of the qb files and doesn't directly relate to any FoF requirements, and any discussions about custom GH3 songs should probably be on ScoreHero anyway.

A call to FoF people: what other info do you need? I think most of the requirements (note charts, timing, beats) has been worked out now and this is all you really need to get the tracks working in FoF, right?

Is there a place where information about the format of the qb files is collected? I've been following this thread but I can't seem to find where the timing information is specified.

I think the specifics of the HOPO algorithm and the HOPO bit are pertinent for FOF as there may be room to incorporate custom HOPO's at some point.

[Curunir]

Does this new hopo/extraction information mean that there will be another release of the GH3 tunes?

[tma]

I'm in the process of writing up some detailed specs on what we know about the mid.qb format, and I plan to post it on ScoreHero once I'm done. That will probably take me a few days to collate all the information and explore a few things. Expect something by the end of the week.

Debr went through some of the specifics on the timing here:

https://www.fretsonfire.org/cgi-bin....y116289

The simple answer to it is, there are two sections somewhere after the co-op charts. They define the time signatures (4/4, 3/4, 6/8, etc) and also the timing of each "beat" of the song. The actual timing value used in the chart sections are based on a fixed 125BPM, which once it's all calculated down equates to the "length" value of each note in the chart section equalling the number of milliseconds since the start of the song. Note this is independent of the time signature and the "beats" track, which are (we believe) just used for display purposes, to determine how quickly Star Power is used, and probably something to do with the score gained from sustained notes.

On the subject of sustained notes, there are some anomalies between what's in the files and what is actually in the game. Some sustains as calculated from the charts are either too short or too long. We (well, Debr, I haven't really looked into it too much yet) still haven't figured this out yet.

In case it hasn't been mentioned already, I believe there are two cases where HO/PO apply:

1) the normal GH2 way (notes close to each other).
2) the new GH3 way, where notes are specifically defined as HO/PO.

The GH3 method is simply the 6th bit of the "notes" value in the charts. If this is set, the note is a HO/PO and will show up in the game with a white ring around it.

I haven't actually played it enough to verify if the old GH2 method still works. I've had a number of notes that I thought I should have been able to hammer but couldn't, so I don't know about that.