04699: cubeqst: Upgrade to chd v5 changes SHA1

ID	Category [?]	Severity [?]	Reproducibility	Date Submitted	Last Update
04699	Core	Major	Always	Feb 20, 2012, 19:41	Mar 27, 2012, 08:37

Tester	Scagazza	View Status	Public	Platform	MAME (Self-compiled)
Assigned To	aaron	Resolution	Fixed	OS	Windows XP (64-bit)
Status [?]	Resolved			Driver
Version	0.145u1	Fixed in Version	0.145u4	Build	64-bit
		Fixed in Git Commit		Github Pull Request #

Summary	04699: cubeqst: Upgrade to chd v5 changes SHA1
Description	I have updated cubeqst.chd with command: chdman copy -i cubeqst.chd -o newcubeqst.chd renamed the new chd and then ran the game: C:\MAME>mame cubeqst cubeqst.chd WRONG CHECKSUMS: EXPECTED: SHA1(d0e24bb5d0ae424e0816110ec7d6b21189044d57) FOUND: SHA1(1c5e002450df6bf83b2a6cda919c06173f2f0e22) WARNING: the game might not run correctly. Average speed: 99.99% (229 seconds) In addition I got this error when I quit the game: Error: attempt to free untracked memory in (null)(0)! 000000000022D408: 00000000010DA4E9 (not found) 000000000022D410: 000007FF7FC79230 (iob+0x0060) 000000000022D418: 000000000489D75E (not found) 000000000022D420: 000000000022D418 (not found) 000000000022D428: 000000000022D418 (not found) 000000000022D430: 000000000022D580 (not found) 000000000022D438: 0000000000000000 (not found) 000000000022D440: 000007FF7250CF40 (SymFunctionTableAccess) 000000000022D448: 000007FF7250D390 (SymGetModuleBase64) 000000000022D450: 000007FF725B6950 (MiniDumpWriteDump+0x8c1f0) 000000000022D4D0: 0000000077D8C0FD (DosDateTimeToFileTime+0x00bd)
Steps To Reproduce
Additional Information
Github Commit

Flags
Regression Version	0.145u1
Affected Sets / Systems	cubeqst

Attached Files	sidebyside.png (101,950 bytes) Feb 22, 2012, 17:51 Uploaded by Haze
Attached Files	chd_convert.7z (72,220 bytes) Feb 23, 2012, 17:06 Uploaded by Firewave CHD conversion information

No.08223 Tafoid Administrator Feb 20, 2012, 20:24 edited on: Feb 20, 2012, 20:31	Before I look into this, I want to make sure your original image verifies correctly. Take a 0.145 version of CHDMAN and -verify your original (version 4) CHD.
No.08224 Scagazza Tester Feb 20, 2012, 20:46	Original chd verified: C:\MAME\roms\cubeqst>chdman -verify cubeqst.chd chdman - MAME Compressed Hunks of Data (CHD) manager 0.145 (Feb 5 2012) Input file: cubeqst.chd SHA1 verification successful!..
No.08225 Tafoid Administrator Feb 20, 2012, 21:36	I was able to confirm it for cubeqst at least.. it gave me even a different SHA1 than you printed.
No.08226 Haze Senior Tester Feb 20, 2012, 22:20	worst case scenario this is critical and will be destroying data in the CHDs, so whatever you do absolutely do not delete the original LD CHDs ;-)
No.08227 Mr_Person Tester Feb 21, 2012, 00:57	It seems that disc games are getting new metadata, and subsequently new SHA1s. ===== Input file: cap-jjk000.chd File Version: 5 Logical size: 65,821,824 bytes Hunk Size: 9,792 bytes Total Hunks: 6,722 Unit Size: 2,448 bytes Total Units: 26,888 Compression: lzma (LZMA), zlib (Deflate), huff (Huffman), cdfl (CD FLAC) CHD size: 39,513,739 bytes Ratio: 60.0% SHA1: bb9fbb462251937cc0a9e78b11226da8719c251d Data SHA1: d5a11315ac21e573ffe78e63602ec2cb420f361f Metadata: Tag='CHT2' Index=0 Length=88 bytes TRACK:1 TYPE:MODE1 SUBTYPE:NONE FRAMES:26888 PREGAP:0 PGTYPE Input file: cap-jjk000.chd.old File Version: 4 Logical size: 65,821,824 bytes Hunk Size: 9,792 bytes Total Hunks: 6,722 Unit Size: 2,448 bytes Total Units: 26,888 Compression: zlib (Deflate) CHD size: 45,108,605 bytes Ratio: 68.5% SHA1: 09869f6d8c032b527e02d815749dc8fab1289e86 Data SHA1: d5a11315ac21e573ffe78e63602ec2cb420f361f Metadata: Tag='CHTR' Index=0 Length=45 bytes TRACK:1 TYPE:MODE1 SUBTYPE:NONE FRAMES:26888. ===== Meanwhile, HDD games such as area51 and kinst are updating fine and have identical metadata.
No.08228 Haze Senior Tester Feb 21, 2012, 02:45	That shouldn't be happening.... the idea was that the SHA1s stayed the same, so why they're getting new / different metadata I don't know. The set in question is a laserdisc game and is a different problem tho, and worrying if Tafoid and the OP get different results. Aaron needs to address both tho, ideally this should have been tested and worked out before U1, because if people convert their CHDs now they risk irreversably damaging them.
No.08229 B2K24 Senior Tester Feb 21, 2012, 02:49	** this is not a complaint as I always back everything up. I'm just trying to help sharing information * I just finished converting a complete set of 0.145u1 CHDs and scanning with CMP 4.03a with chdman properly set to V5 and path to it properly set. 248/559 are incorrect according to the audit. chdman 0.145u1 CRC32: F4D4CFDB MD5: 9499D54BAF3F88385C86D773C65BE5B7 SHA-1: 6064A4875AB7E04D590246B42782BD6D0D27198A http://pastebin.com/NRujHpXV
No.08231 R. Belmont Developer Feb 21, 2012, 04:22 edited on: Feb 21, 2012, 04:44	It is correct behavior for CHTR type CHD-CDs to change SHA1s on conversion because the metadata is updated to CHT2. Such discs were made with a pre-2008 version of CHDMAN and should have been marked as BAD_DUMP once CHDMAN gained the capability to preserve gap data. They all need to be redumped as source becomes available. (Note that the Data SHA1s do match in Mr. Person's report, which proves it's simply the metadata change). I have no idea what's going on with the GD-ROMs yet; none of this stuff should affect those.
No.08233 Scagazza Tester Feb 21, 2012, 08:24	I just wonder if chds introduced in 0.145u1 and dumped directly with v5 could be considered good dump or not. initdv3e\gds-0033.chd jojoba\cap-jjm-120.chd
No.08235 R. Belmont Developer Feb 21, 2012, 13:13 edited on: Feb 21, 2012, 13:16	initdv3e is a v4 image. I didn't add jojoba so I can't speak for that, but it likely is as well. To clarify, if "chdman info" shows this: Metadata: Tag='CHTR' Index=0 Length=45 bytes then it's a bad dump. If it shows this: Metadata: Tag='CHT2' Index=0 Length=88 bytes then it's a good dump. (Not including harddisks, DVDs, and laserdiscs, of course).
No.08236 Haze Senior Tester Feb 21, 2012, 13:40 edited on: Feb 21, 2012, 13:49	and obviously for the cases where CHTR changes to CHT2 and the SHA1 changes as a result the source should be updated with the correct SHA1s (should probably have been done, with appropriate BAD_DUMP tags before u1 to avoid confusion) the GD-ROMs are going to need some investigation as to why they've changed (that accounts for most of the list) Luckily they're easy to spot because they all had gd* names :-) the LD issue definitely needs looking at if the metadata is changing, especially if different people are getting different results as indicated here.
No.08237 R. Belmont Developer Feb 21, 2012, 14:24	Update: the GD-ROMs that changed are also CHTR. So in those cases the SHA1s should be updated, but since we don't get gap information from the common .gdi format there's no need to BAD_DUMP them.
No.08238 Firewave Senior Tester Feb 21, 2012, 14:40	All chdman info: http://pastebin.com/a3hFezbB Aorry I accidentally deleted the are51mx after converting it, so I just have the v5 one.
No.08239 Haze Senior Tester Feb 21, 2012, 15:29	does the bad cubeqst pass -verify? I'd be interested in knowing if it does because the hunk size /4 is greater than 65535 which is the flac block limit.. and flac will refuse to encode if you go over it, hence why I added the saftey checks for it when encoding CD / HDD FLAC ( http://www.sendspace.com/file/ikgp19 ) (having multiple flac blocks in a hunk isn't a problem, the codec handles it transparently) but maybe the AV codec never has a hunk of entirely FLAC/ there is also a lower limit of 16 samples (64 bytes) (feeding it less data isn't a problem either, it will just be a bit wasteful) all depends how the FLAC stuff gets encoded in LDs tho, and might not be the issue at all. It's one avenue I'd investigate tho.
No.08241 B2K24 Senior Tester Feb 21, 2012, 16:50	It does not pass -verify I get error executing verify command (Error reading CHD file (newcubeqst.chd): decompression error) http://pastebin.com/3cGxnxdQ
No.08242 Firewave Senior Tester Feb 21, 2012, 21:59	I was missing some CHDs before. Here's the complete list: http://pastebin.com/c5Ena34r
No.08245 Haze Senior Tester Feb 22, 2012, 12:16	if it doesn't pass -verify it's more than a metadata problem then. the SHA1 used (and written to the header) is calculated from the raw input data during compression. Sounds like the CHD files it makes are basically just invalid. Wouldn't surprise me if it's what I'm saying above + more. btw the radio silence on the other aspects of my patch (optimal hunk + flac block sizes / settings) in addition to the safety checking there is annoying. Has anybody run verify over a complete set of converted images?
No.08246 Firewave Senior Tester Feb 22, 2012, 13:44	I am running a test right now chdman info -i old.chd chdman verify -i old.chd chdman copy -i old.chd -o new.chd chdman info -i new.chd chdman verify -i new.chd It's very time consuming. It's running since half a day and it's just at gauntleg and so far all converted LD CHDs failed to verify. I also had two verification errors with unconverted CHDs (drmn5m/b05jaa02.chd and drmn6m/b16jaa02.chd), but maybe those were damaged to begin with.
No.08247 Haze Senior Tester Feb 22, 2012, 14:57	V4 D:\mame180212>chdman verify -i d:\b05jaa02.chd chdman - MAME Compressed Hunks of Data (CHD) manager 0.145u1 (Feb 19 2012) Raw SHA1 verification successful! Overall SHA1 verification successful! D:\mame180212>chdman verify -i d:\b16jaa02.chd chdman - MAME Compressed Hunks of Data (CHD) manager 0.145u1 (Feb 19 2012) Raw SHA1 verification successful! Overall SHA1 verification successful! check your HDD?
No.08249 Firewave Senior Tester Feb 22, 2012, 16:53	Apparently somebody put bogus versions of them out, since I have at least one other guy with bad ones. Will try to get good ones.
No.08250 Haze Senior Tester Feb 22, 2012, 17:52	the actual metadata in cubeqst is changing tho, yes.. doesn't explain the decompression errors tho and I haven't managed to untangle from the code the meaning of the bits which have changed.
No.08253 Haze Senior Tester Feb 22, 2012, 19:03 edited on: Feb 22, 2012, 19:15	The actual point of failure when decompressing Cube Quest (which is one which reports the right data SHA1) is if (totalsize >= complength) return AVHERR_INVALID_DATA; in decode_data if you remove that check (cubeqst) does at least complete and say that the SHA1 matches what was expected, but it shouldn't be failing in the first place. This doesn't explain why the Metadata has changed either.
No.08256 Firewave Senior Tester Feb 23, 2012, 04:23	I got this error while converting mach3.chd pure virtual method called Will try to get a backtrace later.
No.08257 B2K24 Senior Tester Feb 23, 2012, 05:04	seems to convert and verify here http://pastebin.com/zedm1W30
No.08260 Firewave Senior Tester Feb 23, 2012, 14:56	OK, the "pure virtual" error happens when there is not enough diskspace during a copy. Also the bad CHDs I encountered were just incomplete files, so I suggest a filesize check when loading a CHD (it's in the header anyways and a ftell() call shouldn't hurt).
No.08323 Scagazza Tester Mar 7, 2012, 11:01	I have tested chdman compiled against latest mess svn (rev. 14714) with Aaron's fix, the SHA1 is now correct but the new chd doesn't pass the verify. C:\MAME\roms\cubeqst>chdman info -i cubeqst.chd chdman - MAME Compressed Hunks of Data (CHD) manager 0.145u3 (Mar 7 2012) Input file: cubeqst.chd File Version: 4 Logical size: 41,096,611,968 bytes Hunk Size: 380,496 bytes Total Hunks: 108,008 Unit Size: 380,496 bytes Total Units: 108,008 Compression: avhu (A/V Huffman) CHD size: 11,517,456,445 bytes Ratio: 28.0% SHA1: d0e24bb5d0ae424e0816110ec7d6b21189044d57 Data SHA1: 5266a0a43963628464fac1719e1cefec627042a4 Metadata: Tag='AVAV' Index=0 Length=76 bytes FPS:59.940058 WIDTH:720 HEIGHT:262 INTERLACED:1 CHANNELS:2 S Metadata: Tag='AVLD' Index=0 Length=1728128 bytes C:\MAME\roms\cubeqst>chdman info -i newcubeqst.chd chdman - MAME Compressed Hunks of Data (CHD) manager 0.145u3 (Mar 7 2012) Input file: newcubeqst.chd File Version: 5 Logical size: 41,096,611,968 bytes Hunk Size: 380,496 bytes Total Hunks: 108,008 Unit Size: 380,496 bytes Total Units: 108,008 Compression: avhu (A/V Huffman) CHD size: 11,465,974,734 bytes Ratio: 27.9% SHA1: d0e24bb5d0ae424e0816110ec7d6b21189044d57 Data SHA1: 5266a0a43963628464fac1719e1cefec627042a4 Metadata: Tag='AVAV' Index=0 Length=76 bytes FPS:59.940058 WIDTH:720 HEIGHT:262 INTERLACED:1 CHANNELS:2 S Metadata: Tag='AVLD' Index=0 Length=1728128 bytes C:\MAME\roms\cubeqst>chdman verify -i newcubeqst.chd chdman - MAME Compressed Hunks of Data (CHD) manager 0.145u3 (Mar 7 2012) Error reading CHD file (newcubeqst.chd): decompression error
No.08329 Scagazza Tester Mar 7, 2012, 19:21	IMHO this bug is not solved. Now the SHA1 is correct but the new created chd doesn't verify. Please read the post!
No.08330 Tafoid Administrator Mar 7, 2012, 19:45	The bug is regarding changing the SHA1.. which is no longer does. "verify" is another issue which is being worked on and since it's not part of a U release, doesn't need to be reported in a separate report.. The resultant image appears good, regardless.
No.08331 Scagazza Tester Mar 7, 2012, 19:50	How can you say that the resultant image appears good if it doesn't verify?
No.08394 Darkfalz Tester Mar 19, 2012, 15:06	I did a huge 2-day batch convert. At least half of CHDs now fail. I hope these are updated in new driver and we don't need to reacquire them all. I don't see it mentioned in the thead but all the Beatmania derivatives have also changed SHA-1, not just the GD rips.
No.08395 Haze Senior Tester Mar 19, 2012, 15:38	I can't confirm Beatmania SHA1s changing, I tested several with no change.
No.08411 Darkfalz Tester Mar 26, 2012, 19:58	Pretty much all my CD based games failed (Beatmania, CPS3, Konomi etc.) is there a command I should have used to convert these rather than copy -i -o ? They're backed up so I can try them again, but I'm really confused.
No.08416 Haze Senior Tester Mar 27, 2012, 03:34	A lot of the CD based games will have their SHA1s change unfortunately, this is because the old metadata (track gaps and the like) gets updated to a new format in the process, and to ensure the format is secure this data forms part of the overall hash. The hashes haven't yet been updated in MAME, which will need to happen before 0.146 ships, dunno who is doing that tho. The MESS lists fare better, with only a handful of images having the old tags, thus them not changing when the chds are updated. I'm hoping the ECC compression code I've given to Aaron will be included before 0.146 (Because it helps a LOT with the MESS Saturn sets), although that won't change the SHA1s any furher, just people should be aware that there might be at least one more tweak to the actual format prior to 0.146 all going well.
No.08418 Darkfalz Tester Mar 27, 2012, 08:37	Sounds like another 2 days of recompressing CHDs might be in my future then ;-)