Viewing Issue Advanced Details Jump to Notes ]
ID Category [?] Severity [?] Reproducibility Date Submitted Last Update
04699 Core Major Always 2012-02-20 19:41 2012-03-27 08:37
Tester Scagazza View Status Public Platform MAME (Self-compiled)
Assigned To aaron Resolution Fixed OS Windows XP (64-bit)
Status [?] Resolved   Driver
Version 0.145u1 Fixed in Version 0.145u4 Build 64-bit
Summary 04699: cubeqst: Upgrade to chd v5 changes SHA1
Description I have updated cubeqst.chd with command:
chdman copy -i cubeqst.chd -o newcubeqst.chd
renamed the new chd and then ran the game:

C:\MAME>mame cubeqst
cubeqst.chd WRONG CHECKSUMS:
    EXPECTED: SHA1(d0e24bb5d0ae424e0816110ec7d6b21189044d57)
     FOUND: SHA1(1c5e002450df6bf83b2a6cda919c06173f2f0e22)
WARNING: the game might not run correctly.
Average speed: 99.99% (229 seconds)

In addition I got this error when I quit the game:

Error: attempt to free untracked memory in (null)(0)!
  000000000022D408: 00000000010DA4E9 (not found)
  000000000022D410: 000007FF7FC79230 (iob+0x0060)
  000000000022D418: 000000000489D75E (not found)
  000000000022D420: 000000000022D418 (not found)
  000000000022D428: 000000000022D418 (not found)
  000000000022D430: 000000000022D580 (not found)
  000000000022D438: 0000000000000000 (not found)
  000000000022D440: 000007FF7250CF40 (SymFunctionTableAccess)
  000000000022D448: 000007FF7250D390 (SymGetModuleBase64)
  000000000022D450: 000007FF725B6950 (MiniDumpWriteDump+0x8c1f0)
  000000000022D4D0: 0000000077D8C0FD (DosDateTimeToFileTime+0x00bd)
Steps To Reproduce
Additional Information
Flags
Regression Version 0.145u1
Affected Sets / Systems cubeqst
Attached Files
png file icon sidebyside.png (101,950 bytes)  2012-02-22 17:51 by Haze
Haze
7z file icon chd_convert.7z (72,220 bytes)  2012-02-23 17:06 by Firewave
CHD conversion information


-  Notes
User avatar (08223)
Tafoid (Administrator)
2012-02-20 20:24
edited on: 2012-02-20 20:31

Before I look into this, I want to make sure your original image verifies correctly.
Take a 0.145 version of CHDMAN and -verify your original (version 4) CHD.

User avatar (08224)
Scagazza (Tester)
2012-02-20 20:46

Original chd verified:
C:\MAME\roms\cubeqst>chdman -verify cubeqst.chd
chdman - MAME Compressed Hunks of Data (CHD) manager 0.145 (Feb 5 2012)
Input file: cubeqst.chd
SHA1 verification successful!..
User avatar (08225)
Tafoid (Administrator)
2012-02-20 21:36

I was able to confirm it for cubeqst at least.. it gave me even a different SHA1 than you printed.
User avatar (08226)
Haze (Developer)
2012-02-20 22:20

worst case scenario this is critical and will be destroying data in the CHDs, so whatever you do absolutely do not delete the original LD CHDs ;-)
User avatar (08227)
Mr_Person (Tester)
2012-02-21 00:57

It seems that disc games are getting new metadata, and subsequently new SHA1s.

=====

Input file: cap-jjk000.chd
File Version: 5
Logical size: 65,821,824 bytes
Hunk Size: 9,792 bytes
Total Hunks: 6,722
Unit Size: 2,448 bytes
Total Units: 26,888
Compression: lzma (LZMA), zlib (Deflate), huff (Huffman), cdfl (CD FLAC)
CHD size: 39,513,739 bytes
Ratio: 60.0%
SHA1: bb9fbb462251937cc0a9e78b11226da8719c251d
Data SHA1: d5a11315ac21e573ffe78e63602ec2cb420f361f
Metadata: Tag='CHT2' Index=0 Length=88 bytes
              TRACK:1 TYPE:MODE1 SUBTYPE:NONE FRAMES:26888 PREGAP:0 PGTYPE

Input file: cap-jjk000.chd.old
File Version: 4
Logical size: 65,821,824 bytes
Hunk Size: 9,792 bytes
Total Hunks: 6,722
Unit Size: 2,448 bytes
Total Units: 26,888
Compression: zlib (Deflate)
CHD size: 45,108,605 bytes
Ratio: 68.5%
SHA1: 09869f6d8c032b527e02d815749dc8fab1289e86
Data SHA1: d5a11315ac21e573ffe78e63602ec2cb420f361f
Metadata: Tag='CHTR' Index=0 Length=45 bytes
              TRACK:1 TYPE:MODE1 SUBTYPE:NONE FRAMES:26888.

=====

Meanwhile, HDD games such as area51 and kinst are updating fine and have identical metadata.
User avatar (08228)
Haze (Developer)
2012-02-21 02:45

That shouldn't be happening.... the idea was that the SHA1s stayed the same, so why they're getting new / different metadata I don't know.

The set in question is a laserdisc game and is a different problem tho, and worrying if Tafoid and the OP get different results.

Aaron needs to address both tho, ideally this should have been tested and worked out *before* U1, because if people convert their CHDs now they risk irreversably damaging them.
User avatar (08229)
B2K24 (Senior Tester)
2012-02-21 02:49

**** this is not a complaint as I always back everything up. I'm just trying to help sharing information ***

I just finished converting a complete set of 0.145u1 CHDs and scanning with CMP 4.03a with chdman properly set to V5 and path to it properly set.

248/559 are incorrect according to the audit.

chdman 0.145u1
CRC32: F4D4CFDB
MD5: 9499D54BAF3F88385C86D773C65BE5B7
SHA-1: 6064A4875AB7E04D590246B42782BD6D0D27198A


http://pastebin.com/NRujHpXV
User avatar (08231)
R. Belmont (Developer)
2012-02-21 04:22
edited on: 2012-02-21 04:44

It is correct behavior for CHTR type CHD-CDs to change SHA1s on conversion because the metadata is updated to CHT2. Such discs were made with a pre-2008 version of CHDMAN and should have been marked as BAD_DUMP once CHDMAN gained the capability to preserve gap data. They all need to be redumped as source becomes available. (Note that the Data SHA1s do match in Mr. Person's report, which proves it's simply the metadata change).

I have no idea what's going on with the GD-ROMs yet; none of this stuff should affect those.

User avatar (08233)
Scagazza (Tester)
2012-02-21 08:24

I just wonder if chds introduced in 0.145u1 and dumped directly with v5 could be considered good dump or not.
initdv3e\gds-0033.chd
jojoba\cap-jjm-120.chd
User avatar (08235)
R. Belmont (Developer)
2012-02-21 13:13
edited on: 2012-02-21 13:16

initdv3e is a v4 image. I didn't add jojoba so I can't speak for that, but it likely is as well.

To clarify, if "chdman info" shows this:
Metadata: Tag='CHTR' Index=0 Length=45 bytes

then it's a bad dump.

If it shows this:
Metadata: Tag='CHT2' Index=0 Length=88 bytes

then it's a good dump. (Not including harddisks, DVDs, and laserdiscs, of course).

User avatar (08236)
Haze (Developer)
2012-02-21 13:40
edited on: 2012-02-21 13:49

and obviously for the cases where CHTR changes to CHT2 and the SHA1 changes as a result the source should be updated with the correct SHA1s (should probably have been done, with appropriate BAD_DUMP tags before u1 to avoid confusion)

the GD-ROMs are going to need some investigation as to why they've changed (that accounts for most of the list) Luckily they're easy to spot because they all had gd* names :-)

the LD issue *definitely* needs looking at if the metadata is changing, especially if different people are getting different results as indicated here.

User avatar (08237)
R. Belmont (Developer)
2012-02-21 14:24

Update: the GD-ROMs that changed are *also* CHTR. So in those cases the SHA1s should be updated, but since we don't get gap information from the common .gdi format there's no need to BAD_DUMP them.
User avatar (08238)
Firewave (Developer)
2012-02-21 14:40

All chdman info:
http://pastebin.com/a3hFezbB

Aorry I accidentally deleted the are51mx after converting it, so I just have the v5 one.
User avatar (08239)
Haze (Developer)
2012-02-21 15:29

does the bad cubeqst pass -verify?

 I'd be interested in knowing if it does because the hunk size /4 is greater than 65535 which is the flac block limit.. and flac will refuse to encode if you go over it, hence why I added the saftey checks for it when encoding CD / HDD FLAC ( http://www.sendspace.com/file/ikgp19 )

(having multiple flac blocks in a hunk isn't a problem, the codec handles it transparently)

but maybe the AV codec never has a hunk of entirely FLAC/

there is also a lower limit of 16 samples (64 bytes) (feeding it less data isn't a problem either, it will just be a bit wasteful)

all depends how the FLAC stuff gets encoded in LDs tho, and might not be the issue at all. It's one avenue I'd investigate tho.
User avatar (08241)
B2K24 (Senior Tester)
2012-02-21 16:50

It does not pass -verify

I get error executing verify command (Error reading CHD file (newcubeqst.chd): decompression error)

http://pastebin.com/3cGxnxdQ
User avatar (08242)
Firewave (Developer)
2012-02-21 21:59

I was missing some CHDs before. Here's the complete list:

http://pastebin.com/c5Ena34r
User avatar (08245)
Haze (Developer)
2012-02-22 12:16

if it doesn't pass -verify it's more than a metadata problem then.

the SHA1 used (and written to the header) is calculated from the raw input data during compression. Sounds like the CHD files it makes are basically just invalid. Wouldn't surprise me if it's what I'm saying above + more.

btw the radio silence on the other aspects of my patch (optimal hunk + flac block sizes / settings) in addition to the safety checking there is annoying.

Has anybody run verify over a complete set of converted images?
User avatar (08246)
Firewave (Developer)
2012-02-22 13:44

I am running a test right now

chdman info -i old.chd
chdman verify -i old.chd
chdman copy -i old.chd -o new.chd
chdman info -i new.chd
chdman verify -i new.chd


It's very time consuming. It's running since half a day and it's just at gauntleg and so far all converted LD CHDs failed to verify. I also had two verification errors with unconverted CHDs (drmn5m/b05jaa02.chd and drmn6m/b16jaa02.chd), but maybe those were damaged to begin with.
User avatar (08247)
Haze (Developer)
2012-02-22 14:57

V4

D:\mame180212>chdman verify -i d:\b05jaa02.chd
chdman - MAME Compressed Hunks of Data (CHD) manager 0.145u1 (Feb 19 2012)
Raw SHA1 verification successful!
Overall SHA1 verification successful!

D:\mame180212>chdman verify -i d:\b16jaa02.chd
chdman - MAME Compressed Hunks of Data (CHD) manager 0.145u1 (Feb 19 2012)
Raw SHA1 verification successful!
Overall SHA1 verification successful!

check your HDD?
User avatar (08249)
Firewave (Developer)
2012-02-22 16:53

Apparently somebody put bogus versions of them out, since I have at least one other guy with bad ones. Will try to get good ones.
User avatar (08250)
Haze (Developer)
2012-02-22 17:52

the actual metadata in cubeqst is changing tho, yes..

doesn't explain the decompression errors tho

and I haven't managed to untangle from the code the meaning of the bits which have changed.
User avatar (08253)
Haze (Developer)
2012-02-22 19:03
edited on: 2012-02-22 19:15

The actual point of failure when decompressing Cube Quest (which is one which reports the right data SHA1) is

if (totalsize >= complength)
return AVHERR_INVALID_DATA;

in decode_data

if you remove that check (cubeqst) does at least complete and say that the SHA1 matches what was expected, but it shouldn't be failing in the first place.

This doesn't explain why the Metadata has changed either.

User avatar (08256)
Firewave (Developer)
2012-02-23 04:23

I got this error while converting mach3.chd

pure virtual method called


Will try to get a backtrace later.
User avatar (08257)
B2K24 (Senior Tester)
2012-02-23 05:04

seems to convert and verify here

http://pastebin.com/zedm1W30
User avatar (08260)
Firewave (Developer)
2012-02-23 14:56

OK, the "pure virtual" error happens when there is not enough diskspace during a copy.

Also the bad CHDs I encountered were just incomplete files, so I suggest a filesize check when loading a CHD (it's in the header anyways and a ftell() call shouldn't hurt).
User avatar (08323)
Scagazza (Tester)
2012-03-07 11:01

I have tested chdman compiled against latest mess svn (rev. 14714) with Aaron's fix, the SHA1 is now correct but the new chd doesn't pass the verify.

C:\MAME\roms\cubeqst>chdman info -i cubeqst.chd
chdman - MAME Compressed Hunks of Data (CHD) manager 0.145u3 (Mar 7 2012)
Input file: cubeqst.chd
File Version: 4
Logical size: 41,096,611,968 bytes
Hunk Size: 380,496 bytes
Total Hunks: 108,008
Unit Size: 380,496 bytes
Total Units: 108,008
Compression: avhu (A/V Huffman)
CHD size: 11,517,456,445 bytes
Ratio: 28.0%
SHA1: d0e24bb5d0ae424e0816110ec7d6b21189044d57
Data SHA1: 5266a0a43963628464fac1719e1cefec627042a4
Metadata: Tag='AVAV' Index=0 Length=76 bytes
              FPS:59.940058 WIDTH:720 HEIGHT:262 INTERLACED:1 CHANNELS:2 S
Metadata: Tag='AVLD' Index=0 Length=1728128 bytes

C:\MAME\roms\cubeqst>chdman info -i newcubeqst.chd
chdman - MAME Compressed Hunks of Data (CHD) manager 0.145u3 (Mar 7 2012)
Input file: newcubeqst.chd
File Version: 5
Logical size: 41,096,611,968 bytes
Hunk Size: 380,496 bytes
Total Hunks: 108,008
Unit Size: 380,496 bytes
Total Units: 108,008
Compression: avhu (A/V Huffman)
CHD size: 11,465,974,734 bytes
Ratio: 27.9%
SHA1: d0e24bb5d0ae424e0816110ec7d6b21189044d57
Data SHA1: 5266a0a43963628464fac1719e1cefec627042a4
Metadata: Tag='AVAV' Index=0 Length=76 bytes
              FPS:59.940058 WIDTH:720 HEIGHT:262 INTERLACED:1 CHANNELS:2 S
Metadata: Tag='AVLD' Index=0 Length=1728128 bytes

C:\MAME\roms\cubeqst>chdman verify -i newcubeqst.chd
chdman - MAME Compressed Hunks of Data (CHD) manager 0.145u3 (Mar 7 2012)
Error reading CHD file (newcubeqst.chd): decompression error
User avatar (08329)
Scagazza (Tester)
2012-03-07 19:21

IMHO this bug is not solved.
Now the SHA1 is correct but the new created chd doesn't verify.
Please read the post!
User avatar (08330)
Tafoid (Administrator)
2012-03-07 19:45

The bug is regarding changing the SHA1.. which is no longer does.
"verify" is another issue which is being worked on and since it's not part of a U release, doesn't need to be reported in a separate report.. The resultant image appears good, regardless.
User avatar (08331)
Scagazza (Tester)
2012-03-07 19:50

How can you say that the resultant image appears good if it doesn't verify?
User avatar (08394)
Darkfalz (Tester)
2012-03-19 15:06

I did a huge 2-day batch convert. At least half of CHDs now fail. I hope these are updated in new driver and we don't need to reacquire them all. I don't see it mentioned in the thead but all the Beatmania derivatives have also changed SHA-1, not just the GD rips.
User avatar (08395)
Haze (Developer)
2012-03-19 15:38

I can't confirm Beatmania SHA1s changing, I tested several with no change.
User avatar (08411)
Darkfalz (Tester)
2012-03-26 19:58

Pretty much all my CD based games failed (Beatmania, CPS3, Konomi etc.) is there a command I should have used to convert these rather than copy -i -o ? They're backed up so I can try them again, but I'm really confused.
User avatar (08416)
Haze (Developer)
2012-03-27 03:34

A lot of the *CD* based games will have their SHA1s change unfortunately, this is because the old metadata (track gaps and the like) gets updated to a new format in the process, and to ensure the format is secure this data forms part of the overall hash.

The hashes haven't yet been updated in MAME, which will need to happen before 0.146 ships, dunno who is doing that tho.

The MESS lists fare better, with only a handful of images having the old tags, thus them not changing when the chds are updated.

I'm hoping the ECC compression code I've given to Aaron will be included before 0.146 (Because it helps a LOT with the MESS Saturn sets), although that won't change the SHA1s any furher, just people should be aware that there might be at least one more tweak to the actual format prior to 0.146 all going well.
User avatar (08418)
Darkfalz (Tester)
2012-03-27 08:37

Sounds like another 2 days of recompressing CHDs might be in my future then ;-)