Hello, I have made some verification of this release:
The checksum of the "hht-codiw.rar" file in the shared file is "57C83955".
However, the attached svf in the shared file states: hht-codiw.rar 976bca8b
The same checksum can be found at srrdb and dats.
The checksums of the files located inside of the file "hht-codiw.rar" look correct (on condition srrdb is considered as a standard for comparison).
I am wondering if such a checksum missmatch is caused by a regular compression of the "hht-codiw.rar" file or there is something else.
However, if there is no intention of sticking to the Scene in preservation purposes and the file is kept as is, I would highly appreciate if you could confirm it.
Hello, I have made some verification of this release:
The checksum of the "hht-codiw.rar" file in the shared file is "57C83955".
However, the attached svf in the shared file states: hht-codiw.rar 976bca8b
The same checksum can be found at srrdb and dats.
The checksums of the files located inside of the file "hht-codiw.rar" look correct (on condition srrdb is considered as a standard for comparison).
I am wondering if such a checksum missmatch is caused by a regular compression of the "hht-codiw.rar" file or there is something else.
However, if there is no intention of sticking to the Scene in preservations purposes and the file is kept as is, I would highly appreciate if you could confirm it.
This is the only source I have for this release, a number of things could have happened to the rar for the crc to be changed (if srrdb is correct).
If the content of hht-codiw.rar matches then we are good I guess.
This is the only source I have for this release, a number of things could have happened to the rar for the crc to be changed (if srrdb is correct).
If the content of hht-codiw.rar matches then we are good I guess.
I have an additional queston, if possible, please let me know what kind of method you use to verify releases in your collection/library to be certain they indeed belong to The Scene.
Nevermind, the other upload has the correct SFV and RAR, though the NFO is in the "incorrect" Unix EOF format. There must have been an issue during your download/extraction. If you still have the previous file, please check it against this SHA-256 hash:
They completely match the ones specified at srrdb (including NFO). The most confusing thing is that 7-Zip displays the same checksums as WinRAR does if Palinkat's archive file is just opened with 7-Zip to browse the files. To be honest this is something new to me. I would never imagine that file checksums may be different depending on an application used for data extraction or browsing.
TL;DR: This is a quirk of the RAR5 archive format and not an actual file hash difference.
"That is indeed rather odd. I do not use WinRAR though, so I can only speculate that this may have something to do with the archive using encryption but not for the file names."
- My first guess some time ago now...
I was able to reproduce your findings except for the third sentence. Both WinRAR v5.80 (
) list the "wrong" file hashes in their GUI and from the command-line, but extracting the archive produces files with the same correct CRC32 hashes, most importantly 976BCA8B for hht-codiw.rar.
The cause of this issue is a difference between the RAR4 and RAR5 archive formats, specifically the stored checksum.
Notice how in WinRAR v5.80 the hash column is labeled "Checksum" instead of the usual "CRC32". That is because this archive is a particular combination of two parameters: RAR4/RAR5 and plaintext/encryption/full encryption. To illustrate this, here are commands to create and parse all six possible variations:
Switches:
-ma[4|5] Specify a version of archiving format
-m<0..5> Set compression level (0-store...3-default...5-maximal)
-p[password] Set password
-hp[password] Encrypt both file data and headers
As you can see, only 5b.rar has the wrong hash and MAC was appended to the hash name.
You can see that Call_of_Duty_Infinite_Warfare_Unlocker-HHT.rar is the same combination as 5b.rar because it is RAR5 (see the last paragraph of the spoiler or alternatively click on the "Info" button in WinRAR) and is encrypted (hence the password), though only the data, since opening it lets one see the metadata such as directory and file names without having to enter a password.
Encrypted archives (both RAR and 7-Zip) are never identical even if created with the exact same command and files to prevent certain kinds of cryptographic attacks, but the decrypted contents including the metadata obviously match. In the above-mentioned special case though, the CRC32 MAC changes each time:
is a defensive measure for archives that have encrypted file data but plaintext headers, which is why it is not used for fully encrypted archives. To quote the
File encryption record
This record is present if file data is encrypted.
[...]
0x0002 Use tweaked checksums instead of plain checksums.
If flag 0x0002 is present, RAR transforms the checksum preserving file or service data integrity, so it becomes dependent on encryption key. It makes guessing file contents based on checksum impossible. It affects both data CRC32 in file header and checksums in file hash record in extra area.
# list.cpp:336
if (hd.FileHash.Type==HASH_CRC32)
mprintf(L"\n%12ls: %8.8X",
hd.UseHashKey ? L"CRC32 MAC":hd.SplitAfter ? L"Pack-CRC32":L"CRC32",
hd.FileHash.CRC32);
# headers.hpp:226
// Use HMAC calculated from HashKey and checksum instead of plain checksum.
bool UseHashKey;
// Key to convert checksum to HMAC. Derived from password with PBKDF2
// using additional iterations.
byte HashKey[SHA256_DIGEST_SIZE];
# arcread.cpp:1049
hd->UseHashKey=(Flags & FHEXTRA_CRYPT_HASHMAC)!=0;
# headers5.hpp:103
#define FHEXTRA_CRYPT_HASHMAC 0x02 // Use MAC for unpacked data checksums.
# hash.cpp:274
bool DataHash::Cmp(HashValue *CmpValue,byte *Key)
{
HashValue Final;
Result(&Final);
if (Key!=NULL)
ConvertHashToMAC(&Final,Key);
return Final==*CmpValue;
}
# crypt5.cpp:193
void ConvertHashToMAC(HashValue *Value,byte *Key)
{
if (Value->Type==HASH_CRC32)
{
byte RawCRC[4];
RawPut4(Value->CRC32,RawCRC);
byte Digest[SHA256_DIGEST_SIZE];
hmac_sha256(Key,SHA256_DIGEST_SIZE,RawCRC,sizeof(RawCRC),Digest,NULL,NULL,NULL,NULL);
Value->CRC32=0;
for (uint I=0;I<ASIZE(Digest);I++)
Value->CRC32^=Digest[I] << ((I & 3) * 8);
Value->CRC32&=0xffffffff; // In case the variable size is larger than 32-bit.
}
[...]
}
In combination with the miniscule GUI label change, that does not adequately explain the situation at all, an unsuspecting user is led to assume that the actual file hashes are different.
Though 7-Zip is also quite bad as the column is always called CRC and since
the rows are simply blank with the following bullet point in the changelog: "Speed optimizations for hash caclulation [sic]: CRC-32, CRC-64, BLAKE2sp". I guess it really is that much quicker to just leave it out completely...
In the end this may not have been another case of old programs causing issues, but the real question remains: Why you are still using those ancient software versions. WinRAR v5.80 is
).
2. Go to (CTRL+G) B5A30.
3. Replace the first two bytes at that location (8A C3) with B0 01 by just typing the latter without the space.
4. Save the file (CTRL+S) and close the hex editor.
5. If you plan on using the command-line version, repeat the previous steps with Rar.exe but go to 1ADAA instead.
These offsets are for WinRAR v7.01.