It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
avatar
joveian: It turns out that innoextract --test doesn't helpfully tell you what file has an issue if you are testing a bunch of them so I used:

fd -g 'setup*.exe' -x echo '{}:' \; -x innoextract --test --gog --silent
So this was on linux, to test the Windows installers?

Did this give you the information of which games or even files have the problem? Does it search/test recursively from all subdirectories too, or only the current directory?
I don't remember which thread I had found it in (it has been 3-4 months since I had seen it), but what have helped my case of having corrupted games' files is to disable antivirus while downloading.
Most games' files don't need this action (for the antivirus to be disabled), but the few cases I had corrupt files, this was the solution (for me at least) for a smooth download.
I don't even remember how the person explained it. It was something about the antivirus sees as threat some parts of these files and quarantines them. So when you try to install the game, these parts aren't there and then messages appear during installation about missing things.
avatar
joveian: fd -g 'setup*.exe' -x echo '{}:' \; -x innoextract --test --gog --silent
avatar
timppu: So this was on linux, to test the Windows installers?
Yes, however both fd (zip) and innoextract (zip) are available for Windows so it should just work there as well (or GOGPlus as you mentioned). Well, the --gog thing on innoextract is that some .bin files are just RAR format so you might need unrar (exe) as well, although I'm not sure how often that is actually used.
avatar
timppu: Did this give you the information of which games or even files have the problem?
Yes, it echos the name of the file and then the output from innoextract, which will either be just a newline (if it tests ok) or the error message. You can then run innoextract --test --gog (without --silent) to get detailed information about the contents depending on the error (for some reason I see empty brackets without file names but highlighting between the brackets shows the file name).

Actually, I found one other error: one random file in Kerbal Space Program 1.12.2 Breaking Ground extracted to the wrong size. It didn't say which one was the correct size. Possibly that might be from a very slightly truncated file as well, although it was well before I noticed issues with any other files so possibly it was just an issue with the build (or I ran out of disk space when downloading, which I've also done several times :/). Since I can't get that file any more, I checked the detailed file info and can get that particular file from a different version if necessary (I think 1.12.2 is the last version before they changed something about the docking that some people didn't like so I wanted to keep it just in case).

avatar
timppu: Does it search/test recursively from all subdirectories too, or only the current directory?
The form I gave is recursive, add --max-depth 1 to just search the current directory.

The xe command I gave for testing the Linux installers is not recursive but could be made recursive by:

fd -g '*.sh' | xe -s 'echo -n "$1: "; bash "$1" --check'

I should note that both versions just run all the .sh files with the --check option, so it is good to be careful that all the .sh files are actually GOG installers. It wouldn't be too hard to extract the MD5, figure out the offset (the MD5 applies after the first 519 lines it seems) and check it without executing the script.

avatar
Braggadar: Everyone's first stop should be a simple checksum comparison.
Yeah, that would be ideal. I'm not sure what innoextract tests exactly, although from installer description it sounds like there are checksums for the .bin files at least. Looking at the gogcli description it sounds like the Galaxy API has checksums for the main game files but not the extras. Is that what you mean by the xml files? I checked the API links from gogdb but they don't seem to have file checksums. Most zip files just have a basic CRC I think, which can detect some things but is not reliable.

I've looked at the different downloaders that generous community members have made but I like having everything in one giant directory and that helps me use minimal disk space, although as my collection continues to increase the automated management gets more tempting. There are some used large drives that aren't that expensive.

If anyone from GOG is reading, blake3 and bao are the state of the art (bao is the streaming form of blake3 and would allow testing parts of downloads). But even just basic file name + sha256 hash would be great to be able to easily detect CDN issues.
Post edited February 19, 2023 by joveian
avatar
joveian: Looking at the gogcli description it sounds like the Galaxy API has checksums for the main game files but not the extras. Is that what you mean by the xml files? I checked the API links from gogdb but they don't seem to have file checksums. Most zip files just have a basic CRC I think, which can detect some things but is not reliable.
Yup, the GOG API provides checksums for the main installers (and very occasionally some exes in extras) by providing an xml file. The file contains both chunk and complete MD5 checksums of the file as well as other data like filesize and upload date (I think). If you're using a browser and not a gogcli or other application, then all you need to do is take the download link (which still has a live token) and append ".xml" on the end.

Sometimes though the xml is unaccessible and you need to add a second slash in the filepath (just prior to filename) to get it. Very rarely the xml is missing or damaged.
avatar
Braggadar: If you're using a browser and not a gogcli or other application, then all you need to do is take the download link (which still has a live token) and append ".xml" on the end.
If you allow add-ons to run on the site, you can just install the GOG checksum lookup user script.
avatar
Braggadar: Everyone's first stop should be a simple checksum comparison.
avatar
joveian: Yeah, that would be ideal. I'm not sure what innoextract tests exactly, although from installer description it sounds like there are checksums for the .bin files at least. Looking at the gogcli description it sounds like the Galaxy API has checksums for the main game files but not the extras. Is that what you mean by the xml files?
E.g. gogrepoc.py does md5 checksum comparison for the installers... but the main problem is that there are some issues with that for many of the games, e.g. that data is not provided by GOG (empty xml file or whatever). Or in theory, the md5 checksum GOG provides might just be wrong, for a reason or another. Or, in the future when gog.com is no more and you have hundreds of your downloaded GOG game installers on your hard drive, gog servers can't provide you the md5 checksums anymore. Or, if you want to keep some older version of an installer around, GOG will not provide the md5 checksum for that either, but only for the updated installer. For instance the non-SCUMMV version of Myst 2: Riven that I have.

Hence, I prefer there is some alternative way to test the integrity of the installers, even without the md5 checksums. Like the innoextract test.
Post edited February 19, 2023 by timppu
Thanks everyone! Strange that it isn't reliable but something is better than nothing at least. I might try to put together a simple script that uses curl to download everything for one game, manually following the redirects (resuming as needed) and verifying the download if possible. I'll post a new thread and link to it from here if I do.

avatar
timppu:
Yeah, having alternatives is nice. I've been using b3sum for checking the integrity of backups (of stuff I don't keep encrypted) lately, using:

fd -t f -X b3sum -- > ~/d.b3

to create a file with the hash of all regular files under the current directory (I'm not sure if Windows has ~ but the main point is to create the file somewhere not in the current directory so you don't get hash of part of that file). Then b3sum --check ~/d.b3 to check everything. There is a windows exe of b3sum as well (not GUI though). It is quite fast so always ends up io bound no matter how fast your disk is, which is handy when you want to verify 1TB+. I usually save the hashes when copying stuff to external backups but that would be a good thing to add to the download script also.
Post edited February 19, 2023 by joveian
avatar
Ice_Mage: If you allow add-ons to run on the site, you can just install the GOG checksum lookup user script.
Nice! But while I'm already using Adalia and Barefoot, that's about as many scripts as I want at the moment. And in my case I save the whole xml itself alongside each installer file. ;)
avatar
timppu: Or, in the future when gog.com is no more and you have hundreds of your downloaded GOG game installers on your hard drive, gog servers can't provide you the md5 checksums anymore.
Which is why everyone should be keeping the checksum hash with your archived files as you download them rather than relying on GOG servers to retrieve them on demand.

But I see what you're saying though.

Personally, I've never seen an "empty" xml file. Those files which appear that way are normally retrieved whole by that little added slash trick... something to do with the path not pointing to the file correctly for whatever reason.

I have seen a very rare missing xml. Those with much larger collections likely see them a lot more than I do.
Post edited February 20, 2023 by Braggadar
avatar
timppu: Or, in the future when gog.com is no more and you have hundreds of your downloaded GOG game installers on your hard drive, gog servers can't provide you the md5 checksums anymore.
avatar
Braggadar: Which is why everyone should be keeping the checksum hash with your archived files as you download them rather than relying on GOG servers to retrieve them on demand.
If my GOG archive was static, ie. not receiving any updates anymore, I would probably just run rhash over the whole collection, making sha256 checksums for all of the files. Then I'd know even years from now if they are still intact.

In the meantime, since there seem to be some hiccups here and there GOG providing the md5 checksums, I'd rather rely on some other way to make sure the files are ok. "innoextract test" seems to be the one, to make 100% certain that all the installers are fine. It doesn't have to rely on the existence of external checksums provided by someone else.