Firek: I haven't consulted this with Fallen, so I may be a little off on some points, but here goes...
TL;DR - the Downloader takes up extra space, and the process takes extra time but it's only to make absolutely sure that you download the correct files, for both your peace of mind, and ours. :)
The Downloader downloads 10MB chunk files, which it stores in the game's downloads folder throughout the download process.
After downloading each chunik, it verifies its checksum to make sure that it wasn't corrupt. If it was, the chunk is re-downloaded. If it was corrupt again, and its MD5 is identical, it means that a corrupt copy of the installer is cached somewhere, or we have uploaded the wrong MD5 checksum somehow, or we've just updated the installer mid-download (no longer a problem now that we add universal patches). In such cases, the Downloader asks the user to report this. I would prefer it if the Downloader reported such an event automatically, but we know how sensitive some of our users are towards sending information without their consent, even though calling it "DRM" would have been a long shot.
As soon as all necessary chunks are downloaded, the Downloader merges them into installer files (.exe or .bin).
As soon as all chunks are downloaded (or after each file is merged - not sure at this moment, since I'm writing from home), and in turn all installer files are merged, the Downloader verifies the merged files' integrity. At this point both the chunks and the merged files are present in the folder, to be on the safe side, and take up double the space.
Only after every installer file is confirmed to be uncorrupted, are the chunk files deleted and you're free to enjoy both your game and the temporarily used drive space. ;)
The process takes up extra time and drive space, but is necessary because, even through every chunk is checked, corruption can happen when merging consistent files, due to a borked HDD or RAM.
Troubleshooting such cases (when the Downloader ended up delivering corrupt files), is a real pain both for us Support folk, and for the users, as it takes time and forces users to perform relatively computer-knowledge-intensive tasks. To make our users' lives, and our own, easier, Fallen has added another security measure to the Downloader.
If the merged files are somehow corrupt, the chunks are re-checked on the off chance that the installer was updated mid-transfer (no longer the case), or the chunks were tampered with mid-download (unlikely, but possible).
If this wasn't the case, the files are merged again. If the merged files are corrupt once more, but the checksums were identical on both attempts, the Downloader asks the user to report this.
Once we get such a report we triple-check that we uploaded the proper MD5 checksums. No idea how such a thing might happen if the uploaded checksums were correct, and I'm pretty sure this hasn't happened after the last update. We'll figure this out in case it happens.
Neat, thanks for the detailed information about the entire process and the reasoning behind the decisions. You have answered my question with overkill, thanks.
Quick question: Have you already discussed the possibility of mentioning either in the installer or on the download page, how much space is required to complete the process of preparing the installer? it was just a somewhat nasty surprise when I freed up the amount of space mentioned on the website to find that I actually needed quite a bit more space for the process to complete.
Edit: Also i think the installer wasn't giving a good idea of what it was doing, exactly. I believe at the time I was getting the impression that the process of preparing the install files after the download was complete, was the downloader still downloading stuff and when it failed, I was left with the impression of having an incomplete download and thus believing I needed to restart the download from scratch.
As you can imagine, restarting a 15GB download isn't exactly fun, unless you're in a country that has uber internet.
