Topic: Very, very big maps.

[Maxim]

It is possible to run PNGOUT directly on the files on the webserver, but the host might not be happy for you to run at 100% CPU for a few days (!).

[JonLeung]

Sure, LDK, I'll take any and all of your files again if you're going to be recompressing them.

I guess I should check my PNGGauntlet settings and see if I have to do another massive site recompression.  >_<  At the very least, I should redo the really big ones that could save a few MB.

Thank goodness with GoDaddy I don't ever run out of bandwidth like my previous host sometimes did, but it's not so much about bandwidth now but more so that users can view them as quickly and easily as possible.

[DarkWolf]

Theoretically if you're running on a Linux server you could nice the pngout command to have a very low priority, but still, some hosts might find that annoying.  And if you are on a shared host with lots of traffic it might not even be worth it.

Every once in a while I throw the idea around in my head of writing a distributed PNGout application.  Not something on the scale of BOINC or other grid computing applications, but at least being able to setup a few computers that could do a batch of work in their spare time.  If Jon wanted to re-compress the maps on the site, I would be willing to lend a hand (and some computing power).

[Ryan Ferneau]

Every map I submit is procesed by PNGGauntlet. Simply because Photoshop has slow PNG export so I'm saving every map in BMP and then compress them by PNGGauntlet with Xtreme! option. Advanced options are set to default.

I'm a little scared to use PNGGauntlet again, because the last time I did, it just took up more and more of the CPU time until it finally froze up my computer.     Why do you think that happened?

[Maxim]

It's supposed to use up all the CPU, it's using it to compress the image. It'll use up memory too, which can be an issue for large images.

[DarkWolf]

In my experience PNGGauntlet will sometimes act strangely on large files or large groups of files.  For larger images it's just better to run pngout.  Even for a directory of smaller images I find pngout is just easier from the command line.  Also for smaller images I like to run pngout a few times with the /r switch after the first pass to get rid of a few additional bytes.

[LDK]

I'm currently working on a maps for Alien Shooter 2 which has pretty huge maps. So big that PNGOUT is refusing to compress them and exits with "malloc failed" error. My guess is that I'm hitting memory limitation of 32-bit. Unfortunatelly I can't find 64-bit PNGOUT for windows so I'm stuck. So any ideas how to bypass this error and compress these big files? Photoshop has very bad PNG output and PNG's from PS can be compressed to cca 60% of their original size.

one of the bigger levels from game: 18836 x 13674 pixels, uncopressed PNG: 273,6MB

[DarkWolf]

I think there is a 64-bit version for Linux somewhere, but not for Windows.  You could try increasing your virtual memory limit, but that won't help if the program itself is actually using more than 4 GB of RAM.

For these huge maps perhaps using the Google Maps API (or something similar) would be more appropriate?

[Revned]

Try JPG. See if it looks bad, and how the filesize compares. Otherwise you'll have to divide the map into smaller segments so that people can actually view it.

[Maxim]

Other PNG optimisers might have 64-bit builds (if you have a 64-bit OS). Your example image is only (!) 0.9GB uncompressed (assuming the worst case of 4 bytes per pixel) so that ought not to be an issue, though. (Well, maybe compression takes up more memory than I'd expected.)

Ken Silverman (PNGOUT guy) on 64-bit builds - maybe you can post there and he'll be convinced it's needed. It'll take forever to run, though...

[LDK]

DarkWolf: Physical memory is not problem, I have 4GB RAM and dynamic swap file. Problem is 2GB limit on 32-bit processes. Google Maps API would be ideal but I don't have any experience with it. And I don't know if it could run on vgmaps it's realy not up to me.

Revned: Tried JPG on quality 10 and get 76MB file without significant detail reduction. Now I'm trying color reduction to 16 bit which should lower PNG file sizes little bit. This game is using true color space only for lighting effects. Overal image quality loss is negligible.

Maxim: Compression takes more memory than uncopressed image. 8416 x 12674 px image (200MB in BMP) takes 1,3GB. I've tried Ken's forum few days ago but posting there ended with 403 error.

[DarkWolf]

That's right, I keep forgetting Microsoft doesn't want you to use the available memory in your system.  At any rate, unless pngout is compiled with a special header, the 2 GB limit will stand on a 32-bit MS OS.  I have access to 64-bit Linux machines, so I could try to help you out there once you have the final version of a map done.  A bitmap compressed with 7-zip probably wouldn't be too bad of a size to send across the internet.

[Maxim]

You can use all the available memory in your system, just not more than 2GB in user address space in the same process. They had to split it somewhere... and making it large address aware is more complicated than just a header setting.

Uncompressed images are generally stored at 32bpp for speed reasons, that's 4 bytes per pixel. 8416x12674 therefore takes 407MB. PNG compression at its simplest level would only need another 32KB of RAM for compression, but PNGOUT stores the compressed image in RAM too; so the rest of the RAM is presumably used for compression optimisation purposes. Given how easily it gets near the 2GB limit, he ought to release a 64-bit version anyway.

[LDK]

So I've done some research on Google Maps API and found user friendly way how to open big maps within Google Maps:

- GMap Image Cutter http://www.casa.ucl.ac.uk/richard/GoogleMapImageCutter/gmimgcut-download.asp will create one HTML file and folder with JPG tiles
- Key for domain which will host final map is needed from Google website http://code.google.com/apis/maps/signup.html (keys are free)
- Key is inserted to HTML and everything is uploaded

Unfortunately there are some issues: Tiles are saved in JPEG so quality is not perfect and there is no 100% zoom ratio. For example zoom level 5 is 95% and zoom level 6 is 102%. I've tried to change picture size to multiples of 256 (tile size) but with no luck.

I've coverted that big map from AS2 and uploading it here: http://doc.riotix.com/map/. Size of whole thing is 132MB and almost 22 000 files. Aside some problems I think this is a nice and easy way to store large maps. Even perfect if 100% zoom level would be created from PNG's...