I’ve uploaded a test build of the tile cache work I’ve been doing on the uckelman-working branch. The primary upshot of this is that you can run the Case Blue/GDII module with a 256MB heap now, which means that you can probably run any existing module with 256MB heap.
The test build is 3.2.0-svn7390. Note that you won’t get everything in this build if you check out the uckelman-working branch from SVN, since this build contains a whole of uncommitted changes.
What you should see the first time you load a module is a dialog showing you the progress of slicing map images to tiles and writing those tiles to disk. So long as the images in the module aren’t modified and you don’t delete the tile cache, you’ll never see this dialog again once the tiling is done. (When a module is loaded, it checks that the cached tiles for each image are fresh. Stale or missing tiles are recreated, fresh ones are left alone.)
There are some rough edges still, namely:
-
There’s no way to force the tile cache to be cleared from within VASSAL. The only way to do that right now is to manually delete the cache directory for the module.
-
Canceling tiling leaves you with a “busy” cursor.
-
Writing tiles for modules or image files which contain characters in their names which aren’t legal in filenames on your system will cause an IOException. The solution to this is to sanitize such names before trying to write to disk. (We should really do this for modules, too—there are a few modules which contain files with bad names, which prevents standard ZIP tools from unzipping them.)
-
The tile slicer, which runs as a child process, presently has its max heap fixed at 512MB. This may be too much for some images, too little for others. It would be nice if we could automatically set the max heap for the tile slicer based on the dimensions of the largest image it needs to tile.
-
Displaying progress as a percentage is surprisingly complex.
The most common type of progress dialog, for file downloads, is simple, as in that setting, you know at the start how large the file is and that you’re planning to download all the bytes. By knowing how many bytes you have and how long that took, you can make a completion time estimate.
Tiling is not a single, homogeneous operation. During tiling, there are four operations happening: image loading, image type conversion, tile slicing, and tile writing. Presently, the only one of these which generates progress updates is tile writing. In particular, what you’re seeing in the progress bar is the percentage of tiles written so far. However, the other three processes are much more time-consuming, which is why you’ll see the progress bar advance fitfully instead of smoothly.
Let me describe the process in more detail, because I’d like to get some suggestions for how to display progress more smoothly. For each image to be tiled, the tile slicer
- loads the image, via ImageIO,
- converts the image to TYPE_INT_RGB or TYPE_INT_ARGB,
- slices each tile from the image, scaled to the requested size,
- writes the tile’s image data to the disk cache
Of these, ImageIO has progress hooks, so we can get completion percentage from that, and writing tile data to disk is quite fast (tiles are on the order of 100KB), so the only progress worth tracking there is counting how many have completed. For slicing, it’s the scaling which takes all the time; adding progress listener support to the scaler wouldn’t be too hard. Image conversion is more complicated: In 3.1, we did all image type conversions in memory, which unfortunately means that you need to have enough heap to store two copies of the image simultaneously. For very large images, this isn’t feasible. What the type converter I wrote does is tries an in-memory conversion using Graphics2D.drawImage(), and if that fails, it uses BufferedImage.getRGB() to convert the image data to TYPE_INT_ARGB, writes that to a temporary file one row at a time, lets the original image be garbage collected, creates the destination image, reads the image data back from disk, and writes to the new image using BufferedImage.setRGB(). In the on-disk case, we can count progress by seeing how many rows of pixels have been written to disk, and then by how many have been read. In the in-memory case, the only progress indication we can get from Graphics2D.drawImage() is that it hasn’t started (because we haven’t called it yet) and that it’s finished (because execution has moved to the next line). To make things worse, we have no way of telling which conversion method will be used for any given image, since the on-disk conversion is attempted only when the in-memory one fails with an OutOfMemoryException, or even whether we’ll be doing a conversion at all, since in some (rare?) cases, ImageIO will give us an image which is already the right type.
That said, even if we could get a completion percentage from each part of the process, it’s not the case that 1% of image loading will take the same amount of time as converting 1% of the image rows or writing 1% of the tiles. The scaling time for tiles of the same size at different scale factors isn’t even consistent, so slicing 1% of the tiles might take dramatically longer than slicing some other 1% of the tiles. (Rather perversely, scaling time grows as the scale factor decreases, because you’re filtering more source pixels into each destination pixel.) So, I’m rather perplexed as to how to do a good job with this. Maybe it doesn’t matter that much, but I think that users will suspect that the program has seized up if they see the progress bar remaining at one point for 10-20 seconds.