[messages] Edit: [Developers] tile cache test build

uckelman uckelman at nomic.net
Mon Oct 11 05:56:03 MST 2010

[This message has been edited.]

I've uploaded a test build of the tile cache work I've been doing on the
uckelman-working branch. The primary upshot of this is that you can run
the Case Blue/GDII module with a 256MB heap now, which means that you
can probably run any existing module with 256MB heap.

The test build is 3.2.0-svn7390[1]. Note that you won't get everything
in this build if you check out the uckelman-working branch from SVN,
since this build contains a whole of uncommitted changes.

What you should see the first time you load a module is a dialog showing
you the progress of slicing map images to tiles and writing those tiles
to disk. So long as the images in the module aren't modified and you
don't delete the tile cache, you'll never see this dialog again once the
tiling is done. (When a module is loaded, it checks that the cached
tiles for each image are fresh. Stale or missing tiles are recreated,
fresh ones are left alone.)

There are some rough edges still, namely:

* There's no way to force the tile cache to be cleared from within
VASSAL. The only way to do that right now is to manually delete the
cache directory for the module.

* Canceling tiling leaves you with a "busy" cursor.

* Writing tiles for modules or image files which contain characters in
their names which aren't legal in filenames on your system will cause an
IOException. The solution to this is to sanitize such names before
trying to write to disk. (We should really do this for modules,
too---there are a few modules which contain files with bad names, which
prevents standard ZIP tools from unzipping them.)

* The tile slicer, which runs as a child process, presently has its max
heap fixed at 512MB. This may be too much for some images, too little
for others. It would be nice if we could automatically set the max heap
for the tile slicer based on the dimensions of the largest image it
needs to tile.

* Displaying progress as a percentage is surprisingly complex.

The most common type of progress dialog, for file downloads, is simple,
as in that setting, you know at the start how large the file is and that
you're planning to download all the bytes. By knowing how many bytes you
have and how long that took, you can make a completion time estimate.

Tiling is not a single, homogeneous operation. During tiling, there are
four operations happening: image loading, image type conversion, tile
slicing, and tile writing. Presently, the only one of these which
generates progress updates is tile writing. In particular, what you're
seeing in the progress bar is the percentage of tiles written so far.
However, the other three processes are much more time-consuming, which
is why you'll see the progress bar advance fitfully instead of smoothly.

Let me describe the process in more detail, because I'd like to get some
suggestions for how to display progress more smoothly. For each image to
be tiled, the tile slicer

1) loads the image, via ImageIO,
2) converts the image to TYPE_INT_RGB or TYPE_INT_ARGB,
3) slices each tile from the image, scaled to the requested size,
4) writes the tile's image data to the disk cache

Of these, ImageIO has progress hooks, so we can get completion
percentage from that, and writing tile data to disk is quite fast (tiles
are on the order of 100KB), so the only progress worth tracking there is
counting how many have completed. For slicing, it's the scaling which
takes all the time; adding progress listener support to the scaler
wouldn't be too hard. Image conversion is more complicated: In 3.1, we
did all image type conversions in memory, which unfortunately means that
you need to have enough heap to store two copies of the image
simultaneously. For very large images, this isn't feasible. What the
type converter I wrote does is tries an in-memory conversion using
Graphics2D.drawImage(), and if that fails, it uses
BufferedImage.getRGB() to convert the image data to TYPE_INT_ARGB,
writes that to a temporary file one row at a time, lets the original
image be garbage collected, creates the destination image, reads the
image data back from disk, and writes to the new image using
BufferedImage.setRGB(). In the on-disk case, we can count progress by
seeing how many rows of pixels have been written to disk, and then by
how many have been read. In the in-memory case, the only progress
indication we can get from Graphics2D.drawImage() is that it hasn't
started (because we haven't called it yet) and that it's finished
(because execution has moved to the next line). To make things worse, we
have no way of telling which conversion method will be used for any
given image, since the on-disk conversion is attempted only when the
in-memory one fails with an OutOfMemoryException, or even whether we'll
be doing a conversion at all, since in some (rare?) cases, ImageIO will
give us an image which is already the right type.

That said, even if we could get a completion percentage from each part
of the process, it's not the case that 1% of image loading will take the
same amount of time as converting 1% of the image rows or writing 1% of
the tiles. The scaling time for tiles of the same size at different
scale factors isn't even consistent, so slicing 1% of the tiles might
take dramatically longer than slicing some other 1% of the tiles.
(Rather perversely, scaling time grows as the scale factor decreases,
because you're filtering more source pixels into each destination
pixel.) So, I'm rather perplexed as to how to do a good job with this.
Maybe it doesn't matter that much, but I think that users will suspect
that the program has seized up if they see the progress bar remaining at
one point for 10-20 seconds.

[1] http://www.nomic.net/~uckelman/tmp/vassal

Read this topic online here:

More information about the messages mailing list