UnZip/Zip problems on MacOS

[marktb1961]

I have noticed that pre-set scenario files can become inaccessible from the module (error message reports bad data) after the following process on MacOS....

use Archive to unzip the .vmod file
do stuff, nothing to do with saved files ... e.g. remove a redundant image file
compress (ie zip) the vmod files back up and rename to create a new .vmod file

I have tracked this down to non-standard characters in the pre-scenario file names e.g. accented characters). So the workaround is obvious but the same file names can be used without problem to add the pre-set scenario direct from the editor. Maybe there is a bug whereby Vassal player is less tolerant than it might be of some data format in the recreated zip file.

I imagine this is some sort of issue with Apple, and in fact I already have an issue that I don't seem to be able to resolve easily (.DSstore file being in the image list).

Overall, it would be nice if the Editor allowed replacement of files (images, sounds) directly, so this external tinkering was not necessary. e.g. a tick-box on file-loading functions to force replacement of the file.

[JoelCFC25]

I imagine this is some sort of issue with Apple, and in fact I already have an issue that I don't seem to be able to resolve easily (.DSstore file being in the image list).

Overall, it would be nice if the Editor allowed replacement of files (images, sounds) directly, so this external tinkering was not necessary. e.g. a tick-box on file-loading functions to force replacement of the file.

I want to make sure I'm following on this section of your post. You are getting the hidden .DSstore file included in your module's images folder when zipping up on MacOS?

The Editor already allows loading in replacement images--if the image being loaded in precisely matches the filename of the already existing image, the new will overwrite the existing. If the filename differs, the image will be loaded in and the existing image will linger on unused inside the ZIP archive. If you mean something else by replacement, can you describe what that is and what "external tinkering" it would obviate?

[uckelman]

I'd like to see an example of a module where this problem happens involving accented characters.

[marktb1961]

I want to make sure I'm following on this section of your post. You are getting the hidden .DSstore file included in your module's images folder when zipping up on MacOS?

The Editor already allows loading in replacement images--if the image being loaded in precisely matches the filename of the already existing image, the new will overwrite the existing. If the filename differs, the image will be loaded in and the existing image will linger on unused inside the ZIP archive. If you mean something else by replacement, can you describe what that is and what "external tinkering" it would obviate?

Yes, to the .DSstore. To be more precise, I don't see it in the module's image folder but in the Vassal Editor, it appears first in the list of images whenever selecting an image in the vassal editor.

I thought the opposite to what you are suggesting about loading in replacement images, so I will take another look. Thank you.

[jrwatts]

The Editor already allows loading in replacement images--if the image being loaded in precisely matches the filename of the already existing image, the new will overwrite the existing. If the filename differs, the image will be loaded in and the existing image will linger on unused inside the ZIP archive. If you mean something else by replacement, can you describe what that is and what "external tinkering" it would obviate?

This was not my experience with 3.4.5 (on Linux, Java 11.0.8 ) last week; I was replacing 5 images with new versions, and when I tried to do it with the editor, the old images stayed in place. I had to use an external archive program to replace the images.

[marktb1961]

I'd like to see an example of a module where this problem happens involving accented characters.

Here's a reproducible example...

Take Commands & Colors module, v3.42.
A lot of scenario files in that module contain an accented character. S111 Ocaña (Cavalry action) is one. That opens fine in the module as downloaded from Vassal.

Now, on MacOs (in my example, at least), unzip the vmod file (using Archive).
Then immediately re-zip it (Compress, from the Finder on MacOS).
Here is one I prepared earlier: https://www.dropbox.com/s/2kpjuwtwp4uor ... .vmod?dl=0

Load the re-zipped file into Vassal.

You will find that S111 Ocana Cavalry Action along with all other scenario files that contains an accented character (check in the unzipped folder to find more) will now refuse to load from the startup wizard and will give the following error message if the scenario is
selected from the module itself:
- Bad Data in Module: S111 - Ocaña (Cavalry Action) - 18 November 1809[Pre-defined setup]: Setup not found S111 - Ocaña_Cavalry.vsav


Hope that helps.

Mark"

[JoelCFC25]

This was not my experience with 3.4.5 (on Linux, Java 11.0.8 ) last week; I was replacing 5 images with new versions, and when I tried to do it with the editor, the old images stayed in place. I had to use an external archive program to replace the images.

I guess I'd have to allow the possibility that something has changed in 3.4.x--I can only confidently vouch for the behavior described in my post up through 3.2.17.

[bdgza]

I can confirm I see the same with the file from Dropbox and when I repeat the experiment myself in Finder. My guess is the Archive Utility replaces the characters with substitutes. When I copy/paste the filename from the 'correct' archive to the 'faulty' archive using WinZip for Mac the scenario loads again. I can analyse what the different actual characters in the zip metadata are by writing a little program later today.

[bdgza]

The 'correct' archive:

Code: Select all

dotnet ReadZipHeader.dll /Users/bdegroot/Downloads/CCNapoleonics3_42.vmod.zip
S111 - Ocaña_Cavalry.vsav
0x53 0x31 0x31 0x31 0x20 0x2D 0x20 0x4F 0x63 0x61 0xF1 0x61 0x5F 0x43 0x61 0x76 0x61 0x6C 0x72 0x79 0x2E 0x76 0x73 0x61 0x76

The 'faulty' archive:

Code: Select all

dotnet ReadZipHeader.dll /Users/bdegroot/Downloads/CCNapoleonics3_42\ re-zipped\ on\ Mac.vmod.zip
S111 - Ocaña_Cavalry.vsav
0x53 0x31 0x31 0x31 0x20 0x2D 0x20 0x4F 0x63 0x61 0x6E 0x303 0x61 0x5F 0x43 0x61 0x76 0x61 0x6C 0x72 0x79 0x2E 0x76 0x73 0x61 0x76

As you can see the original zip file has the filename entry using 0xF1 for the ñ char. The macOS re-zipped entry uses 0x6E 0x0303 for the ñ char. VASSAL should compare filenames based on Unicode equivalent characters instead of comparing the actual bytes (or ban these special characters).

[uckelman]

What this looks like to me is that the filename is stored encoded as ISO-8859-1 in the first archive, and as UTF-8 but with the 'ñ' decomposed into an 'n' and a combining tilde in the second one. Whatever is writing the filename out that way is totally broken, because that's just not an equivalent filename.

Looking around a bit, I discovered to my horror that HFS+ stores filenames in UTF-8 _in a decomposed form_:

https://stackoverflow.com/questions/615 ... names-os-x

That is completely bonkers.

[uckelman]

I looked into this a bit more. It turns out there is something fairly low-impact we can do to mitigate this (though it is still completely bonkers that HFS+ was designed this way). Try VASSAL-3.4.7-SNAPSHOT-bug13525-95b51620d, available here:

http://www.vassalengine.org/~uckelman/tmp/

Does that solve the problem for you?

Also, do I remember correctly that one of you reporting this is using a Mac with a Retina display? If so, there's a second thing I'd appreciate it if you'd check in this build. I suspect that in 3.3 and previous 3.4 builds, the image you were seeing
when dragging pieces was half-size, and in this build that should be back to normal. Is it?

(NB: When I said UTF-8 above, it should have been UTF-16, but that makes no difference otherwise.)

[uckelman]

Thus spake jrwatts:
>
> This was not my experience with 3.4.5 (on Linux, Java 11.0.8 ) last
> week; I was replacing 5 images with new versions, and when I tried to do
> it with the editor, the old images stayed in place. I had to use an
> external archive program to replace the images.

Is this something you can reproduce reliably? What are the exact steps
you took?

--
J.

[marktb1961]

Also, do I remember correctly that one of you reporting this is using a Mac with a Retina display? If so, there's a second thing I'd appreciate it if you'd check in this build. I suspect that in 3.3 and previous 3.4 builds, the image you were seeing
when dragging pieces was half-size, and in this build that should be back to normal. Is it?
)

Yes I have a retina display. I hadn't noticed a problem before and now I look at it, the dragged images are 1:1 in v3.4.6 (compared to the pieces on the board). In the snapshot, they look like 2x size.

I will test the zip issue in the morning.

Mark

[marktb1961]

I looked into this a bit more. It turns out there is something fairly low-impact we can do to mitigate this (though it is still completely bonkers that HFS+ was designed this way). Try VASSAL-3.4.7-SNAPSHOT-bug13525-95b51620d, available here:

http://www.vassalengine.org/~uckelman/tmp/

Does that solve the problem for you?
...
(NB: When I said UTF-8 above, it should have been UTF-16, but that makes no difference otherwise.)

Yes; unzipped and re-zippped C&C Nap v3.42; Bailén and other scenarios with accented characters open fine in the snapshot.

Mark

[bdgza]

With VASSAL-3.4.7-SNAPSHOT-bug13525-95b51620d the macOS rezipped modules work for me as well.