Technical questions

Thus spake Flint1b:

Is this all that is used from the modules, only about 50 methods? If
this is true then backwards compatibility is not as big a problem as it
sounded like.

That’s all that’s used which is deprecated. I didn’t run the analysis
on what’s used in total.


J.

Ahh ok, I misunderstood. Right, that tool makes a file of all deprecated fields/methods first, then checks all modules against that file.

I’m curious though, how many modules are there in total, how big is their total size, and how long did it take to run this dependency analysis? Hours? Days?

Thus spake Flint1b:

Ahh ok, I misunderstood. Right, that tool makes a file of all deprecated
fields/methods first, then checks all modules against that file.

What I ran over the modules is

DependencyExtractor -xml -out blah.xml blah.vmod

which gets everything, if I understand correctly.

That output was then fed to

DependencyReporter -out depreport -show-inbounds -scope-includes-list deprecated

which is what filters out everything not in the deprecated list.

I’m curious though, how many modules are there in total, how big is
their total size, and how long did it take to run this dependency
analysis? Hours? Days?

Moments ago, there were 7546 files which match *.{mod,vmod,ext,vext}
(many of which don’t have any custom code in them, many of which also
aren’t current, but that’s hard to tell just from running find over
the tree where the files are). The total size of the tree is around
270GB, which also includes some images and other random stuff, but
the vast majority of which is modules. I let the analysis run overnight,
and it produced 2.6GB of text. The total time was hours, but I didn’t
keep track of how many hours.


J.

That is also only modules that we host though. There are at least some modules using custom code not hosted on vassalengine.org

Interesting info, thanks.

I have yet another question, in src/, next to the VASSAL package, we have what looks like a stripped and possibly modified or manually extended version of BeanShell v. 2.x. It has a subset of asm in bsh.org.objectweb.asm which BeanShell 1.3.0 did not have so I guess it is BeanShell >= 2.0.

Why is the BeanShell libraries code manually copied into the project instead of bringing its .jar as a dependency? And which version of BeanShell is this exactly? What is the process of updating to newer BeanShell versions? Are we aware that the class bsh.XThis.Handler that we have in the code has this neat little arbitrary code execution exploit github.com/beanshell/beanshell/ … /tag/2.0b6) ?

More generally speaking, if I made a module which, apart from allowing to play a boardgame, used some kind of network channel to contact a private server and get new orders from me e.g. for scanning the hard drive for sensitive data, taking photographs with the camera, would there be some mechanism in Vassal to prevent that i.e. prevent “evil” module designers from doing evil things?

And by extension, what if I was a “sloppy” (in the sense of software security) module designer and without having bad intentions wrote custom code for my module that would allow a user with bad intentions to do evil things e.g. in a multiplayer or PBEM setting enter some code into a text field which my module’s custom code would blindly execute on another user’s machine?

Are problems like this solvable at all if modules have the whole Java language as their API, I am not even sure myself. I only had to deal with server-based security so far.

Because we had to make significant modifications to the source, including parser changes to get it to play nicely with Vassal. We had also been having a load of trouble with upgraded external libraries importing bugs into our project, so we essentially took a fork of Beanshell.

2.0b4

It’s not really required except for any major bug fixes which will need to be done manually.

I don’t believe that affects us, but should probably be patched.

I understand, thank you!

I spent some time trying to wrestle the current parts of the makefile that build the various releases into maven. It looks like it could work, but it’s too much hassle, the end result is too verbose, it’s like walking on crutches. Right now my idea is to throw all these experiments away and try using Java 14’s jpackage. Travis offers linux, windows and macos machines, the requirements for jpackage can be installed theoretically.

But some questions came up while I was doing this:

  • the “other” release contains both a bash script and the exe for windows, what is it for exactly? Windows users will most likely not download it at all, linux and mac users would also download the releases for their respective systems, is the “other” release for some obscure niche unixes like freebsd?
  • the linux (and other) release come without installers, while the windows and macos come with installers, why is this not symmetrical? As a linux user I am glad I can just unzip several versions of Vassal next to each other and select the one I want to run, and select the Java version I want to run it with, but are Windows/Mac users not at a disadvantage here because they need to use the installer or can they install several versions of Vassal as well?
  • the makefile’s “module_deps” target, what is the “grep -v split package” for, it seems to not do anything since the output of jdeps is only a single line which only contains the jmods? And the "tr -d ‘\n’, is it necessary? I have not tried it myself but I have seen several examples of this jdeps-jlink chain where the output of jdeps is fed into jlink without grepping and removing newlines, in one case they even put this output into a $VAR and passed it to jlink.
  • the license is Lesser GPL 2.1, I have never dealt with these things before, does it have to stay at 2.1 or can it be updated to the newer 3.0 just like that, without consulting a lawyer?

Also, big respect to those who do 1st level support, I have watched the “bug” reports and other questions and I understand much better now why the users are spoon-fed with installers and bundled JREs…

Thus spake Flint1b:

I spent some time trying to wrestle the current parts of the makefile
that build the various releases into maven. It looks like it could work,
but it’s too much hassle, the end result is too verbose, it’s like
walking on crutches. Right now my idea is to throw all these experiments
away and try using Java 14’s jpackage. Travis offers linux, windows and
macos machines, the requirements for jpackage can be installed
theoretically.

I doubt that you will get jpackage to work for building all the packages
on a single host, for the following reason: When I tried jpackage, I found
that the Windows-specific parts work only on Windows, the Mac-specific
parts only on Macs.

But some questions came up while I was doing this:

  • the “other” release contains both a bash script and the exe for
    windows, what is it for exactly? Windows users will most likely not
    download it at all, linux and mac users would also download the releases
    for their respective systems, is the “other” release for some obscure
    niche unixes like freebsd?

The “other” package is intended for the case where none of the others
work for you. That could be because you’re on some other Unix (though
in that case, it’s likely that the Linux package would work as well),
or it could be because you’re on Windows but can’t install anything,
or you want to run from a USB drive.

  • the linux (and other) release come without installers, while the
    windows and macos come with installers, why is this not symmetrical? As
    a linux user I am glad I can just unzip several versions of Vassal next
    to each other and select the one I want to run, and select the Java
    version I want to run it with, but are Windows/Mac users not at a
    disadvantage here because they need to use the installer or can they
    install several versions of Vassal as well?
  • Bundling Java on Linux is the Wrong Thing. Java’s libjvm dynamically
    links to various other libraries (e.g., libstdc++, libm, libc, libgcc)
    and all these libraries have versioned symbols. So, if you supply a
    libjvm that you want to be sure will work for the user, it has to be
    compiled and linked against libraries which are no newer than the ones
    the user has… which it turns out is a VERY broad range when you start
    considering Long Term Support releases that a lot of distros do. I checked
    one of the few projects that bundles Java for Linux and found that they
    have a libjvm which is compiled against a libc which was released more
    than a decade ago… This is not a good road to go down.

Any Linux user is a single command away from having a version of Java
which works with VASSAL. The Right Thing on Linux is to take advantage of
the nice, mature package management that every distro has and use the
build of Java the distro maintainers have ensured will work for the
distro the user is running.

  • The Windows installer gives you some options for how to install VASSAL.
    The “Standard” install uninstalls older versions and puts VASSAL X.Y.Z into
    C:\Program Files\VASSAL-X.Y.Z. The “Custom” install lets you choose which
    older versions to remove and which to keep, and also lets you choose a
    different install path if you want. There’s nothing preventing you from
    having several versions of VASSAL installed simultaneously.

The pre-3.3 Windows installer also checked for Java and installed Java
for you if you didn’t have it, before a change at java.com broke that.
Before the Windows installer, we had endless problems with Windows users
(a) being confused when VASSAL wouldn’t run because they didn’t have
Java installed, (b) having a screwed up Java installation, (c) not knowing
how to “install” from a ZIP archive. In general, having an installer is
a normal thing on Windows, and not having one was considered weird.

*The Mac app bundle has the version as part of its name, so if you install
two different versions, they’ll end up in different directories, so there’s
nothing preventing you from having multiple versions of VASSAL installed
on a Mac, either.

  • the makefile’s “module_deps” target, what is the “grep -v split
    package” for, it seems to not do anything since the output of jdeps is
    only a single line which only contains the jmods?

That was to filter out something which no longer shows up in the output,
I think since we eliminated the --add-exports. I’m removing it now.

And the "tr -d ‘\n’,
is it necessary? I have not tried it myself but I have seen several
examples of this jdeps-jlink chain where the output of jdeps is fed into
jlink without grepping and removing newlines, in one case they even put
this output into a $VAR and passed it to jlink.

That’s also no longer needed. (Possibly I was wrong that it was ever
needed, but it didn’t harm anything.) It’s removed in PR 51.

  • the license is Lesser GPL 2.1, I have never dealt with these things
    before, does it have to stay at 2.1 or can it be updated to the newer
    3.0 just like that, without consulting a lawyer?

IIRC, there’s an “any later version” provision, but I would need to
check on that to be sure.

Also, big respect to those who do 1st level support, I have watched the
“bug” reports and other questions and I understand much better now why
the users are spoon-fed with installers and bundled JREs…

Are there T-shirts which say “USERS ARE WHY I DRINK” on them?


J.

Yes I know about this, I meant to have a CI build server do this, one that offers linux, windows and macos hosts. Travis does this, and also Github Actions as a backup option. We could theoretically configure it to build on each of the different OSes, so far my research has showed that all the jpackage’s requirements can be installed on Travis, their macos host comes with the required xcode, their windows host comes with a linux-like packaging system called “chocolatey” which is able to install the wix toolset that jpackage needs on windows.

It also seems possible to build windows packages on linux using wine, but that is walking on crutches again if a real windows host is available.

  1. The Vassal documentation (Users Guide, Module Designers Guide) pdfs are from Vassal version 3.1, they are hopefully written in latex, is their source available and can it be put up on github, and included in the release process?

  2. Why don’t the C++ developers learn from the best and write a launcher for Vassal, in C++, using QT or whatever, one with a small memory footprint, portable across the major OSes, and a simple GUI with some fields to select between bundled or any other JRE, set the heap size, set optional JRE arguments, maybe even select a specific Vassal version? One like this minecraft.net/en-us/article … er-is-live?

A possible variation on this, replace the whole ModuleManager with this launcher, and also disallow starting several Players and Editors? If a user wants to start two Player instances, he starts launcher+player, and another launcher+player.

Thus spake Flint1b:

  1. The Vassal documentation (Users Guide, Module Designers Guide) pdfs
    are from Vassal version 3.1, they are hopefully written in latex, is
    their source available and can it be put up on github, and included in
    the release process?
  • The Reference Manual is some very old HTML, in doc/Reference Manual.

  • The Users’ Guide is a Word document in doc/userguide.

  • The Module Designers’ Guide… is not in the repository and might take
    a bit of tracking down, but IIRC it’s also a Word document.

I argued at the time that both the Users’ and Module Designers’ Guides
be done in a format which was diffable, or otherwise they’d be impossible
to maintain… and here we are. I didn’t press for doing them in LaTeX,
though that was my preference, because I was the only one at the time
who knew LaTeX. Something like Pandoc or even HTML5 would have been better
than Word.

(Can anyone point out where the source for the Module Designers’ Guide
is?)

All three of these need updating for 3.3.


J.

Asciidoc is often used in the Java world, and Pandoc is a neat swiss army knife that could convert the Word documents to Asciidoc asciidoctor.org/docs/asciidoctor-maven-plugin/.

I tried to convert the userguide.doc to Asciidoc using Pandoc, the result is ok, it needs to be cleaned up a bit but everything is there, the text, the paragraphs, headers, the images.

If there is interest in maintaining the guide, I would prepare a PR which adds the necessary files and maven build steps. I would prefer to add this as a separate maven module though so it’s best done if/after we PR #57 which switches to maven-multi-module.

If the sources for the module designers manual are found, we could do the same for it as well.

The reference manual seems to exist twice, once as html in the code and a second time on the wiki, we could change it so there is only a single source, probably best to keep it in the code and add a step to the release build which updates the wiki pages, e.g. with github.com/fastily/jwiki.

Hmmm, is there a solution where the master guide becomes the Wiki and then gets slurped up occasionally into a manual for release?

Because the Wiki offers the opportunity for crowdsourcing of some of the best knowledge, AND offers a more tried-and-true way of maintaining a body of information where the knowledge is distributed over a bunch of people.

Speaking of the Wiki I would also love to see us create (as many sites do) a level of privilege below “admin” but above “joe schmoe” – e.g. Expert or Veteran or whatever, and the main important privilege would be posting wiki edits without the moderation cycle (in other words, equal to a “normal user” on most wiki-based sites). I gather the moderation was put in place mainly because of bots, so having a normal wiki level of trust conferred on long-time users doesn’t seem like a big risk. I edit the wiki occasionally but it feels sort of like “editing through molasses” and so I don’t end up contributing nearly as much as I otherwise would.

That would be good too, sometimes we used confluence wiki at work to write the main documentation, then export it to PDF. Not sure of mediawiki has this, but it probably does.

I will probably get rocks thrown at me for asking this, but after being involved with the code for couple months I feel I have to ask… What’s the deal with these “key commands” or “global key commands” or "GKC"s, I see a lot of discussions where these are involved, I have seen how the modules have a hotkey for each and every possible action, I have never used them in any of the modules I play since it’s more convenient with the mouse… Why are these key commands such an important topic, is the whole application glued together by real and virtual key commands being sent back and forth??

Key Command is really just a legacy name from A Time In Ages Past when Vassal worked exclusively on key commands. I think even THEN, the key commands also generally had ways to perform them with the mouse (e.g. from the context menus). And THESE DAYS there don’t even have to be “actual keys” associated with “key” commands, you can just name it something like DoTheAxisThing which is sort of the Vassal evolutionary equivalent of being able to name a memory location with a “name” rather than 0xe458:0x3200 and stuff like that.

So really a “key command” is just a “command” that activates an individual Trait (to you, Decorator) of a Piece (to you, GamePiece), and makes that trait “do a thing”, whatever that trait likes to do when it gets the right signal. And a Global Key Command is a KIND of trait (for once the internal Java code name is the same) that applies a “key command” to all the pieces on a map (or all maps) that match a certain filter. So they’re a powerful building block of modules (since they are almost the only way two different GamePieces can affect each other), but they’re also a big performance hit because every GKC has to first build a list of all the maps it can touch (that’s the easy part), THEN, for each map go through every piece on that map and see if it qualifies (matches the bsh expression filter) to receive the command, and finally for each piece that matches the key command goes through and checks every single Decorator (“trait”) in the piece to see if it happens to be fired off by this “key stroke” (which might just be a “named key stroke” which is the kind of key stroke that isn’t actually a keystroke, like I mentioned earlier), and if it does match, then fire it off.

p.s. GREAT place if you can find any optimizations! I also dream of a future world where we maintain a list of pieces in each Zone (and maybe even in each Region) of a map, so that we could have “Zone Key Commands” and/or “Region Key Commands” and have them run faster because they don’t have to process through every piece on the map – because honestly MOST of the time I fire one of these I actually already know a specific sub-region of the map that I care about. As long as maintaining those lists wouldn’t be a cure worse than the disease, of course. We had a little linked list for every single “square” in Civ2 though, so I know it’s at least in the range of possibility.

Good explanation, thanks a lot.

First thing that comes to mind is to use the stream API’s Collection.parallelStream() to process all pieces in parallel.

Any time! It’s the least I owe you at this point ;-)