Technical questions

[Flint1b]

This tool looks good: https://jeantessier.github.io/dependency-finder/

I tried it like this, and got some results, sorry for the messed up formatting, the "code" tag here seems to replace some spaces with tabs:

- download, install
-- download here: https://sourceforge.net/projects/depfin ... t/download
-- un-tar-gz

- run
-- add dependency finder /bin to path

Code: Select all

export PATH=$DEPENDENCY_FINDER_HOME/bin:$PATH


-- dependency finder needs JAVA_HOME to be set else it looks in /bin/java

Code: Select all

export JAVA_HOME=/usr/lib/jvm/graalvm-java11


-- possibly filter modules, based on version number in filename or version number set inside the module

Code: Select all

???


-- prepare dependency data for modules, this creates a 1-1.5mb xml per module:

Code: Select all

DependencyExtractor -xml -out Paths_of_Glory_9.7.xml $VASSAL_MODULES/Paths_of_Glory_9.7.vmod

DependencyExtractor -xml -out For_the_People_3.2.xml $VASSAL_MODULES/For_the_People_3.2.vmod


-- extract deprecated elements from vassal, then write a report of deprecated elements in use:

Code: Select all

ListDeprecatedElements -out deprecated.txt $VASSAL_HOME/lib/Vengine.jar

DependencyReporter -out report.txt -show-inbounds -scope-includes-list deprecated.txt Paths_of_Glory_9.7.xml For_the_People_3.2.xml


-- result is an empty file, now do a cross-check by doing the same for the current 3.3 codebase:

Code: Select all

ListDeprecatedElements -out deprecated_3.3.txt $VASSAL_COMPILED_CLASSES

DependencyReporter -out report_3.3.txt -show-inbounds -scope-includes-list deprecated_3.3.txt Paths_of_Glory_9.7.xml For_the_People_3.2.xml


-- result:

Code: Select all

$ cat report_3.3.txt

VASSAL.build.module *
    Map *
        componentCoordinates(java.awt.Point) *
            <-- ForThePeople.FTPKeyBufferer.mousePressed(java.awt.event.MouseEvent)
            <-- ForThePeople.FTPMenuDisplayer.mouseReleased(java.awt.event.MouseEvent)
        mapCoordinates(java.awt.Point) *
            <-- ForThePeople.FTPKeyBufferer.mouseReleased(java.awt.event.MouseEvent)


Overall a very convenient tool, can take anything from a jar, zip, vmod, filesystem directory with compiled classes, finds and analyzes every piece of bytecode in it, has lots of other options for analysis and reporting.

Is this a viable tool and should I try writing a full bash script? I'm not a professional bash coder and don't know how the module files are laid out in the filesystem, also don't know the disk space constraints, and wouldn't be able to test this script in a production-like environment. Also don't know if it would be possible to detect the latest versions of the modules by using the filename, the few modules I've looked at have their version in the filename but this might not apply to all modules, and the module version set by the module designers might also not be a reliable criteria for deciding which is latest.

[uckelman]

Thus spake Flint1b:
> [This message has been edited.]
>
> This tool looks good:
> https://jeantessier.github.io/dependency-finder/[1]

I'm giving this a try now. Running it on all the modules is going to
take quite a while; I'll report back when I have some results.

--
J.

[uckelman]

Thus spake Joel Uckelman:
> Thus spake Flint1b:
> > [This message has been edited.]
> >
> > This tool looks good:
> > https://jeantessier.github.io/dependency-finder/[1]
>
> I'm giving this a try now. Running it on all the modules is going to
> take quite a while; I'll report back when I have some results.

See Bug 13124, where I've attached the results:

http://www.vassalengine.org/tracker/sho ... i?id=13124

--
J.

[Flint1b]

Is this all that is used from the modules, only about 50 methods? If this is true then backwards compatibility is not as big a problem as it sounded like.

[uckelman]

Thus spake Flint1b:
> Is this all that is used from the modules, only about 50 methods? If
> this is true then backwards compatibility is not as big a problem as it
> sounded like.

That's all that's used which is _deprecated_. I didn't run the analysis
on what's used in total.

--
J.

[Flint1b]

That's all that's used which is _deprecated_. I didn't run the analysis
on what's used in total.

Ahh ok, I misunderstood. Right, that tool makes a file of all deprecated fields/methods first, then checks all modules against that file.

I'm curious though, how many modules are there in total, how big is their total size, and how long did it take to run this dependency analysis? Hours? Days?

[uckelman]

Thus spake Flint1b:
>
> Ahh ok, I misunderstood. Right, that tool makes a file of all deprecated
> fields/methods first, then checks all modules against that file.

What I ran over the modules is

DependencyExtractor -xml -out blah.xml blah.vmod

which gets everything, if I understand correctly.

That output was then fed to

DependencyReporter -out depreport -show-inbounds -scope-includes-list deprecated

which is what filters out everything not in the deprecated list.

> I'm curious though, how many modules are there in total, how big is
> their total size, and how long did it take to run this dependency
> analysis? Hours? Days?

Moments ago, there were 7546 files which match *.{mod,vmod,ext,vext}
(many of which don't have any custom code in them, many of which also
aren't current, but that's hard to tell just from running find over
the tree where the files are). The total size of the tree is around
270GB, which also includes some images and other random stuff, but
the vast majority of which is modules. I let the analysis run overnight,
and it produced 2.6GB of _text_. The total time was hours, but I didn't
keep track of how many hours.

--
J.

[Brent Easton]

That is also only modules that we host though. There are at least some modules using custom code not hosted on vassalengine.org

[Flint1b]

Interesting info, thanks.

I have yet another question, in src/, next to the VASSAL package, we have what looks like a stripped and possibly modified or manually extended version of BeanShell v. 2.x. It has a subset of asm in bsh.org.objectweb.asm which BeanShell 1.3.0 did not have so I guess it is BeanShell >= 2.0.

Why is the BeanShell libraries code manually copied into the project instead of bringing its .jar as a dependency? And which version of BeanShell is this exactly? What is the process of updating to newer BeanShell versions? Are we aware that the class bsh.XThis.Handler that we have in the code has this neat little arbitrary code execution exploit http://cve.mitre.org/cgi-bin/cvename.cg ... -2016-2510 that has only been fixed in 2.0b6 (https://github.com/beanshell/beanshell/ ... /tag/2.0b6) ?

More generally speaking, if I made a module which, apart from allowing to play a boardgame, used some kind of network channel to contact a private server and get new orders from me e.g. for scanning the hard drive for sensitive data, taking photographs with the camera, would there be some mechanism in Vassal to prevent that i.e. prevent "evil" module designers from doing evil things?

And by extension, what if I was a "sloppy" (in the sense of software security) module designer and without having bad intentions wrote custom code for my module that would allow a user with bad intentions to do evil things e.g. in a multiplayer or PBEM setting enter some code into a text field which my module's custom code would blindly execute on another user's machine?

Are problems like this solvable at all if modules have the whole Java language as their API, I am not even sure myself. I only had to deal with server-based security so far.

[Brent Easton]

Why is the BeanShell libraries code manually copied into the project instead of bringing its .jar as a dependency?

Because we had to make significant modifications to the source, including parser changes to get it to play nicely with Vassal. We had also been having a load of trouble with upgraded external libraries importing bugs into our project, so we essentially took a fork of Beanshell.

And which version of BeanShell is this exactly?

2.0b4

What is the process of updating to newer BeanShell versions?

It's not really required except for any major bug fixes which will need to be done manually.

Are we aware that the class bsh.XThis.Handler that we have in the code has this neat little arbitrary code execution exploit http://cve.mitre.org/cgi-bin/cvename.cg ... -2016-2510 that has only been fixed in 2.0b6 (https://github.com/beanshell/beanshell/ ... /tag/2.0b6) ?

I don't believe that affects us, but should probably be patched.

[Flint1b]

I understand, thank you!

[Flint1b]

I spent some time trying to wrestle the current parts of the makefile that build the various releases into maven. It looks like it could work, but it's too much hassle, the end result is too verbose, it's like walking on crutches. Right now my idea is to throw all these experiments away and try using Java 14's jpackage. Travis offers linux, windows and macos machines, the requirements for jpackage can be installed theoretically.

But some questions came up while I was doing this:
- the "other" release contains both a bash script and the exe for windows, what is it for exactly? Windows users will most likely not download it at all, linux and mac users would also download the releases for their respective systems, is the "other" release for some obscure niche unixes like freebsd?
- the linux (and other) release come without installers, while the windows and macos come with installers, why is this not symmetrical? As a linux user I am glad I can just unzip several versions of Vassal next to each other and select the one I want to run, and select the Java version I want to run it with, but are Windows/Mac users not at a disadvantage here because they need to use the installer or can they install several versions of Vassal as well?
- the makefile's "module_deps" target, what is the "grep -v split package" for, it seems to not do anything since the output of jdeps is only a single line which only contains the jmods? And the "tr -d '\n', is it necessary? I have not tried it myself but I have seen several examples of this jdeps-jlink chain where the output of jdeps is fed into jlink without grepping and removing newlines, in one case they even put this output into a $VAR and passed it to jlink.
- the license is Lesser GPL 2.1, I have never dealt with these things before, does it have to stay at 2.1 or can it be updated to the newer 3.0 just like that, without consulting a lawyer?

Also, big respect to those who do 1st level support, I have watched the "bug" reports and other questions and I understand much better now why the users are spoon-fed with installers and bundled JREs..

[uckelman]

Thus spake Flint1b:
>
> I spent some time trying to wrestle the current parts of the makefile
> that build the various releases into maven. It looks like it could work,
> but it's too much hassle, the end result is too verbose, it's like
> walking on crutches. Right now my idea is to throw all these experiments
> away and try using Java 14's jpackage. Travis offers linux, windows and
> macos machines, the requirements for jpackage can be installed
> theoretically.

I doubt that you will get jpackage to work for building all the packages
on a single host, for the following reason: When I tried jpackage, I found
that the Windows-specific parts work only on Windows, the Mac-specific
parts only on Macs.

> But some questions came up while I was doing this:
> - the "other" release contains both a bash script and the exe for
> windows, what is it for exactly? Windows users will most likely not
> download it at all, linux and mac users would also download the releases
> for their respective systems, is the "other" release for some obscure
> niche unixes like freebsd?

The "other" package is intended for the case where none of the others
work for you. That could be because you're on some other Unix (though
in that case, it's likely that the Linux package would work as well),
or it could be because you're on Windows but can't install anything,
or you want to run from a USB drive.

> - the linux (and other) release come without installers, while the
> windows and macos come with installers, why is this not symmetrical? As
> a linux user I am glad I can just unzip several versions of Vassal next
> to each other and select the one I want to run, and select the Java
> version I want to run it with, but are Windows/Mac users not at a
> disadvantage here because they need to use the installer or can they
> install several versions of Vassal as well?

* Bundling Java on Linux is the Wrong Thing. Java's libjvm dynamically
links to various other libraries (e.g., libstdc++, libm, libc, libgcc)
and all these libraries have versioned symbols. So, if you supply a
libjvm that you want to be sure will work for the user, it has to be
compiled and linked against libraries which are no newer than the ones
the user has... which it turns out is a VERY broad range when you start
considering Long Term Support releases that a lot of distros do. I checked
one of the few projects that bundles Java for Linux and found that they
have a libjvm which is compiled against a libc which was released more
than a decade ago... This is not a good road to go down.

Any Linux user is a single command away from having a version of Java
which works with VASSAL. The Right Thing on Linux is to take advantage of
the nice, mature package management that every distro has and use the
build of Java the distro maintainers have ensured will work for the
distro the user is running.

* The Windows installer gives you some options for how to install VASSAL.
The "Standard" install uninstalls older versions and puts VASSAL X.Y.Z into
C:\Program Files\VASSAL-X.Y.Z. The "Custom" install lets you choose which
older versions to remove and which to keep, and also lets you choose a
different install path if you want. There's nothing preventing you from
having several versions of VASSAL installed simultaneously.

The pre-3.3 Windows installer also checked for Java and installed Java
for you if you didn't have it, before a change at java.com broke that.
Before the Windows installer, we had endless problems with Windows users
(a) being confused when VASSAL wouldn't run because they didn't have
Java installed, (b) having a screwed up Java installation, (c) not knowing
how to "install" from a ZIP archive. In general, having an installer is
a normal thing on Windows, and not having one was considered weird.

*The Mac app bundle has the version as part of its name, so if you install
two different versions, they'll end up in different directories, so there's
nothing preventing you from having multiple versions of VASSAL installed
on a Mac, either.

> - the makefile's "module_deps" target, what is the "grep -v split
> package" for, it seems to not do anything since the output of jdeps is
> only a single line which only contains the jmods?

That was to filter out something which no longer shows up in the output,
I think since we eliminated the --add-exports. I'm removing it now.

> And the "tr -d '\n',
> is it necessary? I have not tried it myself but I have seen several
> examples of this jdeps-jlink chain where the output of jdeps is fed into
> jlink without grepping and removing newlines, in one case they even put
> this output into a $VAR and passed it to jlink.

That's also no longer needed. (Possibly I was wrong that it was ever
needed, but it didn't harm anything.) It's removed in PR 51.

> - the license is Lesser GPL 2.1, I have never dealt with these things
> before, does it have to stay at 2.1 or can it be updated to the newer
> 3.0 just like that, without consulting a lawyer?

IIRC, there's an "any later version" provision, but I would need to
check on that to be sure.

> Also, big respect to those who do 1st level support, I have watched the
> "bug" reports and other questions and I understand much better now why
> the users are spoon-fed with installers and bundled JREs..

Are there T-shirts which say "USERS ARE WHY I DRINK" on them?

--
J.

[Flint1b]

I doubt that you will get jpackage to work for building all the packages
on a single host, for the following reason: When I tried jpackage, I found
that the Windows-specific parts work only on Windows, the Mac-specific
parts only on Macs.

Yes I know about this, I meant to have a CI build server do this, one that offers linux, windows and macos hosts. Travis does this, and also Github Actions as a backup option. We could theoretically configure it to build on each of the different OSes, so far my research has showed that all the jpackage's requirements can be installed on Travis, their macos host comes with the required xcode, their windows host comes with a linux-like packaging system called "chocolatey" which is able to install the wix toolset that jpackage needs on windows.

It also seems possible to build windows packages on linux using wine, but that is walking on crutches again if a real windows host is available.

[Flint1b]

1) The Vassal documentation (Users Guide, Module Designers Guide) pdfs are from Vassal version 3.1, they are hopefully written in latex, is their source available and can it be put up on github, and included in the release process?

2) Why don't the C++ developers learn from the best and write a launcher for Vassal, in C++, using QT or whatever, one with a small memory footprint, portable across the major OSes, and a simple GUI with some fields to select between bundled or any other JRE, set the heap size, set optional JRE arguments, maybe even select a specific Vassal version? One like this https://www.minecraft.net/en-us/article ... er-is-live?

A possible variation on this, replace the whole ModuleManager with this launcher, and also disallow starting several Players and Editors? If a user wants to start two Player instances, he starts launcher+player, and another launcher+player.