Create account / Log in

module python library

Talk about module design ideas and techniques.

Moderators: uckelman, Tim M

Postby pelle » June 3rd, 2009, 1:05 pm

And, yes, an external VASSAL API is probably more useful than a stand-alone implementation in some other language to access a VMOD. Avoiding the problem of keeping up with format updates is a good reason. There is always Jython to be able to make use of an API from python. :)

But for the particular use-case I'm having I need to get to the file directly, and calling Java code isn't an option really in this case. Trying to keep it as small a hack as possible, relying on as little as possible of the file format to stay the same, is probably a good idea.
pelle
 
Posts: 17
Joined: March 3rd, 2008, 3:52 pm

Re: creating Perl/Python module for editing buildFile string

Postby jestew » September 9th, 2020, 11:51 am

pelle wrote:To me it would be much easier to work with XML than a mix of XML and some other format. But I perfectly well see how most VASSAL users and developers have no reason to change the way it is now.

The quoting is quite complex btw, not as simple as suggested above. As an example, here are the data blob for two very simple counters I created using VASSAL for reverse-engineering purposes:

Code: Select all
+/null/emb2;Flip;2;F;;0;;;0;;;;1;false;0;0;c_r.png;;true;;;;false;;1       prototype;PPP\  piece;;;e_d.png;FRONT/1;F       \       null;0;0;0

Code: Select all
+/null/prototype;P2       emb2;Flip;2;F;;0;;;0;;;;1;false;0;0;g_f.png;;true;;;;false;;1\  prototype;PPP\\ piece;;;c_f.png;FRONT2/ -1;\    \\      null;5;0;


The only difference is that one has two Prototypes (PPP and P2) while whe other has only one (PPP). The second was created by copy-paste from the first from within the VASSAL editor, and nothing was edited except adding the second Prototype. This tiny change makes the quoting all different, in ways that are not easy to guess without further studying the source code I think. For instane look at that last \ that suddenly becomes \\. Also it is not obvious why one ends in 0;0;0, while the other ends in 5;0;. There are also a few other minor differences that is in no obvious way connected to the addition of a Prototype.
[/code]


Hi again,

after adding a feature to the Text Label to handle transparency, I realized that for backwards compatibility I was going to need to support people still using 3.2.17 for some time - so I was going to have to provide a Layer trait with transparent images to replace the Text Label when people weren't using 3.4...

I ran into a challenge of needing to modify a number of existing modules to use this feature (around 10) that each contain many hundreds of counters. Not something I'm excited about continuing to do manually. Now regexes go a long way to handling the problem but... there are some niggly details about the string encoding for pieces that I'm still not quite clear about.

I've dug through the module save code a bit... but it wasn't giving me what I was looking for. This post gets me closer than I've been so far - could any of the developers who understand the process of how GamePieces are serialized point me in the right direction so I can backwards engineer a piece of Perl/Python code to parse buildFiles?

For example, I've figured out through trial and error that as traits are added to a piece the number of '\'-character bits at the end of the file increases. So, if I'm removing a trait from a piece (a delete trait) that ends with two '\\', then I need to remove the section at the end which has two '\\' and reduce the number of '\' from all the traits that appear after this one... Yuck. But I've yet to find the java code that actually implements this.

If you can point me forward, I'd be happy to write this up as a doc on the wiki and publish some buildFile util scripts to help people.

Cheers, jas...
User avatar
jestew
 
Posts: 49
Joined: April 5th, 2009, 6:50 pm

Re: module python library

Postby Cattlesquat » September 9th, 2020, 3:37 pm

In VASSAL, reading and writing these sequences (that specify Traits in the buildFile) is handled by the SequenceEncoder. Most traits will have a "myGetType()" and "mySetType()" method that will deal with these strings.

When you add a new bit of data to a trait, then it is best to add it at the END of the list (for minimum disruption if files are loaded by an earlier version, although in practice this won't happen much). And most of the myGetType() methods are designed so that if no data is found for the "new feature" it is automatically initialized to default values (and so then next time it is written, it will be automatically written with a record for the default value -- or any new value the user changed it to in the meantime).

SO... if the main problem you're describing is that you want to be able to easily update a bunch of pieces in the module to use the new default value, all you have to do is load and save the module once with the new version.

BUT... if the main problem is you want to change a bunch of counters to use an updated non-default value, then you would still load and save the module once first, and THEN change all the records. And since you are designing the new feature, you could conceivably give the encoding values for your trait a slightly unique "signature" so that you could easily global-search-and-replace them in your buildFile without accidentally picking up other strings?

Brian

Also: HOLY THREAD NECRO, BATMAN!!! :D
User avatar
Cattlesquat
 
Posts: 957
Joined: December 2nd, 2019, 4:57 pm
Location: Baltimore, Maryland, USA

Re: creating Perl/Python module for editing buildFile string

Postby Flint1b » September 9th, 2020, 5:59 pm

jestew wrote:But I've yet to find the java code that actually implements this.


It's in GameModule:
Code: Select all
  public String encode(Command c) {
    if (c == null) {
      return null;
    }
    String s = encodeSubCommand(c);
    String s2;
    Command[] sub = c.getSubCommands();
    if (sub.length > 0) {
      SequenceEncoder se = new SequenceEncoder(s, COMMAND_SEPARATOR);
      for (Command command : sub) {
        s2 = encode(command); // <------ note the recursion here
        if (s2 != null) {
          se.append(s2);
        }
      }
      s = se.getValue();
    }
    return s;
  }


The traits of a piece are in a tree instead of a list, each trait is a child of the previous trait, hence the necessary recursion and the increasing number of \ escapes. The innermost trait gets encoded, then encoded again by its containing trait, encoded again by the next trait level, and so on, until 90% of the savegame is \.
User avatar
Flint1b
 
Posts: 462
Joined: May 19th, 2020, 12:27 am
Location: Colonia Agrippina

Re: module python library

Postby jestew » September 9th, 2020, 8:12 pm

Cattlesquat wrote:Also: HOLY THREAD NECRO, BATMAN!!! :D

Glad you noticed...

Cattlesquat wrote:In VASSAL, reading and writing these sequences (that specify Traits in the buildFile) is handled by the SequenceEncoder. Most traits will have a "myGetType()" and "mySetType()" method that will deal with these strings.


Right. This given your super description of traits using the Decorator pattern is helpful.

The kinds of edits I want to do are various:
  • Change the keyboard shortcuts for many commands (easy with regex)
  • Rename certain menu commands (easy with regex)
  • Remove certain traits from some counters (doable with complex regex)

Because the text string for game pieces is not documented anywhere, it's a challenge to grok the meanings or importance of things like non-printing ascii characters embedded within the string (lots of tabs), or the meaning of combinations of characters ('emb2' vs. 'emb', or ';;;;'), or the increasing pattern of '\' characters at the end of the piece.
User avatar
jestew
 
Posts: 49
Joined: April 5th, 2009, 6:50 pm

Re: module python library

Postby Flint1b » September 9th, 2020, 9:21 pm

jestew wrote:Because the text string for game pieces is not documented anywhere, it's a challenge to grok the meanings or importance of things like non-printing ascii characters embedded within the string (lots of tabs), or the meaning of combinations of characters ('emb2' vs. 'emb', or ';;;;'), or the increasing pattern of '\' characters at the end of the piece.


What works well is stopping the debugger before decode(), then running custom strings through decode() and looking at the output. Or stopping right after encode and compare the command that was encoded to the string that came out of it.
User avatar
Flint1b
 
Posts: 462
Joined: May 19th, 2020, 12:27 am
Location: Colonia Agrippina

Re: module python library

Postby Cattlesquat » September 11th, 2020, 2:12 pm

'emb2' vs. 'emb'

emb2 is "Embellishment" which is the internal name for the Layer trait. It's the one you will see and the one you want.

emb1 is "Embellishment0", a legacy earlier-version-of-Layer, which exists for support of Modules From Ancient Times (e.g. pre 3.2.x, I think). You will only see it in very old modules which have never been re-saved under 3.2 versions.

There is a lot in VASSAL that exists to keep old modules running, as that has always been a priority for us.

Brian
User avatar
Cattlesquat
 
Posts: 957
Joined: December 2nd, 2019, 4:57 pm
Location: Baltimore, Maryland, USA

Re: module python library

Postby jestew » September 11th, 2020, 2:13 pm

@Flint1b - cheers - super helpful advice...

the debugger might be unnecessary if I can get my head around the bits below for prototypes and pieces (output from my buildFile report script)

Code: Select all
Prototype: name = Division Cmdr
   text = 1+/null/footprint;84,130;Movement Trail;true;true;5;0,204,153;0,204,102;100;50;20;30;1.0
      rotate;12;102,130;100,130;Rotate CW;Rotate CCW;;;Rotate Vertex\
      emb2;Out-Cmd;2;;Out-Cmd;2;;;2;;;;1;false;0;0;,m_out.gif;,Out_Cmd;true;Out of Command;;;false;;1;1;true;79,130;79,130;\\
      emb2;Activate;2;;Div Ineff;2;;;2;;;;;false;-40;0;,inefd.gif;,;true;Division Ineffective;;;false;;1;1;true;65,130;88,130;\\\
      emb2;Orders;2;;Orders;2;;;2;;;;;false;0;-40;,m_attack.png,m_march.gif;,Attack,March;true;Orders;;;false;;1;1;true;82,130;82,130;\\\\
      emb2;Flip;2;;;2;;;2;;;;;false;0;0;,;,;true;Flip;;;false;;1;1;false;70,130;;\\\\\
      markmoved;moved.gif;32;-32;Mark Moved;77,130\\\\\\
      delete;Delete;68,130\\\\\\\
      piece;;;;/true;;0
      0\
      1\\
      1\\\
      1\\\\
      -1\\\\\
      false\\\\\\
      \\\\\\\
      null;0;0;


and

Code: Select all
Piece: id =   1129
   name = D.H. Hill Corps
   text = +/null/delete;Delete;68,130
      footprint;84,130;Movement Trail;true;true;5;0,153,0;0,153,0;100;50;20;30;1.0\
      rotate;12;102,130;100,130;Rotate CW;Rotate CCW;;;Rotate vertex\\
      emb2;Flip;2;;Flip;2;;;2;;;;;false;0;0;,D_H_Hill_back.png;,+ -repl;true;Flip;;;false;;1;1;true;70,130;70,130;\\\
      piece;;;D_H_Hill_front.png;D.H. Hill Corps/
      true;;0\
      0\\
      1\\\
      null;0;0;1129


Each trait is separated by a tab character (in the report I show each on its own line for ease). Some traits end with a recursion marker, '\' and the recursion depth is the number of markers (the zero-th trait has zero '\' markers).

For the prototype we have the
Code: Select all
'piece;;;;/
which shows the end of the piece and for the piece it has the piece image and piece name.

And then a line for each of the traits with either true;;0, 0,1,-1,false, or (empty string),
markmoved - false
delete, submenu, - empty string
movement trails - true;;0
the rest - 0,1,-1

and then the final
Code: Select all
null;0;0;
with the piece ending in the Piece ID (1129).

Are my above dissections correct so far??

I want to be able to modify a buildFile to:
  • remove traits,
  • add traits
  • modify command names
  • modify keyboard shortcuts

Only removing traits and adding traits is dangerous because it changes the number of '\' markers needed for the traits that come after, and things have to be added/removed to the end of the string and not just the middle.

Any clue as to what the piece at the end of the string is? When I get it wrong the module gives me a bad Map ID error.

Thanks again, jas...
User avatar
jestew
 
Posts: 49
Joined: April 5th, 2009, 6:50 pm

Re: module python library

Postby Flint1b » September 11th, 2020, 4:53 pm

jestew wrote:Are my above dissections correct so far??

The serialization code is ancient, and if you continue working on this you are likely to be come the one guy who knows the Vassal's serialization language :lol:

jestew wrote:I want to be able to modify a buildFile to:
  • remove traits,
  • add traits
  • modify command names
  • modify keyboard shortcuts

Sounds so good that it might even make sense to add this to Vassal itself. Either to the module editor, or as an alternative second module editing tool, "Editor++".

Also, there are plans to create a whole new module serialization system which would read/write modules in JSON.
User avatar
Flint1b
 
Posts: 462
Joined: May 19th, 2020, 12:27 am
Location: Colonia Agrippina

Re: module python library

Postby jestew » September 11th, 2020, 5:31 pm

A very dangerous part of my brain wonders if it isn't easier for me to write the JSON (de)encoder and then the create the Perl/Python library based on that than it is to reverse engineer the existing (de)encoding system...
User avatar
jestew
 
Posts: 49
Joined: April 5th, 2009, 6:50 pm

Re: module python library

Postby jestew » September 14th, 2020, 1:39 pm

OK. Field Report 14 September 2020.

After having loaded VASSAL into IntelliJ and debugged the module loading behavior of game pieces and prototype definitions, I now feel confident on how to describe the structure of the strings in the buildFile.
The components of a piece are delimited by the '/' character and consist of 4 pieces:
    [1] The AddPiece token: '+'
    [2] The piece ID - always 'null' in the buildFile
    [3] The trait stack for that piece - with each sub-trait separated by '\t' (and the '\' chars multiply protected by further back '\' chars)
    [4] The state info for each element of the trait stack - with the same caveat above for '\t' and protection of the '\'

So... The long and short of this is there needs to be two stacks - one for the nested traits and the other for the state information for those traits. If any trait is removed, then it's corresponding state information must also be removed before being reconverted into a string value.

et voila.

This is sufficient for the swiss army knife script I'm writing to convert a bunch of modules. Unfortunately, it is NOT sufficient to use for updating the information for pre-defined setups stored in the module... The savedGame files stored inside the .vsav files are obfuscated text files which are just one long game command string to modify the starting setup...
User avatar
jestew
 
Posts: 49
Joined: April 5th, 2009, 6:50 pm

Re: module python library

Postby ardaedhel » October 29th, 2020, 7:34 am

jestew wrote:The components of a piece are delimited by the '/' character and consist of 4 pieces:
    [1] The AddPiece token: '+'
    [2] The piece ID - always 'null' in the buildFile
    [3] The trait stack for that piece - with each sub-trait separated by '\t' (and the '\' chars multiply protected by further back '\' chars)
    [4] The state info for each element of the trait stack - with the same caveat above for '\t' and protection of the '\'

So... The long and short of this is there needs to be two stacks - one for the nested traits and the other for the state information for those traits. If any trait is removed, then it's corresponding state information must also be removed before being reconverted into a string value.


Dude... Just popping in here to drop an enthusiastic THANK YOU from a guy who's foolishly mucking around parsing a buildFile to generate custom .vlogs. <3
ardaedhel
 
Posts: 1
Joined: November 7th, 2017, 5:20 am

Re: module python library

Postby rajputaman04 » November 20th, 2020, 7:30 am

I haven't even bothered to fully understand the format used to store game pieces. I expected it to be XML, but it turned out to be some sort of hack to get nested string values quoted into an XML string. Even a simple thing like adding a second Prototype turned out to completely alter the way the string must look like. I postponed reading DataSequence.java for now. I guess most things I need to know are in there, but I also noticed some other values that seems to randomly vary even when I create a duplicate of a counter.
rajputaman04
 
Posts: 1
Joined: November 20th, 2020, 7:29 am

Previous

Return to Module Design

Who is online

Users browsing this forum: No registered users and 3 guests