Command-line tools to split and/or reassemble .fwdata files

[rmunn]

These tools were originally created in sillsdev/languageforge-lexbox#735, but are being moved to the flexbridge repository in order to simplify the process of distributing them. The intent of these tools is to enable automated test scenarios for Send/Receive as per sillsdev/LfMerge#339, and possibly other as-yet-unknown use cases.

---

This change is 

[rmunn]

AppVeyor build doesn't support net8.0, it seems. Nevertheless, I believe this is ready to be reviewed: there's no reason, IMHO, to downgrade the targeted .NET version to net6.0 or earlier since all the environments where this tool would be run are easily capable of supporting .NET 8. We'll just have to ignore the failing AppVeyor build.

[myieye]

As just discussed in our team meeting:

• This apparently isn't blocked by any Nuget/FLEx/build stuff 🎉
• This is the right repo for both the C# code and the CLI
• We're not trying to include a CLI exe in a FLEx installation in this PR. Perhaps we'll build an exe manually for our own purposes for now. Redundant. See Idea: generate .fwdata file from lexbox? languageforge-lexbox#631 (comment)

[hahn-kev]

Right now we've actually got 2 different cli tools, could we make them into just one exe and control the split vs merge based on the command?

[rmunn]

Right now we've actually got 2 different cli tools, could we make them into just one exe and control the split vs merge based on the command?

Sure. How about fwdatatool.exe split and fwdatatool.exe combine as the commands? I don't want to use "merge" to avoid confusion with the concept of merging a branch.

[myieye]

Some more options 🙃:

• fwdata.exe build (build is shorter and simpler than combine and I think it communicates just as much)
• fwdata.exe split

I feel like this functionality can basically be seen as the mother of the fwdata format, so it's entitled to such a fundamental name (i.e. just fwdata.exe).

[rmunn]

Interesting: when I rebased on top of develop, it included a commit that removed the AppVeyor build. But the PR checks still include AppVeyor.

[rmunn]

@myieye wrote:

Some more options 🙃:

* `fwdata.exe build` (build is shorter and simpler than combine and I think it communicates just as much)

* `fwdata.exe split`

I feel like this functionality can basically be seen as the mother of the fwdata format, so it's entitled to such a fundamental name (i.e. just fwdata.exe).

Good suggestions; I've implemented both of them in commits c313bfe and 36603a3.

[rmunn]

Note: during testing we ran into an issue where the Chorus timeout on HgRunner was too short, hg update tip was killed halfway through checking out a large project with many media files in LinkedFiles, and then the fwdata tool complained about having no Mercurial repo. We'll add a --timeout option to set a timeout in seconds, and maybe default it to 9999 seconds (functionally equivalent to no timeout at all).

Update: Added --timeout option, decided to default to 10 minutes because 9999 seconds, nearly 3 hours, is a little ridiculous. But it can go as high as 9999 if it needs to.

[hahn-kev]

looks good to me, I'd like @jasonleenaylor to have a chance to comment before we merge this in.

[jasonleenaylor]

src/MkFwData/.editorconfig line 1 at r2 (raw file):

# Copyright (c) 2016-2021 SIL International

Why does it need its own editor config? It mostly duplicates the one at the root, I'd rather keep just one for the whole project unless there is a good reason.

[jasonleenaylor]

src/MkFwData/.gitattributes line 1 at r2 (raw file):

* text=auto whitespace=space-before-tab,tab-in-indent,blank-at-eol,tabwidth=4

Same with the .gitattributes, why add one for the project if there is a perfectly good one at the root

[jasonleenaylor]

Reviewed 2 of 5 files at r1, 1 of 1 files at r2, all commit messages.
Reviewable status: 3 of 5 files reviewed, all discussions resolved (waiting on @rmunn)

[rmunn]

True, it doesn't need its own .editorconfig, except for one thing. The flexbridge repo's root .editorconfig has indent_style = tab for .cs files. I originally wrote this code for a different repo where spaces were used for indentation, and so when we ended up bringing it into the flexbridge repo instead, I decided that rather than push a whitespace-change commit, I'd just use .editorconfig to say "Despite what the root .editorconfig says, any project files in this folder should use spaces for indentation."

I really don't want to change indentation to tab characters. I could give a whole presentation on why tab characters for indentation are a bad idea, but the short version is that they make indendation inflexible. Yes, tab characters in theory allow anyone to set the indent level of their code editor to their preferred style (some people like 2-space indents, some like 4, etc.) but in practice, this promise of flexibility is broken, because this kind of thing invariably happens:

If anyone looks at that file with tab width set to something other than 4, then the function parameters will be misaligned from what they're supposed to be.

In the extreme case, you end up with something like the following code that I found in the FW repo at one point (note that I went through some trouble to anonymize the code in this animation, because I thought I'd be using it in a presentation and I didn't want anyone to feel called out, so I replaced all the words in the code with Lorem Ipsum text of the same character length). Here is an animation of me setting the tab display width between 1 and 8 and how the table, which was clearly meant to align, never aligns properly under any tab width:

I believe what happened is that someone originally wrote that code to align, then later some rename refactorings happened and the lengths of some identifiers changed, and nobody went back and re-aligned that table. But the result perfectly encapsulates why I have always disliked tab characters in indentation.

So I'll trim down that .editorconfig to just the differences from the root (space indents instead of tabs). But I would prefer to keep it in there, so that this file, at least, can have correct indentation.

The .gitattributes file is there for a similar reason: because the root .gitattributes as indent-with-non-tab for all text files, which means "flag a whitespace warning if you find a line indented with a non-tab character". Since this project uses spaces for indentation, I needed to change that to tab-in-indent, meaning "flag a whitespace warning if you find tab characters in the indent".

[rmunn]

Commit 418f119 gets rid of the duplicate lines from .editorconfig, making it much more obvious that it's there to override just two settings (indent style and final newline).

[rmunn]

Reviewable status: 3 of 5 files reviewed, all discussions resolved (waiting on @rmunn)

---

src/MkFwData/.editorconfig line 1 at r2 (raw file):

Previously, jasonleenaylor (Jason Naylor) wrote…

Why does it need its own editor config? It mostly duplicates the one at the root, I'd rather keep just one for the whole project unless there is a good reason.

Replied in #403 (comment)

---

src/MkFwData/.gitattributes line 1 at r2 (raw file):

Previously, jasonleenaylor (Jason Naylor) wrote…

Same with the .gitattributes, why add one for the project if there is a perfectly good one at the root

Replied in #403 (comment) — summary is that this is overriding the indent-with-non-tab warning settings from the root, replacing them with tab-in-indent since MkFwData uses spaces for indentation. So this is here in order to override the one from the project root.

[rmunn]

@jasonleenaylor - Decided to just switch to tab indentation here so there's no need for separate .editorconfig files. Commit cda5364 has that change.

If you're happy with this, I'll go ahead and merge on Monday.

[rmunn]

Having heard nothing, I'm going to assume this is okay. Please contact me if there are other changes you need.