Git Repo needs copies of MPS project and Ravenbrook documents

[rptb1]

The MPS is migrating from being "mastered" inside Ravenbrook to living in git (and on GitHub). Ravenbrook's infrastructure is based on Perforce SCM, where the current git repository is just a mirror of a "codeline" and its "branches" that form only part of a wider "project" in a company-wide "repository". As a consequence, the stuff in the git repo refers to important documents and resources that aren't part of it, and sometimes not visible.

(Quite a lot of the MPS "project" part of the Ravenbrook repository is visible at https://www.ravenbrook.com/project/mps/ )

But the style of git (and GitHub) use tends to keep copies and versions of these resources in the git repo alongside the code, and this makes sense especially now the project is FOSS under the BSD licence.

So, we should branch various things from elsewhere in the Ravenbrook repository in to the MPS git repo. This issue can help to track and discuss how to do it.

A really good example is the code review rules. Currently, the contributing guidelines say:

There are a fairly large number of rules, both low- and high-level that your code must follow in order to be accepted. These rules are the result of continuous process improvement to prevent defects. Unfortunately, we do not have many of them published at present. We apologise if you find it frustrating that we do not accept your changes as they are.

Our rules are a valuable result of defect prevention in the development of the MPS, and we not only want to share them, but encourage collaboration, including the ability of other people to review and apply them so that the MPS continues to deploy with its extremely low defect rate.

I encourage other team members, past and present, to add more items to this issue.

[rptb1]

A technical note about how to bring new content into the repo. Skip this if you're not involved in actually doing that bit.

We should not simply copy files. Those files have history, and that history is valuable data when trying to figure out, for example, what rules were applied when. Or simply who said what (e.g. "git blame").

The MPS git repo is exported from Perforce using Perforce Git Fusion. Unfortunately, we can't adjust Git Fusion's view to include the history of new files, because that would cause Git Fusion to essentially "rebase" all of MPS history. That's not really a flaw in Git Fusion, since it was never really designed to maintain a permanent two-way connection between Perforce and git.

Fortunately, there is a way. We can ask git to merge the histories of separate repos exported from Perforce.

Firstly, we create a Perforce client view on the things we want to incorporate. For example, to bring in the Ravenbrook rules we could create a Perforce client called "mps-rules-merge" with a view like this:

  //info.ravenbrook.com/rule/... //mps-rules-merge/rule/...

note that the directory tree on the right-hand side will be merged into the directory tree of the MPS git repo. Then we can use Git Fusion to create a repo for that and fetch its commits:

  git add remote rules ssh://[email protected]:1622/mps-rules-merge
  git fetch rules

This will create a remote ref that points to the imported tree and its history. Now the magic:

  git merge --allow-unrelated-histories rules/master

This will merge the rules tree into the current tree, and create a merge commit whose parents are both histories, conserving all history.

Important If you attempt this, make sure the result can be pushed back to Perforce before you publish it to GitHub (and I don't mean before you merge, I mean before you even push to GitHub). Otherwise we might desynchronise the two in a public way that's hard to fix up without forcing everyone to rebase. See email message Git Fusion user lookup failure for past user.

References

• Stack Overflow: Merge git submodule into main repository
• Perforce sandbox where I verified all this

[rptb1]

A technical note about how to bring new content into the repo. Skip this if you're not involved in actually doing that bit.

An alternative method using Perforce.

• This method is less likely to cause problems with Git Fusion due to trying to push back changes by users who no longer have Perforce accounts.
• This method must be done while Perforce is in still sync with Git via Git Fusion, unlike the method above which could be done later.

In Perforce, integrate the things we can to incorporate directly into the codeline. For example:

p4 integrate //info.ravenbrook.com/rule/... //info.ravenbrook.com/project/mps/master/rule/...
p4 submit -d "Incorporating Ravenbrook rules into MPS codeline. See <https://github.com/Ravenbrook/mps/issues/95>."

Git Fusion will create commits for the entire history of the integrated stuff, and that will then appear in Git.

For good record keeping, it's probably best to create a Perforce branch spec for this operation, with notes and references. And for good isolation, it's probably best to integrate into a development branch in Perforce, not directly to master as shown. That development branch can be merged after review, in either Git or Perforce.

[rptb1]

This issue indicates a failure to meet kpa.scm.goal.avail and the fact that the public can't follow that link exactly illustrates #95.

[rptb1]

Some more documents found that need to go in (possibly heavily modified):

• Defect resolution procedure

[rptb1]

In Perforce, integrate the things we can to incorporate directly into the codeline.

I used this successfully for some documents in ee4d788 . Recording here as an example of how to do it.

rb@plover:~/git/mps/procedure$ git status
On branch branch/2023-01-19/review-procedure
...
rb@plover:~/git/mps$ git push perforce
rb@plover:~/git/mps$ pushd ~/p4/ravenbrook/mps/
rb@plover:~/p4/ravenbrook/mps$ p4 sync branch/2023-01-19/...
rb@plover:~/p4/ravenbrook/mps$ p4 integrate 'doc/2002-06-18/obsolete-mminfo/mminfo/rule/entry/*/index.txt' 'branch/2023-01-19/review-procedure/procedure/entry.*.rst'
rb@plover:~/p4/ravenbrook/mps$ p4 integrate 'doc/2002-06-18/obsolete-mminfo/mminfo/rule/exit/*/index.txt' 'branch/2023-01-19/review-procedure/procedure/exit.*.rst'
rb@plover:~/p4/ravenbrook/mps$ p4 submit -d "Integrating MM Group review entry and exit criteria into branch/2023-01-19/review-procedure."
Submitting change 198763.
rb@plover:~/p4/ravenbrook/mps$ popd
rb@plover:~/git/mps/procedure$ git pull perforce branch/2023-01-19/review-procedure
Perforce: Copying files: 4
Perforce: 100% (1/1) Copying changelists to git...
Perforce: Adding new Git commit objects to Perforce...Perforce: Submitting new Git commit objects to Perforce
remote: Counting objects: 7, done.        
remote: Compressing objects: 100% (7/7), done.        
remote: Total 7 (delta 1), reused 6 (delta 0)        
Unpacking objects: 100% (7/7), 2.58 KiB | 2.58 MiB/s, done.
From ssh://perforce.ravenbrook.com:1622/mps-public
 * branch                branch/2023-01-19/review-procedure -> FETCH_HEAD
   5ab925a8e..ee4d7886f  branch/2023-01-19/review-procedure -> perforce/branch/2023-01-19/review-procedure
Updating 5ab925a8e..ee4d7886f
Fast-forward
 procedure/entry.design.rst    | 14 ++++++++++++++
 procedure/entry.impl.rst      | 16 ++++++++++++++++
 procedure/entry.universal.rst | 34 ++++++++++++++++++++++++++++++++++
 procedure/exit.universal.rst  | 37 +++++++++++++++++++++++++++++++++++++
 4 files changed, 101 insertions(+)
 create mode 100644 procedure/entry.design.rst
 create mode 100644 procedure/entry.impl.rst
 create mode 100644 procedure/entry.universal.rst
 create mode 100644 procedure/exit.universal.rst

[rptb1]

I used this successfully for some documents

I repeated this procedure for #171 but as you can see from the blame this failed to get the entire history of the plan into Git.