Skip to content

Merging from upstream Vim

Jump to bottom
Yee Cheng Chin edited this page Sep 10, 2024 · 27 revisions

Background

MacVim is a fork of Vim in order to provide a GUI, and as such it routinely needs to merge from upstream. The process is not completely trivial as MacVim is an intrusive fork, in that it doesn't simply wraps a Vim executable in a GUI with minimal change, but instead has source code changes sprinkled all over Vim. There are a few reasons for that, but in particular:

  • Vim isn't designed to be embeddable (which NeoVim helps resolve). In particular, the GUI code is quite intermingled with the core source code. Vim doesn't have a native GUI frontend for macOS, as such there needs to be a fair amount of code added to support it.
  • MacVim makes modifications to the documentation and runtime files to provide a more seamless experience rather than simply slapping a raw Vim binary inside terminal emulator.
  • Certain features like trackpad gesture support or Dark Mode support in scripting relies modifying core source code.

Note: Merging from upstream is usually done by the code maintainers. This is necessary because each merge is usually an involved process and requires going through a lot of files/merge conflicts, and isn't very conductive to a pull-request model that are hard to properly review. Please do not file a pull request to merge from upstream. If you notice that MacVim hasn't merged from upstream in a while (2-3 weeks), file an issue.

Steps

To merge from upstream, add both the Vim (use the remote name vim to be consistent with version history) and MacVim remote to your Git repository, then run git merge vim/master. Resolve conflicts and commit/push.

Caveats

Some merges will result in merge conflicts and need some cleanups. Some caveats to pay attention to:

  • While annoying, the committer should review at least the descriptions of every single change in Vim to watch out for potential gotchas. Sometimes there are important changes that introduce subtle bugs while not generating merge conflicts, and human eyes is still the best way to catch such issues. It also helps you notice any interesting changes that you may want to add to release notes, or features that MacVim should integrate. A good command to run is to git log --reverse origin/master..vim/master | vim - to get a list of commits to review.
  • A lot of MacVim-specific code are wrapped under FEAT_GUI_MACVIM, and merge conflicts tend to be changes around those places as the contexts around the code blocks have changed. Watch out for refactored code as Git doesn't do a good job tracking those so if you see mass deletions/additions, make sure the MacVim-specific changes make it across as well.
  • Files that need special handling:
    • src/ex_cmdidxs.h: This file is auto-generated from all the registered Vim Ex commands. Since MacVim has unique commands (e.g. :macmenu) this file is likely to generate a conflict every time Vim adds a command. Use cd src; make cmdidxs to regenerate this file.
    • README.md in Vim's repo is moved to README_vim.md (which is linked to from MacVim's README.md). Because Git doesn't support native rename tracking, it tends to get confused by any change in Vim's README file. Usually you just want to use the MacVim's version (git checkout --ours -- README.md), and copy the file to README_vim.md instead (git show vim/master:README.md >! README_vim.md).
    • .github/workflows are quite different from upstream. Usually, we don't merge from upstream and you can directly pick MacVim's version, but it's useful to see if there are interesting changes that could be applied to MacVim as well.
    • Makefile: At the end of this file are a list of auto-generated file dependencies. This list is different between Vim and MacVim. If you see merge conflicts here, just delete all of them and run cd src; make depend which will recreate the dependencies correctly.
    • src/configure.ac / src/auto/configure: If these were changed, src/auto/configure needs to be re-generated using autoconf. The version of autoconf should match the upstream version (as of this writing it's 2.71). Easiest way to do it is to install the correct version using Homebrew (brew install autoconf), then do cd src; make AUTOCONF=/opt/homebrew/Cellar/autoconf/2.71/bin/autoconf autoconf.
  • Because Vim's repo doesn't always update runtime files, it's possible that help tags (runtime/doc/tags) from the upstream repo is a little outdated. It's nice to use the helptags command to generate help tags when merging from upstream.

It's also sometimes easier to merge small amounts of Vim commits one at a time (e.g. 100-200 commits) instead of all at once if MacVim is a little behind.

⚠️ Be aware of accidentally closed issues after a merge

GitHub allows commits to automatically close issues via the "close #XXXX" markers in commit messages. Unfortunately, this feature is not configurable at all as of this writing and cannot be turned off. If upstream Vim used this to close an issue, it would be referring to an issue in Vim, but the moment we merge the commit and push to GitHub will try to close a random unrelated issue of the same number in MacVim as well.

There is no easy way to work around this issue, because we don't want to rewrite Vim's history to remove the issue markers (as that will change the commit hash). After you have pushed a merge, you should visit https://github.com/macvim-dev/macvim/issues?q=sort%3Aupdated-desc+ to see if there are any recently updated issues, and if some issues were accidentally closed, just go and manually re-open.

Usually, Vim issue numbers are much larger than MacVim issue numbers and we will be fine. You could manually check what issues are referenced when doing an upstream merge by doing the following:

git log master..vim/master | grep -o "#\d\+" | sort -Vr

Then compare that with the largest issue number in MacVim:

curl -sL "https://api.github.com/repos/macvim-dev/macvim/issues?per_page=1" | jq ".[0].number"

If the referenced issues are all larger than MacVim's largest issue number, then we will be fine.

See https://github.com/orgs/community/discussions/17308

Interesting changes to include for release notes

Interesting changes in Vim should be included in MacVim release notes for users who don't pay close attention to Vim development to be able to tell what changed. Generally, features and important bug fixes should be included. It's also useful to mention most newly included options, commands, functions, and bundled plugins.

The list of new commands and functions can be obtained by diff'ing doc/index.txt (under ex-cmd-index) and doc/usr_41.txt (under function-list). New bundled plugins can be found by looking at runtime/pack/dist/opt.

Future

Long term, MacVim should try to minimize the change from Vim, to make each merge easier and less error-prone. As of this writing (2020-01-14) there are ~2000 lines of diff for all of Vim's C code (src/*.c). Possible ways to improve the situation:

  • Some of the changes are because Vim's macOS support is a little lacking and MacVim added macOS-specific features. Integrate some of MacVim's changes back upstream to avoid having to maintain them locally on MacVim's side.
  • Refactor all the FEAT_GUI_MACVIM code blocks to be more merge friendly by moving them to individual lines.
  • If possible, avoid changes in core Vim and prefer doing them in src/MacVim.

Suggestions are welcome for improvements to make the code easier to merge and maintain.

Clone this wiki locally