Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* ## Binary and buildpack Context Here's how we provide build ruby binaries. First this repo will: - Download a ruby source tarball - Build it into a binary (`make install` etc.) - Zip/tar that binary up and upload it to S3 (filename is coupled to buildpack logic) Then later the buildpack has to: - Take the output of `bundle platform --ruby` and turn that into an S3 url - Download and unzip that tarball and place it on the path That means that this repo is coupled to: - Ruby source conventions (like filenames) - The bundler output (`bundle platform --ruby`) - Any logic the buildpack uses to convert `bundle platform --ruby` to a download URL Another big piece of context is that this is the first year we're trying fully automated binary build and upload steps. I knew when I automated the regular builds that we would likely have to come back to this logic. In prior years we've been able to manually adjust file names which meant inconsistencies were worked around manually. ## First problem - `ruby-3.3.0-preview2.tgz` I've tried to release Ruby 3.3.0-preview2 twice now. The first attempt changed no logic for the binary builder or the buildpack. It produced a file that was uploaded to S3: ``` $ curl -I https://heroku-buildpack-ruby.s3.us-east-1.amazonaws.com/heroku-22/ruby-3.3.0-preview2.tgz HTTP/1.1 200 OK ``` This doesn't work for us for several reasons. You cannot actually run `bundle install` with Ruby 3.3.0-preview2 locally or you would get an error: ``` $ cat Gemfile | grep ruby ruby '3.3.0-preview2' $ bundle install Your Ruby version is 3.3.0.preview2, but your Gemfile specified 3.3.0.pre.preview2 ``` Note that it says there's a difference in the input string in the Gemfile and what bundler thinks we put in our gemfile (`-preview2` versus `.pre.preview2`). This is due to bundler replacing the dash with `.pre.`. ## Second problem `ruby-3.3.0.tgz` When we started supporting prerelease versions of Ruby we used to use the eventual version for pre-releases. So in that scenario Ruby 3.3.0-preview2 would be uploaded to `ruby-3.3.0.tgz`. Then we asked people to put that version in their Gemfile like `ruby "3.3.0"`. Seeing problem number one, I recalled this era and thought we had some edgecase tooling for it. However, this strategy of using the plain version stopped working with Ruby 3.2 when bundler was checking and erroring on Ruby versions and a pre-release version. We maintained that strategy until Ruby 3.1, see a 3.1 changelog (https://devcenter.heroku.com/changelog-items/2292). In Ruby 3.2 we could not longer ask people to use `3.2.0` to work around limitations in bundler (that it does not recognize the dash). Our solution below looks a lot like it did for our solution in Ruby 3.2 where we asked people to put an extra specifier in the Gemfile like `ruby "3.2.0.preview3"` (https://devcenter.heroku.com/changelog-items/2499). Note that this is a different string than `3.3.0-preview2` (dot, which is what bundler needs versus a dash which is the source file from the ruby ftp site). Without recalling this change I put in the work to automate generation of a `ruby-3.3.0.tgz` file which uploaded fine: ``` $ curl -I https://heroku-buildpack-ruby.s3.us-east-1.amazonaws.com/heroku-22/ruby-3.3.0-preview2.tgz HTTP/1.1 200 OK ``` And even can be downloaded fine if you put `ruby "3.3.0"` in a Gemfile (and do not try installing locally). However you'll get an error when you try to deploy: ``` remote: -----> Using Ruby version: ruby-3.3.0 remote: -----> Installing dependencies using bundler 2.3.25 remote: Running: BUNDLE_WITHOUT='development:test' BUNDLE_PATH=vendor/bundle BUNDLE_BIN=vendor/bundle/bin BUNDLE_DEPLOYMENT=1 bundle install -j4 remote: Your Ruby version is 3.3.0.preview2, but your Gemfile specified ~> 3.3.0 remote: Bundler Output: Your Ruby version is 3.3.0.preview2, but your Gemfile specified ~> 3.3.0 remote: ``` In essence I had to re-learn what I had already learned last year (in the changelog above) that `ruby "3.2.0.preview3"` will work with both bundler and the buildpack. (Note that we're using a dot instead of a dash). You can see that's the name of the binary on S3: ``` $ curl -I https://heroku-buildpack-ruby.s3.us-east-1.amazonaws.com/heroku-22/ruby-3.2.0.preview3.tgz HTTP/1.1 200 OK ``` This year I refactored and re-wrote the binary build system to add tests (previously there were none) and learn how it all works. Last year I only observed inputs and outputs. I think what happened is that I manually changed the filename to match the output of `bundle platform --ruby` when running those versions locally and just matched the file and instructions to that. I didn't put in the time to understand the intricies of the behavior of all the systems involved. This year, I correctly recalled that there was an edgecase, and remembered one of the solutions, but I failed to remember that specific solution didn't work with newer bundler versions. Instead of simply getting things working, I'm investing in understanding and writing down WHY some of this behavior exists and where certain limitations are coming from. ## Here we are today That brings us to this PR. The solution looks a lot like Ruby 3.2. We will ask customers to put `ruby "3.3.0.preview2" in their Gemfile. Which works locally without raising a bundler error: ``` $ bundle exec rake "generate_image[heroku-22]" $ bash rubies/heroku-22/ruby-3.3.0-preview2.sh $ docker run -it -v $(pwd)/builds/heroku-22:/tmp/output hone/ruby-builder:heroku-22 bash root@440d92753881:/# mkdir /tmp/unzipped && tar xzf /tmp/output/ruby-3.3.0.preview2.tgz -C /tmp/unzipped && export PATH="/tmp/unzipped/bin/:$PATH" && echo "ruby '3.3.0.preview2'" > Gemfile && bundle install Don't run Bundler as root. Installing your bundle as root will break this application for all non-root users on this machine. [DEPRECATED] This Gemfile does not include an explicit global source. Not using an explicit global source may result in a different lockfile being generated depending on the gems you have installed locally before bundler is run. Instead, define a global source in your Gemfile like this: source "https://rubygems.org". The Gemfile specifies no dependencies Resolving dependencies... Bundle complete! 0 Gemfile dependencies, 1 gem now installed. Use `bundle info [gemname]` to see where a bundled gem is installed. ``` And importantly it produces the same version output: ``` $ bundle platform --ruby ruby 3.3.0.preview2 ``` While it's not revelatory, it is not automated and tested. I also wrote the system to convert these version strings between the ruby source download and bundler version strings. That means that a Heroku developer using `3.3.0-preview2` or `3.3.0.preview2` will get the same output (correct) result uploaded to S3 and a working changelog to go with it. * Remove unused method
- Loading branch information
