Extract super scaffolding tests into smaller individual units (#1825)

* first stab at extracting a stand alone test * a little polish * better naming * getting started on converting these scripts to ruby * getting closer * maybe we'll need to run migrations in the setup * working on teardown * working on teardown * disable that completely * linter * remove the bash scripts * extract another test * extract another * some cleanup * fix * moar * inching forward * extract another * extract another * comment out more stuff * run migrations automatically * extract another * extract another * extract another * nothing to test there * add a comment * linter * try running these in CI * oops * the workspace is dirty in ci * extract another * fix * linter * clean up * Remove old stuff * some clean up and docs * beef up the readme * try consolidating * Revert "try consolidating" This reverts commit 6c2b436. * clean up * clean up * tweak docs * clean up * don't want to run db:schema:load because that will wipe the current db * Use the source attribute option * this doesn't seem to do anything useful * use source in another spot * another source useage
bullet-train-co · Dec 20, 2024 · e6b80f9 · e6b80f9
1 parent 9cdb131
commit e6b80f9
Show file tree

Hide file tree

Showing 34 changed files with 1,074 additions and 531 deletions.
diff --git a/.github/workflows/_run_super_scaffolding_tests.yml b/.github/workflows/_run_super_scaffolding_tests.yml
@@ -17,24 +17,38 @@ on:
         default: false
 
 jobs:
-  # NOTE: This is here just to make the workflow visualization layout better.
-  # Without it the layout is really bad and confusing.
   calculate_matrix:
-    name: 🤷 Fake Matrix
+    name: 🏗️ Generate Test List
     runs-on: ubuntu-latest
+    outputs:
+      tests: ${{ steps.generate-test-list.outputs.result }}
     steps:
-      - name: Do Nothing
-        run: echo "This is a useless step that just helps things look nicer..."
-        shell: bash
+      - name: Checkout Repo
+        uses: "actions/checkout@v4"
+
+      - uses: actions/github-script@v7
+        id: generate-test-list
+        name: Generate Test List
+        with:
+          script: |
+            const globber = await glob.create('test/system/super_scaffolding/**/setup.rb')
+            const tests = []
+            for await (const file of globber.globGenerator()) {
+              console.log(file)
+              var fileParts = file.split("/")
+              var test_dir_name = fileParts[fileParts.length - 2];
+              console.log("test_dir_name =", test_dir_name)
+              tests.push(test_dir_name)
+            }
+            return tests
   test:
     name: "🏗️"
     runs-on: ubuntu-latest
     needs: calculate_matrix
     strategy:
       fail-fast: false
       matrix:
-        # For super scaffolding tests we need to have exactly 5 runners.
-        ci_runners: ["TestSite, Project", "Project::Step, Insight", "Personality::Disposition, Personality::Observation", "TestFile, PartialTest", Webhook]
+        test: ${{ fromJson(needs.calculate_matrix.outputs.tests) }}
     services:
       postgres:
         image: postgres:11-alpine
@@ -125,28 +139,17 @@ jobs:
       - run: yarn build:css
         working-directory: tmp/starter
 
-      # TODO: This might be a bad idea. Maybe we should just have spring in the Gemfile all the time.
-      - name: Allow adding of spring
-        run: bundle config unset deployment
-        working-directory: tmp/starter
-
-      - name: Add spring
-        run: bundle add spring
-        working-directory: tmp/starter
-
       - name: 'Setup Super Scaffolding System Test'
-        run: bundle exec test/bin/setup-super-scaffolding-system-test
+        run: bundle exec test/system/super_scaffolding/${{ matrix.test }}/setup.rb --allow-dirty-workspace
         working-directory: tmp/starter
-        env:
-          CIRCLE_NODE_INDEX: ${{ strategy.job-index }}
 
       - name: 'Run Super Scaffolding Test'
-        run: bin/rails test test/system/super_scaffolding/
+        run: bin/rails test test/system/super_scaffolding/${{ matrix.test }}/ test/system/super_scaffolding/open_api_test.rb
         working-directory: tmp/starter
 
       - name: 'Run Super Scaffolding Webhook Test'
         run: bin/rails test test/controllers/webhooks/incoming/some_provider_webhooks_controller_test.rb
-        if: ${{ strategy.job-index == 5 }}
+        if: ${{ matrix.test == 'webhook' }}
         working-directory: tmp/starter
 
       - name: Upload Test Summary Logs

diff --git a/test/bin/setup-super-scaffolding-system-test b/test/bin/setup-super-scaffolding-system-test
diff --git a/test/system/super_scaffolding/README.md b/test/system/super_scaffolding/README.md
@@ -0,0 +1,99 @@
+# About Super Scaffolding Tests
+
+These tests each require some super scaffolding commands to be run _before_ running the tests.
+
+Each subdirectory below contains a `setup.rb` file that you can run to generate the necessary scaffolding.
+
+After running the tests you can run `teardown.rb` from the same directory to remove the super scaffolded code.
+
+**NOTE** We highly reocmmend that you start with a clean git workspace. The super scaffolding commands generate
+a lot of files and the cleanup process is... let's call it "brute force". If you have uncommitted changes in
+your repo before running `setup.rb` they're likely to get clobbered when you run `teardown.rb`
+
+### Example
+
+To run the `insight` test, you'd run these commands from the main project directory.
+
+First run the `setup.rb` script to generate the scaffolding and run migrations.
+
+```
+$ ./test/system/super_scaffolding/insight/setup.rb
+Generating Insight model with 'bin/rails generate model Insight team:references name:string description:text'
+
+Writing './app/controllers/account/insights_controller.rb'.
+Fixing Standard Ruby on './app/controllers/account/insights_controller.rb'.
+Writing './app/views/account/insights/index.html.erb'.
+Writing './app/views/account/insights/_menu_item.html.erb'.
+# snip
+```
+
+Then run the test:
+
+```
+$ rails test test/system/super_scaffolding/insight/
+🌱 Generating global seeds.
+🌱 Generating test environment seeds.
+Not requiring Knapsack Pro.
+If you'd like to use Knapsack Pro make sure that you've set the environment variable KNAPSACK_PRO_CI_NODE_INDEX
+Started with run options --seed 45445
+
+BulletTrain::SuperScaffolding::InsightTest
+Puma starting in single mode...
+* Puma version: 6.5.0 ("Sky's Version")
+* Ruby version: ruby 3.3.6 (2024-11-05 revision 75015d4c1f) [arm64-darwin23]
+*  Min threads: 5
+*  Max threads: 5
+*  Environment: test
+*          PID: 55207
+* Listening on http://127.0.0.1:3001
+Use Ctrl-C to stop
+  test_developers_can_generate_a_Insight_and_a_nested_Personality::CharacterTrait_model PASS (3.51s)
+
+Finished in 3.51351s
+1 tests, 4 assertions, 0 failures, 0 errors, 0 skips
+Coverage report generated for test to /Users/jgreen/projects/bullet-train-co/bullet_train/coverage.
+Line Coverage: 45.18% (150 / 332)
+```
+Then run `teardown.rb` to clean up:
+
+```
+$ ./test/system/super_scaffolding/insight/teardown.rb
+db/schema.rb has changed - we need to rollback
+== 20241219204101 CreatePersonalityCharacterTraits: reverting =================
+-- drop_table(:personality_character_traits)
+   -> 0.0037s
+== 20241219204101 CreatePersonalityCharacterTraits: reverted (0.0073s) ========
+
+== 20241219204056 CreateInsights: reverting ===================================
+-- drop_table(:insights)
+   -> 0.0016s
+== 20241219204056 CreateInsights: reverted (0.0017s) ==========================
+
+Updated 8 paths from the index
+
+Removing app/avo/resources/insight.rb
+Removing app/avo/resources/personality_character_trait.rb
+Removing app/controllers/account/insights_controller.rb
+Removing app/controllers/account/personality/
+Removing app/controllers/api/v1/insights_controller.rb
+Removing app/controllers/api/v1/personality/
+Removing app/controllers/avo/insights_controller.rb
+Removing app/controllers/avo/personality_character_traits_controller.rb
+Removing app/models/insight.rb
+Removing app/models/personality.rb
+Removing app/models/personality/
+Removing app/views/account/insights/
+Removing app/views/account/personality/
+Removing app/views/api/v1/insights/
+Removing app/views/api/v1/personality/
+Removing config/locales/en/insights.en.yml
+Removing config/locales/en/personality/
+Removing db/migrate/20241219204056_create_insights.rb
+Removing db/migrate/20241219204101_create_personality_character_traits.rb
+Removing test/controllers/api/v1/insights_controller_test.rb
+Removing test/controllers/api/v1/personality/
+Removing test/factories/insights.rb
+Removing test/factories/personality/
+Removing test/models/insight_test.rb
+Removing test/models/personality/
+```
diff --git a/test/system/super_scaffolding/insight/insight_test.rb b/test/system/super_scaffolding/insight/insight_test.rb
@@ -0,0 +1,42 @@
+require "application_system_test_case"
+
+class BulletTrain::SuperScaffolding::InsightTest < ApplicationSystemTestCase
+  setup do
+    @jane = create :onboarded_user, first_name: "Jane", last_name: "Smith"
+  end
+
+  # NOTE: This test only runs if the `setup.rb` file in this directory has run.
+  # See ../README.md for details.
+
+  # force autoload.
+  [
+    "Insight",
+    "Personality::CharacterTrait"
+  ].each do |class_name|
+    class_name.constantize
+  rescue
+    nil
+  end
+
+  if defined?(Insight)
+    test "developers can generate a Insight and a nested Personality::CharacterTrait model" do
+      login_as(@jane, scope: :user)
+      visit account_team_path(@jane.current_team)
+
+      click_on "Add New Insight"
+      click_on "Create Insight"
+      assert_text("Name can't be blank.")
+      fill_in "Name", with: "Some New Example Insight"
+      click_on "Create Insight"
+
+      assert_text("Insight was successfully created.")
+
+      click_on "Add New Character Trait"
+      click_on "Create Character Trait"
+      assert_text("Name can't be blank.")
+      fill_in "Name", with: "Some New Example Character Trait"
+      click_on "Create Character Trait"
+      assert_text("Character Trait was successfully created.")
+    end
+  end
+end
diff --git a/test/system/super_scaffolding/insight/setup.rb b/test/system/super_scaffolding/insight/setup.rb
@@ -0,0 +1,18 @@
+#!/usr/bin/env ruby
+
+require_relative "../super_scaffolding_test_setup"
+
+class Setup < SuperScaffoldingTestSetup
+  # This allows us to define helper methods that aren't attached to thor commands
+  no_commands do
+    def setup
+      puts `rails g super_scaffold Insight Team name:text_field description:trix_editor --navbar="ti-world"`
+      puts `rails g super_scaffold Personality::CharacterTrait Insight,Team name:text_field description:trix_editor`
+    end
+  end
+end
+
+# We create our own args array so that we don't have to ask the user to include `scaffolding_setup` on the command line
+args = ["scaffolding_setup"] + ARGV
+
+Setup.start(args)
diff --git a/test/system/super_scaffolding/insight/teardown.rb b/test/system/super_scaffolding/insight/teardown.rb
@@ -0,0 +1,17 @@
+#!/usr/bin/env ruby
+
+require_relative "../super_scaffolding_test_teardown"
+
+class Teardown < SuperScaffoldingTestTeardown
+  # This allows us to define helper methods that aren't attached to thor commands
+  no_commands do
+    def teardown
+      # custom teardown goes here
+    end
+  end
+end
+
+# We create our own args array so that we don't have to ask the user to include `scaffolding_setup` on the command line
+args = ["scaffolding_teardown"] + ARGV
+
+Teardown.start(args)
diff --git a/test/system/super_scaffolding/open_api_test.rb b/test/system/super_scaffolding/open_api_test.rb
@@ -0,0 +1,15 @@
+require "application_system_test_case"
+
+# We run this in CI on each super scaffold test node to ensure that the OpenAPI Document
+# is still valid after generating super scaffolds.
+class SuperScaffoldingOpenAPITest < ApplicationSystemTestCase
+  test "OpenAPI V3 document is still valid" do
+    visit "/" # Make sure the test server is running before linting the file.
+
+    puts(output = `yarn exec redocly lint http://127.0.0.1:#{Capybara.server_port}/api/v1/openapi.yaml 1> /dev/stdout 2> /dev/stdout`)
+    # redocly/openapi-core changed the format of their success message in version 1.2.0.
+    # https://github.com/Redocly/redocly-cli/pull/1239
+    # We use a robust regex here so that we can match both formats.
+    assert output.match?(/Woohoo! Your (Open)?API (definition|description) is valid./)
+  end
+end