E2E tests #640

whilefoo · 2023-08-20T09:02:15Z

Resolves #633

Example of E2E test: ubiquibot-whilefoo-testing/ubiquibot#24

netlify · 2023-08-20T09:02:27Z

✅ Deploy Preview for ubiquibot-staging ready!

Name	Link
🔨 Latest commit	`31be5c6`
🔍 Latest deploy log	https://app.netlify.com/sites/ubiquibot-staging/deploys/6512cf4562ba070008afd03e
😎 Deploy Preview	https://deploy-preview-640--ubiquibot-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

gitcoindev · 2023-08-24T06:41:50Z

This is cool. Not mocked but connecting to an actual staging github application using a personal access token and verifying the results. At some point would be useful to add coverage output.

tests/commands.test.ts

0xcodercrane

I can see @whilefoo you did a lot of works to setup unit tests + e2e.

but I feel like its a little bit away from the original goal.

In terms of unit tests, we should have coverage % in branch and function.
In terms of e2e, need to create a scenario and verify each step -- those need to be in workflow.

src/handlers/payout/post.ts

0xcodercrane · 2023-09-06T13:43:49Z

I am gonna contribute to this task as the codeowner because its an very important task for now and future.
No split in payment even if I contribute. The whole amount would get paid to @whilefoo .

To contribute the task(To be honest, I wanna say this is far away from the completion),

merge @whilefoo 's branch to feat/e2e
I will push my work on feat/e2e. @whilefoo you can continue working on feat/e2e branch cause you're already a member. right?

What do you think @pavlovcik @rndquu @whilefoo ? If you agree about my contribution plan, please react with 👍 .

rndquu

Just to clear the terminology, by "e2e testing" we mean running tests against a staging organization in a real github environment.

The original issue states the test script should file a new issue on a locally configurable repository (.env) that tests all the bot commands. Covering all cases is a huge task so this PR tries to cover at least some cases.

I actually have hard time running the tests getting thrown: "Exceeded timeout of 10000 ms for a hook error all the time:

  commmands test
    ✕ /wallet correct address (5 ms)
    ✕ /wallet wrong address
    ✕ /multiplier
    ✕ /query
    ✕ /query wrong username
    ✕ /help
    ✕ /allow
    ✕ /start and /stop

  ● commmands test › /wallet correct address

    thrown: "Exceeded timeout of 10000 ms for a hook.
    Add a timeout value to this test to increase the timeout, if this is a long-running test. See https://jestjs.io/docs/api#testname-fn-timeout."

      150 |   let issue: Issue;
      151 |
    > 152 |   beforeAll(async () => {
          |   ^
      153 |     const res = await octokitAdmin.rest.issues.create({
      154 |       repo,
      155 |       owner,

      at beforeAll (../tests/commands.test.ts:152:3)
      at Object.<anonymous> (../tests/commands.test.ts:149:9)

Nevertheless I think that the current PR is good to merge and we should run e2e tests introduced in the current PR in a github CI ASAP because every new feature makes the e2e tests implemented in the current PR a bit obsolete.

I would:

Merge the current PR
Add a new CI workflow that runs e2e tests in a staging org on every push to a newly created PR by a bounty hunter (+fix the timeout errors)
Create a new issue to refactor e2e tests step by step to cover all basic cases (i.e. commands)
Start implementing unit tests (which is a totally different epic)

P.S. E2E tests on a staging org are gonna be really brittle but it is better than nothing

0xcodercrane · 2023-09-06T14:25:38Z

If the task is a minimal goal of overall E2E, we can merge the PR. but we should add yarn test to workflow file on the current PR @whilefoo.
So we can quickly review the hunters don't break the commands in review step.

From @rndquu 's comment,
2 should be done before 1.
For 3/4, we already have an epic so we can create subtasks from it.

rndquu · 2023-09-06T14:30:04Z

If the task is a minimal goal of overall E2E, we can merge the PR. but we should add yarn test to workflow file on the current PR @whilefoo. So we can quickly review the hunters don't break the commands in review step.

From @rndquu 's comment, 2 should be done before 1. For 3/4, we already have an epic so we can create subtasks from it.

So we keep the current e2e tests and add a new "test workflow" as a part of the current issue, right?

0xcodercrane · 2023-09-06T14:47:07Z

So we keep the current e2e tests and add a new "test workflow" as a part of the current issue, right?

Right. In that way, we can make the PR valuable after the merge.

whilefoo · 2023-09-06T15:42:26Z

I got rate limited one time during development, so I'm worried if we enable this for every commit, we will get rate limited quickly.

0x4007 · 2023-09-06T20:29:29Z

I got rate limited one time during development, so I'm worried if we enable this for every commit, we will get rate limited quickly.

Different GitHub CI events to consider, including only running on merge to development if rate limiting is a concern.

rndquu · 2023-09-06T21:53:17Z

I got rate limited one time during development, so I'm worried if we enable this for every commit, we will get rate limited quickly.

Yes, perhaps we could run e2e tests only on push to the development branch

0xcodercrane · 2023-09-08T01:43:27Z

I got rate limited one time during development, so I'm worried if we enable this for every commit, we will get rate limited quickly.

Yes, perhaps we could run e2e tests only on push to the development branch

The problem I am seeing if we enable e2e on push event in the development branch is that we (as a reviewer) couldn't verify the hunters don't break before merging the PR right?

Even if we have rate limit, we should allow e2e tests on every pull requests to development branch.

rndquu · 2023-09-12T07:39:16Z

The problem I am seeing if we enable e2e on push event in the development branch is that we (as a reviewer) couldn't verify the hunters don't break before merging the PR right?

Yes. Perhaps we could add the workflow_dispatch handler for the e2e workflow and make it a part of the QA process so that QA engineer could run e2e tests manually before approving.

0x4007 · 2023-09-12T07:40:02Z

The problem I am seeing if we enable e2e on push event in the development branch is that we (as a reviewer) couldn't verify the hunters don't break before merging the PR right?

Yes. Perhaps we could add the workflow_dispatch handler for the e2e workflow and make it a part of the QA process so that QA engineer could run e2e tests manually before approving.

Great idea! Any scope creep (this) can be spun off into a separate task though, so that we can merge asap?

package.json

0x4007 · 2023-09-12T07:44:13Z

The problem I am seeing if we enable e2e on push event in the development branch is that we (as a reviewer) couldn't verify the hunters don't break before merging the PR right?

Yes. Perhaps we could add the workflow_dispatch handler for the e2e workflow and make it a part of the QA process so that QA engineer could run e2e tests manually before approving.

Great idea!

I actually have hard time running the tests getting thrown: "Exceeded timeout of 10000 ms for a hook error all the time:

We have six hours to run tests. I don't see how timeout is an issue. Just increase the timeout time?

whilefoo · 2023-09-12T07:58:44Z

I will fix the tests because there were some changes in between, make timeout 6 hours and add workflow

whilefoo · 2023-09-12T08:54:39Z

Yes. Perhaps we could add the workflow_dispatch handler for the e2e workflow and make it a part of the QA process so that QA engineer could run e2e tests manually before approving.

It seems you can only manually run workflows on a branch and not PR

rndquu · 2023-09-13T16:35:22Z

The problem I am seeing if we enable e2e on push event in the development branch is that we (as a reviewer) couldn't verify the hunters don't break before merging the PR right?

Yes. Perhaps we could add the workflow_dispatch handler for the e2e workflow and make it a part of the QA process so that QA engineer could run e2e tests manually before approving.

Great idea! Any scope creep (this) can be spun off into a separate task though, so that we can merge asap?

As far as I understand the plan for the current PR is to fix current e2e tests and add a workflow that runs them

rndquu · 2023-09-13T16:41:03Z

Yes. Perhaps we could add the workflow_dispatch handler for the e2e workflow and make it a part of the QA process so that QA engineer could run e2e tests manually before approving.

It seems you can only manually run workflows on a branch and not PR

Let's keep it simple, I think it's enough (for start) to have an e2e workflow with the workflow_dispatch handler. At least we would be able to fetch bounty hunter's PRs and run e2e tests against our own test orgs. We can modify the e2e workflow later to run on forked repos.

0x4007 · 2023-09-22T11:15:10Z

Hey @whilefoo its been over a month. Why don't we leave it with a simple implementation so that we can merge asap.

whilefoo · 2023-09-24T14:54:03Z

Successful workflow run: https://github.com/ubiquity-whilefoo/ubiquibot/actions/runs/6290628334

@0xcodercrane if you don't have any comments, let's merge this

0x4007 · 2023-09-24T14:58:38Z

Next we should definitely test every incentive cause those are the most important features.

0x4007 · 2023-09-25T18:51:43Z

Successful workflow run: https://github.com/ubiquity-whilefoo/ubiquibot/actions/runs/6290628334

@0xcodercrane if you don't have any comments, let's merge this

Why do some take so long to run? Like start and stop taking 98 seconds?

0x4007 · 2023-09-25T18:54:10Z

.github/workflows/e2e-test.yml

+          LOG_ENVIRONMENT: "production"
+          SUPABASE_URL: ${{ secrets.SUPABASE_URL }}
+          SUPABASE_KEY: ${{ secrets.SUPABASE_KEY }}
+          X25519_PRIVATE_KEY: "QCDb30UHUkwJAGhLWC-R2N0PiEbd4vQY6qH2Wloybyo"


Should this be exposed? Not sure exactly what it is used for.

it's okay, it must be the same because the test's config wallet private key in encrypted with this private key

whilefoo · 2023-09-26T08:35:01Z

Why do some take so long to run? Like start and stop taking 98 seconds?

It's a long test. Also there is some delay between because Github sometimes doesn't include latest comments right away so sometimes even with the delay it fails

feat: e2e test setup

92b5b86

whilefoo added 3 commits August 22, 2023 13:47

feat: add more tests

fe36b25

fix: ignore kebab-case in tests

78bb917

feat: added tests

b3081cf

0x4007 reviewed Aug 24, 2023

View reviewed changes

tests/commands.test.ts Outdated Show resolved Hide resolved

0x4007 mentioned this pull request Aug 24, 2023

Tests: Code Coverage Output #667

Open

0x4007 added ping and removed ping labels Aug 29, 2023

whilefoo added 2 commits September 3, 2023 19:35

Merge remote-tracking branch 'upstream/development' into e2e-tests

db1850f

feat: more tests

8fb1bce

whilefoo marked this pull request as ready for review September 4, 2023 20:13

whilefoo requested a review from 0xcodercrane as a code owner September 4, 2023 20:13

rndquu self-requested a review September 6, 2023 13:14

0xcodercrane suggested changes Sep 6, 2023

View reviewed changes

rndquu reviewed Sep 6, 2023

View reviewed changes

src/handlers/payout/post.ts Outdated Show resolved Hide resolved

rndquu previously approved these changes Sep 6, 2023

View reviewed changes

feat: rate limit

2639a8f

0x4007 reviewed Sep 12, 2023

View reviewed changes

package.json Show resolved Hide resolved

whilefoo added 4 commits September 12, 2023 11:14

feat: longer timeout

fc7cdf0

Merge remote-tracking branch 'upstream/development' into e2e-tests

064cdc0

Merge remote-tracking branch 'upstream/development' into e2e-tests

794f793

feat: update tests

51bc95f

whilefoo dismissed rndquu’s stale review via 51bc95f September 12, 2023 10:34

whilefoo added 3 commits September 15, 2023 10:32

Merge remote-tracking branch 'upstream/development' into e2e-tests

5e14775

fix: types

29e2cb4

feat: workflow

cc89117

whilefoo added 2 commits September 24, 2023 15:47

Merge remote-tracking branch 'upstream/development' into e2e-tests

308b6c2

feat: longer delays

8ba4743

0x4007 mentioned this pull request Sep 25, 2023

Tests: Share Strings #837

Open

0x4007 reviewed Sep 25, 2023

View reviewed changes

0x4007 enabled auto-merge September 25, 2023 18:55

whilefoo added 2 commits September 26, 2023 14:29

feat: update workflow

56e4427

Merge remote-tracking branch 'upstream/development' into e2e-tests

31be5c6

rndquu mentioned this pull request Sep 29, 2023

Unit Testing #176

Closed

Keyrxng mentioned this pull request Oct 17, 2023

Strings #862

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

E2E tests #640

E2E tests #640

whilefoo commented Aug 20, 2023 •

edited

Loading

netlify bot commented Aug 20, 2023 •

edited

Loading

gitcoindev commented Aug 24, 2023

0xcodercrane left a comment

0xcodercrane commented Sep 6, 2023 •

edited

Loading

rndquu left a comment

0xcodercrane commented Sep 6, 2023 •

edited

Loading

rndquu commented Sep 6, 2023

0xcodercrane commented Sep 6, 2023

whilefoo commented Sep 6, 2023

0x4007 commented Sep 6, 2023

rndquu commented Sep 6, 2023

0xcodercrane commented Sep 8, 2023

rndquu commented Sep 12, 2023

0x4007 commented Sep 12, 2023 •

edited

Loading

0x4007 commented Sep 12, 2023 •

edited

Loading

whilefoo commented Sep 12, 2023

whilefoo commented Sep 12, 2023

rndquu commented Sep 13, 2023 •

edited

Loading

rndquu commented Sep 13, 2023

0x4007 commented Sep 22, 2023

whilefoo commented Sep 24, 2023

0x4007 commented Sep 24, 2023

0x4007 commented Sep 25, 2023

0x4007 Sep 25, 2023

whilefoo Sep 26, 2023

whilefoo commented Sep 26, 2023

E2E tests #640

Are you sure you want to change the base?

E2E tests #640

Conversation

whilefoo commented Aug 20, 2023 • edited Loading

netlify bot commented Aug 20, 2023 • edited Loading

✅ Deploy Preview for ubiquibot-staging ready!

gitcoindev commented Aug 24, 2023

0xcodercrane left a comment

Choose a reason for hiding this comment

0xcodercrane commented Sep 6, 2023 • edited Loading

rndquu left a comment

Choose a reason for hiding this comment

0xcodercrane commented Sep 6, 2023 • edited Loading

rndquu commented Sep 6, 2023

0xcodercrane commented Sep 6, 2023

whilefoo commented Sep 6, 2023

0x4007 commented Sep 6, 2023

rndquu commented Sep 6, 2023

0xcodercrane commented Sep 8, 2023

rndquu commented Sep 12, 2023

0x4007 commented Sep 12, 2023 • edited Loading

0x4007 commented Sep 12, 2023 • edited Loading

whilefoo commented Sep 12, 2023

whilefoo commented Sep 12, 2023

rndquu commented Sep 13, 2023 • edited Loading

rndquu commented Sep 13, 2023

0x4007 commented Sep 22, 2023

whilefoo commented Sep 24, 2023

0x4007 commented Sep 24, 2023

0x4007 commented Sep 25, 2023

0x4007 Sep 25, 2023

Choose a reason for hiding this comment

whilefoo Sep 26, 2023

Choose a reason for hiding this comment

whilefoo commented Sep 26, 2023

whilefoo commented Aug 20, 2023 •

edited

Loading

netlify bot commented Aug 20, 2023 •

edited

Loading

0xcodercrane commented Sep 6, 2023 •

edited

Loading

0xcodercrane commented Sep 6, 2023 •

edited

Loading

0x4007 commented Sep 12, 2023 •

edited

Loading

0x4007 commented Sep 12, 2023 •

edited

Loading

rndquu commented Sep 13, 2023 •

edited

Loading