CodeceptJS behaving differently in the CI compared to local environment

[sethvanwykJET]

I opened this as an issue before but saw that there is a discussion section and thought this might be better suited here.

#3639

Contents:

What are you trying to achieve?

Running tests in the pipeline to give the same results as the local environment.

What do you get instead?

So for many tests we've needed to disable them in the CI as the test is unable to find the xpath with the value when trying to see the data on screen.

Provide test source code if related

  I.waitForElement(`//div[@data-qa='delivery-costs-indicator']//div[contains(text(),'Free')]`);
  I.seeElement(`//div[@data-qa='delivery-costs-indicator']//div[contains(text(),'Free')]`);

For this sample above, the wait for element passes correctly, but it's not able to see the same element it just waited for. The allure report mentions it's undefined.

It's also inconsistent. It might pass in one test and then fail in another. Checking the screenshot, the element is clearly present and it always passes in the local environment.

Example from another test

The content it was looking for was present in the screenshot

Details

• CodeceptJS version: 3.4.1 (From 3.2.2)
• NodeJS Version: 16
• Operating System: Uncertain (docker image running in the cloud)
• puppeteer

We recently also updated the docker image running in our CI and since then the problems have started.

# To debug build process locally run the command below in the project root directory.
# `docker build -f ./tools/docker/end2end-tests/Dockerfile -t debug --progress=plain --build-arg WORKDIR=$(pwd) .`

# Download Puppeteer image along with its dependencies
FROM ghcr.io/puppeteer/puppeteer:19.8.3

# Set working directory to current terminal path when initialized with the right build argument.
ARG WORKDIR
WORKDIR $WORKDIR

# Elevated privileges to install packages and run commands
USER root

# Add group first to make sure the ID is assigned consistently
RUN groupadd --system nightmare && useradd --system --create-home --gid nightmare nightmare

# Installing the pre-required packages and libraries
RUN apt-get update
RUN apt-get install -y libgtk2.0-0 libgconf-2-4 \
      libasound2 libxtst6 libxss1 libnss3 xvfb \
      libcairo2-dev libgif-dev libjpeg-dev libpango1.0-dev \
      libpng-dev libgbm-dev curl google-chrome-unstable \
      inetutils-ping default-jdk

RUN export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/

# Install dumb-init to help prevent zombie chrome processes.
ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64 /usr/local/bin/dumb-init
RUN chmod +x /usr/local/bin/dumb-init

# Install Node.js 16
RUN curl -fsSL https://deb.nodesource.com/setup_16.x | bash - \
    && apt-get install -y nodejs

# Install Yarn
RUN curl -sL https://dl.yarnpkg.com/debian/pubkey.gpg | gpg --dearmor | tee /usr/share/keyrings/yarnkey.gpg >/dev/null \
    echo "deb [signed-by=/usr/share/keyrings/yarnkey.gpg] https://dl.yarnpkg.com/debian stable main" | tee /etc/apt/sources.list.d/yarn.list \
    apt-get update && apt-get install yarn -y

# Clean up
RUN  apt-get clean \
  && rm -rf /var/lib/apt/lists/ \
  && rm -rf /var/cache/oracle-jdk8-installer

# Set yarn version needed for repo
RUN yarn set version 3.2.2

# Copy Repo into the image
COPY . /codecept

# Give run permissions to the entrypoint
RUN chmod -R +x /codecept/tools/docker/end2end-tests

# Install dependencies
RUN yarn --cwd /codecept

RUN ln -s /codecept/node_modules/.bin/codeceptjs /usr/local/bin/codeceptjs
RUN ln -s /codecept/node_modules/.bin/allure /usr/local/bin/allure
RUN ln -s /codecept/node_modules/.bin/cross-env /usr/local/bin/cross-env
RUN ln -s /codecept/node_modules/.bin/electron /usr/local/bin/electron

# Create a directory for the test runner
RUN mkdir /tests
WORKDIR /tests

# Allows us to pass arguments to CodeceptJS run via environment variables
ENV CODECEPT_ARGS=""
ENV RUN_MULTIPLE=false
ENV NO_OF_WORKERS=""

# Set HOST ENV variable for Selenium Server
ENV HOST=selenium

# Set the entrypoint for Nightmare and dumb-init.
ENTRYPOINT ["dumb-init", "--", "/codecept/tools/docker/end2end-tests/entrypoint"]

# Initialize the container with bash
CMD ["bash"]

Debugging

1

I've also tried replacing

RUN ln -s /codecept/node_modules/.bin/codeceptjs /usr/local/bin/codeceptjs

with

RUN ln -s /codecept/node_modules/codeceptjs/bin/codecept.js /usr/local/bin/codeceptjs

But the results were the same, so I don't think it's the difference in how the environment is mounted.

2

I see that between the upgrades that the assertions that were taking about 155ms to validate is immediately failing 4ms in.

waitForAction is currently set to 1500, and waitForTimeout is 10000.

But these 2 don't seem to have an impact on what's happening here.

I tried to look into assertion timeouts but couldn't find anything in the documentation. Am I missing something?

3

Another example of passing locally, but failing in the CI

In this example puppeteer gave a different error:

Execution context was destroyed, most likely because of a navigation.

But the icon was still clearly visible in the screenshot

[adudyak]

What's the error in console? Did you try waitForVisible()?

[sethvanwykJET]

Error in console is mostly just a failed assertion.

cannot find the element yet.

From what I can tell though it's due to the assertions happening much faster than they were before the update. The allure report under 2 shows the difference in the assertion being able to pass after 155ms, while the failure happens in 4ms.

Adding a wait and then finding the element seems to be more stable, but there are 100s of tests with different assertions that now need new waits added that weren't needed before.

[sethvanwykJET]

Worth a shot, but it's not something we'll be able to configure overnight.

I'll bring this up to my colleagues to see what their take is.

For now I want to see what can be done to make Puppeteer work for us without making infrastructural changes.

[adudyak]

Is there a way to adjust the see / seeElement / click operators with an await from the steps file to adjust it in one place instead of adding it to all tests?

ChatGPT says we can use wrapper, but it does not make a lot of sense, if you need custom I.seeElement() in multiple files:

const { I } = inject();

I._seeElement = I.seeElement; // store the original implementation

I.seeElement = function(selector, timeout) {
  // add your custom behavior here
  console.log(`Checking for element ${selector}`);
  
  // call the original implementation
  return this._seeElement(selector, timeout);
}

I can suggest to create in steps_file.js alternative method and replace all seeElement() with it:

seeElementAdv(selector, timeout) {
  this.waitForVisible(selector, timeout);
  this.seeElement(selector, timeout);
}

Or be pro and introduce new class/object Y, extended from I with seeElement() and import to required files.

[sethvanwykJET]

For now it seems like it's probably smarter to just refactor existing tests to have proper waits after actions are performed that might impact the DOM.

As adjusting every seeElement feels a bit Excessive, and might introduce bad behaviour waiting for something that should already be present on the page.

[sethvanwykJET]

Found something else that helped

https://codecept.io/plugins/#steptimeout

We didn't have this configured before, and it's working like the older version after configuring it.