Megalinter apply fixes for Gitlab

[MouadRabihi07]

Hello, i'm using megalinter in a Gitlab ci pipeline, and i was wondering if the option to auto apply fixes with a commit created by megalinter is only possible in github workflows, or i can implement this also in my pipeline ?

[nvuillam]

@MouadRabihi07 it has indeed never been implemented on default gitlab-ci.yml workflow

If you succeed to do it, would you like to make a PR on https://github.com/oxsecurity/megalinter/blob/main/mega-linter-runner/generators/mega-linter/templates/.gitlab-ci.yml ?

If it can help, it could be something like the following (requiring to create an access token and define it in an env variable) , that I used on sfdx-hardis monitoring script

https://github.com/hardisgroupcom/sfdx-hardis/blob/f7cf207d5c8334a9bcc3190e9594c7378b80938c/defaults/monitoring/.gitlab-ci.yml#L28

[alexanderbazhenoff]

Hi, I tried the same thing on GitLab CI agent running in k8s cluster. My .gitlab-ci.yml looks like:

megalinter:
  # ... tags, image, stage... you know
  script:
    # I have pasted these commands for sure:
    - git config --global user.name "${GITLAB_USER_NAME}"
    - git config --global user.email "${GITLAB_USER_EMAIL}"
    - git fetch origin
    - git checkout -B "${CI_COMMIT_REF_NAME}" origin/"${CI_COMMIT_REF_NAME}"
    - git pull origin "${CI_COMMIT_REF_NAME}"
    - git branch --set-upstream-to=origin/"${CI_COMMIT_REF_NAME}" "${CI_COMMIT_REF_NAME}"
    - git remote set-url origin https://${GITLAB_ACCESS_TOKEN_MEGALINTER}@gitlab.domain/${CI_PROJECT_PATH}.git
    # then run megalinter:
    - bash /entrypoint.sh
  variables:
    DEFAULT_WORKSPACE: $CI_PROJECT_DIR
    LOG_LEVEL: INFO
    PARALLEL_PROCESS_NUMBER: 6
    SHOW_ELAPSED_TIME: 'true'
    PRINT_ALIGNED: 'true'
    LINTER_RULES_PATH: .linters
    SHOW_FLAVORS_INFO: 'false'
    GITLAB_COMMENT_REPORTER: 'true'
    VALIDATE_ALL_CODEBASE: 'true'
    APPLY_FIXES: all
    APPLY_FIXES_EVENT: none
    APPLY_FIXES_MODE: pull_request
    DEFAULT_BRANCH: $CI_COMMIT_REF_NAME
    GIT_SOURCE_BRANCH: $CI_COMMIT_REF_NAME
    FORMATTERS_DISABLE_ERRORS: true
    CONSOLE_REPORTER: 'true'
    CONSOLE_REPORTER_SECTIONS: 'true'
    PROCESS_ON_GIT_DIFF: 'false'
    OUTPUT_ARTIFACTS_URL: 'true'
    GITLAB_ACCESS_TOKEN_MEGALINTER: # ... my token with API and repository r/w permissions

But I got: [Updated Sources Reporter] Failed to retrieve git source branch.

I also pasted debug commands before ML run:

$ echo "Current branch $(git rev-parse --abbrev-ref HEAD)"
Current branch updates
$ git branch -r
  origin/main
  origin/updates

Looks good, but ML still can't find the "original branch".

How it works? Can you send me a URL to the code where this "source branch" detects?

Or I need to do it manually like in this issue?

I would like to to it a native way with a merge request rather than using postscript commands.

[alexanderbazhenoff]

I did my own research to workaround this in GitLab CI directly.

Here is a solution how to apply merge requests via merge request to current working branch (not default). If you would like to push it your working branch directly you should do it like @nvuillam described.

1. Create GitLab PAT token with permissions for r/w repository. You can also make an only token to apply MR comments and merge requests.
2. Grant 'maintainer' permissions to a project for the user with PAT token.
3. Create GitLab CI extends file for your library:

---
.megalinter_resulting_script: &megalinter_resulting_script |
  printf "%100s" | tr " " "-"
  if $ANY_ERR; then
    printf "\n \n \n\e[1mPlease check:\e[0m\n \n"
    printf "\nAll linters included in MegaLinter: %s\n" "https://megalinter.io/latest/reporters/ConsoleReporter/"
    printf "\nMegaLinter configuration file: %s\n" "https://megalinter.io/latest/config-file/"
  fi
  if $ANY_ERR; then
    printf "\nMegaLinter \e[41m  FAILED  \e[0m...\n"
    exit 1
  else
    printf "\nMegaLinter \e[42m  SUCCESS  \e[0m...\n"
  fi


.processing_apply_fixes_logic: &processing_apply_fixes_logic |
  if [ "$GITLAB_CI_APPLY_FIXES_MODE" == "always" ] || \
    [[ "$GITLAB_CI_APPLY_FIXES_MODE" == "on_demand" && "$CI_COMMIT_MESSAGE" =~ ^auto.?fix ]]; then
      GITLAB_CI_APPLY_FIXES=true
  else
      GITLAB_CI_APPLY_FIXES=false
  fi

.print_apply_fixes_hints: &print_apply_fixes_hints |
  if ! $GITLAB_CI_APPLY_FIXES; then
    if [[ "$GITLAB_CI_APPLY_FIXES_MODE" != "always" && "$GITLAB_CI_APPLY_FIXES_MODE" != "on_demand" ]]; then
      MSG_AUTO_FIX_REASON="by variable 'GITLAB_CI_APPLY_FIXES_MODE='${GITLAB_CI_APPLY_FIXES_MODE}'"
      MSG_AUTO_FIX_HINT="Set GITLAB_CI_APPLY_FIXES_MODE='always' or Set GITLAB_CI_APPLY_FIXES_MODE='on_demand'"
      MSG_AUTO_FIX_HINT="$MSG_AUTO_FIX_HINT then commit changes with 'auto fix' message"
    elif [[ "$GITLAB_CI_APPLY_FIXES_MODE" == "on_demand" && ! "$CI_COMMIT_MESSAGE" =~ ^auto.?fix ]]; then
      MSG_AUTO_FIX_REASON="by commit message"
      MSG_AUTO_FIX_HINT="Commit changes with 'auto fix' message to apply automated fixes"
    fi
    printf "\e[45m  GitLab CI  \e[0m | Auto fixes are disabled by %s. %s.\n" "$MSG_AUTO_FIX_REASON" "$MSG_AUTO_FIX_HINT"
  fi

.set_up_git: &set_up_git |
  if $GITLAB_CI_APPLY_FIXES; then
    git config --global user.name "$GITLAB_USER_NAME"
    git config --global user.email "$GITLAB_USER_EMAIL"
    git fetch origin
    git checkout -B "$CI_COMMIT_REF_NAME" origin/"$CI_COMMIT_REF_NAME"
  fi

.apply_ci_fixes: &apply_ci_fixes |
  function output_message_source {
    printf "\e[45m  GitLab CI  \e[0m | "
  }
  if $GITLAB_CI_APPLY_FIXES; then
    git add --all
    if git diff --cached --quiet; then
      output_message_source
      printf "No auto fixes to apply.\n"
    else
      if [ -n "$CI_COMMIT_REF_NAME" ]; then
        CURRENT_BRANCH="$CI_COMMIT_REF_NAME"
      else
        CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD)
      fi
      FIX_BRANCH="${CURRENT_BRANCH}-auto-fix"
      output_message_source
      printf "Auto fixes detected, pushing to '%s' branch...\n" "$FIX_BRANCH"
      git checkout -b "$FIX_BRANCH"
      COMMIT_PASS=true
      git commit -m "[MegaLinter] Apply linters auto fixes [skip ci]" || COMMIT_PASS=false
      if $COMMIT_PASS; then
        git push -o ci-skip --force http://root:$GITLAB_ACCESS_TOKEN_MEGALINTER@$CI_SERVER_HOST/$CI_PROJECT_PATH.git \
          HEAD:$FIX_BRANCH
        output_message_source
        printf "Creating merge request assigned to '%s' user ID.\n" "$GITLAB_USER_ID"
        MR_DESCRIPTION="This changes was generated automatically by MegaLinter"
        MR_DESCRIPTION="Review all MegaLinter changes before merging to '${CURRENT_BRANCH}'."
        MR_RESPONSE=$(curl --silent --request POST \
          --header "PRIVATE-TOKEN: $GITLAB_ACCESS_TOKEN_MEGALINTER" \
          --data "source_branch=$FIX_BRANCH" \
          --data "target_branch=$CURRENT_BRANCH" \
          --data "title=[MegaLinter] Apply linters automated fixes" \
          --data "description=$MR_DESCRIPTION" \
          --data "assignee_id=$GITLAB_USER_ID" \
          --data "reviewer_ids[]=$GITLAB_USER_ID" \
          --data "squash=true" \
          --data "remove_source_branch=true" \
          "http://${CI_SERVER_HOST}/api/v4/projects/${CI_PROJECT_ID}/merge_requests")
        MR_ID=$(echo "$MR_RESPONSE" | grep -o '"iid":[0-9]*' | sed 's/"iid"://' || printf "")
        if [ -n "$MR_ID" ]; then
          MR_STATE_MSG="was successfully created"
        else
          MR_STATE_MSG="already exists"
        fi
        MR_URL="http://${CI_SERVER_HOST}/${CI_PROJECT_PATH}/-/merge_requests/${MR_ID}"
        output_message_source
        printf "The merge request %s. %s:\n \n\e[1;37;1m%s%s\e[0m\n \n" "$MR_STATE_MSG" \
          "You can find them now to review automated changes" "$(printf '%*s' 16 '')" "$MR_URL"
      else
        output_message_source
        printf "Nothing to commit.\n"
      fi
    fi
  fi

.ci_megalinter:
  image: ghcr.io/oxsecurity/megalinter:latest
  stage: lint  # or any other you like
  tags:
    - your_tag
  script:
    - *processing_apply_fixes_logic
    - *set_up_git
    - bash /entrypoint.sh && ANY_ERR=false || ANY_ERR=true
    - *apply_ci_fixes
    - *print_apply_fixes_hints
    - *megalinter_resulting_script
  variables:
    APPLY_FIXES: all
    APPLY_FIXES_EVENT: none
    VALIDATE_ALL_CODEBASE: 'true'
    # 'on_demand' for commit message 'autofix' or 'auto fix'
    # otherwise 'always' for applying each commit.
    GITLAB_CI_APPLY_FIXES_MODE: 'on_demand'
    GITLAB_CI_APPLY_FIXES: 'true'
    # Other MegaLinter default variables you wish here
    # ...

.gitlab-ci.yml inside the project:

---
include:
  - project: namespace/your-gitlab-extends-project
    ref: main
    file:
      - your_extends_file.yml
      
mega linter:
  extends: .ci_megalinter
  variables:
    GITLAB_ACCESS_TOKEN_MEGALINTER: # ... your PAT token

• This extends push auto fixes every tome with --force flag (I think you don't care about the previous automated changes in {your_working_branch}-auto-fix branch).
• If MR already exists, you'll see a corresponding message with an URL to all your merge requests.
• If you close this MR it creates a new one.

Hope it helps others.