avoid duplicated job in the gitlab-ci

Question

my gitlab pipeline consist of 4 stages and 5 jobs:

1. formatting
2. test
3. build

• build-frontend
• build-backend

4. deploy

the first 2 stages are running on any branch, the issue is when I create a MR the first two stages are rerun regardless of being passed in the source branch.

what I want is to pass the first tow stages unless I am pushing directly to the main branch.

second thing is that the build-frontendjob is creating an artifact which I am using for the build-backend, I am changing the front end once per year so It just a waste of time. so how I can use the previous created artifiact unless the fronend directory has changed create a new artifact. cause currently my work flow is taking around 1h if I could skip these three jobs formatting, testing, build-front end It will go down to half that time

this is a snippet of my yaml file:

stages:
  - formatting
  - test
  - build
  - deploy

variables:
  DOCKER_REGISTRY: **********
  DOCKER_HOST: **********
  DOCKER_TLS_CERTDIR: *********
  POSTGRES_DB: ******************

formatting:
  image: python:3.9
  stage: formatting
  before_script:
    - cd backend/
    - pip install black==22.3.0
    - pip install bandit
  script:
    - black . --check --diff --color
    - bandit -c pyproject.toml --recursive .

test-backend:
  image: python:3.9
  services:
    - postgres:latest
  stage: test
  before_script:
    - cd backend//
    - pip install -r requirements/base.txt
  script:
    - pytest 
  variables:
    DATABASE_URL: ********
    POSTGRES_HOST_AUTH_METHOD: ******
  coverage: '/TOTAL.*\s+(\d+%)$/'

build-frontend:
  image: node:14.8.0
  stage: build
  before_script:
    - cd web/
  script:
    - yarn install
    - yarn build
  artifacts:
    paths:
      - backend/build/
    expire_in: 20 minutes
  only:
    - master

build-backend:
  image:
    name: amazon/aws-cli:2.15.1
    entrypoint: [""]
  stage: build
  services:
    - docker:dind
  before_script:
    - cd backend/
  needs: [build-frontend]
  script:
    - docker build . 
  only:
    - master

deploy-backend:
  environment:
  stage: deploy
  image:
    name: bitnami/kubectl:1.28
    entrypoint: [ "" ]
  needs: [build-backend]
  before_script:
    - cd backend/
  script:
    - kubectl apply .
  only:
    - master

currently the forntend artifact ends in 20 min, I want it to be reusable.

I tried the following rules for the format and test stages: but either they cancel the job at all or do nothing.

  rules:
    - if: $CI_MERGE_REQUEST_APPROVED
      when: never
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: never
    - if: $CI_PIPELINE_SOURCE == "push"
      when: always

tried to skip using the approval but didn't changed anything.

  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME && $CI_COMMIT_REF_NAME == "master"'
      when: never
    - when: always

    - if: $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"
      when: never
    - if: $CI_PIPELINE_SOURCE == "push" 
      when: always

Answer 1

You can use this rule for first two steps.

rules:
  - if: '$CI_MERGE_REQUEST && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"'
    when: never
  - when: always

And for only frontend changes just filter the folder structure like this;

  only:
    - changes:
        - web/**/*
      - master