Parallelism in CI/CD Pipelines like GitHub Actions

Question

Hello there and thank you for reading my question, its my first one here.

I am working with CI/CD pipelines for a year now and I think they are pretty nice and convinient for developing Websites and Stuff. But in the last months I have more and more problems creating fast, efficient and smart pipelines without redundant dependency installs or similar. So I want to use as less computation ressources as possible while still have fast builds. I want to parallelize steps and use theire artifacts in another final step. For example the following GitHub Actions workflow:

My goal with this workflow is to just build a VueJS Single Page App and deploy it to the IBM Cloud. For that I need to install the npm dependencies and build the Vue App and also install the IBM Cloud CLI. After these two steps are finished the builded App should be pushed to the IBM Cloud.

I could just simply run all steps sequentially like this:

name: Deploy
on:
  push:
    branches: [main]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v2
      - name: Use Node.js 10.X
        uses: actions/setup-node@v1
        with:
          node-version: '10.x'
      - name: Cache Node Modules
        uses: actions/cache@v2
        env:
          cache-name: cache-node-modules
        with:
          # npm cache files are stored in `~/.npm` on Linux/macOS
          path: ~/.npm
          key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
          restore-keys: |
            ${{ runner.os }}-build-${{ env.cache-name }}-
            ${{ runner.os }}-build-
            ${{ runner.os }}-
      - name: Install Dependencies
        run: npm ci
      - name: Build Page
        run: npm run build
      - name: Install IBM Cloud CLI
        run: curl -fsSL https://clis.cloud.ibm.com/install/linux | sh
        shell: bash
      - name: Install Cloud Foundry CLI
        run: ibmcloud cf install
        shell: bash
      - name: Authenticate with IBM Cloud CLI
        run: ibmcloud login --apikey "${{ secrets.IBM_CLOUD_API_KEY }}" --no-region -g Default
        shell: bash
      - name: Target a Cloud Foundry org and space
        run: ibmcloud target --cf-api "${{ secrets.IBM_CLOUD_CF_API }}" -o "${{ secrets.IBM_CLOUD_CF_ORG }}" -s "${{ secrets.IBM_CLOUD_CF_SPACE }}"
        shell: bash
      - name: Deploy to Cloud Foundry
        run: ibmcloud cf push
        shell: bash

But in my opinion this is very ugly and can be improved. So I tried to split the job into 3 parts: build, predeploy and deploy. The build job installs and builds the Vue App. The Predeploy job install the IBM CLI. These two jobs doesn't depend on each other so they can be parallized. But the last job, deploy, depends on both so I added the needs: [build, predeploy] value to it. So I have the following workflow to archive this:

### This will not work!
name: Deploy
    on:
      push:
        branches: [main]
    jobs:
      build:
        runs-on: ubuntu-latest
        steps:
          - name: Checkout Code
            uses: actions/checkout@v2
          - name: Use Node.js 10.X
            uses: actions/setup-node@v1
            with:
              node-version: '10.x'
          - name: Cache Node Modules
            uses: actions/cache@v2
            env:
              cache-name: cache-node-modules
            with:
              # npm cache files are stored in `~/.npm` on Linux/macOS
              path: ~/.npm
              key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
              restore-keys: |
                ${{ runner.os }}-build-${{ env.cache-name }}-
                ${{ runner.os }}-build-
                ${{ runner.os }}-
          - name: Install Dependencies
            run: npm ci
          - name: Build Page
            run: npm run build
      predeploy:
        runs-on: ubuntu-latest
        defaults:
          run:
            shell: bash
        steps:
          - name: Install IBM Cloud CLI
            run: curl -fsSL https://clis.cloud.ibm.com/install/linux | sh
          - name: Install Cloud Foundry CLI
            run: ibmcloud cf install
          - name: Authenticate with IBM Cloud CLI
            run: ibmcloud login --apikey "${{ secrets.IBM_CLOUD_API_KEY }}" --no-region -g Default
          - name: Target a Cloud Foundry org and space
            run: ibmcloud target --cf-api "${{ secrets.IBM_CLOUD_CF_API }}" -o "${{ secrets.IBM_CLOUD_CF_ORG }}" -s "${{ secrets.IBM_CLOUD_CF_SPACE }}"
      deploy:
        needs: [build, predeploy]
        runs-on: ubuntu-latest
        steps:
          - name: Deploy to Cloud Foundry
            # Error: 'ibmcloud: command not found'
            run: ibmcloud cf push
            shell: bash

Which looks on the GUI like:

[![My GitHub Workflow on the GUI][1]][1]

But this workflow will error since the last job doesn't share the same environment as the other jobs. I am aware that I could use the up/download Artifact feature of GitHub Actions but this seems to me like using a lot of resources. But I dont want to use a lot of ressources for my pipeline, I dont need a lot of different virtual environments or build matrixes. (I know they are very good for large projects, but they seem a little overkill for my little site)

So here are my two final Questions:

Why is parallelism in CI/CD often complication and not straight forward?

How can I improve my current pipeline with parallelism and without redundant executions?

I am glad about every helpful advice or link. Thank you. :) [1]: https://i.stack.imgur.com/qEqLs.png

ITChap ITChap · Accepted Answer · 2021-03-01T12:44:29

I think your original workflow was already pretty efficient. As you mentioned, different jobs are executed on different runners and sometime the additional complexity and effort put into the synchronization/logic between workflows outweighs the benefits of parallelism. In your case I don't think it would make much sense to run your jobs in parallel.

For your first question, I don't think it's an issue specific to CI/CD pipelines. I am getting a bit out of scope here but you have similar issues in any code that does work in parallel or as a matter of fact in any work in general that is done in parallel anywhere. Being factories, teams, code, CI pipelines, as soon as the work is split up, there will be some sort of mechanism to manage the allocation of work and track its progress. Which will make it more complex. Why GH workflows might seem less straightforward than other systems seem to be a better question and I think it comes does to how long it has been around. It's a pretty recent addition to github and as new features are progressively being added it gets easier and easier to work with.

Regarding other optimizations for your workflow, I would recommend trying to avoid redoing the same work every time the workflow run if it's not needed. You already do this with the cache action for npm. But you could, for example build a docker image, or even better an action ,with your IBM CLI in it and remove the pre-deploy stage entirely. Simply having:

name: Deploy
on:
  push:
    branches: [main]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v2
      - name: Use Node.js 10.X
        uses: actions/setup-node@v1
        with:
          node-version: '10.x'
      - name: Cache Node Modules
        uses: actions/cache@v2
        env:
          cache-name: cache-node-modules
        with:
          # npm cache files are stored in `~/.npm` on Linux/macOS
          path: ~/.npm
          key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
          restore-keys: |
            ${{ runner.os }}-build-${{ env.cache-name }}-
            ${{ runner.os }}-build-
            ${{ runner.os }}-
      - name: Install Dependencies
        run: npm ci
      - name: Build Page
        run: npm run build
      - name: Deploy to Cloud Foundry
        uses: my-action:v1
        with:
          api-key: ${{ secrets.IBM_CLOUD_API_KEY }}
          cf-api: ${{ secrets.IBM_CLOUD_CF_API }}
          cf-org: ${{ secrets.IBM_CLOUD_CF_ORG }}
          cf-space: ${{ secrets.IBM_CLOUD_CF_SPACE }}

Parallelism in CI/CD Pipelines like GitHub Actions

1 Answers