Posts Circle CI Commands
Post
Cancel

Circle CI Commands

Circle CI command

Cricle CI now as the option to create reusable steps in your pipeline that act almost as functions. Using commands it is now possible to create standard parts of the pipeline without needing to use external tools.

Commands are a great feature that seem to be a bit buried in the rest of the documentation. Here I will try to talk about them in some more detail.

The problem that this solves

Commands are particularly useful when used which a monorepo. A monorepo is a practice for development teams to keep all their code together in a single git repo. When designing a CI pipeline for a monorepo we often need to deal with each directory separately.

Imagine a Circle CI workflow like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
version: 2.1
orbs:
  node: circleci/node@2.0.3


workflows:
  build:
    jobs:
    - project1
    - project2

jobs:

  project1:
    executor:
      name: node/default
      tag: '12.16'
    steps:
    - checkout
    - node/install-yarn
    - restore_cache:
        key: dependency-cache-
    - run:
        name: Yarn install
        command: yarn
        working_directory: project1
    - save_cache:
        key: dependency-cache-
        paths:
          - project1/node_modules
    - run:
        name: Yarn test
        command: yarn test
        working_directory: project1
        environment:
          CI: true


  project2:
    executor:
      name: node/default
      tag: '12.16'
    steps:
      - checkout
      - node/install-yarn
      - restore_cache:
          key: dependency-cache-
      - run:
          name: Yarn install
          command: yarn
          working_directory: project1
      - save_cache:
          key: dependency-cache-
          paths:
            - project2/node_modules
      - run:
          name: Yarn test
          command: yarn test
          working_directory: project1
          environment:
            CI: true

In this example we have 2 javascript projects in the repo, for both of them we need to checkout the code, install yarn, fetch the dependencies and then run the test. The details of the pipeline do not really matter, the important part is that we have a pipeline that has 2 builds with almost the same steps.

As more projects get added our pipeline will grow with repeating code, which any engineer will feel is wrong. There are 2 main ways to solve this problem, build scripts and commands.

Build scripts

The first option to remove the duplication is to introduce build scripts. These could be done with almost any language but common choices would be bash or make. We could create a javascript tool but the principle is the same, we could combine the build steps into a single step to remove some of the duplication in the pipeline.

Our new pipeline might be more like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
version: 2.1
orbs:
  node: circleci/node@2.0.3


workflows:
  build:
    jobs:
    - project1
    - project2

jobs:

  project1:
    executor:
      name: node/default
      tag: '12.16'
    steps:
      - checkout
      - node/install-yarn
      - restore_cache:
          key: dependency-cache-
      - run:
          name: Test
          command: ./test.sh project1
      - save_cache:
          key: dependency-cache-
          paths:
            - project1/node_modules


  project2:
    executor:
      name: node/default
      tag: '12.16'
    steps:
      - checkout
      - node/install-yarn
      - restore_cache:
          key: dependency-cache-
      - run:
          name: Test
          command: ./test.sh project2
      - save_cache:
          key: dependency-cache-
          paths:
            - project2/node_modules

Using a build script we have been able to remove a few lines but there is a lot that we cannot reduce. The problem is that steps like checkout, restore_cache or save_cache are all part of Circle CI that cannot be easily used in a custom script. This problem is not unique to Circle CI, every CI tool has special commands that form part of their DSL and we cannot easily use in a custom script.

The other problem that this solution presents is the location of an error. If the dependency installation or the tests or a syntax problem in the test.sh occur the error will appear in the same place. The onus is on the developers to ensure the error reporting is clear enough that we can see the real cause of the problem in the logs.

Build scripts offer some improvements, particularly for complex builds, but they come with a lot of limitations.

Circle CI Commands

Commands offers the best of both worlds. We can create a command to run our test, complete with custom commands.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
version: 2.1
orbs:
  node: circleci/node@2.0.3


workflows:
  build:
    jobs:
    - project1
    - project2

commands:
    yarn_test:
        description: Configure Yarn
        parameters:
          directory:
            type: string
        steps:
          - checkout
          - node/install-yarn
          - restore_cache:
              key: dependency-cache-
          - run:
              name: Yarn install
              command: yarn
              working_directory: << parameters.directory >>
          - save_cache:
              key: dependency-cache-
              paths:
                - << parameters.directory >>/node_modules
        - run:
              name: Yarn test
              command: yarn test
              working_directory: << parameters.directory >>
              environment:
                CI: true


jobs:

  project1:
    executor:
      name: node/default
      tag: '12.16'
    steps:
        - yarn_test:
            directory: project1

  project2:
    executor:
      name: node/default
      tag: '12.16'
    steps:
        - yarn_test:
            directory: project2

Now we have all the steps to checkout the code, configure the cache and dependencies as well as test in a single command. Command parameters make it act more like a function with input that make it reusable throughout the pipeline. In addition there is the option for commands to use Circle CI bespoke keys and even other commands.

As the pipeline grows the command can be reused again and again to keep the total size of the code down. With the opportunely to compose commands by breaking the steps down and having one command use another we get a flexible and powerful CI syntax.

When the pipeline is run, each step in the command is executed separately so it becomes really easy to find problems.

Conclusion

Commands in Circle CI is, in my opinion, on of the best features in the platform. They allow reusable code as part of the pipeline and enable a large, complex pipeline to remain manageable as it grows. Commands are one of the reasons why Circle CI remains my first choice for CI services.

Now if they would just allow us to split a pipeline over more than one file…

This post is licensed under CC BY 4.0 by the author.