Private GitHub Actions templates for your organization

GitHub Actions is the new kid on the CI/CD block, but it’s quickly outgunning traditional open source automation servers. For a while now, the dev team at Sinch Mailgun has known some of the tools and technologies we set this up with have a shrinking community and risk becoming too niche to reasonably maintain. It’s for these reasons that we are currently undergoing a migration from our Jenkins-based CI/CD to GitHub Actions CI/CD. Full disclosure: Sometimes batteries are not included – unless you’re a GitHub Enterprise customer that is…

I say this because until very recently, workflow templates in private repositories were not supported, so we had to use a workaround. Luckily, GitHub recently released an update that now supports this functionality on its Enterprise plans. But for those of you on GitHub’s Free or Team plans, the issue still persists. We don’t know if or when GitHub will roll out this update to other plans, so in the meantime, here’s what we went through and how we solved it.

My prior experience with GitLab CI made for a shorter learning curve with GitHub Actions, but the finer details proved to differ. For example, including a set of steps from one project in another is much different. GitHub Actions and GitLab CI both support reusing steps from one repository in another, but GitHub Actions Free and Team plans require the steps you want included to be publicly accessible. In Gitlab CI, one can use authentication to import a private repo’s jobs, stages, variables, etc., using the includes directive.

At the time of writing this, GitHub Actions had issues (make it two) tracking this desired functionality of private repos. These issues had been open for almost 18 months and it just recently got added for those on Enterprise plans. I’m not sure when (or if) this functionality will be available to everyone, but it seems like a core bit of functionality to support private reuse of workflows.

I scoured the Stack Overflow posts and dug through GitHub Issues comment threads, but found no good workarounds for the missing ability to include workflows from other private repos in the same organization. 

I thought that maybe git submodules would be a possible solution. After all, it allows a repo to include another repo. Could we include our workflow templates as a submodule in our repo and use git’s submodule management to keep the templates up to date? We could, but not if we wanted to have templates irrelevant to some repos in the template repo. Git treats a repo as the smallest unit in submodule management, so our solution will either have a repo with only relevant workflows or avoid using submodules altogether.

We opted to not go with submodules because we want one workflow repo to hold Go and Python (among others) templates, but I gleaned some valuable information from that train of thought. Some people use symlinks to include parts of a repo in another repo instead of submodules.

So we tried to link our template to our app’s workflow. ln $DIR/workflows/go/build_and_push.yml $DIR/my-app/.github/workflows/build_and_push.yml works to keep our workflow template in sync with our app… until we run a git command that modifies the template (git changes the inodes of files it modifies). Since we want to be able to git pull for updates to the templates, we would need to find something that could automatically apply changes from the template repo to an app’s file.

Without the native ability to include private templates in our workflows and us wanting to have a single repo storing our workflow templates, we opted for this simple solution that optimizes the ease of applying template updates. Until GitHub Free and Team plans support private templates, you can migrate, but this is one possible solution we have tried and tested.

We include git hooks samples in our template repository (e.g. hooks/post-checkout.sample) because any git commands that are going to break links need to re-establish them. The hooks for post-checkout and post-merge both look like this:

                            

                                #!/bin/sh
#
# An example hook script to keep your application workflow up-to-date with this
# template. For example, this updates my-app's workflow with the go template.
#
# To activate, this must have execute permission and be copied like so:
#
# $ cp hooks/post-checkout.sample .git/hooks/post-checkout
# $ chmod +x .git/hooks/post-checkout

# setup our project directory, assumes my-app is sibling
export PROJ_DIR=..
# remove existing workflow for clean install
rm $PROJ_DIR/my-app/.github/workflows/build_and_push.yml
# link app workflow to template
ln -v $PROJ_DIR/workflows/go/build_and_push.yml $PROJ_DIR/my-app/.github/workflows/build_and_push.yml
                            
                        

git merge and git checkout are common operations that break links, so we execute this script after those actions.

So, for users to apply workflow templates to their apps, they must:

1. Clone our workflows repo.
2. Edit the hooks/post-checkout.sample and hooks/post-merge.sample to match their app(s).
3. Copy them to their respective .git/hooks locations and give them execute permission.
4. Perform a git checkout or git pull (git checkout master from master works).

Unfortunately, this does not keep the template repo up-to-date on its own and requires some interaction from the project owners to adopt updates – the changes need to be applied to the remote, after all. They can do so via the following commands:

                            

                                # from the workflow templates repo
$ git pull
# from the project repo
$ git add .github/workflows/
$ git commit -m "upgrade workflow template"
$ git push
                            
                        

And after that, the workflow will run with the most up-to-date changes from the template repo.

Since we optimized for ease of applying templates, we made other tradeoffs with our workaround. The most obvious is the setup required. We are asking users to organize their repos in a particular way and manually copy and paste files. These requirements contribute to a certain amount of fragility, where the system can break if files are moved or inodes change without our awareness.

However, there is an interesting benefit that we gain from this workaround. We’ve discovered a toolset and scripts that can work in a generic context. We created a git-based generic package manager that utilizes linking and git hooks to update files between git repos:

                            

                                #!/bin/sh

# set paths/files (relative to workflows repo)
export TEMPLATE_DIR=.
export APP_DIR=../my-app
export DOCKER_TEMPL=go/Dockerfile
export WKFLW_TEMPL=go/build.yml

#remove files for clean install
rm $APP_DIR/Dockerfil $APP_DIR/.github/workflows/build.yml

# link app workflow to template
ln -v $TEMPLATE_DIR/$DOCKER_TEMPL $APP_DIR/Dockerfile
ln -v $TEMPLATE_DIR/$WKFLW_TEMPL $APP_DIR/.github/workflows/build.yml

                            
                        

So why GithHub Actions? Well, our current system requires modifications, pull requests, deployments, and more to onboard a new service. And as a GitHub Enterprise user, we have a large allocation of execution minutes within GitHub Actions, and so we decided to evaluate that as an alternative. 

Now that we’ve spent some time with GitHub Actions, here are my takeaways:

As you can see, the grass is not always greener. But we predict that GitHub, with its endless resources from Microsoft, will outpace Jenkins in the long run. 

While our workaround is not pretty, it’s a solution that works until GitHub supports private templates for non-Enterprise plans. If this happens, you can migrate templates over into a single repository. 

With that said, we have a GitHub Enterprise account but don’t have the ability to turn on the private repo functionality that GitHub added yet. We’re still determining the source of this problem. Until then, we’re currently using the workaround and will check back in the future to see if we can get access to this new functionality. Watch this space!

Author: Jacob Simonson Software Developer III at Sinch Mailgun