How we have configured GitLab CI-CD with a private npm registry based on Verdaccio, to automatically release and update a Volto site

Since 2020 we are working, together with Bilbomatica, for the EEA to publish the new site of the Copernicus Land Monitoring Service. The new site has been built with the latest Plone 5.2 version in the backend and Volto, the new react based frontend for Plone in the frontend.

During the development of the new website, we have used the automation that EEA already provides using its GitHub account and Jenkins builders. Thanks to that automated procedures, creating new versions of packages and shipping them to a staging environment is very easy: we just have to merge some pull-requests on GitHub, and the release pipeline is launched, which updates the underlying rancher catalog, creates docker images and applies the updates to the staging environment. 

In parallel we have also been working on updating one of our client's site to Volto. The work comprises updating the old site from Plone 4.3.x to Plone 5.2.x and the frontend to the new react-based frontend built with Volto. With the lessons learned during the development at the EEA, we wanted to have a similar development and release management scenario, without the need of running buildouts or Volto build operations on the servers. We use GitLab as a git repo for our client's private code and we are already using GitLab CI-CD to do the boring stuff during the deploy some of our sites, so we decide to go with it and give it a try to build and ship our Volto sites.

As a summary we have done the following:

1. Create a private NPM registry
2. Configure a GitLab CI-CD pipeline to release and publish a volto-xxx package privately in Verdaccio
3. Configure a GitLab CI-CD pipeline to release and build a xxxx-frontend Volto package that installs the volto-xxx package from the private NPM registry
4. Configure a GitLab CI-CD pipeline to scp the result of the xxxx-frontend Volto package build into the staging environment

These are the details of the process.

A private NPM registry

Our first need was to have a private NPM-like registry to publish our client's packages, so that they could be used when building the frontend. This step is important if you want to have control over what you ship (what you generally want 😉) and don't want to ship uncontrolled code directly from your git repo. Sometimes it is easier just to clone a git repo and do git pull and ./bin/supervisorctl reload to restart the site and you are done. But when you have to do that on several servers, sometimes you need to run some tests and wait, or run a buildout that takes ages to run, it is better and less error-prone to have a package built and released. 

We Googled and found Verdaccio: a lightweight private NPM registry which proxies to npmjs.com to download packages that are not uploaded to it. After some research, trial and errors, we configured it using docker, and reversed-proxied with an nginx server, which handles the TLS negotiation. The configuration files we used are available on our GitHub account. Check that repo for full configuration information.

Building a npm package and publishing it in Verdaccio

With a private npm registry configured, we needed to tag, create and publish the npm package that has our Volto customization and stylings to our private npm registry. We will be using release-it to automate that procedure.

To do so we have faced several problems on how npm and yarn handle the authentication tokens to be used with the private npm registry. Finally we found a working way to handle it and that's what we are presenting here.

1. Create a .gitlab-ci file

We added a .gitlab-ci.yml file in our Volto addon called volto-xxxx, with the following contents:

2. Add a .release-it.json file

We also added a .release-it.json file in our volto-xxxx package with the following contents:

3. Modify our package's package.json file

In our package's package.json file we had to add the following configuration bits at the end of the file:

4. Details

Our git workflow is similar to gitflow, so all features are developed in a branch called develop or branches created from develop and merged back first to develop and then into main when the package is ready to be created. That's why our CI-CD file is restricted to branch main. We use several environment variables, configured either at package or organisation level on GitLab, which mean the following:

• VOLTO_DEPLOY_SSH_PRIVATE_KEY: a RSA private key of the public key added as a deploy-key to the repo, so that release-it can push back to the repo when it bumps the version and modifies the changelog.
• CI_GITLAB_USERNAME: a dummy username to commit release-it's changes
• CI_GITLAB_USER_EMAIL: a dummy e-mail addres to commit release-it's changes
• CODESYNTAX_NPM_REPO_URL: the full URL (without https://) of our private NPM registry
• CODESYNTAX_REPO_NPM_TOKEN: the authentication token to publish in our private NPM registry (see Verdaccio configuration for details)

The important bits are the following:

• GitLab CI-CD is run from a detached-HEAD state, which means that the cloned repo has no back reference to your GitLab repo. That's why we set the remote url, manually checkout the branch and pull the origin into it. This is needed to run release-it and get the changes it makes back to GitLab.
• Dump the private npm repo URL and the always-auth=true bits into a .npmrc file and the registry URL into .yarnrc. These are needed to require npm and yarn to use authentication when downloading packages from our private npm registry. Our private npm registry is configured to always require authentication, so this step is needed, otherwise it will not be possible to upload the package and install any package during the CI-CD process.
• In the .release-it.json file it is important to add the pushArgs so that the created tag is pushed to GitLab and also that push operation does not create a new GitLab CI-CD process.
• In the package.json file it is important to point to the relevant private npm registry url (full url with https://), so that release-it knows where to publish the package.

Moreover we have configured our package's GitLab repo (under Settings -> General), adding the said VOLTO_DEPLOY_SSH_PRIVATE_KEY with a RSA private key and with its public key as a Deploy key; and we have also protected the main branch from any push operation and allow them only to the deploy key. This way, only the CI-CD process will be allowed to push into main.

With this configuration, when a new merge-request gets merged into main, the CI-CD process will start, will create a new minor version of the package and publish it into our private NPM registry.

Building the Volto project and publishing it into staging

When we have the volto-package already released in our NPM private repository, it is time to publish the frontend package. The xxxx-frontend package is the main Volto project for our website.

1. Add a .gitlab-ci.yml file

To do so, we have created the following .gitlab-ci.yml file:

2. Add a .release-it.json file

And the following .release-it.json file:

3. Details

The important bits here are the following:

• The jsconfig.json file dance: this file is used by mrs-developer to know which packages are in "develop mode" and should be cloned from git while in development. The jsconfig.json.prod file has a default configuration where all those "develop mode" packages are disregarded and thus their correct version is downloaded from the npm registry. That's why we are doing the jsconfig.json file dance there, not to lose the development information that is stored in GitLab.
• The Variables configured in GitLab are similar (or the same) if you compare them to the ones created in the volto-xxxx package. There is a new one: SSH_KNOWN_HOSTS. This variable has the rsa key of the destination server where the code will be published. It can be obtained executing ssh-keyscan -t rsa your.destination.server.com In this destination server the public key that matches the VOLTO_DEPLOY_SSH_PRIVATE_KEY must be installed in ~/.ssh/authorized_keys otherwise the scp process will fail.
• You need to provide the SSH_SERVER_USER that will be used to scp the code and the SSH_SERVER_NAME to do the connection. You will also need to provide the path to your buildout. We had long ago decided to install our buildouts in a given path on the servers. Now we have decided to serve the Volto application from inside that folder, specifically from $SSH_SERVER_BUILDOUT_PATH/frontend/build. This way we have configured a single supervisor instance that handles both Plone and Volto. So you need to set it as a variable in GitLab CI.

Like in the previous case with the volto-xxxx package, we have configured our package's GitLab repo (under Settings -> General), adding the said VOLTO_DEPLOY_SSH_PRIVATE_KEY with a RSA private key and with its public key as a Deploy key; and we have also protected the main branch from any push operation and allow them only to the deploy key. This way, only the CI-CD process will be allowed to push into main.

So with this configuration when a new merge-request is opened from develop to main, the first step of the CI process is run, just to be sure that the package can be built without issues. When the build is green, the merge-request can be merged into main, and the GitLab CI process will tag it, build the Volto application (it takes about 10 minutes), save it as an artifact and then scp it to the server. And all of it automatically.

Conclusions

We really think that this level of automation will simplify our working process, and really concentrate on developing new features and customising the sites properly, instead of doing ssh to the servers or waiting the yarn build command to finish.

If you have any comment on our set-up or want to provide some comments, please feel free to use the comments.

Thanks to the EEA and Eau de Web, because we have learnt a lot about release process automation while working for them. Most of the ideas that we have implemented in GitLab CI-CD have been taken from the work that has been done for the Plone and Volto projects at the EEA.

There are several improvement that can be made now that we have this degree of automation, like automatically upgrading the version of the volto-xxx addon package in the xxx-frontend package when a new release is published (like it is done in the EEA repos). But for now we are not doing this automatically.