Mirror repository to Gitlab for External CI usage

Until recently was possible to use Gitlab as an external CI for a repository hosted somewhere else. In the same vein of Cloud CI providers such as Travis or Circle.

The way it worked was using Gitlab’s repo mirroring feature to mirror all repository changes to Gitlab, then the pipelines will run from the Gitlab repo.

All was fine until Gitlab decided to make it a paid feature. And last Friday they finally pulled the plug, by simply disabling mirroring. Some people were caught by surprised, myself included. Just see the posts below

Pull based Mirroring from GitHub for CI is no longer mirrored

Gitlab mirroring from github project is not working

I would be a happy Gitlab customer if it wasn’t for my employer requirements to use Bitbucket, furthermore we already self host our own Gitlab runners. Being required to upgrade and pay over 200 USD per month and not so many additional benefits is a no-go for the time being.

Therefore, we decide to do the mirroring on our side.

1. Generate Access Token (read/write permissions to repo) (https://gitlab.com/profile/personal_access_tokens)
2. Take a small server and install repository (git clone original-repo.com)
3. Install Github Hub (https://github.com/github/hub)
4. Create file with the following contents (it ensures we can mirror all branches and tags).

echo "updating repo"
echo "checking out all remote branches locally"
git branch -r | grep -v '\->' | while read remote; do git branch --track "${remote#origin/}" "$remote"  2>&1 | grep -v "already exists"; done
echo "fetching updates"
git fetch --all
git pull --all
echo "updating all branches from remote"
hub sync 2>&1 | grep -v  "was deleted on origin, but appears not merged into 'master'"
echo "push to gitlab"
git push --all https://oath2:[token]@gitlab.com/[user]/[project].git
git push --tags https://oath2:[token]@gitlab.com/[user]/[project].git

4. Install cron job crontab -e

* * * * * cd /var/www/[project-dir]; bash mirror.sh; sleep 25; bash mirror.sh;

Simply mirroring everything to Gitlab doesn’t seem to work as it tries to mirror merge requests and will face some incompatibilities. Example of errors resulting from “mirroring”.

remote: error: cannot lock ref 'refs/heads/feature/[branch-name]': unable to resolve reference 'refs/heads/feature/[branch-name]'
remote: error: cannot lock ref 'refs/merge-requests/11/merge': unable to resolve reference 'refs/merge-requests/11/merge'
remote: error: cannot lock ref 'refs/pipelines/996296': unable to resolve reference 'refs/pipelines/996296'

TODO: this was just some quick notes, need to be properly rewritten.

Let’s say the post is tagged with #hack #wip.