I’ve installed GitLab on my server, and it hasn’t exactly been a smooth ride. So here goes, what went wrong in my case — and how to fix it.
First off, I should mention the server is a LXC container running Debian 9 (Stretch) 64bits.
Prerequisites install failures
Following along the GitLab install guide, the second step is to download a script from
https://packages.gitlab.com and blindly pipe it into
sudo bash, then… wait, what?
Well, since I’m not one for running as
root random shit downloaded over the Internet without prior extensive scrutiny, I first took a look at the script. Basically it:
- checks the host distro
- fetches an APT sources configuration file from the GitLab repo
- fetches the GitLab package repo’s key
Sadly, there is very little in the way of error-checking in this script. More specifically, all executions of
apt-get don’t bother checking the return code, and hence fail to notice any errors that may have occurred.
In my case, what failed was step 4 (since both
debian-archive-keyring were already installed, otherwise they would have failed too). This was because I use
etckeeper with stricter settings that the default, by which it will refuse to install stuff if there are uncommitted changes in
There were uncommitted changes in my
etckeeper failed the install attempt, which was silently ignored by GitLab’s script. Thus in the end, even though I got the success message from the end of the script, nothing much had actually happened.
This was easily fixed by just properly committing changes in
/etc before running the script, and also afterwards (since the script does drop its sources configuration file in
APT proxy issues
The last step in the above script is to run
apt-get update to fetch the
InRelease file from the GitLab package repo. On my box, this failed with the following errors:
Ign:7 https://packages.gitlab.com/gitlab/gitlab-ee/debian stretch InRelease Err:8 https://packages.gitlab.com/gitlab/gitlab-ee/debian stretch Release Received HTTP code 403 from proxy after CONNECT Reading package lists... Done E: The repository 'https://packages.gitlab.com/gitlab/gitlab-ee/debian stretch Release' does no longer have a Release file. N: Updating from such a repository can't be done securely, and is therefore disabled by default.
Easy enough, this must be because of my
apt-cacher-ng proxy which is not configured for HTTPS proxying.
So I added the following to my APT proxy configuration:
But the error remained, even though reaching the file directly in a browser worked fine… After much fiddling with APT’s proxy configuration, and much cursing, but still no joy, I brought out the big guns:
And there it was:
read(6, "103 Redirect\nURI: https://packages.gitlab.com/gitlab/gitlab-ee/debian/dists/stretch/InRelease\nNew-URI: https://packages-gitlab-com.s3-accelerate.amazonaws.com/7/11/debian/dists/stretch/InRelease?AWSAccessKeyId=AKIAJ74R7IHMTQVGFCEA&Signature=oo10HdjIUUaV5Ms2OPTS7hPUsPo=&Expires=1516749086\n\n", 64000) = 290 […] read(13, "400 URI Failure\nURI: https://packages-gitlab-com.s3-accelerate.amazonaws.com/7/11/debian/dists/stretch/InRelease?AWSAccessKeyId=AKIAJ74R7IHMTQVGFCEA&Signature=oo10HdjIUUaV5Ms2OPTS7hPUsPo=&Expires=1516749086\nMessage: Received HTTP code 403 from proxy after CONNECT\n\n", 64000) = 265
Indeed, in the browser, I had noticed the redirect to packages-gitlab-com.s3-accelerate.amazonaws.com. However, its significance with regards to my proxy issue had eluded me: while the no-proxy directive I had set in the APT configuration did prevent going through the proxy when reaching out to
packages.gitlab.com, it didn’t apply to the redirected URL at
So at last the solution became clear:
# GitLab repo (first domain redirects to the second, so both need the proxy bypass) Acquire::https::Proxy::packages.gitlab.com DIRECT; Acquire::https::Proxy::packages-gitlab-com.s3-accelerate.amazonaws.com DIRECT;
apt-get update finally managed to fetch the GitLab repo’s
sysctl settings failure
After downloading and installing the
gitlab-ee package, GitLab tries to configure itself.
sysctl-related failures happened during this step, all due to the fact that in an LXC container,
sysfs is read-only and hence
sysctl variables can’t be set. This is a common issue and it even has its own entry in GitLab’s list of common installation problems. The solution is to gather those parameters after GitLab fails to set them, and actually set them on the host:
# Gitlab settings (see forge:/opt/gitlab/embedded/etc/90-omnibus-gitlab*) kernel.shmall = 4194304 kernel.shmmax = 17179869184 kernel.sem = 250 32000 32 262 net.core.somaxconn = 1024
Another solution mentioned at the end of this GitHub issue would be to use
dpkg-divert to override the installed
sysctl configuration files, but I didn’t try this.
Some more info on the
sysctl variables being set:
shmall: a blog post, and this extract from the Linux kernel’s Documentation/sysctl/kernel.txt:
shmall: This parameter sets the total amount of shared memory pages that can be used system wide. Hence, SHMALL should always be at least ceil(shmmax/PAGE_SIZE). If you are not sure what the default PAGE_SIZE is on your Linux system, you can run the following command: # getconf PAGE_SIZE ============================================================== shmmax: This value can be used to query and set the run time limit on the maximum shared memory segment size that can be created. Shared memory segments up to 1Gb are now supported in the kernel. This value defaults to SHMMAX.
also, from the Linux kernel’s include/uapi/linux/shm.h:
/* * SHMMNI, SHMMAX and SHMALL are default upper limits which can be * modified by sysctl. The SHMMAX and SHMALL values have been chosen to * be as large possible without facilitating scenarios where userspace * causes overflows when adjusting the limits via operations of the form * "retrieve current limit; add X; update limit". It is therefore not * advised to make SHMMAX and SHMALL any larger. These limits are * suitable for both 32 and 64-bit systems. */ #define SHMMIN 1 /* min shared seg size (bytes) */ #define SHMMNI 4096 /* max num of segs system wide */ #define SHMMAX (ULONG_MAX - (1UL << 24)) /* max shared seg size (bytes) */ #define SHMALL (ULONG_MAX - (1UL << 24)) /* max shm system wide (pages) */ #define SHMSEG SHMMNI /* max shared segs per process */
sem: a random wiki page, and the Linux kernel’s include/uapi/linux/sem.h:
/* * SEMMNI, SEMMSL and SEMMNS are default values which can be * modified by sysctl. * The values has been chosen to be larger than necessary for any * known configuration. * * SEMOPM should not be increased beyond 1000, otherwise there is the * risk that semop()/semtimedop() fails due to kernel memory fragmentation when * allocating the sop array. */ #define SEMMNI 32000 /* <= IPCMNI max # of semaphore identifiers */ #define SEMMSL 32000 /* <= INT_MAX max num of semaphores per id */ #define SEMMNS (SEMMNI*SEMMSL) /* <= INT_MAX max # of semaphores in system */ #define SEMOPM 500 /* <= 1 000 max num of ops per semop call */ #define SEMVMX 32767 /* <= 32767 semaphore maximum value */ #define SEMAEM SEMVMX /* adjust on exit max value */
somaxconn: a NASA wiki page, and this extract from the Linux kernel’s Documentation/networking/ip-sysctl.txt:
somaxconn - INTEGER Limit of socket listen() backlog, known in userspace as SOMAXCONN. Defaults to 128. See also tcp_max_syn_backlog for additional tuning for TCP sockets.