Minor annoyances while installing GitLab

I’ve installed GitLab on my server, and it hasn’t exactly been a smooth ride. So here goes, what went wrong in my case — and how to fix it.

First off, I should mention the server is a LXC container running Debian 9 (Stretch) 64bits.

Prerequisites install failures

Following along the GitLab install guide, the second step is to download a script from https://packages.gitlab.com and blindly pipe it into sudo bash, then… wait, what?

Well, since I’m not one for running as root random shit downloaded over the Internet without prior extensive scrutiny, I first took a look at the script. Basically it:

  1. checks the host distro
  2. installs curl
  3. installs debian-archive-keyring
  4. installs apt-transport-https
  5. fetches an APT sources configuration file from the GitLab repo
  6. fetches the GitLab package repo’s key
  7. runs apt-get update

Sadly, there is very little in the way of error-checking in this script. More specifically, all executions of apt-get don’t bother checking the return code, and hence fail to notice any errors that may have occurred.

In my case, what failed was step 4 (since both curl and debian-archive-keyring were already installed, otherwise they would have failed too). This was because I use etckeeper with stricter settings that the default, by which it will refuse to install stuff if there are uncommitted changes in /etc.
There were uncommitted changes in my /etc, so etckeeper failed the install attempt, which was silently ignored by GitLab’s script. Thus in the end, even though I got the success message from the end of the script, nothing much had actually happened.

This was easily fixed by just properly committing changes in /etc before running the script, and also afterwards (since the script does drop its sources configuration file in /etc/apt).

APT proxy issues

The last step in the above script is to run apt-get update to fetch the InRelease file from the GitLab package repo. On my box, this failed with the following errors:

Ign:7 https://packages.gitlab.com/gitlab/gitlab-ee/debian stretch InRelease
Err:8 https://packages.gitlab.com/gitlab/gitlab-ee/debian stretch Release
  Received HTTP code 403 from proxy after CONNECT
Reading package lists... Done
E: The repository 'https://packages.gitlab.com/gitlab/gitlab-ee/debian stretch Release' does no longer have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.

Easy enough, this must be because of my apt-cacher-ng proxy which is not configured for HTTPS proxying.
So I added the following to my APT proxy configuration:

Acquire::https::Proxy::packages.gitlab.com DIRECT;

But the error remained, even though reaching the file directly in a browser worked fine… After much fiddling with APT’s proxy configuration, and much cursing, but still no joy, I brought out the big guns: strace.

And there it was:

read(6, "103 Redirect\nURI: https://packages.gitlab.com/gitlab/gitlab-ee/debian/dists/stretch/InRelease\nNew-URI: https://packages-gitlab-com.s3-accelerate.amazonaws.com/7/11/debian/dists/stretch/InRelease?AWSAccessKeyId=AKIAJ74R7IHMTQVGFCEA&Signature=oo10HdjIUUaV5Ms2OPTS7hPUsPo=&Expires=1516749086\n\n", 64000) = 290
[…]
read(13, "400 URI Failure\nURI: https://packages-gitlab-com.s3-accelerate.amazonaws.com/7/11/debian/dists/stretch/InRelease?AWSAccessKeyId=AKIAJ74R7IHMTQVGFCEA&Signature=oo10HdjIUUaV5Ms2OPTS7hPUsPo=&Expires=1516749086\nMessage: Received HTTP code 403 from proxy after CONNECT\n\n", 64000) = 265

Indeed, in the browser, I had noticed the redirect to packages-gitlab-com.s3-accelerate.amazonaws.com. However, its significance with regards to my proxy issue had eluded me: while the no-proxy directive I had set in the APT configuration did prevent going through the proxy when reaching out to packages.gitlab.com, it didn’t apply to the redirected URL at packages-gitlab-com.s3-accelerate.amazonaws.com.

So at last the solution became clear:

# GitLab repo (first domain redirects to the second, so both need the proxy bypass)
Acquire::https::Proxy::packages.gitlab.com DIRECT;
Acquire::https::Proxy::packages-gitlab-com.s3-accelerate.amazonaws.com DIRECT;

And indeed, apt-get update finally managed to fetch the GitLab repo’s InRelease file.

sysctl settings failure

After downloading and installing the gitlab-ee package, GitLab tries to configure itself.
Several sysctl-related failures happened during this step, all due to the fact that in an LXC container, sysfs is read-only and hence sysctl variables can’t be set. This is a common issue and it even has its own entry in GitLab’s list of common installation problems. The solution is to gather those parameters after GitLab fails to set them, and actually set them on the host:

# Gitlab settings (see forge:/opt/gitlab/embedded/etc/90-omnibus-gitlab*)
kernel.shmall = 4194304
kernel.shmmax = 17179869184
kernel.sem = 250 32000 32 262
net.core.somaxconn = 1024

Another solution mentioned at the end of this GitHub issue would be to use dpkg-divert to override the installed sysctl configuration files, but I didn’t try this.

Some more info on the sysctl variables being set:

  • shmmax and shmall: a blog post, and this extract from the Linux kernel’s Documentation/sysctl/kernel.txt: 
    shmall:

This parameter sets the total amount of shared memory pages that
can be used system wide. Hence, SHMALL should always be at least
ceil(shmmax/PAGE_SIZE).

If you are not sure what the default PAGE_SIZE is on your Linux
system, you can run the following command:

# getconf PAGE_SIZE

==============================================================

shmmax:

This value can be used to query and set the run time limit
on the maximum shared memory segment size that can be created.
Shared memory segments up to 1Gb are now supported in the
kernel.  This value defaults to SHMMAX.

    also, from the Linux kernel’s include/uapi/linux/shm.h:

    /*
 * SHMMNI, SHMMAX and SHMALL are default upper limits which can be
 * modified by sysctl. The SHMMAX and SHMALL values have been chosen to
 * be as large possible without facilitating scenarios where userspace
 * causes overflows when adjusting the limits via operations of the form
 * "retrieve current limit; add X; update limit". It is therefore not
 * advised to make SHMMAX and SHMALL any larger. These limits are
 * suitable for both 32 and 64-bit systems.
 */
#define SHMMIN 1            /* min shared seg size (bytes) */
#define SHMMNI 4096             /* max num of segs system wide */
#define SHMMAX (ULONG_MAX - (1UL << 24)) /* max shared seg size (bytes) */
#define SHMALL (ULONG_MAX - (1UL << 24)) /* max shm system wide (pages) */
#define SHMSEG SHMMNI           /* max shared segs per process */
  • sem: a random wiki page, and the Linux kernel’s include/uapi/linux/sem.h: 
    /*
 * SEMMNI, SEMMSL and SEMMNS are default values which can be
 * modified by sysctl.
 * The values has been chosen to be larger than necessary for any
 * known configuration.
 *
 * SEMOPM should not be increased beyond 1000, otherwise there is the
 * risk that semop()/semtimedop() fails due to kernel memory fragmentation when
 * allocating the sop array.
 */


#define SEMMNI  32000           /* <= IPCMNI  max # of semaphore identifiers */
#define SEMMSL  32000           /* <= INT_MAX max num of semaphores per id */
#define SEMMNS  (SEMMNI*SEMMSL) /* <= INT_MAX max # of semaphores in system */
#define SEMOPM  500            /* <= 1 000 max num of ops per semop call */
#define SEMVMX  32767           /* <= 32767 semaphore maximum value */
#define SEMAEM  SEMVMX          /* adjust on exit max value */
  • somaxconn: a NASA wiki page, and this extract from the Linux kernel’s Documentation/networking/ip-sysctl.txt: 
    somaxconn - INTEGER
    Limit of socket listen() backlog, known in userspace as SOMAXCONN.
    Defaults to 128.  See also tcp_max_syn_backlog for additional tuning
    for TCP sockets.

Leave a Reply

Your email address will not be published. Required fields are marked *