Deploy Mattermost in Air-Gapped Environments

An air-gapped environment is one that is isolated from the public internet, requiring all necessary components to be available locally. This guide outlines what you’ll need to deploy Mattermost in a self-hosted air-gapped environment, focusing on appropriate preparation, deployment guidance and configurations required for a successful deployment.

Overview

At a high level, deploying Mattermost in an air-gapped environment requires preparing all necessary software, container images, and configuration resources in advance, since the target system has no direct internet access; transferring these artifacts to the isolated network using secure media; and then installing, configuring, and validating the deployment within the air-gapped environment. This is a summary of the steps involved:

  1. Select your preferred Mattermost deployment option:

This step is often dictated by the infrastruture already running in your air-gapped environment. If you’re deploying from scratch, we recommend reviewing our server deployment documentation to select the optimal option given your organizations needs.

  1. Setup a private container registry or package mirror:

Ideally the air-gapped environment already has a private container registry or package mirror available. If not, we recommend following our frequently asked questions or referencing online resources specific to your environment.

  1. Prepare your Bill of Materials:

Depending on your deployment method method, you’ll need to download, tag, and push required materials into your private registry or mirror.

  1. Transfer materials into the air-gapped environment:

If the private registry cannot access the public internet, you can prepare an archive of the registry data on your internet connected machine and securely transfer it using approved data transfer methods - for example, burning to a disk.

  1. Install Mattermost

Once you have all the necessary resources in your air-gapped environment, you can move forward with deployment following the instructions for Linux, Kubernetes, or Docker.

  1. Configure Mattermost for air-gapped operation

The configuration settings recommended in this document accomodate for the lack of internet access to operate Mattermost in an air-gapped environment.

Note

Consider talking to a Mattermost expert if your organization needs support deploying Mattermost and supporting services in an air-gapped environment.

Bill of Materials

On an internet connected machine, you must gather all required packages, container images, and dependencies needed for the installation process. The resources you’ll need will depend on your deployment method, specifically:

Using our provided tarball is recommeded as the simplest installation method for air-gapped environments. You can install the Mattermost Server in a few minutes on any air-gapped 64-bit Linux system.

Prerequisites

  • Mattermost tarball. We recommend using the latest ESR for extended support where server upgrades may be infrequent.

  • Database: PostgreSQL installation packages or container images for your Linux distribution

  • File Storage: Local filesystem storage is sufficient for deployments under 2,000 users. For larger deployments requiring high availability, we recommend using an S3-compatible object storage solution, such as MinIO. NFS can also be considered as an alternative for shared storage needs.

  • Load balancer: If you already have a load balancer running in your air-gapped environment you can skip this resource, otherwise we recommend deploying NGINX from these Linux packages.

(Optional) Supporting Services Consider downloading these additional resources if you plan to enable these optional components:

Kubernetes is recommended for a highly scalable and robust deployment if your organization is already running a Kubernetes cluster in the air-gapped environment.

Prerequisites

(Optional) Supporting Services Consider downloading these additional resources if you plan to enable these optional components:

Docker can be used if you don’t have a running Kubernetes cluster in the air-gapped environment, but want to use containers for simplified installation and dependency management. Docker is not recommended for production environments at high scale, as it doesn’t support clustered deployments or High Availability (HA) configurations out-of-the-box.

Prerequisites

(Optional) Supporting Services Consider downloading these additional resources if you plan to enable these optional components:

Mattermost Plugins

Mattermost includes a number of pre-built integrations for mission-critical tools. If you’d like to use any plugins beyond those that are pre-built in the Mattermost package you’ll need to download the plugin binaries from the Mattermost Marketplace. Once you have Mattermost deployed, these plugin binaries can be uploaded directly in the System Console.

SSL/TLS Certificates and Keys

If your deployment requires SSL, ensure you have the necessary certificates. This includes certificates and keys for enabling HTTPS with Mattermost, as well as any CA files or certificates needed to access internal services such as LDAP or SAML.

Local Documentation

Mattermost documenation can be built locally so you’ll have access to installation and configuration documentation in the air-gapped environment. Otherwise, you can download the necessary deployment and configuration documents directly from the GitHub docs repository.

Prerequisites The following software is required to build the documentation locally:

Server configuration

After successful deployment, you’ll need to configure Mattermost for air-gapped operation. The following sections describe these configuration options and offers recommendations for settings.

Mobile push notifications

Mattermost can use mobile push notifications to notify users of new messages and activity. These notifications require a server component to be deployed to send the notifications to the mobile devices. By default, Mattermost will use the public push notification service which is not available in an air-gapped environment. We recommend disabling push notifications in System Console > Environment > Push Notification Server.

Email

Unless you have setup an internal air-gapped email service, we recommend disabling email invitations and email verification from System Console > Authentication > Signup.

GIF picker

The GIF picker relies on a third-party service which has a dependency on external internet access. You can disable it in System Console > Integrations > GIF.

Notices

In-product notices require internet access to periodcally inform administrators and end users of new product improvements, features, and releases. You can disable notices in System Console > Site Configuration > Notices.

Telemetry

To avoid log errors we recommend disabling Telemetry-related features, including the security update check, and error and diagnostics reporting features.

Frequently Asked Questions

What if my air-gapped environment doesn’t have a private container registry or package mirror?

A private container registry securely stores the Docker images necessary for air-gapped deployments, ensuring compliance with data isolation requirements. Similarly, a private package mirror stores operating system packages necessary for air-gapped deployments in Ubuntu or RHEL/CentOS Linux environments. Setting up a local registry or mirror is a critical step in deploying Mattermost to ensure all images, dependencies and packages are available to you in the air-gapped environment. The steps below outline the process required to setup a local registry or mirror, depending on the deployment method you are using. These steps are a rough guide, and can be supplemented with online resources depending on your specific deployment needs.

(Ubuntu) Set up a private Debian package mirror

We will use Aptly to create a local mirror, although you can also use other options such as debmirror.

  1. Install Aptly (on an internet-connected machine):

    apt-get update
apt-get install aptly gnupg

  2. Create GPG key for signing packages:

    gpg --gen-key

  3. Create a mirror configuration:

    aptly mirror create -architectures=amd64 debian-bullseye http://deb.debian.org/debian bullseye main contrib non-free

  4. Update the mirror to download packages:

    aptly mirror update debian-bullseye

  5. Create and publish a snapshot:

    aptly snapshot create debian-bullseye-$(date +%Y%m%d) from mirror debian-bullseye
aptly publish snapshot debian-bullseye-$(date +%Y%m%d)

  6. Serve the repository:

    aptly serve

  7. Client configuration: Configure apt to use your local mirror:

    cat > /etc/apt/sources.list << EOF
deb http://mirror.example.com/debian bullseye main contrib non-free
EOF

(RHEL/CentOS) Set up a private RHEL package mirror

We will use reprosync for a local mirror.

  1. Install required tools (on an internet-connected RHEL system):

    yum install yum-utils createrepo

  2. Download packages:

    mkdir -p /var/www/html/repos/rhel8
reposync -p /var/www/html/repos/rhel8 --download-metadata --repo=rhel-8-for-x86_64-baseos-rpms
reposync -p /var/www/html/repos/rhel8 --download-metadata --repo=rhel-8-for-x86_64-appstream-rpms

  3. Create repository metadata:

    createrepo /var/www/html/repos/rhel8/rhel-8-for-x86_64-baseos-rpms
createrepo /var/www/html/repos/rhel8/rhel-8-for-x86_64-appstream-rpms

  4. Set up a web server:

    yum install httpd
systemctl enable httpd
systemctl start httpd

  5. Client configuration: Disable existing repositories:

    cd /etc/yum.repos.d/
mkdir backup
mv *.repo backup/

  6. Client configuration: Create new repository files:

    cat > /etc/yum.repos.d/local-baseos.repo << EOF
[local-baseos]
name=Red Hat Enterprise Linux 8 BaseOS
baseurl=http://mirror.example.com/repos/rhel8/rhel-8-for-x86_64-baseos-rpms
enabled=1
gpgcheck=0
EOF

cat > /etc/yum.repos.d/local-appstream.repo << EOF
[local-appstream]
name=Red Hat Enterprise Linux 8 AppStream
baseurl=http://mirror.example.com/repos/rhel8/rhel-8-for-x86_64-appstream-rpms
enabled=1
gpgcheck=0
EOF

  7. Client configuration: Clear cache and test:

    yum clean all
yum repolist

Set up a self-hosted private container registry

  1. Install Docker Registry:

    docker run -d -p 5000:5000 --restart=always --name registry registry:2

  2. Configure persistent storage:

    docker run -d -p 5000:5000 --restart=always --name registry \
-v /mnt/registry:/var/lib/registry \
registry:2

  3. Add TLS security (recommended):

    1. Generate self-signed certificates:

      mkdir -p certs
openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key \
-x509 -days 365 -out certs/domain.crt

    2. Run the registry with TLS:

      docker run -d -p 5000:5000 --restart=always --name registry \
-v /mnt/registry:/var/lib/registry \
-v $(pwd)/certs:/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
registry:2

Configure Kubernetes to use private image registries

When using Kubernetes in an air-gapped environment, you need to configure it to use your private registry.

  1. Create a kubernetes secret for registry authentication:

    kubectl create secret docker-registry regcred \
--docker-server=registry.example.com:5000 \
--docker-username=your_username \
--docker-password=your_password \
--docker-email=your_email@example.com

  2. Reference the secret in pod specifications:

    apiVersion: v1
kind: Pod
metadata:
  name: mattermost-pod
spec:
  containers:
  - name: mattermost
    image: registry.example.com:5000/mattermost/mattermost-enterprise-edition:latest
  imagePullSecrets:
  - name: regcred

  3. For Helm deployments, specify the registry in values.yaml:

    image:
  repository: registry.example.com:5000/mattermost/mattermost-enterprise-edition
  tag: latest
  pullPolicy: IfNotPresent

imagePullSecrets:
  - name: regcred

Set up a self-hosted private container registry

  1. Install Docker Registry:

    docker run -d -p 5000:5000 --restart=always --name registry registry:2

  2. Configure persistent storage:

    docker run -d -p 5000:5000 --restart=always --name registry \
-v /mnt/registry:/var/lib/registry \
registry:2

  3. Add TLS security (recommended):

    1. Generate self-signed certificates:

      mkdir -p certs
openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key \
-x509 -days 365 -out certs/domain.crt

    2. Run the registry with TLS:

      docker run -d -p 5000:5000 --restart=always --name registry \
-v /mnt/registry:/var/lib/registry \
-v $(pwd)/certs:/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
registry:2

Populate your private registry

Ensure the required images from the Bill of Materials are downloaded and pushed to the private registry.

Configure Docker to use private image registries

Configure Docker on all hosts to trust and use your private registry.

  1. Add your registry to Docker’s trusted registries:

    Edit or create /etc/docker/daemon.json:

    {
  "insecure-registries": ["registry.example.com:5000"]
}

    For registries using self-signed certificates:

    mkdir -p /etc/docker/certs.d/registry.example.com:5000
cp domain.crt /etc/docker/certs.d/registry.example.com:5000/ca.crt

  2. Restart Docker daemon:

    systemctl restart docker

  3. Test the configuration:

    docker pull registry.example.com:5000/mattermost/mattermost-enterprise-edition:latest