Mattermost Kubernetes Operator

What is an operator?

In Kubernetes, an operator is a set of product- or application-specific instructions packaged into its own program. It tells an application what to run, where to find it, and how to configure it. An operator can automate complex application deployment and operation activities, such as installation, configuration changes, software updates, failure recovery, and more.

Warning

If you used the Mattermost Kubernetes Operator prior to v1.12.x, or you’re still using the ClusterInstallation Custom Resource, check out this guide to learn how to migrate to thenew Mattermost resource.

Confirm resource requirements for production

Running Mattermost in Kubernetes requires different resources based on your total number of users. Minimum requirements are listed in the following table:

User Count	Node Count	Memory per Node	vCPU per Node
5,000	6	8 GB	4
10,000	8	16 GB	4
25,000	14	16 GB	4

These resources take into account all components required for Mattermost, including proxy, database, and file storage.
Requirements may be significantly lower when using an external database and filestore (recommended).
Resource requirements may vary depending on user usage and bot activity.
For larger installations, it may be beneficial to use nodes for the databases that have more memory and/or are optimized for memory.
For installations of more than 25,000 users please contact us for sizing guidelines.

Try out the Mattermost Operator

The MySQL operator and MinIO operator are a good way to try out the Mattermost Operator or develop it on a local cluster; however, it’s not recommended for production deployments.

Install the MySQL operator:

$ kubectl create ns mysql-operator
$ kubectl apply -n mysql-operator -f https://raw.githubusercontent.com/mattermost/mattermost-operator/master/docs/mysql-operator/mysql-operator.yaml

Install the MinIO operator:

$ kubectl create ns minio-operator
$ kubectl apply -n minio-operator -f https://raw.githubusercontent.com/mattermost/mattermost-operator/master/docs/minio-operator/minio-operator.yaml

Rolling upgrades

The Mattermost Kubernetes Operator supports rolling upgrades, so you can upgrade your Mattermost deployment with zero downtime. This process requires at least two replicas as a rolling upgrade cannot be performed if there is only one pod. Replicas are created when a user count is selected and exceeds 100.

New Mattermost releases are announced via our community server, as well as social media and email.

Perform a rolling upgrade

Log in to your Kubernetes instance.
Open the mattermost-installation.yaml manifest (the one created during installation).
Update the spec.version value to the new version.
Save the changes.

Apply the changes with kubectl:

$ kubectl apply -n mattermost -f [PATH_TO_MATTERMOST_MANIFEST]

The operator initiates a job in the Kubernetes cluster and once migration is complete the pods are restarted. If necessary, a database migration is also performed.

To view information about the running job, use

$ kubectl -n mattermost get jobs

At least one pod is available at all times and once all pods are restarted with the new version the upgrade is complete.

To view the status of the pods and to confirm their state, use

$ kubectl -n mattermost get pods

The STATUS of the pods should be running/ready, with an AGE of 10-15 seconds.

Restore an existing Mattermost MySQL database

The Mattermost Operator can be used in a backup and restore scenario to apply an existing Mattermost MySQL database to a new Mattermost installation, in its own namespace. This can also be helpful in the event that you need to revert your Mattermost instance’s database to the most recent backup point, on your existing installation. In both cases, you will need a backup of your database.

The steps you follow to create and upload your backup depends on the provider you’re using and your use case. It’s recommended that you consult the relevant documentation or, if your deployment is managed in a different way, consult your Administrator.

It is important to note that this process requires the creation of a new Mattermost installation - editing the existing .yaml files is not recommended and can result in data loss.

The process described below needs to be completed prior to proceeding with the Mattermost deployment.

Create a backup of your database (e.g. using mysqldump).
Deploy a new server (e.g. an AWS instance).
Install a backup program and back up the database on the new server/instance.
Upload the backed up database to your cloud storage provider (e.g. Amazon S3).
Create a secret.yaml file:

Open a text editor and create a text file containing your credentials which will be used to access the uploaded database.

Save the file as secret.yaml. The example below is for AWS/S3, where name indicates the name of this manifest referenced in the install manifest.
apiVersion: v1
kind: Secret
metadata:
  name: test-restore
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: XXXXXXXXXXXX
  AWS_SECRET_ACCESS_KEY: XXXXXXXXXXXX/XXXXXXXXXXXX
  AWS_REGION: us-east-1
  S3_PROVIDER: AWS

Create a Mattermost cluster installation manifest by opening a text editor and creating a text file with the following details. The Mattermost installation manifest contains fields which must be edited in line with your configuration and environment requirements. Save the file as mattermost-installation.yaml:

apiVersion: mattermost.com/v1alpha1
kind: ClusterInstallation
metadata:
  name: mm-example-full
spec:
  size: ""
  ingressName: example.mattermost-example.com
  ingressAnnotations:
    kubernetes.io/ingress.class: nginx
  version: 5.14.0
  mattermostLicenseSecret: ""
  database:
    storageSize: 50Gi
    minio:
    storageSize: 50Gi
  elasticSearch:
    host: ""
    username: ""
    password: ""

Create a restore manifest by opening a text editor and creating a text file with the following details. Save the file as restore.yaml:

apiVersion: mattermost.com/v1alpha1
kind: MattermostRestoreDB
metadata:
  name: example-mattermostrestoredb
spec:
  initBucketURL: s3://my-sample/my-backup.gz
  mattermostClusterName: example-clusterinstallation
  mattermostDBName: mattermostdb
  mattermostDBPassword: supersecure
  mattermostDBUser: ""
  restoreSecret: ""

Parameters

mattermostClusterName: The ClusterInstallation file name.
restoreSecret: The location of the backup file.
mattermostDBPassword: The password used to access the database.
mattermostDBUser: The username required to access the database.
initBucketURL: The URL of the storage instance/server where the backed up DB is stored.

To initiate deployment, apply the file and specify the path where the newly-created files have been saved:

$ kubectl create ns mattermost
$ kubectl apply -n mattermost -f /path/to/secret.yaml
$ kubectl apply -n mattermost -f /path/to/mattermost-installation.yaml
$ kubectl apply -n mattermost -f /path/to/restore.yaml
The deployment process can be monitored in the Kubernetes user interface. If errors or issues are experienced, review the Mattermost, Operator, and MySQL logs for guidance including error messages. If remediation is not successful, contact Mattermost customer support for assistance.

Once complete, access your Mattermost instance and confirm that the database has been restored.

Migrate From Helm Chart

If you’ve installed Mattermost as helm chart deployment, you can easily move to an operator-installed deployment by following the steps below.

Create a namespace dedicated to the operator.

$ kubectl create ns mattermost-operator

Save the secrets from the current installation. If you have extra secrets mounted as a volume, save them as well. To begin, set the NAMESPACE environment variable to the namespace your Mattermost instance is running in:

$ export NAMESPACE=mattermost
$ kubectl get secrets -n $NAMESPACE mattermost-db-secret -o yaml > mattermost-db-secret.yaml
$ kubectl get secrets -n $NAMESPACE mattermost-license-secret -o yaml > mattermost-license-secret.yaml
$ kubectl get secrets -n $NAMESPACE cert -o yaml > cert.yaml

Change the database secret to match the operator. The helm installation used mattermost.dbsecret while the operator uses DB_CONNECTION_STRING.

$ vim mattermost-db-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: mattermost-db-secret
data:
  DB_CONNECTION_STRING: [YOUR_CONNECTION_STRING]

Create an S3 Secret. Mattermost operator uses secret while helm uses environment variables.

apiVersion: v1
metadata:
  name: s3-secret
kind: Secret
stringData:
  accesskey: [S3_ACCESS_KEY]
  secretkey: [S3_ACCESS_KEY]

Scale the current installation to 0 and deploy the operator.

$ kubectl apply -n mattermost-operator -f https://raw.githubusercontent.com/mattermost/mattermost-operator/master/docs/mattermost-operator/mattermost-operator.yaml
$ kubectl scale deployment -n $NAMESPACE mattermost-deployment --replicas=0

Apply the new secrets in the new namespace.

$ kubectl apply -f mattermost-db-secret.yaml -n $NEW_NAMESPACE
$ kubectl apply -f mattermost-license-secret.yaml -n $NEW_NAMESPACE
$ kubectl apply -f cert.yaml -n $NEW_NAMESPACE
$ kubectl apply -f s3-secret -n $NEW_NAMESPACE

Create a Mattermost installation in the same namespace as the new secrets.

apiVersion: installation.mattermost.com/v1beta1
kind: Mattermost
metadata:
  name: mm-example
spec:
  image: mattermost/mattermost-enterprise-edition
  imagePullPolicy: IfNotPresent
  version: 6.0.0
  size: 5000users
  ingress:
    enabled: true
    host: example.mattermost-example.com
    tlsSecret: "cert"
  licenseSecret: "mattermost-db-secret"
  database:
    external:
      secret: "mattermost-db-secret"
  fileStore:
    external:
      url: s3.amazonaws.com
      bucket: my-s3-bucket
      secret: s3-secret
  elasticSearch: {}
  volumeMounts:
  - name: certificates
    mountPath: /etc/ssl/certs/mycert.pem
    subPath: mycert.pem
  volumes:
  - name: certificates
    secret:
      defaultMode: 420
      secretName: cert

Align the manifest to your needs, then save the file as mattermost-installation.yaml. See the documentation for details about supported fields.: While recommended filenames are provided, your naming conventions may differ.

Apply the new manifest in the relevant namespace.

$ kubectl apply -n $NEW_NAMESPACE -f mattermost-installation.yaml

The deployment process can be monitored in the Kubernetes user interface or using the command line by running:

$ kubectl -n $NEW_NAMESPACE get mm -w

The installation should be deployed successfully when the Custom Resource reaches a stable state.

Configure DNS and use Mattermost.

When the deployment is complete, obtain the hostname or IP address of your Mattermost deployment using the following command:
$ kubectl -n $NEW_NAMESPACE get ingress
Copy the resulting hostname or IP address from the ADDRESS column, open your browser, and connect to Mattermost.

Use your domain registration service to create a canonical name or IP address record for the ingress.host in your manifest, pointing to the address you just copied. For example, on AWS you would do this within a hosted zone in Route53.

Navigate to the ingress.host URL in your browser and use Mattermost.

If you just want to try it out on your local machine without configuring the domain, run:
$ kubectl -n $NEW_NAMESPACE port-forward svc/[YOUR_MATTERMOST_NAME] 8065:8065
Then navigate to http://localhost:8065.

Frequently Asked Questions

What’s the difference between the Mattermost Operator and Helm Charts?

The Mattermost Operator is a self-contained set of application/product-specific instructions that runs inside Kubernetes and facilitates application management and deployment.

Helm is a tool used to deploy Kubernetes manifests to a cluster, but does not facilitate application management.

Does the Mattermost Operator replace the Helm Chart?

No. Helm is a good mechanism to deploy operators and other applications but doesn’t facilitate application management.

Is minIO the only available storage option?

No, other options and operators can be added.

Do I have to install a separate SQL server to use the Mattermost Operator?

No, you can use the operator on top of your existing deployment without setting up another database. We will shortly be providing steps for this configuration.

Can you use blue-green deployments with different database schemas?

Currently this is not supported as it introduces the possibility of missing a data entry in the database.

Are environment variables supported?

Yes. However, config.json file settings will be overridden if the $MM_SQLSETTINGS_DATASOURCE environment variable is set. See the Environment Variables configuration settings documentation for details.

Issues configuring Login with SAML on Kubernetes

For some SAML authentication configurations, 502 status code response can appear during login attempts due to requests being too large. This can be caused by the default proxy-buffer-size setting for NGINX Ingress being too low. To fix this issue, configure an appropriate buffer size (8k or 16k should be sufficient for most cases) with NGINX annotation by adding it to the Mattermost manifest under spec.ingressAnnotations:

...
spec:
...
  ingress:
  ...
    annotations:
      nginx.ingress.kubernetes.io/proxy-buffer-size: 16k
...

Use caution when changing the buffer size as it may slightly impact NGINX performance. Exact values are machine-dependent.