Manage and monitor AlloyDB Omni

Single-server

AlloyDB Omni logs its activity in two locations:

• AlloyDB Omni logs database activity at data/log/postgres, relative to the directory that you defined in your dataplane.conf configuration file.

  You can customize the name and format of this log file through the various log_* directives defined in /var/alloydb/config/postgresql.conf. For more information, see Error Reporting and Logging.

• AlloyDB Omni logs its installation, startup, and shutdown activity at /var/alloydb/logs/alloydb.log.

To check the immediate running status of your server, see Check the status of AlloyDB Omni.

Kubernetes

Find your database cluster log files

You can find postgresql.audit and postgresql.log files on the file system of the database pod. To access these files, follow these steps:

1. Define an environment variable containing the name of the database pod.

   export DB_POD=`kubectl get pod -l alloydbomni.internal.dbadmin.goog/dbcluster=DB_CLUSTER_NAME,alloydbomni.internal.dbadmin.goog/task-type=database -o jsonpath='{.items[0].metadata.name}'`
   

   Replace DB_CLUSTER_NAME with the name of your database cluster. It's the same database cluster name you declared when you created it.

2. Run a shell on the database pod as root.

   kubectl exec ${DB_POD} -it -- /bin/bash
   

3. Find the log files in the /obs/diagnostic/ directory:

   • /obs/diagnostic/postgresql.audit
   • /obs/diagnostic/postgresql.log

List monitoring services

kubectl get svc -n NAMESPACE | grep monitoring

al-1060-dbc-monitoring-system   ClusterIP   10.0.15.227   <none>        9187/TCP   7d20h
al-3de6-dbc-monitoring-system   ClusterIP   10.0.5.205    <none>        9187/TCP   7d19h
al-4bc0-dbc-monitoring-system   ClusterIP   10.0.15.92    <none>        9187/TCP   7d19h

kubectl get svc -n NAMESPACE | grep monitoring

al-2953-dbcluster-foo7-monitoring-db           ClusterIP   10.36.3.243    <none>        9187/TCP   44m
al-2953-dbcluster-foo7-monitoring-system       ClusterIP   10.36.7.72     <none>        9187/TCP   44m

View Prometheus metrics from the command line

The port 9187 is named as metricsalloydbomni for all monitoring services.

1. Set up port forwarding from your local environment to the monitoring service.

   kubectl port-forward service/MONITORING_SERVICE -n NAMESPACE MONITORING_METRICS_PORT:metricsalloydbomni
   

   Replace the following:

   • MONITORING_SERVICE: The name of the monitoring service that you want to forward—for example, al-1060-dbc-monitoring-system.

   • NAMESPACE: The namespace where your cluster belongs.

   • MONITORING_METRICS_PORT: A local available TCP port.

   The following response shows that the services are being forwarded.

   Forwarding from 127.0.0.1:9187 -> 9187
   Forwarding from [::1]:9187 -> 9187
   

2. While the previous command runs, you can access monitoring metrics through HTTP on the port that you specified. For example, you can use curl to see all of the metrics as plain text:

   curl http://localhost:MONITORING_METRICS_PORT/metrics
   

View metrics using the Prometheus API

The alloydbomni.internal.dbadmin.goog/task-type label key and the metricsalloydbomni port is available as a default for all monitoring services in AlloyDB Omni. You can use them together with a single serviceMonitor custom resource to select all the services for all namespaces in your database cluster.

For more information about using the Prometheus API, see the Prometheus Operator documentation.

The following is an example spec field of the serviceMonitor custom resource that includes the alloydbomni.internal.dbadmin.gdc.goog/task-type label key and metricsalloydbomni port. The serviceMonitor custom resource monitors and collects all the kubernetes services in all namespaces

For more information about the complete ServiceMonitor definition, see the ServiceMonitor custom resource definition .

    spec:
      selector:
        matchLabels:
          alloydbomni.internal.dbadmin.goog/task-type: monitoring
      namespaceSelector:
        any: true
      endpoints:
        - port: metricsalloydbomni

    spec:
      selector:
        matchExpressions:
        - key: alloydbomni.internal.dbadmin.gdc.goog/task-type
          operator: Exists
          values: []
      namespaceSelector:
        any: true
      endpoints:
      - port: metricsalloydbomni

Single-server

These instructions apply only to AlloyDB Omni version 15.2.0 and later.

Before you begin

Your machine must have version 1.2 or later of the AlloyDB Omni CLI installed.

Perform the upgrade

To upgrade your AlloyDB Omni installation, run the following command:

sudo alloydb database-server upgrade

Kubernetes

The steps that you take to upgrade AlloyDB Omni in Kubernetes depend upon the version of AlloyDB Omni that you are running, and the version that you are upgrading to.

Determine your current version numbers

To check the version of AlloyDB Omni used by your database cluster, run this command:

kubectl get dbclusters.alloydbomni.dbadmin.goog DB_CLUSTER_NAME -n NAMESPACE -o jsonpath='{.status.primary.currentDatabaseVersion}'

Replace the following:

• DB_CLUSTER_NAME: the name of your database cluster. It's the same database cluster name you declared when you created it.

• NAMESPACE: the Kubernetes namespace of your database cluster.

If you are running version 1.0.0 or later of the AlloyDB Omni Operator, then this command prints the version of AlloyDB Omni in use by your database cluster.

To check the version of the AlloyDB Omni Operator installed on your Kubernetes cluster, run this command:

kubectl get dbclusters.alloydbomni.dbadmin.goog DB_CLUSTER_NAME -n NAMESPACE -o jsonpath='{.status.primary.currentControlPlaneAgentsVersion}'

If you are running version 1.0.0 or later of the AlloyDB Omni Operator, then this command prints the version number of the AlloyDB Omni Operator running on your Kubernetes cluster.

If you are running a version of AlloyDB Omni Operator earlier than 1.0.0, then you can't perform an in-place upgrade of your database cluster or of the AlloyDB Omni Operator. Instead, you must follow the instructions in Upgrade from a pre-1.0.0 AlloyDB Omni Operator.

Otherwise, continue to the next section.

Determine your target version numbers

If you are running a version of AlloyDB Omni Operator 1.0.0 or later, then your next steps depend upon the version of AlloyDB Omni that you want to upgrade to. This, in turn, requires understanding of the AlloyDB Omni version number.

The AlloyDB Omni version number has three parts:

• The major version number of its PostgreSQL compatibility
• The minor version number of its PostgreSQL compatibility
• The patch version number of this AlloyDB Omni release

For example, AlloyDB Omni version 15.5.2 is patch version 2 of the AlloyDB Omni that supports PostgreSQL version 15.5.

If you want to upgrade to a version of AlloyDB Omni that supports a newer version of PostgreSQL, then you must upgrade the AlloyDB Omni Operator itself, alongside your database cluster. Each set of AlloyDB Omni releases that support a particular PostgreSQL minor version has its own associated AlloyDB Omni Operator version number, which you can find in the release note for the AlloyDB Omni version.

If you want to upgrade only to a newer patch version of AlloyDB Omni, then you can upgrade only your database cluster, with no need to upgrade the AlloyDB Omni Operator itself. You can skip to the instructions in Upgrade your database cluster.

Otherwise, continue to the next section.

Upgrade the AlloyDB Omni Operator

To upgrade the AlloyDB Omni Operator, do the following:

1. Define necessary environment variables:

   export GCS_BUCKET=alloydb-omni-operator
   export OPERATOR_VERSION=OPERATOR_VERSION
   export HELM_PATH=$OPERATOR_VERSION/alloydbomni-operator-$OPERATOR_VERSION.tgz
   

   Replace OPERATOR_VERSION with the version of the AlloyDB Omni Operator that you are upgrading to—for example, 1.1.0.

2. Download the newer AlloyDB Omni Operator:

   gsutil cp -r gs://$GCS_BUCKET/$HELM_PATH ./
   tar -xvzf alloydbomni-operator-${OPERATOR_VERSION}.tgz
   

3. Apply the newer AlloyDB Omni Operator custom resource definitions:

   kubectl apply -f alloydbomni-operator/crds
   

4. Upgrade the AlloyDB Omni Operator Helm chart:

   helm upgrade alloydbomni-operator alloydbomni-operator-${OPERATOR_VERSION}.tgz \
   --namespace alloydb-omni-system \
   --atomic \
   --timeout 5m
   

To follow up on your AlloyDB Omni Operator upgrade by upgrading both your Kubernetes manifest and your database cluster, follow the instructions in the next section immediately after completing the previous steps.

Upgrade your database cluster

To upgrade your database cluster, update the following fields in the spec section of the manifest that defines it:

• Set databaseVersion to the full version number of AlloyDB Omni that you want to upgrade this database cluster to.

• If you have also upgraded AlloyDB Omni Operator, then set controlPlaneAgentsVersion to the full version number of the AlloyDB Omni Operator that you have upgraded.

If you are upgrading only the patch version of AlloyDB Omni—for example, updating databaseVersion from 15.5.1 to 15.5.2—then this step is all that you need to accomplish.

If you are upgrading the minor version of PostgreSQL compatibility—for example, updating databaseVersion from 15.4.1 to 15.5.2‐then you must also update controlPlaneAgentsVersion. In that case, make sure that you have followed the extra steps listed in Upgrade the AlloyDB Omni Operator.

As an example, the following manifest excerpt defines a database cluster running AlloyDB Omni Operator version 15.5.2, with AlloyDB Omni Operator version 1.0.0:

apiVersion: alloydbomni.dbadmin.goog/v1 
kind: DBCluster
metadata:
  name: dbcluster-sample
spec:
  databaseVersion: 15.5.2
  controlPlaneAgentsVersion: 1.0.0

In this example, to upgrade the database cluster to run AlloyDB Omni version 15.5.3, change databaseVersion: 15.5.2 to databaseVersion: 15.5.3.

Upgrade from a pre-1.0.0 AlloyDB Omni Operator

If you are running a version of AlloyDB Omni Operator earlier than 1.0.0, then upgrading a Kubernetes-based AlloyDB Omni installation involves uninstalling and then re-installing the AlloyDB Omni Operator after backing up your data. Follow these steps:

1. List all of your database clusters:

   kubectl get dbclusters.alloydbomni.dbadmin.goog --all-namespaces
   

2. For each database cluster, use the pg_dumpall command to export all of its data.

3. Uninstall the AlloyDB Omni Operator. This includes deleting all of your database clusters.

4. Reinstall the AlloyDB Omni Operator. You can use the same commands that you used to install the previous version of the AlloyDB Omni Operator, with no need to specify a new version number.

5. Recreate your database clusters. You can adapt the same manifest files that you used when previously creating your database clusters. You might need to update the files to reflect API changes that version 1.0.0 of the AlloyDB Omni Operator introduced, such as the databaseVersion attribute replacing the older version attribute.

6. Use pg_restore or the \i command in psql to import your previously exported data into the recreated clusters.

Single-server

To uninstall AlloyDB Omni, run the following command:

sudo alloydb database-server uninstall

Your data directory remains on your file system after you uninstall AlloyDB Omni. You can move, archive, or delete this directory, depending on whether and how you want to preserve your data after uninstalling AlloyDB Omni.

Kubernetes

Delete your database cluster

To delete your database cluster, set isDeleted to true in its manifest. You can accomplish this with the following command.

kubectl patch dbclusters.alloydbomni.dbadmin.goog DB_CLUSTER_NAME -p '{"spec":{"isDeleted":true}}' --type=merge

Replace DB_CLUSTER_NAME with the name of your database cluster. It's the same database cluster name you declared when you created it.

Uninstall the AlloyDB Omni Operator

To uninstall the AlloyDB Omni Kubernetes Operator from your Kubernetes cluster, take the following steps:

1. Delete all of your database clusters:

   for ns in $(kubectl get dbclusters.alloydbomni.dbadmin.goog --all-namespaces -o=jsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}'); do
   for cr in $(kubectl get dbclusters.alloydbomni.dbadmin.goog -n $ns -o=jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
   kubectl patch dbclusters.alloydbomni.dbadmin.goog $cr -n $ns --type=merge -p '{"spec":{"isDeleted":true}}'
   done
   done
   

2. Wait for the AlloyDB Omni Kubernetes Operator to delete all of your database clusters. Use the following command to check whether any database resources remain:

   kubectl get dbclusters.alloydbomni.dbadmin.goog --all-namespaces
   

3. Delete other resources that the AlloyDB Omni Kubernetes Operator created:

   kubectl delete failovers.alloydbomni.dbadmin.goog --all --all-namespaces
   kubectl delete restores.alloydbomni.dbadmin.goog --all --all-namespaces
   kubectl delete switchovers.alloydbomni.dbadmin.goog --all --all-namespaces
   

4. Uninstall the AlloyDB Omni Kubernetes Operator:

   helm uninstall alloydbomni-operator --namespace alloydb-omni-system
   

5. Clean up secrets, custom resource descriptions, and namespaces related to the AlloyDB Omni Kubernetes Operator:

   kubectl delete certificate -n alloydb-omni-system --all
   kubectl get secrets --all-namespaces -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,ANNOTATION:.metadata.annotations.cert-manager\.io/issuer-name | grep -E 'alloydbomni|dbs-al' | awk '{print $1 " " $2}' | xargs -n 2 kubectl delete secret -n
   kubectl delete crd -l alloydb-omni=true
   kubectl delete ns alloydb-omni-system