When a service works with a database its core is tied with the database schema. Usually, that means that service models are represented in the database. Usually as tables, but not necessary.

Anyway, from that comes a requirement to update database schema when the core of service is changed. If you version your service then you should version your database schema as well. For this purpose, the Liquibase is a good tool(or at least one that I like). In this example I use the Postgres.

Workflow

Whatever path you choose your workflow will be the same:

  1. Update database schema
  2. Update service

Strategies

How:

  1. Apply changes manual
  2. Use initContainers
  3. Use separate Job

Setup & Requirements

If you want to try this you’ll need to install liquibase cli, k3d and kustomize(if you have kubectl you already have it). Get the code.

First start k3d cluster:

k3d cluster create -p "9999:80@loadbalancer" playground

This will start one node Kubernetes cluster with loadbalancer. Localhost port 9999 is mapped to the loadbalancer port 80, see details here.

Now we can deploy the Postgres database:

kubectl apply -k base/postgres

This will create a Deployment Postgres in the namespace Postgres, with schema docker, user root, and password password.

Setup Echo application:

kubectl -n app apply -k base/app

This application has two endpoints:

  • /echo, repeats message
  • /counter, for increasing and retrieving counter

I am using httpie, but cult will be good as well. You can check if the application is working with:

http POST localhost:9999/echo msg="hello ddd"

You should get:

HTTP/1.1 200 OK
Content-Length: 21
Content-Type: text/plain; charset=utf-8
Date: Sun, 13 Feb 2022 15:34:00 GMT

{
    "echo": "hello ddd"
}

Try:

http POST localhost:9999/counter

Result is:

HTTP/1.1 200 OK
Content-Length: 51
Content-Type: text/plain; charset=utf-8
Date: Sun, 13 Feb 2022 15:34:57 GMT

{
    "msg": "pq: relation \"counters\" does not exist"
}

That means the database is there, the application is connected but the table is missing.

Manual changes

I would not recommend this approach.
In this case, do the port forward:

kubectl -n postgres port-forward svc/postgres 5432

If you have installed Liquibase CLI, and checkout the code:

cd liquibase
liquibase update

####################################################
##   _     _             _ _                      ##
##  | |   (_)           (_) |                     ##
##  | |    _  __ _ _   _ _| |__   __ _ ___  ___   ##
##  | |   | |/ _` | | | | | '_ \ / _` / __|/ _ \  ##
##  | |___| | (_| | |_| | | |_) | (_| \__ \  __/  ##
##  \_____/_|\__, |\__,_|_|_.__/ \__,_|___/\___|  ##
##              | |                               ##
##              |_|                               ##
##                                                ##
##  Get documentation at docs.liquibase.com       ##
##  Get certified courses at learn.liquibase.com  ##
##  Free schema change activity reports at        ##
##      https://hub.liquibase.com                 ##
##                                                ##
####################################################
Starting Liquibase at 16:46:06 (version 4.7.1 #1239 built at 2022-01-20 20:31+0000)
Liquibase Version: 4.7.1
Liquibase Community 4.7.1 by Liquibase
Do you want to see this operation's report in Liquibase Hub, which improves team collaboration?
If so, enter your email. If not, enter [N] to no longer be prompted, or [S] to skip for now, but ask again next time [S]:
n
No operations will be reported. Simply add a liquibase.hub.apiKey setting to generate free deployment reports. Learn more at https://hub.liquibase.com
* Updated properties file liquibase.properties to set liquibase.hub.mode=off
Running Changeset: changelog.xml::init::rnemet
Liquibase command 'update' was executed successfully.

Now validate:

❯ http POST localhost:9999/counter

HTTP/1.1 200 OK
Content-Length: 13
Content-Type: text/plain; charset=utf-8
Date: Sun, 13 Feb 2022 15:51:58 GMT

{
    "msg": "OK"
}

or:

❯ http GET localhost:9999/counter

HTTP/1.1 200 OK
Content-Length: 14
Content-Type: text/plain; charset=utf-8
Date: Sun, 13 Feb 2022 15:52:08 GMT

{
    "counter": 1
}

This approach works but is not visible, not automated, not versioned, and error-prone.

Init containers

In the Deployment definition, you can specify init containers. That will be a set of containers that have to be successfully executed before other containers are started. We can put there a liquibase container. It will be run before the Echo application:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: echo-app
  name: echo-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: echo-app
  template:
    metadata:
      labels:
        app: echo-app
    spec:
      containers:
      - env:
        - name: DB_HOST
          value: postgres.postgres.svc
        - name: APP_PORT
          value: "9999"
        image: rnemet/echo:latest
        name: echo
        ports:
        - containerPort: 9999
          name: http
        volumeMounts:
        - mountPath: /app/configs
          name: app-cfg
          readOnly: true
      initContainers:
      - command:
        - liquibase
        - --defaults-file=/liquibase/changelog/liquibase.properties
        - update
        image: liquibase/liquibase:4.7.0
        name: liquibase
        volumeMounts:
        - mountPath: /liquibase/changelog
          name: app-db
          readOnly: true
      volumes:
      - name: app-db
        secret:
          secretName: app-db
      - name: app-cfg
        secret:
          secretName: app-cfg

To try this scenario delete cluster:

k3d cluster delete playground

Repeat steps from the manual procedure for creating a cluster, deploying a Postgres and Echo app. Validate that tables are not created.

Deploy secret which will contain liquibase.properties:

kubectl -n app apply -k base/liquibase

Then apply changes where init containers are added:

kubectl -n app apply -k overlays/app

wait for the deployment to finish:

kubectl -n app rollout status deployment echo-app

This approach is much better than manual deployment. The problem with this approach is that a database schema will be run for each Pod. Not only for rollout but as well when HPA adds a new Pod. The schema will be updated only the first time, liquibase is smart enough to recognize what changes are applied. Only one liquibase instance can make the change. When the liquibase instance makes changes, it sets the lock on the database, which prevents other liquibase instances to modify the database. Once it is done lock is released. The next one takes database, set lock, and check if changes are needed. This is done for each Pod. This means if you want to update two or more Pods at once, it is not possible. One Pod at the time will be updated because each liquibase container has access to the database. But only one can have a lock at the time.

Job

If we decouple the liquibase, and run it as a separate Job before we deploy the Echo application:

apiVersion: batch/v1
kind: Job
metadata:
  name: liquibase
spec:
  template:
    spec:
      containers:
      - command:
        - liquibase
        - --defaults-file=/liquibase/changelog/liquibase.properties
        - update
        image: liquibase/liquibase:4.7.0
        name: liquibase
        volumeMounts:
        - mountPath: /liquibase/changelog
          name: app-db
          readOnly: true
      restartPolicy: Never
      volumes:
      - name: app-db
        secret:
          secretName: app-db

To try this scenario delete and recreate the cluster. Add Postgres and echo app. Check if the counter endpoint is working. It should not. Then deploy secret:

kubectl -n app apply -k base/liquibase

This is the same Secret required for init containers. Then deploy Job:

kubectl -n app apply -k overlays/liquibase

With Kubernetes Job database changes are decoupled. They run only once. Ci/cd pipeline can detect if there are changes to the database and run them before deploying the application. Changes are applied separately from the code. If there are no code changes only the database will be updated.

Resources