Automating TeamCity installation in Kubernetes on AWS

TeamCity is a great product, but while the normal method of configuring it through the UI is relatively easy, it's also not the best option in these days of automation and "everything-as-code". In this post I'll explore just how automated a TeamCity installation can be made.

What you'll need

  • A Kubernetes cluster - if you don't have one already then try kops
  • An empty database schema and the credentials to access it
  • Helm - the Kubernetes package manager
  • A GitHub repository to store your TeamCity Kotlin DSL in and a private deploy-key with write access to this repo.

Automation areas

There are basically 4 areas that need to be looked at:

  • Instance/container provisioning
  • The TeamCity server and agent software installation
  • The TeamCity server configuration settings (plugins, users, groups)
  • The project and build configuration

The first two are quite easy. We'll use Helm to spin up both server and agents, but it's also easily archivable with something like Cloudformation and Chef cookbooks or Ansible roles if that is more to your liking.

The third step and fourth step is where it kind of falls apart. The server configuration settings are stored in a combination of xml files and the database and while TeamCity has an option to pull project configuration in from git, for that to work you have to configure a VCS root (typically through the UI), including the secrets needed to access it and switch configuration synchronization on.

Assuming you have the prerequisites, lets begin.

First let's create a new namespace in Kubernetes for teamcity:
kubectl create namespace teamcity

Then add two k8n secrets containing the private key (here github-key.pem) and our database properties file.
kubectl -n teamcity create secret generic github-key-secret --from-file=github-key.pem

Create a file from this template:


Then store it as a secret:
kubectl -n teamcity create secret generic tc-db-properties --from-file=./

Remember to delete the file afterwards and see for the risks associated with storing sensitive information as k8n secrets.

The Helm Chart

In you're unfamiliar with Helm charts, it's a way to define, install and upgrade kubernetes based applications. This one is made up of several yaml files that decribe the various components using templating.

└── teamcity
    ├── Chart.yaml
    ├── templates
    │   ├── agent_deployment.yaml
    │   ├── _helpers.tpl
    │   ├── server_configmap.yaml
    │   ├── server_deployment.yaml
    │   ├── server_pvc.yaml
    │   └── server_service.yaml
    └── values.yaml

We'll be creating two build agents (agent_deployment.yaml), a TeamCity server (server_deployment.yaml), a service to expose it to the world (server_service.yaml) and a persistant volume chain (server_pvc.yaml).

Configuration is stored in values.yaml

# Default values for teamcity.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
  name: tc-server
  replicaCount: 1
    repository: jetbrains/teamcity-server
    tag: latest
    pullPolicy: IfNotPresent
    name: teamcity-server
    type: LoadBalancer
    servicePort: 8111
    enabled: true
    - ReadWriteOnce
    annotations: {}
    # If defined, PVC must be created manually before volume will be bound
    existingClaim: ""
    mountPath: /data/teamcity_server/datadir
    size: 5Gi
    storageClass: ""
    subPath: ""
  resources: {}
    #  cpu: 100m
    #  memory: 128Mi
    #  cpu: 100m
    #  memory: 128Mi
  name: tc-agent
  replicaCount: 2
    repository: jetbrains/teamcity-agent
    tag: latest
    pullPolicy: IfNotPresent

We specify a service to expose the resulting server with a type: LoadBalancer. After we're done, we'll be able to access the TeamCity service through an AWS ELB.

In addition to this, we have a server_configmap.yaml defining the script that is used to download the plugins specified above.

We'll use these in server_deployment.yaml:

apiVersion: extensions/v1beta1
kind: Deployment
  name: {{ template "fullname" . }}
    app: {{ template "name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    component: {{ }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
  replicas: {{ .Values.server.replicaCount }}
        app: {{ template "name" . }}
        component: {{ }}
        release: {{ .Release.Name }}
        checksum/config: {{ include (print $.Template.BasePath "/server_configmap.yaml") . | sha256sum }}
        - name: download-plugins
          image: busybox
            - "sh"
            - "/"
            - name: plugins
              mountPath: /plugins
            - name: config
              mountPath: /
        - name: {{ .Chart.Name }}
          image: "{{ .Values.server.image.repository }}:{{ .Values.server.image.tag }}"
          imagePullPolicy: {{ .Values.server.image.pullPolicy }}
            - name: TEAMCITY_SERVER_OPTS
              value: ""
            - containerPort: 8111
{{ toYaml .Values.server.resources | indent 12 }}
          - name: data
            mountPath: /data/teamcity_server/datadir
            readOnly: false
          - name: config
            mountPath: /data/teamcity_server/datadir/config/
            readOnly: true
          - name: plugins
            mountPath: /data/teamcity_server/datadir/plugins
            readOnly: false
          - name: secret-volume
            readOnly: false
            mountPath: /data/teamcity_server/secrets
      - name: secret-volume
          secretName: github-key-secret
      - name: plugins
        emptyDir: {}
      - name: config
          secretName: tc-db-properties
      - name: data
      {{- if .Values.server.persistentVolume.enabled }}
          claimName: {{ if .Values.server.persistentVolume.existingClaim }}{{ .Values.server.persistentVolume.existingClaim }}{{- else }}{{ template "fullname" . }}{{- end }}
      {{- else }}
        emptyDir: {}
      {{- end -}}

What we are doing here is first running a busybox container to download the plugins specified in values.yaml. Then the actual TeamCity container and the volumes mounted in it.

There are a few special volumeMounts specified here:

Our secret is exposed to the server:

- name: config
            mountPath: /data/teamcity_server/datadir/config/
            readOnly: true

The downloaded plugins are mounted and will be available in the TeamCity server

          - name: plugins
            mountPath: /data/teamcity_server/datadir/plugins
            readOnly: false

Finally the private part of the deploy key is mounted

          - name: secret-volume
            readOnly: false
            mountPath: /data/teamcity_server/secrets

Spinning it all up with Helm
helm install teamcity/ --namespace teamcity

You should see output similar to:

NAME:   idolized-squirrel
LAST DEPLOYED: Wed Aug  1 09:49:35 2018
NAMESPACE: teamcity

And quite a few resources in status <pending>

Wait a little bit for AWS to create your EBS volume and the loadbalancer, then run
kubectl -n teamcity get services -o wide

You should now see the external address of the loadbalancer:

NAME       TYPE           CLUSTER-IP      EXTERNAL-IP                                                              PORT(S)          AGE       SELECTOR
teamcity   LoadBalancer   8111:31735/TCP   42m       app=teamcity,component=tc-server,release=idolized-squirrel

Configure TeamCity to use project settings from VCS

Now we'll unfortunately have to do a few steps by hand

Paste the loadbalancer address into a browser (remember the port) and you should be presented with a screen asking you to accept the license and input an Administrator username and password

Having logged in as our new Administrator user, we'll configure TeamCity to use our predefined VCS root and stores it's project configuration there

Click Administration in the top right corner

Select the <Root Project> and click VCS Roots in the left menu.
Setup a new VCS root using a custom key and specify /data/teamcity_server/secrets/github-key.pem as private key path.

Having done that, select Show more and then Versioned settings in the left menu.

You should see a screen like below, but with Synchronization disabled, change that to enabled and configure it like shown


Click the Apply button and TeamCity will store the current settings in your git repo. I've used Kotlin in the example, but you can also store the settings as XML.

To add subprojects, clone this repo and edit the settings in your favorite IDE. For an introduction to using Kotlin to setup projects in TeamCity, see: however the documentation is severely lacking in examples and in reality you'll want to resort to making the change in the UI and then viewing the generated DSL (look for the "view DSL" button") to get familiar with the syntax.

As a minimum you'll probably also want to add some users and go to Agents in the top menu and authorize the two build agents we've installed.


Unfortunately that is about as far as TeamCity automation gets when it comes to installation. While you can edit the XML that defines the server configuration, it needs to be in sync with the data stored in the database. It's worth pointing out that the settings XML is so tied to the DB that you can't even install a new TeamCity server and reuse an existing database without also restoring a backup of the XML. Backups can be scheduled under Administration -> Backups

Does that mean you shouldn't automate it? Not at all. While the initial installation have a few manual steps, running TeamCity in a container and having as much as possible defined in code is still a great step up from a manually provisioned server and the real wins come when you start scripting your projects with kotlin and placing the code in a git repo instead of using the UI. More on that in a later article.