Building Multiple Cassandra Data Centers on Kubernetes using StatefulSets
Kubernetes is one of the popular container orchestration tools. It automates the deployment, scaling and management of containerized (usually docker) applications. It’s so flexible that it let you run your apps on-premises, hybrid, public/private cloud infrastructure (e.g. google cloud platform, GCP or Amazon Web Services, AWS). It’s developed and open-sourced by google based on their decade’s worth of experience building container-management.
Kubernetes is still under active development. Currently its latest release is 1.5.1. One of most important beta feature for this release is the StatefulSets (previously known for PetSets). StatefulSets wass developed due to the need to persist the data storage even if a pod die/restarted, new pod which replaces the old pod should use back the same persistent data storage. There is a unique identity between the pod and the persistent data that the kubernetes is aware of. This aid in the term of scaling the cassandra instance without the need to keep track of the storage attached to it.
For this post, we will try to build a multi data centers cluster of cassandra using both statefulsets and replication controls. The replications controls will build a cluster which only consists of a single node for ease of data backup purpose. So it’s not strictly required and you can use just StatefulSets.
Let set up the service + Replication control which will serve as seed nodes to bootstrap another Cassandra data center. We will change from replication controller example given by the kubernetes
cassandra-controller.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: cassandrabk
name: cassandrabk ## this name will be used as the seed ip in another cluster
spec:
clusterIP: None
ports:
- port: 9042
port: 7000 ## we added this for intra-node communication
selector:
app: cassandrabk
---
apiVersion: v1
kind: ReplicationController
metadata:
name: cassandrabk
# The labels will be applied automatically
# from the labels in the pod template, if not set
# labels:
# app: cassandra
spec:
replicas: 1
# The selector will be applied automatically
# from the labels in the pod template, if not set.
# selector:
# app: cassandra
template:
metadata:
labels:
app: cassandrabk
spec:
containers:
- command:
- /run.sh
resources:
limits:
cpu: 100m
memory: 768Mi
env:
- name: MAX_HEAP_SIZE
value: 256M
- name: HEAP_NEWSIZE
value: 100M
- name: CASSANDRA_SERVICE
## need to set this value as we are not using default "cassandra" for the end points to populate ip
value: cassandrabk
- name: CASSANDRA_SEED_PROVIDER
value: "io.k8s.cassandra.KubernetesSeedProvider"
- name: CASSANDRA_CLUSTER_NAME
value: "mycluster-demo"
- name: CASSANDRA_DC
value: "dc-bk"
- name: CASSANDRA_RACK
value: "rack01"
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
image: gcr.io/google-samples/cassandra:v11
name: cassandrabk
ports:
- containerPort: 7000
name: intra-node
- containerPort: 7001
name: tls-intra-node
- containerPort: 7199
name: jmx
- containerPort: 9042
name: cql
volumeMounts:
- name: cs-host-bk
mountPath: /cassandra_data
volumes:
- name: cs-host-bk
hostPath:
path: /home/cassandra/cs-bk
You may notice that we are setting the memory and cpu to a lower value as we try to put everything into a single node for this demo. You should use the recommended value. Let create the resources.
kubectl create -f cassandra-controller.yaml
Now, you may now start populating data into this new C* node.
Let set up the service + StatefulSets yaml file. Again, We will change from the sample given by the kubernetes. A few things to take note:
- we will first set the auto-bootstrap to “false”. Once the node has restored the data successfully, we will set it to “true”
- set the seed to service name to the one we just set up (i.e. cassandrabk)
cassandra-statefulsets.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: cassandra
name: cassandra
spec:
clusterIP: None
ports:
- port: 9042
port: 7000
selector:
app: cassandra
---
apiVersion: "apps/v1beta1"
kind: StatefulSet
metadata:
name: cassandra
spec:
serviceName: cassandra
replicas: 1
template:
metadata:
labels:
app: cassandra
spec:
containers:
- name: cassandra
image: gcr.io/google-samples/cassandra:v11
# imagePullPolicy: Always
ports:
- containerPort: 7000
name: intra-node
- containerPort: 7001
name: tls-intra-node
- containerPort: 7199
name: jmx
- containerPort: 9042
name: cql
resources:
limits:
cpu: 100m
memory: 768Mi
securityContext:
capabilities:
add:
- IPC_LOCK
env:
- name: MAX_HEAP_SIZE
value: 256M
- name: HEAP_NEWSIZE
value: 100M
- name: CASSANDRA_SEEDS
# value: "cassandra-0.cassandra.default.svc.cluster.local"
value: cassandrabk
- name: CASSANDRA_CLUSTER_NAME
value: "mycluster-demo"
- name: CASSANDRA_DC
value: "dc01"
- name: CASSANDRA_RACK
value: "rack01"
- name: CASSANDRA_AUTO_BOOTSTRAP
value: "false" # when insert into new center
# value: "true" # after data has been populated
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
readinessProbe:
exec:
command:
- /bin/bash
- -c
- /ready-probe.sh
initialDelaySeconds: 300
timeoutSeconds: 30
# These volume mounts are persistent. They are like inline claims,
# but not exactly because the names need to match exactly one of
# the stateful pod volumes.
volumeMounts:
- name: cassandra-data
mountPath: /cassandra_data
# These are converted to volume claims by the controller
# and mounted at the paths mentioned above.
# do not use these in production until ssd GCEPersistentDisk or other ssd pd
volumeClaimTemplates:
- metadata:
name: cassandra-data
annotations:
volume.alpha.kubernetes.io/storage-class: anything
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1G
Once the StatefulSets cassandra data center is live, we can create the database schema for it.
CREATE KEYSPACE mydatabase WITH replication = {'class': 'NetworkTopologyStrategy', 'dc01': 1, 'dc-bk': 1}
Remember to do the same for the cassandrabk cluster.
You can check the
nodetool status
to make sure you see both data centers can discover each other.
root@cassandrabk-bpqct:/# nodetool status
Datacenter: dc-bk
============================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 10.0.2.10 241.81 KiB 32 100.0% xxxx-xxxx rack01
Datacenter: dc01
===============================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 10.0.2.11 427.32 KiB 32 100.0% yyyy-yyyy rack01
Now you are run the
nodetool rebuild -- cassandrabk
to get your data populated!
If you met error “unable to source for streaming range”, make sure to check on this page. Altering the keyspace to use network topology with proper replication factor will fix it.
We hope you have learned something and enjoyed this post. Do follow our Facebook (update more often) and our Twitter . We will continue to share more on latest technology update & usage besides showcasing our work.
References:
Posted on Dec 22, 2016
Leave a comment or suggestion below