Strimzi on Oracle Kubernetes Engine (OKE)+ Kafka Connect on Oracle Streaming Service(OSS) — A Match Made in Heaven

_____ _                            _             
/ ___| | (_)
\ `--.| |_ _ __ ___ __ _ _ __ ___ _ _ __ __ _
`--. \ __| '__/ _ \/ _` | '_ ` _ \| | '_ \ / _` |
/\__/ / |_| | | __/ (_| | | | | | | | | | | (_| |
\____/ \__|_| \___|\__,_|_| |_| |_|_|_| |_|\__, |
__/ |
|___/
Photo by Joao Branco on Unsplash

For the Impatient

Even if Oracle Cloud takes the complexity away from managing Kafka, Zookeeper and allied Infrastructure. Kafka Connect and other connectors bring a fresh set of problems.

1) Who manages the HA and scalability of these connectors ?
Connector tasks running inside containers orchestrated by Kubernetes emerged as a pattern for scalability & HA of connectors running on k8s.
Photo by Jenna Beekhuis on Unsplash
2) The toil of writing Dockerfiles and kubernetes manifests ?
Building connector plugins defining the right class paths , making sure the connector property files are mounted as volumes etc.
TOIL — A Very Real thing in Kafka

Along came a Strimzi, to solve these issues.

What are these components ?

----------------------------------
1) OKE - Oracle Kubernetes Engine
----------------------------------
* Managed k8s Service running on OCI
* Multi-Availability Domains capbility ( Data Centers )
* Highly available Master obfuscated from end-user
* Tightly integrated with Oracle Cloud Infrastructure
* Helm & Tiller Built in
* RBAC Integration with Cloud IAM
* One-Click provisioning & scaling
----------
2) Strimzi
----------
*
A CNCF Sandbox project
* Used to remove plumbing complexity of Kafka Management in k8s
* Follows Operator pattern to extend k8s functionality to Kafka
* Custom Resource Definitions for kafka connect, mirror maker
* RBAC, Clusterroles, Bindings for several Kafka functionality
---------------------------
3) Oracle Streaming Service
---------------------------
*
Fully managed , high throughput streaming service
* Scale Mechanism through Partition
* Horizontally Scalable Service
* Kafka Compatible

Putting it all together

#1 — Setup Oracle Streaming Service

Step 1: Create working instance of Oracle Streaming Service with Kafka Connect, capture some informationHow : Instructions 

#2 — Setup Oracle Kubernetes Engine

Go to Developer ServicesContainer Clusters

Step 2: Setup OKE - Oracle Kubernetes Engine with Helm Enabled
How : Instructions

#3- Setup Strimzi Kafka Operator

The Strimzi kafka Cluster operator is a deployment that defines all the necessary Custom Resource Definitions, Clusterrole, ClusterroleBindings

Step 3: Setup Strimzi Operator using a Helm chart in a given namespace
How :
******* SETUP STRIMZI WITH HELM *********kubectl create namespace strimzi
helm repo add strimzi https://strimzi.io/charts/
helm install strimzi --namespace strimzi strimzi/strimzi-kafka-operator

#4- Setup Custom Kafka Connect Image

The Kafka Connect Image from strimzi is configured to expect all the Plugins ( Eg. JDBC Source, JDBC Sink, MQTT Source jar files etc ) in /opt/kafka/plugins path.

So when you write a Dockerfile you simply create a folder called plugins where you paste all your plugins . That way the kafka connect image stays light.

mkdir -p strimzi
vi Dockerfile
FROM strimzi/kafka:0.15.0-kafka-2.3.1
USER root:root
COPY ./plugins/ /opt/kafka/plugins/
USER 1001
docker build -t <docker-Username>/<docker-registry-path>:<tag> .
docker push <image-name>:<tag>

#5- Download TLS Certificate from Streaming service

Replace the url’s appropriately depending on the endpoint that is available for eg. mumbai would be ap-mumbai-1 , look at list of streaming end points here

Click on Kafka Connection Settings
Copy the URL
echo | openssl s_client -connect cell-1.streaming.us-phoenix-1.oci.oraclecloud.com:9092 |  sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > mycertfile.pem

#6 — Setup Secrets & Prepare YAML File

Secrets

TLS Secret
kubectl -n strimzi create secret generic bootstrap-tls-secret --from-file=mycertfile.pem
SASL_SSL Auth Secret
kubectl -n strimzi create secret generic oss-auth-token --from-literal=password="<Your Auth Token>"

Kafka Connect YAML file

vi kafka-connect.yaml

Start deployment

kubectl -n strimzi apply -f .

Voila !!

Your Kafka Connect managed by Strimzi & Kubernetes is Ready

2020-01-04 16:32:48,269 INFO [Worker clientId=connect-1, groupId=sesym2019-mqtt-connect] Finished reading to end of log and updated config snapshot, new config log offset: 3082179196929 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1]
2020-01-04 16:32:48,269 INFO [Worker clientId=connect-1, groupId=sesym2019-mqtt-connect] Starting connectors and tasks using config offset 3082179196929 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1]
2020-01-04 16:32:48,269 INFO [Worker clientId=connect-1, groupId=sesym2019-mqtt-connect] Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1]

Other Advanced Customizations

Using Strimzi to manage Kafka Connectors instead of REST Proxy
https://strimzi.io/docs/master/#assembly-kafka-connect-external-configuration-deployment-configuration-kafka-connect-s2i

What Problems does this setup solve

Photo by Markus Spiske on Unsplash

#1 : Infra Complexity

Oracle Streaming Service removes the complexity of having Zookeepers to manage your Kafka Pets.

#2 Application Rewrite

Oracle Streaming Service Kafka Connect harness removes the need to rewrite applications written on Kafka using kafka connect

#3 Vendor Lock-In

Unlike an SQS or any other PubSub, the kafka connect harness makes sure you are not locked into one cloud vendor

#4 Being Kafka Aware

With the operator pattern, Strimzi on Oracle Kubernetes Engine connecting to Kafka Connect makes 

#5 Inadvertent Errors

Accidental Deletes of Deployments, Pods, Services as Strimzi uses CRDs to manage these components. 

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store