前回の続きで、今度はOperator SDKを使ってGoでControllerを書くかたちでOperatorを作ってみる。
下記を参照しながら作っていくが、例ではMemcachedのDeployment(レプリカ数:3)となっているが、これをmackerel-agentのDaemonSetとして作ってみる。
operator-sdk/user-guide.md at master · operator-framework/operator-sdk · GitHub
SDKのインストールまでは済んでいるので、プロジェクトの雛形を生成する。
ᐅ operator-sdk new mackerel-operator
INFO[0000] Creating new Go operator 'mackerel-operator'.
INFO[0000] Created cmd/manager/main.go
INFO[0000] Created build/Dockerfile
(中略)
(68/70) Wrote k8s.io/apiextensions-apiserver@0fe22c71c47604641d9aa352c785b7912c200562
(69/70) Wrote github.com/coreos/prometheus-operator@v0.26.0
(70/70) Wrote sigs.k8s.io/controller-tools@v0.1.8
INFO[0095] Run dep ensure done
INFO[0095] Run git init ...
Initialized empty Git repository in /Users/yamadanaoyuki/go/src/github.com/mackerel-operator/.git/
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
INFO[0103] Run git init done
INFO[0103] Project creation complete.
ある程度まで、コードが自動生成される。
ᐅ operator-sdk add api --api-version=kirishikistudios.com/v1alpha1 --kind=Mackerel
INFO[0000] Generating api version kirishikistudios.com/v1alpha1 for kind Mackerel.
INFO[0000] Created pkg/apis/kirishikistudios/v1alpha1/mackerel_types.go
INFO[0000] Created pkg/apis/addtoscheme_kirishikistudios_v1alpha1.go
INFO[0000] Created pkg/apis/kirishikistudios/v1alpha1/register.go
INFO[0000] Created pkg/apis/kirishikistudios/v1alpha1/doc.go
INFO[0000] Created deploy/crds/kirishikistudios_v1alpha1_mackerel_cr.yaml
INFO[0001] Created deploy/crds/kirishikistudios_v1alpha1_mackerel_crd.yaml
INFO[0005] Running deepcopy code-generation for Custom Resource group versions: [kirishikistudios:[v1alpha1], ]
INFO[0007] Code-generation complete.
INFO[0008] Running OpenAPI code-generation for Custom Resource group versions: [kirishikistudios:[v1alpha1], ]
INFO[0009] Created deploy/crds/kirishikistudios_v1alpha1_mackerel_crd.yaml
INFO[0009] Code-generation complete.
INFO[0009] API generation complete.
Controllerを追加・編集する
ᐅ operator-sdk add controller --api-version=kirishikistudios.com/v1alpha1 --kind=Mackerel
INFO[0000] Generating controller version kirishikistudios.com/v1alpha1 for kind Mackerel.
INFO[0000] Created pkg/controller/mackerel/mackerel_controller.go
INFO[0000] Created pkg/controller/add_mackerel.go
INFO[0000] Controller generation complete.
メインロジックである pkg/controller/mackerel/mackerel_controller.go の中身を少しみてみると
(コードはこちら)
found := &appsv1.DaemonSet{}
err = r.client.Get(context.TODO(), types.NamespacedName{Name: Mackerel.Name, Namespace: Mackerel.Namespace}, found)
if err != nil && errors.IsNotFound(err) {
dep := r.daemonsetForMackerel(Mackerel)
reqLogger.Info("Creating a new Daemonset", "Daemonset.Namespace", dep.Namespace, "Daemonset.Name", dep.Name)
err = r.client.Create(context.TODO(), dep)
if err != nil {
reqLogger.Error(err, "Failed to create new Daemonset", "Daemonset.Namespace", dep.Namespace, "Daemonset.Name", dep.Name)
return reconcile.Result{}, err
}
return reconcile.Result{Requeue: true}, nil
}
r.clientというのはKubernetesのCRUDをできるクライアントを内蔵していて、実際に必要なリソース（今回はMackerelというCR）があるかどうかチェックして無ければ作成する、ということをやっている。
DaemonSetのリソースは下記のようにGoの構造体として生成するのだが、慣れないとYamlを書いたほうが簡単に感じる。コード補完に頼りつつ、ドキュメントを見つつ書いた。
func (r *ReconcileMackerel) daemonsetForMackerel(m *apiv1alpha1.Mackerel) *appsv1.DaemonSet {
ls := labelsForMackerel(m.Name)
dep := &appsv1.DaemonSet{
TypeMeta: metav1.TypeMeta{
APIVersion: "extensions/v1beta1",
Kind: "DaemonSet",
},
ObjectMeta: metav1.ObjectMeta{
Name: m.Name,
Namespace: m.Namespace,
},
Spec: appsv1.DaemonSetSpec{
Selector: &metav1.LabelSelector{
MatchLabels: ls,
},
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: ls,
},
Spec: corev1.PodSpec{
Containers: []corev1.Container{{
Image: "mackerel/mackerel-agent:latest",
Name: "mackerel-agent",
Env: []corev1.EnvVar{
{
Name: "apikey",
TODO
Value: "xxxxxxxxxxxxxx",
},
{
TODO
Name: "opts",
Value: "-role=minikube:mbp13",
},
{
Name: "enable_docker_plugin",
Value: "1",
},
},
VolumeMounts: []corev1.VolumeMount{
{
Name: "docker-sock",
MountPath: "/var/run/docker.sock",
},
{
Name: "mackerel-id",
MountPath: "/var/lib/mackerel-agent/",
},
},
}},
Volumes: []corev1.Volume{
{
Name: "docker-sock",
VolumeSource: corev1.VolumeSource{
HostPath: &corev1.HostPathVolumeSource{Path: "/var/run/docker.sock"},
},
},
{
Name: "mackerel-id",
VolumeSource: corev1.VolumeSource{
HostPath: &corev1.HostPathVolumeSource{Path: "/var/lib/mackerel-agent/"},
},
},
},
},
},
},
}
_ = controllerutil.SetControllerReference(m, dep, r.scheme)
return dep
}
CRDをデプロイする
ᐅ kubectl create -f deploy/crds/kirishikistudios_v1alpha1_mackerel_crd.yaml
customresourcedefinition.apiextensions.k8s.io "mackerels.kirishikistudios.com" created
Operatorのイメージをビルド&DockerHubにPushする
ᐅ operator-sdk build chokkoy/mackerel-operator:v0.0.1
INFO[0004] Building Docker image chokkoy/mackerel-operator:v0.0.1
Sending build context to Docker daemon 197.9MB
Step 1/7 : FROM alpine:3.8
---> 491e0ff7a8d5
Step 2/7 : ENV OPERATOR=/usr/local/bin/mackerel-operator USER_UID=1001 USER_NAME=mackerel-operator
---> Using cache
---> 23212d49d23b
Step 3/7 : COPY build/_output/bin/mackerel-operator ${OPERATOR}
---> 508598d68109
Step 4/7 : COPY build/bin /usr/local/bin
---> 82ea47e91962
Step 5/7 : RUN /usr/local/bin/user_setup
---> Running in 20736001278d
+ mkdir -p /root
+ chown 1001:0 /root
+ chmod ug+rwx /root
+ chmod g+rw /etc/passwd
+ rm /usr/local/bin/user_setup
Removing intermediate container 20736001278d
---> 495bc12d5160
Step 6/7 : ENTRYPOINT ["/usr/local/bin/entrypoint"]
---> Running in b3dbe6930311
Removing intermediate container b3dbe6930311
---> d159180ac21f
Step 7/7 : USER ${USER_UID}
---> Running in 3cf2c8f425b3
Removing intermediate container 3cf2c8f425b3
---> 99d8a5dbf5f3
Successfully built 99d8a5dbf5f3
Successfully tagged chokkoy/mackerel-operator:v0.0.1
INFO[0012] Operator build complete.
ᐅ docker push chokkoy/mackerel-operator:v0.0.1
このイメージ名：タグで deploy/operator.yamlのIMAGEを置き換えて、デプロイ
ᐅ kubectl create -f deploy/service_account.yaml
serviceaccount "mackerel-operator" created
ᐅ kubectl create -f deploy/role.yaml
role.rbac.authorization.k8s.io "mackerel-operator" created
ᐅ kubectl create -f deploy/role_binding.yaml
rolebinding.rbac.authorization.k8s.io "mackerel-operator" created
ᐅ kubectl create -f deploy/operator.yaml
deployment.apps "mackerel-operator" created
この状態で、OperatorのDeployment(Pod)が一台起動している
ᐅ kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
mackerel-operator 1 1 1 1 6s
ᐅ kubectl get pods
NAME READY STATUS RESTARTS AGE
mackerel-operator-557ff88b57-gjh9w 1/1 Running 0 9s
CR(カスタムリソース)をデプロイする
ᐅ kubectl apply -f deploy/crds/kirishikistudios_v1alpha1_mackerel_cr.yaml
mackerel.kirishikistudios.com "example-mackerel" created
すると、mackerel-agentのDaemonSetが起動する。
ᐅ kubectl get pods
NAME READY STATUS RESTARTS AGE
example-mackerel-4m7d7 1/1 Running 0 7s
mackerel-operator-557ff88b57-gjh9w 1/1 Running 0 30s
ᐅ kubectl get ds
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
example-mackerel 1 1 1 1 1 <none> 10s
Mackerel上で監視できている
SDKを使ってOperatorを作る流れがなんとなくわかった。GoでKubernetesのリソースを制御できるのは面白い！
ソースコードはこちら
https://github.com/chokkoyamada/mackerel-operator-go
