前回の続きで、今度はOperator SDKを使ってGoでControllerを書くかたちでOperatorを作ってみる。
下記を参照しながら作っていくが、例ではMemcachedのDeployment(レプリカ数:3)となっているが、これをmackerel-agentのDaemonSetとして作ってみる。 operator-sdk/user-guide.md at master · operator-framework/operator-sdk · GitHub
SDKのインストールまでは済んでいるので、プロジェクトの雛形を生成する。
ᐅ operator-sdk new mackerel-operator INFO[0000] Creating new Go operator 'mackerel-operator'. INFO[0000] Created cmd/manager/main.go INFO[0000] Created build/Dockerfile (中略) (68/70) Wrote k8s.io/apiextensions-apiserver@0fe22c71c47604641d9aa352c785b7912c200562 (69/70) Wrote github.com/coreos/prometheus-operator@v0.26.0 (70/70) Wrote sigs.k8s.io/controller-tools@v0.1.8 INFO[0095] Run dep ensure done INFO[0095] Run git init ... Initialized empty Git repository in /Users/yamadanaoyuki/go/src/github.com/mackerel-operator/.git/ Auto packing the repository in background for optimum performance. See "git help gc" for manual housekeeping. INFO[0103] Run git init done INFO[0103] Project creation complete.
ある程度まで、コードが自動生成される。
APIを追加する
ᐅ operator-sdk add api --api-version=kirishikistudios.com/v1alpha1 --kind=Mackerel INFO[0000] Generating api version kirishikistudios.com/v1alpha1 for kind Mackerel. INFO[0000] Created pkg/apis/kirishikistudios/v1alpha1/mackerel_types.go INFO[0000] Created pkg/apis/addtoscheme_kirishikistudios_v1alpha1.go INFO[0000] Created pkg/apis/kirishikistudios/v1alpha1/register.go INFO[0000] Created pkg/apis/kirishikistudios/v1alpha1/doc.go INFO[0000] Created deploy/crds/kirishikistudios_v1alpha1_mackerel_cr.yaml INFO[0001] Created deploy/crds/kirishikistudios_v1alpha1_mackerel_crd.yaml INFO[0005] Running deepcopy code-generation for Custom Resource group versions: [kirishikistudios:[v1alpha1], ] INFO[0007] Code-generation complete. INFO[0008] Running OpenAPI code-generation for Custom Resource group versions: [kirishikistudios:[v1alpha1], ] INFO[0009] Created deploy/crds/kirishikistudios_v1alpha1_mackerel_crd.yaml INFO[0009] Code-generation complete. INFO[0009] API generation complete.
Controllerを追加・編集する
ᐅ operator-sdk add controller --api-version=kirishikistudios.com/v1alpha1 --kind=Mackerel INFO[0000] Generating controller version kirishikistudios.com/v1alpha1 for kind Mackerel. INFO[0000] Created pkg/controller/mackerel/mackerel_controller.go INFO[0000] Created pkg/controller/add_mackerel.go INFO[0000] Controller generation complete.
メインロジックである pkg/controller/mackerel/mackerel_controller.go の中身を少しみてみると
(コードはこちら)
found := &appsv1.DaemonSet{}
err = r.client.Get(context.TODO(), types.NamespacedName{Name: Mackerel.Name, Namespace: Mackerel.Namespace}, found)
if err != nil && errors.IsNotFound(err) {
// Define a new Daemonset
dep := r.daemonsetForMackerel(Mackerel)
reqLogger.Info("Creating a new Daemonset", "Daemonset.Namespace", dep.Namespace, "Daemonset.Name", dep.Name)
err = r.client.Create(context.TODO(), dep)
if err != nil {
reqLogger.Error(err, "Failed to create new Daemonset", "Daemonset.Namespace", dep.Namespace, "Daemonset.Name", dep.Name)
return reconcile.Result{}, err
}
// Daemonset created successfully - return and requeue
return reconcile.Result{Requeue: true}, nil
}
r.clientというのはKubernetesのCRUDをできるクライアントを内蔵していて、実際に必要なリソース（今回はMackerelというCR）があるかどうかチェックして無ければ作成する、ということをやっている。
DaemonSetのリソースは下記のようにGoの構造体として生成するのだが、慣れないとYamlを書いたほうが簡単に感じる。コード補完に頼りつつ、ドキュメントを見つつ書いた。
func (r *ReconcileMackerel) daemonsetForMackerel(m *apiv1alpha1.Mackerel) *appsv1.DaemonSet { ls := labelsForMackerel(m.Name) dep := &appsv1.DaemonSet{ TypeMeta: metav1.TypeMeta{ APIVersion: "extensions/v1beta1", Kind: "DaemonSet", }, ObjectMeta: metav1.ObjectMeta{ Name: m.Name, Namespace: m.Namespace, }, Spec: appsv1.DaemonSetSpec{ Selector: &metav1.LabelSelector{ MatchLabels: ls, }, Template: corev1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Labels: ls, }, Spec: corev1.PodSpec{ Containers: []corev1.Container{{ Image: "mackerel/mackerel-agent:latest", Name: "mackerel-agent", Env: []corev1.EnvVar{ { Name: "apikey", //TODO get from ConfigMap or Secret Value: "xxxxxxxxxxxxxx", }, { //TODO get from ConfigMap or Secret Name: "opts", Value: "-role=minikube:mbp13", }, { Name: "enable_docker_plugin", Value: "1", }, }, VolumeMounts: []corev1.VolumeMount{ { Name: "docker-sock", MountPath: "/var/run/docker.sock", }, { Name: "mackerel-id", MountPath: "/var/lib/mackerel-agent/", }, }, }}, Volumes: []corev1.Volume{ { Name: "docker-sock", VolumeSource: corev1.VolumeSource{ HostPath: &corev1.HostPathVolumeSource{Path: "/var/run/docker.sock"}, }, }, { Name: "mackerel-id", VolumeSource: corev1.VolumeSource{ HostPath: &corev1.HostPathVolumeSource{Path: "/var/lib/mackerel-agent/"}, }, }, }, }, }, }, } // Set Mackerel instance as the owner and controller _ = controllerutil.SetControllerReference(m, dep, r.scheme) return dep }
CRDをデプロイする
ᐅ kubectl create -f deploy/crds/kirishikistudios_v1alpha1_mackerel_crd.yaml customresourcedefinition.apiextensions.k8s.io "mackerels.kirishikistudios.com" created
Operatorのイメージをビルド&DockerHubにPushする
ᐅ operator-sdk build chokkoy/mackerel-operator:v0.0.1
INFO[0004] Building Docker image chokkoy/mackerel-operator:v0.0.1
Sending build context to Docker daemon 197.9MB
Step 1/7 : FROM alpine:3.8
---> 491e0ff7a8d5
Step 2/7 : ENV OPERATOR=/usr/local/bin/mackerel-operator USER_UID=1001 USER_NAME=mackerel-operator
---> Using cache
---> 23212d49d23b
Step 3/7 : COPY build/_output/bin/mackerel-operator ${OPERATOR}
---> 508598d68109
Step 4/7 : COPY build/bin /usr/local/bin
---> 82ea47e91962
Step 5/7 : RUN /usr/local/bin/user_setup
---> Running in 20736001278d
+ mkdir -p /root
+ chown 1001:0 /root
+ chmod ug+rwx /root
+ chmod g+rw /etc/passwd
+ rm /usr/local/bin/user_setup
Removing intermediate container 20736001278d
---> 495bc12d5160
Step 6/7 : ENTRYPOINT ["/usr/local/bin/entrypoint"]
---> Running in b3dbe6930311
Removing intermediate container b3dbe6930311
---> d159180ac21f
Step 7/7 : USER ${USER_UID}
---> Running in 3cf2c8f425b3
Removing intermediate container 3cf2c8f425b3
---> 99d8a5dbf5f3
Successfully built 99d8a5dbf5f3
Successfully tagged chokkoy/mackerel-operator:v0.0.1
INFO[0012] Operator build complete.
ᐅ docker push chokkoy/mackerel-operator:v0.0.1
このイメージ名：タグで deploy/operator.yamlのIMAGEを置き換えて、デプロイ
ᐅ kubectl create -f deploy/service_account.yaml serviceaccount "mackerel-operator" created ᐅ kubectl create -f deploy/role.yaml role.rbac.authorization.k8s.io "mackerel-operator" created ᐅ kubectl create -f deploy/role_binding.yaml rolebinding.rbac.authorization.k8s.io "mackerel-operator" created ᐅ kubectl create -f deploy/operator.yaml deployment.apps "mackerel-operator" created
この状態で、OperatorのDeployment(Pod)が一台起動している
ᐅ kubectl get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE mackerel-operator 1 1 1 1 6s
ᐅ kubectl get pods NAME READY STATUS RESTARTS AGE mackerel-operator-557ff88b57-gjh9w 1/1 Running 0 9s
CR(カスタムリソース)をデプロイする
ᐅ kubectl apply -f deploy/crds/kirishikistudios_v1alpha1_mackerel_cr.yaml mackerel.kirishikistudios.com "example-mackerel" created
すると、mackerel-agentのDaemonSetが起動する。
ᐅ kubectl get pods NAME READY STATUS RESTARTS AGE example-mackerel-4m7d7 1/1 Running 0 7s mackerel-operator-557ff88b57-gjh9w 1/1 Running 0 30s
ᐅ kubectl get ds NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE example-mackerel 1 1 1 1 1 <none> 10s
Mackerel上で監視できている
SDKを使ってOperatorを作る流れがなんとなくわかった。GoでKubernetesのリソースを制御できるのは面白い！
ソースコードはこちら
