GKE に kustomize を使ってデプロイしてみる

2020-05-04

この記事は Bazel で Go のソースコードをビルドするぞ の延長ですが、 内容が Bazel とはあまり関係ない部分なので別記事としました。

上記の記事でも使った以下のリポジトリを使います。

righ/bazel-sample-project - Github

目次

今回の記事では Deployment を使った普通のデプロイについては書かないのでご了承ください。

下準備 preparing

デプロイするにあたって必要なツールをセットアップしていきます。

GCP のアカウント作成はここでは説明しません。 新規作成の場合は無料枠が3万円分もらえるはずなのでとりあえず作ってみるとよいでしょう。

下準備が大半を占めてるのではよ本題は入れって人は 飛ばしてください

kubectl Installation

この記事の目的である kustomize を使ってデプロイするだけであれば kubectl は必要ありませんが、 色々確認するために使うのでインストールしておきましょう。

Mac や Windows の場合は DockerDesktop をインストールして Kubernetes を有効にすれば OK です。

settings-kubernetes.png

それ以外の場合は kubectlのインストールおよびセットアップ - Kubernetes を参照してください。

Google Cloud SDK の Setup

GKE にデプロイするには Google Cloud SDK をインストールする必要があります。

環境に合わせてセットアップします。

Mac や Linux の場合は少々面倒で アーカイブをダウンロードしてスクリプトを実行する必要があります。

私の場合はホームディレクトリ配下に解凍して google-cloud-sdk/install.sh を実行しました。 google-cloud-sdk/bin/ 配下にコマンドがあるので PATH を設定すれば gcloud コマンドが使えるようになります。

最後に gcloud init を実行すれば OAuth の認証が行われ、GCP のアカウント(プロジェクト) とひも付きます。

クラスタを作成する creating cluster

Kubernetes で管理する最も大きな単位が クラスタという入れ物です。

クラスタは GUI で作成することもできますが、コマンドでも作成できます。 今回は gcloud コマンドを使って sample-cluster というクラスタを作りました。

$ gcloud container clusters create --num-nodes=2 sample-cluster \                                                                                                                        [master]
--zone asia-northeast1-a \
--machine-type g1-small \
--min-nodes=2 --max-nodes=3
WARNING: Currently VPC-native is not the default mode during cluster creation. In the future, this will become the default mode and can be disabled using `--no-enable-ip-alias` flag. Use `--[no-]enable-ip-alias` flag to suppress this warning.
WARNING: Newly created clusters and node-pools will have node auto-upgrade enabled by default. This can be disabled using the `--no-enable-autoupgrade` flag.
WARNING: Starting in 1.12, default node pools in new clusters will have their legacy Compute Engine instance metadata endpoints disabled by default. To create a cluster with legacy instance metadata endpoints disabled in the default node pool, run `clusters create` with the flag `--metadata disable-legacy-endpoints=true`.
WARNING: Your Pod address range (`--cluster-ipv4-cidr`) can accommodate at most 1008 node(s).
This will enable the autorepair feature for nodes. Please see https://cloud.google.com/kubernetes-engine/docs/node-auto-repair for more information on node autorepairs.
Creating cluster sample-cluster in asia-northeast1-a... Cluster is being health-checked (master is healthy)...done.
Created [https://container.googleapis.com/v1/projects/righm9/zones/asia-northeast1-a/clusters/sample-cluster].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/asia-northeast1-a/sample-cluster?project=righm9
kubeconfig entry generated for sample-cluster.
NAME            LOCATION           MASTER_VERSION  MASTER_IP     MACHINE_TYPE  NODE_VERSION    NUM_NODES  STATUS
sample-cluster  asia-northeast1-a  1.14.10-gke.27  34.85.37.248  g1-small      1.14.10-gke.27  2          RUNNING

(おそらく)コマンドでの作成が終わると自動的にコンテキストが切り替わります。(少し自信なし) 切り替わっていない場合は kubectl config use-context でスイッチしましょう。

コンテキストとは kubectl が指すクラスタです。

kubectl で確認してみましょう。

$ kubectl config get-contexts                                                                                                                                                            [master]
CURRENT   NAME                                          CLUSTER                                       AUTHINFO                                      NAMESPACE
          docker-desktop                                docker-desktop                                docker-desktop
          docker-for-desktop                            docker-desktop                                docker-desktop
*         gke_righm9_asia-northeast1-a_sample-cluster   gke_righm9_asia-northeast1-a_sample-cluster   gke_righm9_asia-northeast1-a_sample-cluster

無事に GKE に作成したクラスタにスイッチされていますね。

この記事では gke_righm9_asia-northeast1-a_sample-cluster コンテキストに スイッチされている前提で話を進めていきます。

備考

ローカルの開発でも Kubernetes を使っている場合はコンテキストを戻すことを忘れないようにしましょう。

kubectl config use-context docker-desktop
Switched to context "docker-desktop".

GCR に Push する

GKE で利用するイメージはどこかしらの Dockerレジストリに Push する必要があります。

Bazel で Go のソースコードをビルドするぞ で Docker Hub に Push してあったのですが、なぜかそれを指定しても Pull できなかった(ImagePullBackOff)ので おとなしく GCR に Push することにしました。

docker コマンドで gcr に PUSH するために 認証ヘルパー を使います。

$ gcloud auth configure-docker                                          [master]
Adding credentials for all GCR repositories.
WARNING: A long list of credential helpers may cause delays running 'docker build'. We recommend passing the registry name to configure only the registry you are using.
After update, the following will be written to your Docker config file
 located at [/Users/righ/.docker/config.json]:
 {
  "credHelpers": {
    "gcr.io": "gcloud",
    "marketplace.gcr.io": "gcloud",
    "eu.gcr.io": "gcloud",
    "us.gcr.io": "gcloud",
    "staging-k8s.gcr.io": "gcloud",
    "asia.gcr.io": "gcloud"
  }
}

Do you want to continue (Y/n)?  Y

Docker configuration file updated.

無事認証が終わったら 以下のようにタグを付けて Push するのが一般的な方法です。

$ docker build -t gcr.io/プロジェクトID/イメージ名 ビルドコンテキスト
$ docker push gcr.io/プロジェクトID/イメージ名
Bazel を使って GCR にPUSHする

今回使うリポジトリはもともと Bazel を使うためのサンプルなので GCR へ PUSH するための設定もしてあります。

bazel-sample-project/service.bzl
load("@bazel_tools//tools/build_defs/pkg:pkg.bzl", "pkg_tar")
load("@io_bazel_rules_docker//container:container.bzl", "container_image", "container_push")
load("@io_bazel_rules_docker//go:image.bzl", "go_image")

def service_image(name, **kwargs):
    pkg_tar(
        name = "tar",
        srcs = ["@grpc_health_probe//file"],
        mode = "0o755",
        package_dir = "/bin",
        visibility = ["//visibility:public"],
    )

    container_image(
        name = "base_image",
        base = "@go_image_base//image",
        tars = [":tar"],
        visibility = ["//visibility:public"],
    )

    go_image(
        name = "image",
        base = ":base_image",
        embed = ["//" + name + ":go_default_library"],
        goarch = "amd64",
        goos = "linux",
        visibility = ["//visibility:public"],
        **kwargs
    )

    container_push(
        name = "push_to_dockerhub",
        format = "Docker",
        image = ":image",
        registry = "docker.io",
        repository = "righm9/" + name,
        tag = "{BUILD_TIMESTAMP}",
    )

    container_push(
        name = "push_to_gcr",
        format = "Docker",
        image = ":image",
        registry = "gcr.io",
        repository = "righm9/" + name,
        tag = "latest",
    )

これを使って Push してみます。

$ bazel run //gateway:push_to_gcr
INFO: Analyzed target //gateway:push_to_gcr (1 packages loaded, 7 targets configured).
INFO: Found 1 target...
Target //gateway:push_to_gcr up-to-date:
  bazel-bin/gateway/push_to_gcr.digest
  bazel-bin/gateway/push_to_gcr
INFO: Elapsed time: 0.385s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 2 total actions
INFO: Build completed successfully, 2 total actions
2020/05/03 22:46:35 Successfully pushed Docker image to gcr.io/righm9/gateway:latest
$ bazel run //echo:push_to_gcr
INFO: Analyzed target //echo:push_to_gcr (1 packages loaded, 7 targets configured).
INFO: Found 1 target...
Target //echo:push_to_gcr up-to-date:
  bazel-bin/echo/push_to_gcr.digest
  bazel-bin/echo/push_to_gcr
INFO: Elapsed time: 0.318s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 2 total actions
INFO: Build completed successfully, 2 total actions
2020/05/03 22:46:55 Successfully pushed Docker image to gcr.io/righm9/echo:latest

こんな感じになっていれば OK です。

repos.png

kustomize というリポジトリはこの後のセクションで作ります。

kustomize

GKE へのデプロイで kustomize を使うには GoogleCloudPlatform/cloud-builders-community - Github を使います。

Clone して、移動して

$ git clone https://github.com/GoogleCloudPlatform/cloud-builders-community
Cloning into 'cloud-builders-community'...
remote: Enumerating objects: 3866, done.
remote: Total 3866 (delta 0), reused 0 (delta 0), pack-reused 3866
Receiving objects: 100% (3866/3866), 965.20 KiB | 660.00 KiB/s, done.
Resolving deltas: 100% (1748/1748), done.

$ cd cloud-builders-community/kustomize

以下のように gcloud builds submit でビルドすると kustomize という GCR にリポジトリが作られます。 (順番が前後しますが先程のリポジトリはこれによって作られたものだったのです)

$ gcloud builds submit --config cloudbuild.yaml .
Creating temporary tarball archive of 5 file(s) totalling 5.8 KiB before compression.
Uploading tarball of [.] to [gs://righm9_cloudbuild/source/1588435618.22-b17a4c75902448f98d85fa42a590cb86.tgz]
API [cloudbuild.googleapis.com] not enabled on project [846141039915].
 Would you like to enable and retry (this will take a few minutes)?
(y/N)?  y

Enabling service [cloudbuild.googleapis.com] on project [846141039915]...
Operation "operations/acf.51554e69-6b85-41df-b101-79b6366136a0" finished successfully.
Created [https://cloudbuild.googleapis.com/v1/projects/righm9/builds/42caf256-9873-4886-88b8-0096d239cf71].
Logs are available at [https://console.cloud.google.com/cloud-build/builds/42caf256-9873-4886-88b8-0096d239cf71?project=846141039915].
------------------------------------------------------------------------------------------------------ REMOTE BUILD OUTPUT ------------------------------------------------------------------------------------------------------
starting build "42caf256-9873-4886-88b8-0096d239cf71"

FETCHSOURCE
Fetching storage object: gs://righm9_cloudbuild/source/1588435618.22-b17a4c75902448f98d85fa42a590cb86.tgz#1588435620139411
Copying gs://righm9_cloudbuild/source/1588435618.22-b17a4c75902448f98d85fa42a590cb86.tgz#1588435620139411...
/ [1 files][  2.3 KiB/  2.3 KiB]
Operation completed over 1 objects/2.3 KiB.
BUILD
Starting Step #0
Step #0: Already have image (with digest): gcr.io/cloud-builders/docker
Step #0: Sending build context to Docker daemon  11.78kB
Step #0: Step 1/7 : FROM gcr.io/cloud-builders/gcloud
Step #0:  ---> f340f9885ea0
Step #0: Step 2/7 : ENV VER 3.4.0
Step #0:  ---> Running in d59b849b9bd4
Step #0: Removing intermediate container d59b849b9bd4
Step #0:  ---> 5cbc9f7b5b08
Step #0: Step 3/7 : ENV VERSION v${VER}
Step #0:  ---> Running in f151f29f5ac5
Step #0: Removing intermediate container f151f29f5ac5
Step #0:  ---> a0773d5aa34a
Step #0: Step 4/7 : COPY kustomize.bash /builder/kustomize.bash
Step #0:  ---> 63bd42008292
Step #0: Step 5/7 : RUN apt-get update &&     apt-get install -y wget &&     wget https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2F${VERSION}/kustomize_${VERSION}_linux_amd64.tar.gz &&     tar xvzf kustomize_${VERSION}_linux_amd64.tar.gz &&     mkdir /builder/kustomize &&     mv kustomize /builder/kustomize/kustomize &&     chmod +x /builder/kustomize/kustomize &&     chmod +x /builder/kustomize.bash &&     rm kustomize_${VERSION}_linux_amd64.tar.gz &&     apt-get remove --purge -y wget &&     apt-get --purge -y autoremove &&     apt-get clean &&     rm -rf /var/lib/apt/lists/*
Step #0:  ---> Running in cc446cdec4e6
Step #0: Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
Step #0: Get:2 http://ppa.launchpad.net/git-core/ppa/ubuntu xenial InRelease [23.8 kB]
Step #0: Get:3 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
Step #0: Get:4 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [861 kB]
Step #0: Get:5 http://security.ubuntu.com/ubuntu xenial-security/main Translation-en [323 kB]
Step #0: Get:6 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages [7204 B]
Step #0: Get:7 http://security.ubuntu.com/ubuntu xenial-security/restricted Translation-en [2152 B]
Step #0: Get:8 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [490 kB]
Step #0: Get:9 http://security.ubuntu.com/ubuntu xenial-security/universe Translation-en [200 kB]
Step #0: Get:10 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [6088 B]
Step #0: Get:11 http://security.ubuntu.com/ubuntu xenial-security/multiverse Translation-en [2888 B]
Step #0: Get:12 http://ppa.launchpad.net/git-core/ppa/ubuntu xenial/main amd64 Packages [3372 B]
Step #0: Get:13 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Step #0: Get:14 http://ppa.launchpad.net/git-core/ppa/ubuntu xenial/main Translation-en [2432 B]
Step #0: Get:15 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]
Step #0: Get:16 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1201 kB]
Step #0: Get:17 http://archive.ubuntu.com/ubuntu xenial/main Translation-en [568 kB]
Step #0: Get:18 http://archive.ubuntu.com/ubuntu xenial/restricted amd64 Packages [8344 B]
Step #0: Get:19 http://archive.ubuntu.com/ubuntu xenial/restricted Translation-en [2908 B]
Step #0: Get:20 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [7532 kB]
Step #0: Get:21 http://archive.ubuntu.com/ubuntu xenial/universe Translation-en [4354 kB]
Step #0: Get:22 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages [144 kB]
Step #0: Get:23 http://archive.ubuntu.com/ubuntu xenial/multiverse Translation-en [106 kB]
Step #0: Get:24 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [1141 kB]
Step #0: Get:25 http://archive.ubuntu.com/ubuntu xenial-updates/main Translation-en [432 kB]
Step #0: Get:26 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [7616 B]
Step #0: Get:27 http://archive.ubuntu.com/ubuntu xenial-updates/restricted Translation-en [2272 B]
Step #0: Get:28 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [797 kB]
Step #0: Get:29 http://archive.ubuntu.com/ubuntu xenial-updates/universe Translation-en [333 kB]
Step #0: Get:30 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [17.1 kB]
Step #0: Get:31 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse Translation-en [8632 B]
Step #0: Get:32 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages [7280 B]
Step #0: Get:33 http://archive.ubuntu.com/ubuntu xenial-backports/main Translation-en [4456 B]
Step #0: Get:34 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages [8064 B]
Step #0: Get:35 http://archive.ubuntu.com/ubuntu xenial-backports/universe Translation-en [4328 B]
Step #0: Fetched 19.2 MB in 4s (4123 kB/s)
Step #0: Reading package lists...
Step #0: Reading package lists...
Step #0: Building dependency tree...
Step #0: Reading state information...
Step #0: The following packages were automatically installed and are no longer required:
Step #0:   cpp cpp-5 gcc-5 libasan2 libatomic1 libc-dev-bin libc6-dev libcc1-0
Step #0:   libcilkrts5 libexpat1-dev libgcc-5-dev libgomp1 libisl15 libitm1 liblsan0
Step #0:   libmpc3 libmpfr4 libmpx0 libpython-dev libpython2.7 libpython2.7-dev
Step #0:   libquadmath0 libtsan0 libubsan0 linux-libc-dev manpages-dev
Step #0:   python-pkg-resources python2.7-dev
Step #0: Use 'apt autoremove' to remove them.
Step #0: The following NEW packages will be installed:
Step #0:   wget
Step #0: debconf: delaying package configuration, since apt-utils is not installed
Step #0: 0 upgraded, 1 newly installed, 0 to remove and 67 not upgraded.
Step #0: Need to get 0 B/299 kB of archives.
Step #0: After this operation, 905 kB of additional disk space will be used.
Step #0: Selecting previously unselected package wget.
(Reading database ... 17441 files and directories currently installed.)
Step #0: Preparing to unpack .../wget_1.17.1-1ubuntu1.5_amd64.deb ...
Step #0: Unpacking wget (1.17.1-1ubuntu1.5) ...
Step #0: Setting up wget (1.17.1-1ubuntu1.5) ...
Step #0: --2020-05-02 16:08:04--  https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv3.4.0/kustomize_v3.4.0_linux_amd64.tar.gz
Step #0: Resolving github.com (github.com)... 140.82.114.3
Step #0: Connecting to github.com (github.com)|140.82.114.3|:443... connected.
Step #0: HTTP request sent, awaiting response... 302 Found
Step #0: Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/133067498/e3329480-04c6-11ea-8ace-611b2004e838?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200502%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200502T160804Z&X-Amz-Expires=300&X-Amz-Signature=28be438ecd32e7901d3dfe375331ab6072323917f10164bc2c82f3f6114ea49a&X-Amz-SignedHeaders=host&actor_id=0&repo_id=133067498&response-content-disposition=attachment%3B%20filename%3Dkustomize_v3.4.0_linux_amd64.tar.gz&response-content-type=application%2Foctet-stream [following]
Step #0: --2020-05-02 16:08:04--  https://github-production-release-asset-2e65be.s3.amazonaws.com/133067498/e3329480-04c6-11ea-8ace-611b2004e838?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200502%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200502T160804Z&X-Amz-Expires=300&X-Amz-Signature=28be438ecd32e7901d3dfe375331ab6072323917f10164bc2c82f3f6114ea49a&X-Amz-SignedHeaders=host&actor_id=0&repo_id=133067498&response-content-disposition=attachment%3B%20filename%3Dkustomize_v3.4.0_linux_amd64.tar.gz&response-content-type=application%2Foctet-stream
Step #0: Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.89.148
Step #0: Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.89.148|:443... connected.
Step #0: HTTP request sent, awaiting response... 200 OK
Step #0: Length: 8496221 (8.1M) [application/octet-stream]
Step #0: Saving to: 'kustomize_v3.4.0_linux_amd64.tar.gz'
Step #0:
Step #0:      0K .......... .......... .......... .......... ..........  0%  870K 9s
Step #0:     50K .......... .......... .......... .......... ..........  1% 1.68M 7s
Step #0:    100K .......... .......... .......... .......... ..........  1% 99.3M 5s
Step #0:    150K .......... .......... .......... .......... ..........  2%  194M 4s
Step #0:    200K .......... .......... .......... .......... ..........  3% 1.64M 4s
Step #0:    250K .......... .......... .......... .......... ..........  3% 68.2M 3s
Step #0:    300K .......... .......... .......... .......... ..........  4%  207M 3s
Step #0:    350K .......... .......... .......... .......... ..........  4%  218M 2s
Step #0:    400K .......... .......... .......... .......... ..........  5% 1.74M 3s
Step #0:    450K .......... .......... .......... .......... ..........  6%  229M 2s
Step #0:    500K .......... .......... .......... .......... ..........  6%  219M 2s
Step #0:    550K .......... .......... .......... .......... ..........  7%  229M 2s
Step #0:    600K .......... .......... .......... .......... ..........  7%  216M 2s
Step #0:    650K .......... .......... .......... .......... ..........  8%  231M 2s
Step #0:    700K .......... .......... .......... .......... ..........  9%  236M 1s
Step #0:    750K .......... .......... .......... .......... ..........  9%  224M 1s
Step #0:    800K .......... .......... .......... .......... .......... 10%  221M 1s
Step #0:    850K .......... .......... .......... .......... .......... 10% 1.67M 1s
Step #0:    900K .......... .......... .......... .......... .......... 11%  184M 1s
Step #0:    950K .......... .......... .......... .......... .......... 12%  232M 1s
Step #0:   1000K .......... .......... .......... .......... .......... 12%  227M 1s
Step #0:   1050K .......... .......... .......... .......... .......... 13%  196M 1s
Step #0:   1100K .......... .......... .......... .......... .......... 13%  196M 1s
Step #0:   1150K .......... .......... .......... .......... .......... 14%  219M 1s
Step #0:   1200K .......... .......... .......... .......... .......... 15%  231M 1s
Step #0:   1250K .......... .......... .......... .......... .......... 15%  221M 1s
Step #0:   1300K .......... .......... .......... .......... .......... 16%  254M 1s
Step #0:   1350K .......... .......... .......... .......... .......... 16% 8.22M 1s
Step #0:   1400K .......... .......... .......... .......... .......... 17%  193M 1s
Step #0:   1450K .......... .......... .......... .......... .......... 18%  232M 1s
Step #0:   1500K .......... .......... .......... .......... .......... 18%  255M 1s
Step #0:   1550K .......... .......... .......... .......... .......... 19%  264M 1s
Step #0:   1600K .......... .......... .......... .......... .......... 19%  239M 1s
Step #0:   1650K .......... .......... .......... .......... .......... 20%  262M 1s
Step #0:   1700K .......... .......... .......... .......... .......... 21%  274M 1s
Step #0:   1750K .......... .......... .......... .......... .......... 21% 2.68M 1s
Step #0:   1800K .......... .......... .......... .......... .......... 22% 75.3M 1s
Step #0:   1850K .......... .......... .......... .......... .......... 22% 13.5M 1s
Step #0:   1900K .......... .......... .......... .......... .......... 23%  140M 1s
Step #0:   1950K .......... .......... .......... .......... .......... 24%  133M 1s
Step #0:   2000K .......... .......... .......... .......... .......... 24% 76.4M 1s
Step #0:   2050K .......... .......... .......... .......... .......... 25%  255M 1s
Step #0:   2100K .......... .......... .......... .......... .......... 25%  274M 1s
Step #0:   2150K .......... .......... .......... .......... .......... 26%  263M 1s
Step #0:   2200K .......... .......... .......... .......... .......... 27%  235M 1s
Step #0:   2250K .......... .......... .......... .......... .......... 27%  268M 1s
Step #0:   2300K .......... .......... .......... .......... .......... 28%  284M 1s
Step #0:   2350K .......... .......... .......... .......... .......... 28%  265M 1s
Step #0:   2400K .......... .......... .......... .......... .......... 29% 41.8M 1s
Step #0:   2450K .......... .......... .......... .......... .......... 30%  246M 0s
Step #0:   2500K .......... .......... .......... .......... .......... 30% 11.3M 0s
Step #0:   2550K .......... .......... .......... .......... .......... 31% 77.5M 0s
Step #0:   2600K .......... .......... .......... .......... .......... 31%  225M 0s
Step #0:   2650K .......... .......... .......... .......... .......... 32%  284M 0s
Step #0:   2700K .......... .......... .......... .......... .......... 33%  273M 0s
Step #0:   2750K .......... .......... .......... .......... .......... 33%  270M 0s
Step #0:   2800K .......... .......... .......... .......... .......... 34%  257M 0s
Step #0:   2850K .......... .......... .......... .......... .......... 34%  268M 0s
Step #0:   2900K .......... .......... .......... .......... .......... 35%  287M 0s
Step #0:   2950K .......... .......... .......... .......... .......... 36%  283M 0s
Step #0:   3000K .......... .......... .......... .......... .......... 36%  221M 0s
Step #0:   3050K .......... .......... .......... .......... .......... 37%  266M 0s
Step #0:   3100K .......... .......... .......... .......... .......... 37%  281M 0s
Step #0:   3150K .......... .......... .......... .......... .......... 38%  283M 0s
Step #0:   3200K .......... .......... .......... .......... .......... 39%  254M 0s
Step #0:   3250K .......... .......... .......... .......... .......... 39%  270M 0s
Step #0:   3300K .......... .......... .......... .......... .......... 40%  271M 0s
Step #0:   3350K .......... .......... .......... .......... .......... 40%  278M 0s
Step #0:   3400K .......... .......... .......... .......... .......... 41%  233M 0s
Step #0:   3450K .......... .......... .......... .......... .......... 42% 10.1M 0s
Step #0:   3500K .......... .......... .......... .......... .......... 42% 9.59M 0s
Step #0:   3550K .......... .......... .......... .......... .......... 43%  199M 0s
Step #0:   3600K .......... .......... .......... .......... .......... 43%  222M 0s
Step #0:   3650K .......... .......... .......... .......... .......... 44%  273M 0s
Step #0:   3700K .......... .......... .......... .......... .......... 45%  277M 0s
Step #0:   3750K .......... .......... .......... .......... .......... 45% 7.97M 0s
Step #0:   3800K .......... .......... .......... .......... .......... 46% 7.59M 0s
Step #0:   3850K .......... .......... .......... .......... .......... 47%  259M 0s
Step #0:   3900K .......... .......... .......... .......... .......... 47%  276M 0s
Step #0:   3950K .......... .......... .......... .......... .......... 48%  286M 0s
Step #0:   4000K .......... .......... .......... .......... .......... 48%  254M 0s
Step #0:   4050K .......... .......... .......... .......... .......... 49% 17.4M 0s
Step #0:   4100K .......... .......... .......... .......... .......... 50%  259M 0s
Step #0:   4150K .......... .......... .......... .......... .......... 50%  268M 0s
Step #0:   4200K .......... .......... .......... .......... .......... 51%  228M 0s
Step #0:   4250K .......... .......... .......... .......... .......... 51%  279M 0s
Step #0:   4300K .......... .......... .......... .......... .......... 52%  270M 0s
Step #0:   4350K .......... .......... .......... .......... .......... 53%  275M 0s
Step #0:   4400K .......... .......... .......... .......... .......... 53%  227M 0s
Step #0:   4450K .......... .......... .......... .......... .......... 54%  271M 0s
Step #0:   4500K .......... .......... .......... .......... .......... 54%  283M 0s
Step #0:   4550K .......... .......... .......... .......... .......... 55%  266M 0s
Step #0:   4600K .......... .......... .......... .......... .......... 56%  231M 0s
Step #0:   4650K .......... .......... .......... .......... .......... 56%  285M 0s
Step #0:   4700K .......... .......... .......... .......... .......... 57%  273M 0s
Step #0:   4750K .......... .......... .......... .......... .......... 57%  280M 0s
Step #0:   4800K .......... .......... .......... .......... .......... 58%  241M 0s
Step #0:   4850K .......... .......... .......... .......... .......... 59%  272M 0s
Step #0:   4900K .......... .......... .......... .......... .......... 59%  281M 0s
Step #0:   4950K .......... .......... .......... .......... .......... 60%  286M 0s
Step #0:   5000K .......... .......... .......... .......... .......... 60%  227M 0s
Step #0:   5050K .......... .......... .......... .......... .......... 61%  270M 0s
Step #0:   5100K .......... .......... .......... .......... .......... 62%  286M 0s
Step #0:   5150K .......... .......... .......... .......... .......... 62%  253M 0s
Step #0:   5200K .......... .......... .......... .......... .......... 63%  244M 0s
Step #0:   5250K .......... .......... .......... .......... .......... 63%  275M 0s
Step #0:   5300K .......... .......... .......... .......... .......... 64%  282M 0s
Step #0:   5350K .......... .......... .......... .......... .......... 65%  270M 0s
Step #0:   5400K .......... .......... .......... .......... .......... 65%  234M 0s
Step #0:   5450K .......... .......... .......... .......... .......... 66% 48.6M 0s
Step #0:   5500K .......... .......... .......... .......... .......... 66% 14.2M 0s
Step #0:   5550K .......... .......... .......... .......... .......... 67%  274M 0s
Step #0:   5600K .......... .......... .......... .......... .......... 68%  229M 0s
Step #0:   5650K .......... .......... .......... .......... .......... 68%  285M 0s
Step #0:   5700K .......... .......... .......... .......... .......... 69%  285M 0s
Step #0:   5750K .......... .......... .......... .......... .......... 69%  269M 0s
Step #0:   5800K .......... .......... .......... .......... .......... 70% 18.7M 0s
Step #0:   5850K .......... .......... .......... .......... .......... 71% 32.6M 0s
Step #0:   5900K .......... .......... .......... .......... .......... 71%  278M 0s
Step #0:   5950K .......... .......... .......... .......... .......... 72%  285M 0s
Step #0:   6000K .......... .......... .......... .......... .......... 72%  258M 0s
Step #0:   6050K .......... .......... .......... .......... .......... 73%  285M 0s
Step #0:   6100K .......... .......... .......... .......... .......... 74%  253M 0s
Step #0:   6150K .......... .......... .......... .......... .......... 74%  281M 0s
Step #0:   6200K .......... .......... .......... .......... .......... 75%  173M 0s
Step #0:   6250K .......... .......... .......... .......... .......... 75%  128M 0s
Step #0:   6300K .......... .......... .......... .......... .......... 76%  141M 0s
Step #0:   6350K .......... .......... .......... .......... .......... 77%  267M 0s
Step #0:   6400K .......... .......... .......... .......... .......... 77%  244M 0s
Step #0:   6450K .......... .......... .......... .......... .......... 78%  287M 0s
Step #0:   6500K .......... .......... .......... .......... .......... 78% 10.0M 0s
Step #0:   6550K .......... .......... .......... .......... .......... 79%  262M 0s
Step #0:   6600K .......... .......... .......... .......... .......... 80%  212M 0s
Step #0:   6650K .......... .......... .......... .......... .......... 80%  282M 0s
Step #0:   6700K .......... .......... .......... .......... .......... 81%  285M 0s
Step #0:   6750K .......... .......... .......... .......... .......... 81%  283M 0s
Step #0:   6800K .......... .......... .......... .......... .......... 82%  255M 0s
Step #0:   6850K .......... .......... .......... .......... .......... 83%  267M 0s
Step #0:   6900K .......... .......... .......... .......... .......... 83%  267M 0s
Step #0:   6950K .......... .......... .......... .......... .......... 84%  288M 0s
Step #0:   7000K .......... .......... .......... .......... .......... 84%  237M 0s
Step #0:   7050K .......... .......... .......... .......... .......... 85% 7.36M 0s
Step #0:   7100K .......... .......... .......... .......... .......... 86%  102M 0s
Step #0:   7150K .......... .......... .......... .......... .......... 86%  257M 0s
Step #0:   7200K .......... .......... .......... .......... .......... 87%  248M 0s
Step #0:   7250K .......... .......... .......... .......... .......... 87%  262M 0s
Step #0:   7300K .......... .......... .......... .......... .......... 88%  276M 0s
Step #0:   7350K .......... .......... .......... .......... .......... 89%  267M 0s
Step #0:   7400K .......... .......... .......... .......... .......... 89%  213M 0s
Step #0:   7450K .......... .......... .......... .......... .......... 90%  280M 0s
Step #0:   7500K .......... .......... .......... .......... .......... 90%  262M 0s
Step #0:   7550K .......... .......... .......... .......... .......... 91% 6.19M 0s
Step #0:   7600K .......... .......... .......... .......... .......... 92% 11.0M 0s
Step #0:   7650K .......... .......... .......... .......... .......... 92%  240M 0s
Step #0:   7700K .......... .......... .......... .......... .......... 93% 37.9M 0s
Step #0:   7750K .......... .......... .......... .......... .......... 94%  256M 0s
Step #0:   7800K .......... .......... .......... .......... .......... 94%  229M 0s
Step #0:   7850K .......... .......... .......... .......... .......... 95%  280M 0s
Step #0:   7900K .......... .......... .......... .......... .......... 95%  254M 0s
Step #0:   7950K .......... .......... .......... .......... .......... 96%  274M 0s
Step #0:   8000K .......... .......... .......... .......... .......... 97%  240M 0s
Step #0:   8050K .......... .......... .......... .......... .......... 97%  280M 0s
Step #0:   8100K .......... .......... .......... .......... .......... 98%  277M 0s
Step #0:   8150K .......... .......... .......... .......... .......... 98%  257M 0s
Step #0:   8200K .......... .......... .......... .......... .......... 99%  234M 0s
Step #0:   8250K .......... .......... .......... .......... .......   100%  271M=0.3s
Step #0:
Step #0: 2020-05-02 16:08:05 (27.3 MB/s) - 'kustomize_v3.4.0_linux_amd64.tar.gz' saved [8496221/8496221]
Step #0:
Step #0: kustomize
Step #0: Reading package lists...
Step #0: Building dependency tree...
Step #0: Reading state information...
Step #0: The following packages were automatically installed and are no longer required:
Step #0:   cpp cpp-5 gcc-5 libasan2 libatomic1 libc-dev-bin libc6-dev libcc1-0
Step #0:   libcilkrts5 libexpat1-dev libgcc-5-dev libgomp1 libisl15 libitm1 liblsan0
Step #0:   libmpc3 libmpfr4 libmpx0 libpython-dev libpython2.7 libpython2.7-dev
Step #0:   libquadmath0 libtsan0 libubsan0 linux-libc-dev manpages-dev
Step #0:   python-pkg-resources python2.7-dev
Step #0: Use 'apt autoremove' to remove them.
Step #0: The following packages will be REMOVED:
Step #0:   wget*
Step #0: 0 upgraded, 0 newly installed, 1 to remove and 67 not upgraded.
Step #0: After this operation, 905 kB disk space will be freed.
(Reading database ... 17451 files and directories currently installed.)
Step #0: Removing wget (1.17.1-1ubuntu1.5) ...
Step #0: Purging configuration files for wget (1.17.1-1ubuntu1.5) ...
Step #0: Reading package lists...
Step #0: Building dependency tree...
Step #0: Reading state information...
Step #0: The following packages will be REMOVED:
Step #0:   cpp* cpp-5* gcc-5* libasan2* libatomic1* libc-dev-bin* libc6-dev* libcc1-0*
Step #0:   libcilkrts5* libexpat1-dev* libgcc-5-dev* libgomp1* libisl15* libitm1*
Step #0:   liblsan0* libmpc3* libmpfr4* libmpx0* libpython-dev* libpython2.7*
Step #0:   libpython2.7-dev* libquadmath0* libtsan0* libubsan0* linux-libc-dev*
Step #0:   manpages-dev* python-pkg-resources* python2.7-dev*
Step #0: 0 upgraded, 0 newly installed, 28 to remove and 63 not upgraded.
Step #0: After this operation, 133 MB disk space will be freed.
(Reading database ... 17440 files and directories currently installed.)
Step #0: Removing cpp (4:5.3.1-1ubuntu1) ...
Step #0: Removing gcc-5 (5.4.0-6ubuntu1~16.04.12) ...
Step #0: Removing cpp-5 (5.4.0-6ubuntu1~16.04.12) ...
Step #0: Removing libgcc-5-dev:amd64 (5.4.0-6ubuntu1~16.04.12) ...
Step #0: Removing libasan2:amd64 (5.4.0-6ubuntu1~16.04.12) ...
Step #0: Removing libatomic1:amd64 (5.4.0-6ubuntu1~16.04.12) ...
Step #0: Removing python2.7-dev (2.7.12-1ubuntu0~16.04.9) ...
Step #0: Removing libpython-dev:amd64 (2.7.12-1~16.04) ...
Step #0: Removing libpython2.7-dev:amd64 (2.7.12-1ubuntu0~16.04.9) ...
Step #0: Removing libexpat1-dev:amd64 (2.1.0-7ubuntu0.16.04.5) ...
Step #0: Removing libc6-dev:amd64 (2.23-0ubuntu11) ...
Step #0: Removing libc-dev-bin (2.23-0ubuntu11) ...
Step #0: Removing libcc1-0:amd64 (5.4.0-6ubuntu1~16.04.12) ...
Step #0: Removing libcilkrts5:amd64 (5.4.0-6ubuntu1~16.04.12) ...
Step #0: Removing libgomp1:amd64 (5.4.0-6ubuntu1~16.04.12) ...
Step #0: Removing libisl15:amd64 (0.16.1-1) ...
Step #0: Removing libitm1:amd64 (5.4.0-6ubuntu1~16.04.12) ...
Step #0: Removing liblsan0:amd64 (5.4.0-6ubuntu1~16.04.12) ...
Step #0: Removing libmpc3:amd64 (1.0.3-1) ...
Step #0: Purging configuration files for libmpc3:amd64 (1.0.3-1) ...
Step #0: Removing libmpfr4:amd64 (3.1.4-1) ...
Step #0: Removing libmpx0:amd64 (5.4.0-6ubuntu1~16.04.12) ...
Step #0: Removing libpython2.7:amd64 (2.7.12-1ubuntu0~16.04.9) ...
Step #0: Removing libquadmath0:amd64 (5.4.0-6ubuntu1~16.04.12) ...
Step #0: Removing libtsan0:amd64 (5.4.0-6ubuntu1~16.04.12) ...
Step #0: Removing libubsan0:amd64 (5.4.0-6ubuntu1~16.04.12) ...
Step #0: Removing linux-libc-dev:amd64 (4.4.0-176.206) ...
Step #0: Removing manpages-dev (4.04-2) ...
Step #0: Removing python-pkg-resources (20.7.0-1) ...
Step #0: Processing triggers for libc-bin (2.23-0ubuntu10) ...
Step #0: Removing intermediate container cc446cdec4e6
Step #0:  ---> 1324a1aa600e
Step #0: Step 6/7 : ENV PATH=/builder/kustomize:$PATH
Step #0:  ---> Running in b71c623cc1b7
Step #0: Removing intermediate container b71c623cc1b7
Step #0:  ---> 63187d0f3177
Step #0: Step 7/7 : ENTRYPOINT ["/builder/kustomize.bash"]
Step #0:  ---> Running in 8de8de4ea6a1
Step #0: Removing intermediate container 8de8de4ea6a1
Step #0:  ---> bd67e188d43b
Step #0: Successfully built bd67e188d43b
Step #0: Successfully tagged gcr.io/righm9/kustomize:latest
Finished Step #0
Starting Step #1
Step #1: Already have image: gcr.io/righm9/kustomize
Step #1: {Version:kustomize/v3.4.0 GitCommit:2c9635967a2b1469d605a91a1d040bd27c73ca7d BuildDate:2019-11-12T05:00:57Z GoOs:linux GoArch:amd64}
Finished Step #1
PUSH
Pushing gcr.io/righm9/kustomize
The push refers to repository [gcr.io/righm9/kustomize]
0dfc862af1d8: Preparing
fba1ae934d4f: Preparing
bf56ea947895: Preparing
b40c899868d8: Preparing
84ff92691f90: Preparing
713b651b89c5: Preparing
2bccb5ac4b09: Preparing
713b651b89c5: Waiting
2bccb5ac4b09: Waiting
bf56ea947895: Mounted from cloud-builders/git
84ff92691f90: Mounted from cloud-builders/wget
b40c899868d8: Mounted from cloud-builders/gsutil
713b651b89c5: Mounted from cloud-builders/wget
2bccb5ac4b09: Mounted from cloud-builders/wget
fba1ae934d4f: Pushed
0dfc862af1d8: Pushed
latest: digest: sha256:ff21699d3581aa19ce249e695270df2e4c1919c2ee1b9d0851d5dc982cb8bae0 size: 1790
DONE
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

ID                                    CREATE_TIME                DURATION  SOURCE                                                                            IMAGES                             STATUS
42caf256-9873-4886-88b8-0096d239cf71  2020-05-02T16:07:46+00:00  32S       gs://righm9_cloudbuild/source/1588435618.22-b17a4c75902448f98d85fa42a590cb86.tgz  gcr.io/righm9/kustomize (+1 more)  SUCCESS

これで kustomize リポジトリができているはずなので確認してみてください。

GKE に Deploy する

ようやくここからが本番です。

ここでも gcloud builds submit コマンドを使ってビルドをしていきます。 先程使った cloudbuild.yaml と同じファイル名ですが中身は異なり、前のセクションで作成した kustomize リポジトリを使って production の Kustomization をビルドしています。

bazel-sample-project/cloudbuild.yaml
steps:
- id: deploy gateway
  name: 'gcr.io/$PROJECT_ID/kustomize'
  args:
  - 'build'
  - 'kube/services/gateway/overlays/production'
  env:
  - 'APPLY=true'
  - 'CLOUDSDK_COMPUTE_ZONE=asia-northeast1-a'
  - 'CLOUDSDK_CONTAINER_CLUSTER=sample-cluster'
  - 'GCLOUD_PROJECT=righm9'

- id: deploy echo
  name: 'gcr.io/$PROJECT_ID/kustomize'
  args:
  - 'build'
  - 'kube/services/echo/overlays/production'
  env:
  - 'APPLY=true'
  - 'CLOUDSDK_COMPUTE_ZONE=asia-northeast1-a'
  - 'CLOUDSDK_CONTAINER_CLUSTER=sample-cluster'
  - 'GCLOUD_PROJECT=righm9'
$ gcloud builds submit --config cloudbuild.yaml
Creating temporary tarball archive of 33 file(s) totalling 28.8 KiB before compression.
Some files were not included in the source upload.

Check the gcloud log [/Users/righ/.config/gcloud/logs/2020.05.03/22.58.35.984271.log] to see which files and the contents of the
default gcloudignore file used (see `$ gcloud topic gcloudignore` to learn
more).

Uploading tarball of [.] to [gs://righm9_cloudbuild/source/1588514316.06-6d59d2dd01774271bd0131c108cbf44b.tgz]
Created [https://cloudbuild.googleapis.com/v1/projects/righm9/builds/b22a262a-6792-427f-aba8-c19b09d1b2a3].
Logs are available at [https://console.cloud.google.com/cloud-build/builds/b22a262a-6792-427f-aba8-c19b09d1b2a3?project=846141039915].
------------------------------------------------- REMOTE BUILD OUTPUT -------------------------------------------------
starting build "b22a262a-6792-427f-aba8-c19b09d1b2a3"

FETCHSOURCE
Fetching storage object: gs://righm9_cloudbuild/source/1588514316.06-6d59d2dd01774271bd0131c108cbf44b.tgz#1588514317616235
Error: Exec command aa533ebfef3c9a50f5b5ee4efd886e8c7f2f80b45f8378cc2480eb70dbc9fee3 is already running
BUILD
Starting Step #0 - "deploy gateway"
Step #0 - "deploy gateway": Pulling image: gcr.io/righm9/kustomize
Step #0 - "deploy gateway": Using default tag: latest
Step #0 - "deploy gateway": latest: Pulling from righm9/kustomize
Step #0 - "deploy gateway": 75f546e73d8b: Already exists
Step #0 - "deploy gateway": 0f3bb76fc390: Already exists
Step #0 - "deploy gateway": 3c2cba919283: Already exists
Step #0 - "deploy gateway": 5a992b2091ae: Already exists
Step #0 - "deploy gateway": dc685810031f: Already exists
Step #0 - "deploy gateway": 0325363c9dcf: Pulling fs layer
Step #0 - "deploy gateway": cfe0bb10a3dc: Pulling fs layer
Step #0 - "deploy gateway": 0325363c9dcf: Verifying Checksum
Step #0 - "deploy gateway": 0325363c9dcf: Download complete
Step #0 - "deploy gateway": 0325363c9dcf: Pull complete
Step #0 - "deploy gateway": cfe0bb10a3dc: Verifying Checksum
Step #0 - "deploy gateway": cfe0bb10a3dc: Download complete
Step #0 - "deploy gateway": cfe0bb10a3dc: Pull complete
Step #0 - "deploy gateway": Digest: sha256:0a4a22993c7380cc57e145112a22e0cc9ccb2dfb1e1ec376e0ee6d141179b04c
Step #0 - "deploy gateway": Status: Downloaded newer image for gcr.io/righm9/kustomize:latest
Step #0 - "deploy gateway": gcr.io/righm9/kustomize:latest
Step #0 - "deploy gateway": Running: gcloud container clusters get-credentials --project="righm9" --zone="asia-northeast1-a" "sample-cluster"
Step #0 - "deploy gateway": Fetching cluster endpoint and auth data.
Step #0 - "deploy gateway": kubeconfig entry generated for sample-cluster.
Step #0 - "deploy gateway": configmap/nginx-conf unchanged
Step #0 - "deploy gateway": service/gateway unchanged
Step #0 - "deploy gateway": deployment.apps/gateway created
Finished Step #0 - "deploy gateway"
Starting Step #1 - "deploy echo"
Step #1 - "deploy echo": Already have image (with digest): gcr.io/righm9/kustomize
Step #1 - "deploy echo": service/echo unchanged
Step #1 - "deploy echo": deployment.apps/echo created
Finished Step #1 - "deploy echo"
PUSH
DONE
-----------------------------------------------------------------------------------------------------------------------

ID                                    CREATE_TIME                DURATION  SOURCE                                                                            IMAGES  STATUS
b22a262a-6792-427f-aba8-c19b09d1b2a3  2020-05-03T13:58:39+00:00  18S       gs://righm9_cloudbuild/source/1588514316.06-6d59d2dd01774271bd0131c108cbf44b.tgz  -       SUCCESS

これだけです。

ワークロードを見てみると Push されていることがわかりますね。

workload.png

備考

production の kustomization.yaml では GCR に Push したイメージを使うようにベースの定義にパッチしています。

  • 📁 kube
    • 📁 skaffold
      • 🗒 kustomization.yaml
    • 📁 services
      • 📁 echo
        • 📁 overlays
          • 📁 local
            • 🗒 kustomization.yaml
          • 📁 production
            • 🗒 kustomization.yaml
            • 🗒 patch.yaml
        • 📁 base
          • 🗒 deployment.yaml
          • 🗒 kustomization.yaml
          • 🗒 service.yaml
      • 📁 gateway
        • 📁 overlays
          • 📁 local
            • 🗒 kustomization.yaml
            • 🗒 service.yaml
          • 📁 production
            • 🗒 kustomization.yaml
            • 🗒 service.yaml
            • 🗒 patch.yaml
        • 📁 base
          • 🗒 deployment.yaml
          • 🗒 kustomization.yaml
          • 🗒 service.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
  - ../services/echo/overlays/local
  - ../services/gateway/overlays/local
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base
patchesJson6902:
- path: patch.yaml
  target:
    kind: Deployment
    name: echo
    group: apps
    version: v1
- op: replace
  path: /spec/template/spec/containers/0/image
  value: gcr.io/righm9/echo:latest
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
spec:
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
      name: echo
    spec:
      containers:
      - image: righm9/echo
        name: echo
        ports:
          - containerPort: 8001
            name: grpc
            protocol: TCP
        resources:
          limits:
            cpu: 50m
            memory: 200Mi
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
apiVersion: v1
kind: Service
metadata:
  name: echo
spec:
  ports:
    - name: grpc
      port: 8001
      protocol: TCP
      targetPort: 8001
  selector:
    app: echo
  type: ClusterIP
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base
patchesStrategicMerge:
- service.yaml
apiVersion: v1
kind: Service
metadata:
  name: gateway
spec:
  type: LoadBalancer
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base
patchesStrategicMerge:
- service.yaml
patchesJson6902:
- path: patch.yaml
  target:
    kind: Deployment
    name: gateway
    group: apps
    version: v1
apiVersion: v1
kind: Service
metadata:
  name: gateway
spec:
  type: LoadBalancer
- op: replace
  path: /spec/template/spec/containers/0/image
  value: gcr.io/righm9/gateway:latest
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gateway
spec:
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: gateway
  template:
    metadata:
      labels:
        app: gateway
      name: gateway
    spec:
      containers:
      - image: righm9/gateway
        name: gateway
        env:
          - name: ECHO_HOST
            value: echo.default.svc.cluster.local
        ports:
          - containerPort: 8000
            name: grpc
            protocol: TCP
        resources:
          limits:
            cpu: 50m
            memory: 200Mi
      
      - image: nginx:1.17
        name: nginx
        ports:
          - containerPort: 80
            name: http
            protocol: TCP
        resources:
          limits:
            cpu: 50m
            memory: 200Mi
        volumeMounts:
          - name: nginx-conf
            mountPath: /etc/nginx/nginx.conf
            subPath: nginx.conf
      volumes:
        - name: nginx-conf
          configMap:
            name: nginx-conf

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-conf
data:
  nginx.conf: |
    user nginx;
    worker_processes  3;
    error_log  /var/log/nginx/error.log;
    events {
      worker_connections  10240;
    }
    http {
      log_format  main
              'remote_addr:$remote_addr\t'
              'time_local:$time_local\t'
              'method:$request_method\t'
              'uri:$request_uri\t'
              'host:$host\t'
              'status:$status\t'
              'bytes_sent:$body_bytes_sent\t'
              'referer:$http_referer\t'
              'useragent:$http_user_agent\t'
              'forwardedfor:$http_x_forwarded_for\t'
              'request_time:$request_time';
      access_log	/var/log/nginx/access.log main;
      server {
          listen       80;
          server_name  _;
          location / {
              root   html;
              proxy_pass http://localhost:8000;
          }
      }
    }
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
apiVersion: v1
kind: Service
metadata:
  name: gateway
spec:
  ports:
    - name: app
      port: 8000
      protocol: TCP
      targetPort: 8000
    - name: server
      port: 80
      protocol: TCP
      targetPort: 80
  selector:
    app: gateway
  type: ClusterIP

最後に gateway の IP にアクセス&動作確認してみます。

gke-helloworld.png

(もうアクセスできません)

無事動いたのでおしまい。

参考