[GCP] Cloud Run으로 배포된 Application을 Canary 배포테스트

0. 개요

자동화된 카나리아 테스트 및 백분율 기반 트래픽 관리를 통해 개발자 분기에서 프로덕션으로 코드 진행을 실행하는 Cloud Run용 배포 파이프라인을 구현합니다.

 

1. 환경준비

GCP Console의 Cloud Shell 에서 진행합니다.

export PROJECT_ID=$(gcloud config get-value project)
export PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format='value(projectNumber)')
export REGION=us-east1
gcloud config set compute/region $REGION

Project, Region 변수 선언 및 gcloud region 세팅

 

gcloud services enable \
cloudresourcemanager.googleapis.com \
container.googleapis.com \
sourcerepo.googleapis.com \
cloudbuild.googleapis.com \
containerregistry.googleapis.com \
run.googleapis.com

API 활성화

• Cloud Resource Manager
• GKE
• Cloud Source Repositories
• Cloud Build
• Container Registry
• Cloud Run

gcloud projects add-iam-policy-binding $PROJECT_ID \
--member=serviceAccount:$PROJECT_NUMBER@cloudbuild.gserviceaccount.com \
--role=roles/run.admin

Cloud Build Service Accountd에 Cloud Run Admin 권한 부여

 

gcloud iam service-accounts add-iam-policy-binding \
$PROJECT_NUMBER-compute@developer.gserviceaccount.com \
--member=serviceAccount:$PROJECT_NUMBER@cloudbuild.gserviceaccount.com \
--role=roles/iam.serviceAccountUser

Cloud Run Runtime Service Account의 Cloud Build Service Account에 IAM 사용자역할 부

git config --global user.email "majjangjjang@example.com"
git config --global user.name "majjangjjang"

git 환경설정

 

git clone https://github.com/GoogleCloudPlatform/software-delivery-workshop cloudrun-progression
cd cloudrun-progression/labs/cloudrun-progression
rm -rf ../../.git

sample App 복사

 

vi branch-cloudbuild.yaml
vi master-cloudbuild.yaml
vi tag-cloudbuild.yaml

substitutions:
  _SERVICE_NAME: hello-cloudrun
  _REGION: us-east1

Sample code중 branch-cloudbuild.yaml, master-cloudbuild.yaml, tag-cloudbuild.yaml의 region이 us-central1로 되어있어 해당 예시는 us-east1 이므로 수정해줘야함

 

sed "s/PROJECT/${PROJECT_ID}/g" branch-trigger.json-tmpl > branch-trigger.json
sed "s/PROJECT/${PROJECT_ID}/g" master-trigger.json-tmpl > master-trigger.json
sed "s/PROJECT/${PROJECT_ID}/g" tag-trigger.json-tmpl > tag-trigger.json

trigger시에 필요한 변수 선언

 

gcloud source repos create cloudrun-progression
git init
git config credential.helper gcloud.sh
git remote add gcp https://source.developers.google.com/p/$PROJECT_ID/r/cloudrun-progression
git branch -m master
git add . && git commit -m "initial commit"
git push gcp master

Google Source Repository를 만들고 sample app push 하기

 

 

1. Sample App Cloud Build로 이미지 생성후 Cloud Run으로 배포

gcloud builds submit --tag gcr.io/$PROJECT_ID/hello-cloudrun
gcloud run deploy hello-cloudrun \
--image gcr.io/$PROJECT_ID/hello-cloudrun \
--platform managed \
--region $REGION \
--tag=prod -q

공개 서비스로 생성시에는 --allow-unauthenticated  옵션추가

https://cloud.google.com/run/docs/authenticating/public 관련링크

 

배포된 모습

 

 

PROD_URL=$(gcloud run services describe hello-cloudrun --platform managed --region $REGION --format=json | jq --raw-output ".status.url")
echo $PROD_URL
curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" $PROD_URL

### Result
Hello World v1.0

인증된 사용자로 응답 확인 시 URL이 나타남

 

 

2. Dynamic Developer 배포활성화

Git의 개발 브랜치에 대한 고유한 URL을 사용하여 개발자를 활성화합니다.각 분기는 분기 이름으로 식별되는 URL로 표시됩니다. 브랜치에 커밋하면 배포가 트리거되고 동일한 URL에서 업데이트에 액세스할 수 있습니다.

 

gcloud beta builds triggers create cloud-source-repositories --trigger-config branch-trigger.json

### branch-trigger.json

{
  "triggerTemplate": {
    "projectId": "qwiklabs-gcp-03-6a484735a043",
    "repoName": "cloudrun-progression",
    "branchName": "[^(?!.*master)].*"
  },
  "name": "branch",
  "description": "Trigger dev build/deploy for any branch other than master",

  "filename": "branch-cloudbuild.yaml"

### branch-cloudbuild.yaml

# Copyright 2021 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
### branch-cloudbuild.yaml

# Default Values
substitutions:
  _SERVICE_NAME: hello-cloudrun
  _REGION: us-east1

steps:

### Build
  - id: "build image"
    name: "gcr.io/cloud-builders/docker"
    args: ["build", "-t", "gcr.io/${PROJECT_ID}/${_SERVICE_NAME}", "."]

### Push
  - id: "push image"
    name: "gcr.io/cloud-builders/docker"
    args: ["push", "gcr.io/${PROJECT_ID}/${_SERVICE_NAME}"]

### Deploy
  - id: "deploy prod service"
    name: "gcr.io/google.com/cloudsdktool/cloud-sdk"
    entrypoint: "bash"
    args:
      - '-c'
      - |
          gcloud run deploy ${_SERVICE_NAME} \
            --platform managed \
            --region ${_REGION} \
            --image gcr.io/${PROJECT_ID}/${_SERVICE_NAME} \
            --tag=${BRANCH_NAME} \
            --no-traffic

Cloud Build Trigger 생성

 

git checkout -b new-feature-1

git에서 새로운 branch 만들기

 

###vi app.py

#!/usr/bin/python
#
# Copyright 2021 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os

from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello_world():
    #return 'Hello World v1.0'
    return 'Hello World v1.1'
    

if __name__ == "__main__":
    app.run(debug=True,host='0.0.0.0',port=int(os.environ.get('PORT', 8080)))

app.py에서 version 수정

 

git add . && git commit -m "updated" && git push gcp new-feature-1

Source Repository로 Push

 

 

Trigger가 동작하며 Cloud Run version Update 하여 배포

 

 

BRANCH_URL=$(gcloud run services describe hello-cloudrun --platform managed --region $REGION --format=json | jq --raw-output ".status.traffic[] | select (.tag==\"new-feature-1\")|.url")
echo $BRANCH_URL

Cloud Run의 Application URL을 변수 저장

 

curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" $BRANCH_URL

### Result
Hello World v1.1

인증된  URL로 호출하면 업데이트 된 결과를 얻을 수 있음 

 

3. Canary Deploy 자동화 테스트

코드가 기본 분기에 커밋될 때 활성화되는 트리거를 구현합니다. 트리거는 고유한 카나리아 URL에 코드를 배포하고 모든 라이브 트래픽의 10%를 여기로 라우팅합니다.

 

gcloud beta builds triggers create cloud-source-repositories --trigger-config master-trigger.json

테스트를 위한 Trigger 생성

### master-trigger.json

{
  "triggerTemplate": {
    "projectId": "qwiklabs-gcp-00-9a91239a3b88",
    "repoName": "cloudrun-progression",
    "branchName": "master"
  },
  "name": "master",
  "description": "Trigger canary build/deploy for any commit to the master branch",

### master-cloudbuild.yaml

# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Default Values
substitutions:
  _SERVICE_NAME: hello-cloudrun
  _REGION: us-east1

steps:

### Build
  - id: "build image"
    name: "gcr.io/cloud-builders/docker"
    args: ["build", "-t", "gcr.io/${PROJECT_ID}/${_SERVICE_NAME}", "."]

### Push
  - id: "push image"
    name: "gcr.io/cloud-builders/docker"
    args: ["push", "gcr.io/${PROJECT_ID}/${_SERVICE_NAME}"]

### Deploy
  - id: "deploy canary service"
    name: "gcr.io/google.com/cloudsdktool/cloud-sdk"
    entrypoint: "bash"
    args:
      - '-c'      - |
          gcloud run deploy ${_SERVICE_NAME} \            --platform managed \
            --region ${_REGION} \
            --image gcr.io/${PROJECT_ID}/${_SERVICE_NAME} \
            --tag=canary \
            --no-traffic


  # Route Traffic
  - id: "route prod traffic"
    name: "gcr.io/google.com/cloudsdktool/cloud-sdk"
    entrypoint: "bash"
    args:
      - '-c'
      - |
          apt-get install -y jq
          export CANARY=$$(gcloud run services describe hello-cloudrun --platform managed  --region ${_REGION} --format=json | jq --raw-output ".spec.traffic[] | select (.tag==\"canary\")|.revisionName")
          export PROD=$$(gcloud run services describe hello-cloudrun --platform managed  --region ${_REGION} --format=json | jq --raw-output ".spec.traffic[] | select (.tag==\"prod\")|.revisionName")

          echo SHORT_SHA is $SHORT_SHA
          echo Canary is $${CANARY}
          echo gcloud beta run services update-traffic  ${_SERVICE_NAME} --update-tags=sha-$SHORT_SHA=$${CANARY}  --platform managed  --region ${_REGION}
          gcloud beta run services update-traffic  ${_SERVICE_NAME} --update-tags=sha-$SHORT_SHA=$${CANARY}  --platform managed  --region ${_REGION}
          gcloud run services update-traffic  ${_SERVICE_NAME} --to-revisions=$${PROD}=90,$${CANARY}=10  --platform managed  --region ${_REGION}

 

 

git checkout master
git merge new-feature-1
git push gcp master

분기를 병합하고 Source Repository로 Push

 

Cloud Deploy 모습

 

Clodu Run의 Revision을 보면 prod 환경으로 90% 그리고 10%는 canary 배포된 이미지로 가는것으로 설정되었습니다.

 

그 이유로는 master-cloudbuild.yaml을 보면 알 수있는데

gcloud run deploy ${_SERVICE_NAME} \
--platform managed \
--region ${_REGION} \
--image gcr.io/${PROJECT_ID}/${_SERVICE_NAME} \
--tag=canary \
--no-traffic

39-45줄
새 버전을 배포하고 태그 플래그를 사용하여 고유한 카나리아 URL에서 트래픽을 라우팅

 

gcloud beta run services update-traffic ${_SERVICE_NAME} --update-tags=sha-$SHORT_SHA=$${CANARY} --platform managed --region ${_REGION}

61줄
배포의 Git 짧은 SHA를 기록하는 개정에 정적 태그를 추가

 

gcloud run services update-traffic ${_SERVICE_NAME} --to-revisions=$${PROD}=90,$${CANARY}=10 --platform managed --region ${_REGION}

62줄

90%를 프로덕션으로, 10%를 카나리아로 라우팅하도록 트래픽을 업데이트

 

CANARY_URL=$(gcloud run services describe hello-cloudrun --platform managed --region $REGION --format=json | jq --raw-output ".status.traffic[] | select (.tag==\"canary\")|.url")
echo $CANARY_URL

해당 명령어를 통해 Canary로 배포된 URL을 찾습니다.

 

curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" $CANARY_URL

### Result 
Hello World v1.1

테스트 결과

 

LIVE_URL=$(gcloud run services describe hello-cloudrun --platform managed --region $REGION --format=json | jq --raw-output ".status.url")
for i in {0..20};do
curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" $LIVE_URL; echo \n
done

백분율 기반으로 테스트

 

4. Canary 배포 후 이제 Production 으로 적용

 

트래픽의 작은 하위 집합으로 카나리아 배포가 검증된 후 나머지 라이브 트래픽에 대한 배포를 해제합니다.

 

리포지토리에서 태그를 생성할 때 활성화되는 트리거를 설정합니다. 트리거는 태그의 커밋 SHA를 기반으로 이미 배포된 개정으로 트래픽의 100%를 마이그레이션합니다. 커밋 SHA를 사용하면 카나리아 트래픽으로 검증된 개정이 나머지 프로덕션 트래픽에 사용되는 개정이 됩니다.

 

gcloud beta builds triggers create cloud-source-repositories --trigger-config tag-trigger.json

트리거 생성

### tag-trigger.json

{
  "triggerTemplate": {
    "projectId": "qwiklabs-gcp-00-ed465f3b673d",
    "repoName": "cloudrun-progression",
    "tagName": ".*"
  },
  "name": "tag",
  "description": "Migrate from canary to prod triggered by creation of any tag",

  "filename": "tag-cloudbuild.yaml"

### tag-cloudbuild.yaml

# Copyright 2021 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Default Values
substitutions:
  _SERVICE_NAME: hello-cloudrun
  _REGION: us-east1

steps:

  # Route Traffic
  - id: "route prod traffic"
    name: "gcr.io/google.com/cloudsdktool/cloud-sdk"
    entrypoint: "bash"
    args:
      - '-c'
      - |
          apt-get install -y jq
          export CANARY=$$(gcloud run services describe hello-cloudrun --platform managed  --region ${_REGION} --format=json | jq --raw-output ".spec.traffic[] | select (.tag==\"canary\")|.revisionName")
          export PROD=$$(gcloud run services describe hello-cloudrun --platform managed  --region ${_REGION} --format=json | jq --raw-output ".spec.traffic[] | select (.tag==\"prod\")|.revisionName")


          echo gcloud beta run services update-traffic  ${_SERVICE_NAME} --update-tags=prod=$${CANARY}  --platform managed  --region ${_REGION}
          echo gcloud run services update-traffic  ${_SERVICE_NAME} --to-revisions=$${PROD}=100  --platform managed  --region ${_REGION}

          gcloud beta run services update-traffic  ${_SERVICE_NAME} --update-tags=prod=$${CANARY}  --platform managed  --region ${_REGION}
          export NEW_PROD=$$(gcloud run services describe hello-cloudrun --platform managed  --region ${_REGION} --format=json | jq --raw-output ".spec.traffic[] | select (.tag==\"prod\")|.revisionName")
          gcloud run services update-traffic  ${_SERVICE_NAME} --to-revisions=$${NEW_PROD}=100  --platform managed  --region ${_REGION}

 

 

 

git tag 1.1
git push gcp 1.1

tag 생성후 Push

 

Cloud Run에서 Revisions를 보면 v1.1 에 traffic 100%을 볼 수 있다.

 

 

tag-cloudbuild.yaml을 살펴보면

 

gcloud beta run services update-traffic ${_SERVICE_NAME} --update-tags=prod=$${CANARY} --platform managed --region ${_REGION}

37줄

prod 태그를 추가하여 canary 개정을 업데이트합니다. 배포된 버전은 이제 prod 및 canary 모두에 대해 태그가 지정

 

gcloud run services update-traffic ${_SERVICE_NAME} --to-revisions=$${NEW_PROD}=100 --platform managed --region ${_REGION}

39줄

기본 서비스 URL에 대한 트래픽을 업데이트하여 트래픽의 100%를 prod로 태그가 지정된 개정으로 라우팅

 

---

출처

https://www.cloudskillsboost.google/focuses/52827?catalog_rank=%7B%22rank%22%3A1%2C%22num_filters%22%3A0%2C%22has_search%22%3Atrue%7D&parent=catalog&search_id=22355499