장고 앱 CI/CD, 무중단 배포 구현기

개요

AS-IS

1. 서버에서 직접 도커 이미지를 빌드
• 서버 자원을 더 사용합니다

2. 도커 컨테이너 종료후 빌드와 재실행
• 빌드하는 동안 시간이 많이 걸려 서버가 다운타운이 길어집니다
• 빌드후 서버 종료-재실행을 진행해 중단 시간을 줄여도, 여전히 무중단은 아닙니다

3. Nginx 관련
• Nginx가 같이 재시작하고 있어 무중단 배포가 어려운점
• 그리고 도커에서 실행되고 있어 TLS 인증서 자동 갱신이 어려웠던 점

TO-BE

도커 이미지 저장소

Github Action

Nginx와 무중단 배포

• AWS ELB

• Route 53 DNS

• AWS API Gateway

• Nginx 사용

서버 인스턴스 사전 구성에 대해서

알림

구현

ECR Registry 설정

서버 사전 구성

sudo apt-get update

• Docker, Docker compose 설치

sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker

• nginx 설치

sudo apt-get install -y nginx

• AWS cli 설치

sudo apt install unzip
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

Github Actions workflow

name: Build, Push to ECR, and Deploy

on:
  push:
    branches: [release]
  workflow_dispatch:

jobs:
  build-and-push:
    runs-on: ubuntu-latest

    outputs:
      REGISTRY: ${{ steps.login-ecr.outputs.registry }}

    steps:
      - name: Checkout the code
        uses: actions/checkout@v4
        with:
          submodules: true
          token: ${{ secrets.PAT_TOKEN }}

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ vars.AWS_REGION }}

      # ECR 로그인
      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v2

      # Docker 이미지 빌드 및 푸시
      - name: Build, tag, and push docker image to Amazon ECR
        env:
          REGISTRY: ${{ steps.login-ecr.outputs.registry }}
          REPOSITORY: ${{ secrets.ECR_REPOSITORY }}
          IMAGE_TAG: ${{ github.sha }}
        run: |
          docker build -t $REGISTRY/$REPOSITORY:$IMAGE_TAG .
          docker push $REGISTRY/$REPOSITORY:$IMAGE_TAG

  deploy:
    runs-on: ubuntu-latest
    needs: build-and-push

    steps:
      - name: Checkout the code
        uses: actions/checkout@v4
        with:
          submodules: true
          token: ${{ secrets.PAT_TOKEN }}

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ vars.AWS_REGION }}

      # 필요한 파일을 서버로 전송
      - name: Copy files to server
        uses: appleboy/scp-action@v0.1.7
        with:
          host: ${{ secrets.EC2_HOST }}
          username: ${{ secrets.EC2_USER }}
          key: ${{ secrets.EC2_SSH_KEY }}
          port: ${{ secrets.EC2_SSH_PORT }}
          source: "./deploy/"
          target: /home/${{ secrets.EC2_USER }}/

      - name: Create .env on server
        uses: appleboy/ssh-action@v0.1.5
        with:
          host: ${{ secrets.EC2_HOST }}
          username: ${{ secrets.EC2_USER }}
          key: ${{ secrets.EC2_SSH_KEY }}
          port: ${{ secrets.EC2_SSH_PORT }}
          script: |
            echo "REGISTRY=${{ secrets.ECR_REGISTRY }}" > /home/${{ secrets.EC2_USER }}/deploy/.env
            echo "REPOSITORY=${{ secrets.ECR_REPOSITORY }}" >> /home/${{ secrets.EC2_USER }}/deploy/.env
            echo "AWS_REGION=${{ vars.AWS_REGION }}" >> /home/${{ secrets.EC2_USER }}/deploy/.env
            echo "IMAGE_TAG=${{ github.sha }}" >> /home/${{ secrets.EC2_USER }}/deploy/.env
            echo "AWS_ACCESS_KEY_ID=${{ secrets.AWS_ACCESS_KEY_ID }}" >> /home/${{ secrets.EC2_USER }}/deploy/.env
            echo "AWS_SECRET_ACCESS_KEY=${{ secrets.AWS_SECRET_ACCESS_KEY }}" >> /home/${{ secrets.EC2_USER }}/deploy/.env

      # 배포 실행
      - name: Deploy via SSH
        uses: appleboy/ssh-action@v0.1.5
        env:
          REGISTRY: ${{ needs.build-and-push.outputs.REGISTRY }}
          REPOSITORY: ${{ secrets.ECR_REPOSITORY }}
          AWS_REGION: ${{ vars.AWS_REGION }}
          IMAGE_TAG: ${{ github.sha }}
        with:
          host: ${{ secrets.EC2_HOST }}
          username: ${{ secrets.EC2_USER }}
          key: ${{ secrets.EC2_SSH_KEY }}
          port: ${{ secrets.EC2_SSH_PORT }}
          envs: REGISTRY, REPOSITORY, AWS_REGION, IMAGE_TAG
          script: |
            cd /home/${{ secrets.EC2_USER }}/deploy
            chmod +x deploy.sh
            ./deploy.sh

GitHub - aws-actions/amazon-ecr-login: Logs into Amazon ECR with the local Docker client.

Logs into Amazon ECR with the local Docker client. - aws-actions/amazon-ecr-login

https://github.com/aws-actions/amazon-ecr-login

블루/그린 배포 실행 스크립트

#!/bin/bash

set -e

# 환경 변수 로드
source .env

export AWS_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY

# AWS ECR 로그인
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $REGISTRY

# IMAGE_TAG는 GitHub Actions에서 전달된 값을 사용
IMAGE_TAG=${IMAGE_TAG}

# 현재 활성화된 버전 확인
if [ -f active_version ]; then
  CURRENT_VERSION=$(cat active_version)
else
  CURRENT_VERSION="green"  # 초기 배포 시 기본값
fi

# 다음 배포할 버전 결정
if [ "$CURRENT_VERSION" = "blue" ]; then
  NEW_VERSION="green"
  ACTIVE_PORT=8002
else
  NEW_VERSION="blue"
  ACTIVE_PORT=8001
fi

# Docker Compose 환경 변수 설정
export REGISTRY
export REPOSITORY
export IMAGE_TAG

# 새로운 이미지 풀
docker pull $REGISTRY/$REPOSITORY:$IMAGE_TAG

# 새로운 서비스 시작
docker compose up -d app_${NEW_VERSION}

# 헬스 체크 대기
echo "새로운 버전의 컨테이너 헬스 체크 중..."
sleep 10

# 헬스 체크 수행
if ! curl -f http://localhost:${ACTIVE_PORT}/v1/health; then
  echo "새로운 컨테이너가 헬스 체크에 실패했습니다. 롤백합니다."
  docker compose stop app_${NEW_VERSION}
  exit 1
fi

# Nginx 설정 업데이트
sed "s/\${ACTIVE_UPSTREAM}/app_${NEW_VERSION}/g" nginx.conf.template > nginx.conf
sudo cp nginx.conf /etc/nginx/conf.d/default.conf
sudo systemctl reload nginx

# 이전 서비스 중지
docker compose stop app_${CURRENT_VERSION}

# 활성화된 버전 업데이트
echo "${NEW_VERSION}" > active_version

echo "${NEW_VERSION} 버전으로 배포 완료."

Nginx

upstream app_blue {
    server 127.0.0.1:8001;
}

upstream app_green {
    server 127.0.0.1:8002;
}

server {
    listen 80;
    server_name api.misiklog.com;
    
    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl;
    server_name api.misiklog.com;

    ssl_certificate /etc/letsencrypt/live/blog.wibaek.net/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/blog.wibaek.net/privkey.pem;

    location / {
        proxy_pass http://${ACTIVE_UPSTREAM};
        proxy_set_header Host $host:$server_port;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
    
    location /static/ {
        alias /home/ubuntu/static/;
    }
}

• $host와 $request_uri
• $host: in this order of precedence: host name from the request line, or host name from the “Host” request header field, or the server name matching a request
• $request_uri: full original request URI (with arguments)
• https://nginx.org/en/docs/http/ngx_http_core_module.html

경험한 문제

The workflow is not valid. .github/workflows/cd-prod.yml: A mapping was not expected

# 문제 발생
- name: Deploy via SSH
  uses: appleboy/ssh-action@v0.1.5
  with:
    env:
      REGISTRY: ${{ needs.build-and-push.outputs.REGISTRY }}
      REPOSITORY: ${{ secrets.ECR_REPOSITORY }}
      AWS_REGION: ${{ vars.AWS_REGION }}
      IMAGE_TAG: ${{ github.sha }}
      
# 수정
- name: Deploy via SSH
  uses: appleboy/ssh-action@v0.1.5
  env:
    REGISTRY: ${{ needs.build-and-push.outputs.REGISTRY }}
    REPOSITORY: ${{ secrets.ECR_REPOSITORY }}
    AWS_REGION: ${{ vars.AWS_REGION }}
    IMAGE_TAG: ${{ github.sha }}
  with:
    envs: REGISTRY, REPOSITORY, AWS_REGION, IMAGE_TAG

scp 전송중 'tar: empty archive'

source: |
  ./docker-compose.yml
  ./deploy/nginx.conf.template
  ./deploy/deploy.sh

1. 여러번 반복해서 전송해준다

2. 한 폴더에 넣어서 전송해준다

error copy file to dest: ***, error message: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

err: bash: line x: ./deploy.sh: Permission denied

deploy.sh 실행중 문제

err: Unable to locate credentials. You can configure credentials by running "aws configure".
err: Error: Cannot perform an interactive login from a non TTY device

Nginx 정적파일 403 Forbidden 문제

server {
    location /static/ {
        alias /home/ubuntu/static/;
    }
}

drwxrwxr-x 6 ubuntu ubuntu 4096 Dec  1 00:00 static
# 아래와 같이 /ubuntu 폴더에는 자신과 자신의 그룹 외의 others 에게는 아무 권한도 부여하지 않고 있다
drwxr-x---  8 ubuntu ubuntu 4096 Dec  1 00:00 ubuntu

$ sudo chmod 755 ubuntu/
$ ls -al
drwxr-xr-x  8 ubuntu ubuntu 4096 Dec  1 00:00 ubuntu