Бэкапирование terraform.tfstate в CI

На работе мы храним terraform в хранилище типа s3 (не amazon). В нашем хранилище нет версионности, поэтому бэкап и lifecycle реализуем сами через утилиту rclone.

Создание конфигурационного файла:

.terraform_rclone_generate_config:
  script:
    - |
        cat > /tmp/rclone.conf <<EOF
        [s3]
        type = s3
        provider = Other
        endpoint = ${S3_HOST}
        access_key_id = ${AWS_ACCESS_KEY_ID}
        secret_access_key = ${AWS_SECRET_ACCESS_KEY}
        EOF

Шаблон для бэкапа:

.terraform_backup_state:
  script:
    - !reference [".terraform_rclone_generate_config", "script"]
    - |
        BACKUP_DATE=$(date +%Y%m%d_%H%M%S)
        BACKUP_PATH="backups/terraform.tfstate_${BACKUP_DATE}"
        rclone --config /tmp/rclone.conf copyto \
          s3:${S3_BUCKET_NAME}/terraform.tfstate \
          s3:${S3_BUCKET_NAME}/${BACKUP_PATH} -P \
          && echo -e "\033[0;32mState backup created: s3://${S3_BUCKET_NAME}/${BACKUP_PATH}\e[0m" \
          || echo -e "\033[0;33mNo state file to backup\e[0m"

Бэкап вызываем каждый раз при terraform init:

.terraform_init:
  <<: *terraform_common
  script:
    - terraform init
    - !reference [".terraform_backup_state", "script"]
  artifacts:
    name: "terraform-init-${CI_JOB_ID}"
    paths:
      - "**/.terraform"
      - "**/.terraform.lock.hcl"
    expire_in: 1 hours

Якорь terraform_common тут не сильно важен.

Далее нужно реализовать очистку бэкапа от старых файлов, сохраняя последние 10 бэкапов.

.terraform_cleanup_old_tfstate_backup:
  <<: *terraform_common
  script:
    - !reference [".terraform_rclone_generate_config", "script"]
    - |
        set +o pipefail
        if [[ "$CI_PROJECT_NAME" == "elk" ]]; then
          PROJECT_NAME=$ELK_CLUSTER_NAME
        fi;
        echo -e "\033[0;36m=== Starting backup cleanup for: ${PROJECT_NAME} ===\e[0m"
        BACKUPS=$(rclone --config /tmp/rclone.conf lsf \
          s3:${S3_BUCKET_NAME}/backups/ \
          --files-only 2> /dev/null |grep -E "terraform\.tfstate_[0-9]{8}_[0-9]{6}$" | sort -r
        )
        if [[ -z "$BACKUPS" ]];then
          echo -e "\033[0;33mNo backups found in s3://${S3_BUCKET_NAME}/backups/\e[0m"
          exit 0
        fi;
        TOTAL=$(echo "$BACKUPS"| wc -l)
        KEEP=10

        echo -e "\033[0;36mTotal backups found: ${TOTAL}\e[0m"
        echo -e "\033[0;36mKeeping last: ${KEEP}\e[0m"

        if [[ "$TOTAL" -gt "$KEEP" ]];then
          TO_DELETE_COUNT=$((TOTAL - KEEP))
          TO_DELETE=$(echo "$BACKUPS" | tail -n "$TO_DELETE_COUNT")
          echo -e "\033[0;33mDeleting ${TO_DELETE_COUNT} old backups:\e[0m"
          DELETED=0
          while IFS= read -r backup; do
            if [[ -n "$backup" ]];then
              if rclone --config /tmp/rclone.conf delete \
                "s3:${S3_BUCKET_NAME}/backups/${backup}" 2>/dev/null; then
                echo "  ✓ Deleted: ${backup}"
                DELETED=$((DELETED +1))
              else
                echo "  ✗ Failed to delete: ${backup}"
              fi;
            fi;
          done <<< "$TO_DELETE"
          echo -e "\033[0;32m=== Cleanup completed: ${DELETED}/${TO_DELETE_COUNT} backups deleted ===\e[0m"
        else
          echo -e "\033[0;32mNo cleanup needed (total: ${TOTAL}, keeping: ${KEEP})\e[0m"
        fi;

terraform под ELK чуть отличается, поэтому имеем небольшой костыль в начале. Финальная джоба в проекте по очистке будет запускаться через schedule pipeline примерно так:

terraform:schedule:cleanup:backups:
  stage: clean
  extends: [".terraform_cleanup_old_tfstate_backup"]
  parallel:
    matrix:
      - ELK_CLUSTER_NAME:
          - <cluster_1>
          - <cluster_2>
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule" && $SCHEDULE_TYPE == "backup_cleanup"'

Более полную картину по реализации управления объектами ELK через terraform опишу в статье на сайте несколько позже.

#terraform