O8 Secrets Management Guide

Overview

Orchestr8 now provides enterprise-grade secrets management using External Secrets Operator (ESO) with support for AWS Secrets Manager and GCP Secret Manager. This follows GitOps best practices by ensuring no secrets are stored in Git repositories.

Quick Start

1. Initialize Secrets Management

# Deploy External Secrets Operator and configure AWS
o8 secrets init --provider aws --deploy-eso

# Or for GCP
o8 secrets init --provider gcp --deploy-eso

# Or for both providers
o8 secrets init --provider both --deploy-eso

2. Configure Cloud Provider Access

AWS Setup

# Set AWS credentials
export AWS_REGION=us-east-1
export AWS_ACCOUNT_ID=123456789012

# Create IAM role for External Secrets (EKS with IRSA)
aws iam create-role --role-name orchestr8-external-secrets \
  --assume-role-policy-document file://trust-policy.json

# Attach policy
aws iam attach-role-policy --role-name orchestr8-external-secrets \
  --policy-arn arn:aws:iam::aws:policy/SecretsManagerReadWrite

GCP Setup

# Set GCP project
export GOOGLE_PROJECT=my-project

# Create service account
gcloud iam service-accounts create o8-external-secrets \
  --display-name="O8 External Secrets"

# Grant permissions
gcloud projects add-iam-policy-binding $GOOGLE_PROJECT \
  --member="serviceAccount:o8-external-secrets@$GOOGLE_PROJECT.iam.gserviceaccount.com" \
  --role="roles/secretmanager.admin"

# Enable Workload Identity (GKE)
kubectl annotate serviceaccount external-secrets-gcp-sa \
  -n external-secrets \
  iam.gke.io/gcp-service-account=o8-external-secrets@$GOOGLE_PROJECT.iam.gserviceaccount.com

3. Create Secrets

# Interactive mode (prompts for values)
o8 secrets create my-app-db --provider aws

# From JSON file
o8 secrets create my-app-db --provider aws --data @secrets.json

# From environment variables (PREFIX: SECRET_)
export SECRET_USERNAME=admin
export SECRET_PASSWORD=secure123
o8 secrets create my-app-db --provider aws --from-env

# Inline JSON
o8 secrets create my-app-api --provider gcp \
  --data '{"api_key": "sk-123456", "endpoint": "https://api.example.com"}'

4. Deploy ExternalSecret Resources

Create an ExternalSecret resource to fetch secrets into Kubernetes:

# my-app-secrets.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-app-secrets
  namespace: my-app
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
    kind: SecretStore
  target:
    name: my-app-secrets
    creationPolicy: Owner
  data:
    - secretKey: db_password
      remoteRef:
        key: o8/my-app-db
        property: password

Apply it:

kubectl apply -f my-app-secrets.yaml

5. Verify Secret Synchronization

# Check ExternalSecret status
kubectl get externalsecrets -n my-app

# Check if Kubernetes secret was created
kubectl get secrets -n my-app

# Force sync all secrets
o8 secrets sync --all

CLI Commands Reference

o8 secrets init

Initialize secrets management infrastructure.

Options:

• --provider / -p: Cloud provider (aws, gcp, both)
• --deploy-eso: Deploy External Secrets Operator

o8 secrets create

Create a new secret in the cloud provider.

Arguments:

• NAME: Secret name (without prefix)

Options:

• --provider / -p: Cloud provider (aws, gcp)
• --data / -d: JSON data or @filename
• --from-env: Read from environment variables
• --description: Secret description

o8 secrets list

List all secrets from providers.

Options:

• --provider / -p: Filter by provider (aws, gcp, both)
• --prefix: Filter secrets by prefix

o8 secrets describe

Show secret details.

Arguments:

• NAME: Secret name

Options:

• --provider / -p: Cloud provider
• --show-values: Display secret values (masked)

o8 secrets delete

Delete a secret.

Arguments:

• NAME: Secret name

Options:

• --provider / -p: Cloud provider
• --yes / -y: Skip confirmation

o8 secrets sync

Force synchronization of ExternalSecrets.

Options:

• --namespace / -n: Specific namespace
• --all: Sync all namespaces

o8 secrets migrate

Migrate existing hardcoded secrets to cloud providers.

Migration from Hardcoded Secrets

Step 1: Run Migration

# Migrate existing secrets to AWS
o8 secrets migrate
# Choose provider when prompted: aws

Step 2: Update Applications

Replace direct secret references with ExternalSecret resources:

# Before (hardcoded secret)
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
data:
  password: cGFzc3dvcmQxMjM=  # base64 encoded

# After (ExternalSecret)
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  secretStoreRef:
    name: aws-secrets
  target:
    name: db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: o8/app/database
        property: password

Step 3: Verify and Clean Up

# Verify secrets are working
kubectl get secrets -A

# Remove old secret files
git rm *.secret.yaml
git commit -m "chore: migrate to external secrets management"

Best Practices

1. Secret Naming Convention

Use hierarchical naming:

• o8/argocd/admin-password
• o8/langfuse/database
• o8/voicefuse/api-keys

2. Secret Rotation

# Rotate a specific secret
o8 secrets create my-app-db --provider aws \
  --data '{"password": "new-secure-password"}'

# Force immediate sync
o8 secrets sync --namespace my-app

3. Environment Separation

Use different secret paths for environments:

• o8/dev/app/database
• o8/staging/app/database
• o8/prod/app/database

4. Access Control

Limit SecretStore access per namespace:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: app-secrets
  namespace: my-app
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: my-app-sa

5. Monitoring

Monitor ExternalSecret sync status:

# Check sync status
kubectl get externalsecrets -A -o wide

# View events
kubectl describe externalsecret my-app-secrets -n my-app

Troubleshooting

Secret Not Syncing

1. Check ExternalSecret status:

   kubectl describe externalsecret my-secret -n my-namespace

2. Verify SecretStore configuration:

   kubectl get secretstore -n my-namespace

3. Check ESO logs:

   kubectl logs -n external-secrets deploy/external-secrets

Permission Errors

1. AWS: Verify IAM role and policies
2. GCP: Check service account permissions
3. Ensure IRSA/Workload Identity is configured

Force Refresh

# Annotate to force refresh
kubectl annotate externalsecret my-secret -n my-namespace \
  force-sync=$(date +%s) --overwrite

Security Considerations

1. Never commit secrets to Git - Always use external providers
2. Use least privilege - Grant minimal required permissions
3. Enable audit logging - Track secret access in cloud providers
4. Rotate regularly - Implement secret rotation policies
5. Encrypt at rest - Ensure providers encrypt secrets
6. Use RBAC - Limit who can create/modify ExternalSecrets

Integration with O8 Platform

The secrets management integrates seamlessly with:

• ArgoCD: Repository credentials, admin passwords
• OAuth2 Proxy: Client secrets, cookie secrets
• Keycloak: Admin credentials, client secrets
• Databases: Connection strings, passwords
• APIs: Keys, tokens, certificates

Next Steps

1. Set up cloud provider authentication
2. Migrate existing secrets using o8 secrets migrate
3. Create ExternalSecret resources for applications
4. Configure monitoring and alerting
5. Implement secret rotation policies

For more information, see:

• External Secrets Operator Documentation
• AWS Secrets Manager
• GCP Secret Manager