Skip to main content

CUE-Based Module Architecture

Overview​

Orchestr8 uses a hybrid approach combining CUE (Configure, Unify, Execute) for type-safe resource generation and Kustomize for environment-specific overlays. This provides the best of both worlds: compile-time validation with runtime flexibility.

Architecture Components​

🎯 CUE: Configuration Generation Engine​

  • Purpose: Generate tenant-specific Kubernetes resources
  • When: At configuration time during o8 tenant operations
  • What: Type-safe, validated, tenant-contextualized resources
  • Benefits: Type safety, constraint validation, tenant isolation

🔧 Kustomize: Environment Management​

  • Purpose: Environment-specific overlays and patches
  • When: At deployment time via ArgoCD
  • What: Dev/staging/prod configurations, multi-cluster patches
  • Benefits: GitOps-native, kubectl integration, environment promotion

🚀 ArgoCD: GitOps Delivery​

  • Purpose: Continuous deployment of generated configurations
  • When: At deployment time, watching Git repository
  • What: Automated sync of tenant configurations to clusters
  • Benefits: Git-based workflows, rollback capabilities, multi-cluster

Data Flow Architecture​

Implementation Details​

CUE Schema Structure​

schemas/
├── tenant/
│   └── tenant.cue           # Main tenant configuration schema
├── keycloak/
│   └── keycloak.cue         # Keycloak realm generation
└── kubernetes/
    └── kubernetes.cue       # Kubernetes resource templates

Generated Tenant Structure​

tenants/
└── {tenant-name}/
    ├── tenant.yaml              # Source configuration
    ├── keycloak-realm.json      # Generated Keycloak realm
    ├── tenant-bundle.json       # Complete bundle
    ├── kubernetes/              # Base tenant resources
    │   ├── namespace.yaml
    │   ├── resource-quota.yaml
    │   ├── limit-range.yaml
    │   └── network-policies.yaml
    └── modules/                 # Generated module resources
        ├── langfuse/
        │   ├── namespace.yaml
        │   ├── deployment.yaml
        │   └── service.yaml
        └── clickhouse/
            ├── namespace.yaml
            ├── deployment.yaml
            └── service.yaml

Module Generation Process​

1. CLI Command​

o8 tenant add-module test-company langfuse --auth --roles "admin,analyst"

2. CUE Resource Generation​

The CUE engine generates tenant-specific resources:

def generate_tenant_module_resources(
    tenant_config: Dict[str, Any],
    module_name: str,
    target_namespace: str
) -> Dict[str, Any]:
    """Generate K8s resources specific to tenant context."""

    # Module-specific generation logic
    if module_name == "langfuse":
        return self._generate_langfuse_resources(tenant_config, target_namespace)
    elif module_name == "clickhouse":
        return self._generate_clickhouse_resources(tenant_config, target_namespace)

3. Tenant Context Integration​

Generated resources include:

  • Tenant Labels: orchestr8.platform/tenant: {tenant-name}
  • Module Labels: orchestr8.platform/module: {module-name}
  • Namespace: {tenant-name}-{module-name}
  • Domain: {module}.{tenant-domain}
  • Resource Quotas: Applied from tenant configuration
  • Security Policies: Network policies, RBAC, pod security

4. ArgoCD Application Generation​

Each module gets an ArgoCD Application pointing to generated resources:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-company-langfuse
  labels:
    orchestr8.platform/tenant: test-company
    orchestr8.platform/module: langfuse
spec:
  source:
    path: tenants/test-company/modules/langfuse
  destination:
    namespace: test-company-langfuse

Benefits of This Approach​

✅ Type Safety​

  • CUE schemas catch configuration errors at generation time
  • Module configurations validated against tenant constraints
  • Prevents invalid resource combinations

✅ Tenant Isolation​

  • Each module deployed in tenant-specific namespace
  • Network policies enforce isolation boundaries
  • Resource quotas applied per tenant
  • Security contexts tenant-specific

✅ GitOps Native​

  • All configurations stored in Git
  • ArgoCD provides automated deployment
  • Environment promotion through Git workflows
  • Rollback capabilities via Git history

✅ Environment Flexibility​

  • Kustomize overlays for environment-specific configs
  • Multi-cluster deployments supported
  • Development/staging/production pipelines

✅ Developer Experience​

  • Simple CLI commands: o8 tenant add-module
  • Instant feedback on configuration issues
  • Generated resources inspectable before deployment
  • Clear separation of concerns

Module Types and Templates​

AI/ML Modules​

  • Langfuse: LLM observability platform
  • Ollama: Local LLM serving
  • Jupyter: Notebook environments

Data Modules​

  • ClickHouse: Analytics database
  • PostgreSQL: Relational database
  • Redis: Caching layer

Application Modules​

  • Web Apps: Frontend applications
  • APIs: Backend services
  • Utilities: Cron jobs, workers

Environment Overlays with Kustomize​

While CUE generates base resources, Kustomize handles environment-specific variations:

# kustomization.yaml for staging environment
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base

patches:
  - target:
      kind: Deployment
      name: langfuse
    patch: |-
      - op: replace
        path: /spec/replicas
        value: 3
      - op: replace
        path: /spec/template/spec/containers/0/resources/limits/memory
        value: 2Gi

configMapGenerator:
  - name: env-config
    literals:
      - ENVIRONMENT=staging
      - LOG_LEVEL=debug

Security Model​

Network Isolation​

  • Default deny network policies
  • Explicit allow rules for required communication
  • Tenant-to-tenant traffic blocked
  • Internet access controlled per tenant

RBAC Integration​

  • Service accounts per module
  • Role bindings scoped to tenant namespaces
  • Keycloak realm integration for user authentication
  • GitHub/OAuth provider integration

Pod Security​

  • Restricted pod security standards enforced
  • Non-root containers required
  • Read-only root filesystems
  • Capabilities dropped (ALL)
  • Seccomp profiles enabled

Testing Strategy​

Unit Tests​

  • CUE schema validation
  • Resource generation logic
  • Tenant configuration parsing
  • Module-specific resource templates

Integration Tests​

  • End-to-end tenant creation
  • Module deployment workflows
  • ArgoCD application generation
  • Kubernetes resource validation

E2E Tests​

  • Full GitOps pipeline
  • Multi-tenant isolation verification
  • Security policy enforcement
  • Resource quota compliance

Future Enhancements​

Advanced CUE Features​

  • Complex constraint validation
  • Cross-module dependency resolution
  • Dynamic resource sizing based on tenant tier
  • Cost optimization recommendations

Enhanced Module System​

  • Module marketplace/registry
  • Dependency management
  • Version compatibility checking
  • Automated testing pipelines

Multi-Cluster Support​

  • Cross-cluster tenant distribution
  • Global load balancing
  • Disaster recovery automation
  • Compliance framework integration

Best Practices​

Configuration Management​

  1. Keep CUE schemas simple - Complex logic in code, not config
  2. Version your schemas - Maintain backward compatibility
  3. Test configurations - Validate before deployment
  4. Document constraints - Clear error messages for validation failures

Module Development​

  1. Follow naming conventions - Consistent resource naming
  2. Include health checks - Proper readiness/liveness probes
  3. Set resource limits - Prevent resource exhaustion
  4. Implement observability - Metrics, logging, tracing

Operational Guidelines​

  1. Monitor resource usage - Track tenant consumption
  2. Regular security audits - Review RBAC and network policies
  3. Backup configurations - Git repository backups
  4. Document procedures - Incident response playbooks

Conclusion​

The CUE-based module architecture provides a robust foundation for multi-tenant Kubernetes management. By combining CUE's type safety with Kustomize's flexibility and ArgoCD's GitOps capabilities, we achieve:

  • Developer Productivity: Simple CLI commands for complex operations
  • Operational Safety: Type-checked configurations prevent errors
  • Scalability: Automated tenant and module management
  • Security: Built-in isolation and access controls
  • Maintainability: Clear separation of concerns and GitOps workflows

This architecture scales from single-tenant development to enterprise multi-tenant production environments while maintaining security, compliance, and operational excellence.