Structum Configuration Philosophy¶

Complete guide to Structum’s layered configuration philosophy.

Feature	Status	Details
Version	0.1.0	Alpha
Last Updated	2026-01-13

Index¶

Fundamental Principle
The 5-Layer Model
Separation of Concerns
Resolution Flow
Diagrams and Visualizations
Usage Scenarios
Best Practices

1. Fundamental Principle¶

Structum adopts a layered configuration approach to separate responsibilities between:

Developers → Define structure and defaults
DevOps/SRE → Configure infrastructure and secrets
End Users → Customize runtime experience

1.1 Core Philosophy: “Configuration as Code”¶

Configuration in Structum follows the same principles as code:

Versioned (defaults in Git)
Validated (Pydantic schemas)
Testable (mock providers for unit tests)
Overridable (override layers without modifying current configuration)

1.2 Separation of Concerns¶

Business logic should not know where configuration comes from.

# ✅ Decoupled business code
config = get_config()
db_host = config.get("database.host")  
# Can come from: TOML, ENV, Secrets, Runtime, ...

# ❌ Anti-pattern: coupling
db_host = os.getenv("DB_HOST", "localhost")  
# Hard-coded to the override mechanism

This allows switching backends (files → Redis → Vault) without touching a single line of business logic.

2. The 5-Layer Model¶

Structum uses a priority hierarchy to resolve every configuration:

┌──────────────────────────────────────────────────┐
│ Layer 1: RUNTIME (In-Memory)                     │  ← HIGHEST PRIORITY
│ • Changes via config.set() not persisted         │
│ • Valid only for the current process             │
├──────────────────────────────────────────────────┤
│ Layer 2: USER PERSISTENT (Runtime Saved)         │
│ • File: ~/.structum/{namespace}_saved.json       │
│ • Changes persisted via config.save()            │
│ • Specific to user/instance                      │
├──────────────────────────────────────────────────┤
│ Layer 3: ENVIRONMENT VARIABLES                   │
│ • Format: STRUCTUM_{NAMESPACE}__{KEY}            │
│ • Infrastructure configuration (Docker/K8s)      │
│ • Set by DevOps/deployment scripts               │
├──────────────────────────────────────────────────┤
│ Layer 4: SECRETS                                 │
│ • File: config/.secrets.toml (git-ignored)       │
│ • Contains passwords, API keys, tokens           │
│ • Automatically merged with Layer 5              │
├──────────────────────────────────────────────────┤
│ Layer 5: APPLICATION DEFAULTS                    │  ← LOWEST PRIORITY
│ • File: config/app/{namespace}.toml              │
│ • Defaults declared by developer                 │
│ • Versioned in Git, baseline for everyone        │
└──────────────────────────────────────────────────┘

2.1 Resolution: “First Found Wins”¶

When you request config.get("database.host"):

# Pseudo-code of the resolver
def resolve_value(key):
    if key in runtime_layer:
        return runtime_layer[key]  # Layer 1
    
    if key in user_persistent_layer:
        return user_persistent_layer[key]  # Layer 2
    
    if key in environment_variables:
        return environment_variables[key]  # Layer 3
    
    if key in secrets_layer:
        return secrets_layer[key]  # Layer 4
    
    if key in application_defaults:
        return application_defaults[key]  # Layer 5
    
    return default_value  # If no layer has the value

The first value found wins. Higher layers overwrite lower ones.

3. Separation of Concerns¶

3.1 Layer 5: Developer Layer (Application Defaults)¶

Who: Software Engineer, Library Author
Format: TOML
Location: config/app/{namespace}.toml (in repository, versioned)
Scope: Define structure, valid keys, safe default values

Example:

# config/app/database.toml
[default]
host = "localhost"
port = 5432
pool_size = 10
timeout = 30

[default.auth]
user = "app_user"
# ⚠️ DO NOT put passwords here

[production]
host = "db.prod.example.com"
pool_size = 50

Features:

Versioned in Git (part of code)
Contains only safe values (no secrets)
Supports multiple environments ([default], [production], [development])
Validated by Pydantic schema (optional but recommended)

Access:

config.get("database.host")       # "localhost" (from [default])
config.get("database.pool_size")  # 10

3.2 Layer 4: Secrets Layer¶

Who: DevOps, Security Team
Format: TOML (local) or Secret Manager (production)
Location: config/.secrets.toml (git-ignored) or Vault/K8s Secrets
Scope: Isolate sensitive credentials from code

Example:

# config/.secrets.toml (DO NOT version!)
[database.auth]
password = "SuperSecretPassword123"

[api_client]
api_key = "sk-proj-abc123xyz"
client_secret = "very-secret-value"

⚠️ CRITICAL: Security Checklist

[ ] Added to .gitignore
[ ] File permissions: chmod 600 config/.secrets.toml
[ ] In production: use Vault/K8s Secrets instead of files
[ ] Periodic credential rotation

Automatic Merge: The plugin automatically merges secrets with corresponding namespaces:

# Merged in-memory:
config.get("database.auth.user")      # "app_user" (from database.toml)
config.get("database.auth.password")  # "SuperSecret..." (from .secrets.toml)

3.3 Layer 3: Environment Variables (Infrastructure)¶

Who: DevOps, SRE, CI/CD Pipeline
Format: Environment Variables
Location: Shell, Docker Compose, K8s ConfigMaps
Scope: Specific environment override without modifying files

Convention:

{ENV_PREFIX}__{NAMESPACE}__{PATH__TO__KEY}

Example:

# Single value override
export STRUCTUM__DATABASE__HOST=prod-db.example.com
export STRUCTUM__DATABASE__PORT=3306

# Nested value override
export STRUCTUM__DATABASE__POOL__MAX_SIZE=100

# Lists (comma-separated)
export STRUCTUM__DATABASE__HOSTS=db1.com,db2.com,db3.com

Typical Use: Docker Compose

# docker-compose.yml
version: '3.8'
services:
  app:
    image: myapp:latest
    environment:
      - STRUCTUM__DATABASE__HOST=postgres
      - STRUCTUM__DATABASE__PORT=5432
      - STRUCTUM__LOG_LEVEL=INFO

Typical Use: Kubernetes

# k8s-deployment.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp-config
data:
  STRUCTUM__DATABASE__HOST: "prod-db.svc.cluster.local"
  STRUCTUM__DATABASE__PORT: "5432"
---
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: myapp
        envFrom:
        - configMapRef:
            name: myapp-config

3.4 Layer 2: User Persistent (Runtime Saved)¶

Who: Instance Operator, End User, Application Runtime
Format: JSON
Location: ~/.structum/{namespace}_saved.json (user home)
Scope: Save user preferences between restarts

Features:

Automatically created on first config.save()
Contains only modified keys (does not duplicate defaults)
User/instance specific (not shared)

Example:

# Runtime modification
config = get_config()
config.set("ui.theme", "dark_mode")
config.set("ui.font_size", 14)

# Persist to disk
config.save()

# File created: ~/.structum/ui_saved.json
# {
#   "theme": "dark_mode",
#   "font_size": 14
# }

On restart:

config = get_config()
theme = config.get("ui.theme")  # "dark_mode" (from saved file)
# Even if config/app/ui.toml says theme = "light"

Use Cases:

UI preferences (theme, font size, layout)
Last opened files
Window dimensions
Persistent caching

3.5 Layer 1: Runtime (In-Memory)¶

Who: Application Runtime
Format: In-Memory Dict
Location: RAM (volatile)
Scope: Temporary changes without persistence

Features:

Lives only during process execution
Does not persist after exit() (unless save() is called)
Useful for tests, debugging, temporary feature flags

Example:

config = get_config()

# Temporary modification (in-memory only)
config.set("debug.verbose", True)
print(config.get("debug.verbose"))  # True

# DO NOT call config.save()

# Application restart
config = get_config()
print(config.get("debug.verbose"))  # False (default from TOML)

Use Cases:

Temporary feature flags
Debug mode via CLI flag
Runtime A/B tests
Session specific overrides

4. Resolution Flow¶

4.1 Complete Example: Multi-Layer Resolution¶

Setup:

# config/app/database.toml
[default]
host = "localhost"
port = 5432
pool_size = 10

# config/.secrets.toml
[database.auth]
password = "secret123"

# Environment
export STRUCTUM__DATABASE__HOST=staging-db.com
export STRUCTUM__DATABASE__PORT=3306

// ~/.structum/database_saved.json
{
  "host": "my-custom-db.local"
}

# Runtime
config.set("database.pool_size", 50)

Resolution:

config = get_config()

config.get("database.host")
# Layer 1 (Runtime): ❌ Not set
# Layer 2 (User): ✅ "my-custom-db.local" ← WINS

config.get("database.port")
# Layer 1 (Runtime): ❌ Not set
# Layer 2 (User): ❌ Not in saved file
# Layer 3 (ENV): ✅ 3306 ← WINS

config.get("database.pool_size")
# Layer 1 (Runtime): ✅ 50 ← WINS

config.get("database.auth.password")
# Layer 1-3: ❌ Not set
# Layer 4 (Secrets): ✅ "secret123" ← WINS

4.2 Flow Diagram¶

                    ┌─────────────────────┐
                    │ config.get("key")   │
                    └──────────┬──────────┘
                               │
                               ▼
                    ┌──────────────────────┐
                    │ Layer 1: Runtime?    │
                    │ (in-memory via set())│
                    └──────┬───────────────┘
                           │
                     NO   ┌┴┐   YES
                    ┌─────┤ ├─────┐
                    │     └─┘     │
                    ▼             ▼
         ┌──────────────────┐   [Return Value]
         │ Layer 2: User?   │
         │ (*_saved.json)   │
         └────┬─────────────┘
              │
        NO   ┌┴┐   YES
       ┌─────┤ ├─────┐
       │     └─┘     │
       ▼             ▼
┌──────────────┐   [Return Value]
│ Layer 3: ENV?│
│ (STRUCTUM__) │
└────┬─────────┘
     │
NO  ┌┴┐   YES
  ┌─┤ ├──┐
  │ └─┘  │
  ▼      ▼
┌─────────────┐   [Return Value]
│ Layer 4:    │
│ Secrets?    │
└──┬──────────┘
   │
NO ┌┴┐   YES
 ┌─┤ ├──┐
 │ └─┘  │
 ▼      ▼
┌──────────────┐   [Return Value]
│ Layer 5:     │
│ Defaults?    │
│ (.toml)      │
└──┬───────────┘
   │
NO ┌┴┐   YES
 ┌─┤ ├──┐
 │ └─┘  │
 ▼      ▼
[Return    [Return Value]
 Default
 Arg]

5. Diagrams and Visualizations¶

5.1 Complete Architecture¶

┌────────────────────────────────────────────────────────────────┐
│                     APPLICATION CODE                           │
│                   config.get("db.host")                        │
└────────────────────────┬───────────────────────────────────────┘
                         │
                         ▼
┌────────────────────────────────────────────────────────────────┐
│                  CONFIG FACADE (Singleton)                     │
│  • Manages layer priority                                      │
│  • Resolves dot-notation (a.b.c → nested dict)                 │
│  • In-memory cache for performance                             │
└────────────────────────┬───────────────────────────────────────┘
                         │
                         ▼
┌────────────────────────────────────────────────────────────────┐
│              DYNACONF CONFIG PROVIDER (Strategy)               │
│                                                                │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐          │
│  │  Namespace   │  │  Namespace   │  │  Namespace   │          │
│  │  "database"  │  │ "api_client" │  │    "ui"      │          │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘          │
│         │                 │                  │                 │
│         ▼                 ▼                  ▼                 │
│  ┌─────────────────────────────────────────────────┐           │
│  │         LAYER RESOLVER (Priority Stack)         │           │
│  ├─────────────────────────────────────────────────┤           │
│  │ 1. Runtime (volatile)                           │           │
│  │ 2. User Persistent (~/.structum/*.json)         │           │
│  │ 3. Environment Variables (STRUCTUM__*)          │           │
│  │ 4. Secrets (.secrets.toml)                      │           │
│  │ 5. Application Defaults (.toml)                 │           │
│  └─────────────────────────────────────────────────┘           │
│                                                                │
│  ┌─────────────────────────────────────────────────┐           │
│  │   PYDANTIC VALIDATOR (Optional)                 │           │
│  │   • Strong typing                               │           │
│  │   • Field validation                            │           │
│  │   • Custom rules                                │           │
│  └─────────────────────────────────────────────────┘           │
└────────────────────────────────────────────────────────────────┘

5.2 Data Flow (Mermaid)¶

        graph TB
    subgraph "Configuration Layers (Priority ↓)"
        L1[Layer 1: Runtime In-Memory]
        L2[Layer 2: User Persistent JSON]
        L3[Layer 3: Environment Variables]
        L4[Layer 4: Secrets TOML]
        L5[Layer 5: Application Defaults]
    end
    
    subgraph "Validation & Merging"
        V[Pydantic Validator]
        M[Layer Merger]
    end
    
    subgraph "Application Access"
        APP[config.get - key -]
        FACADE[Config Facade]
    end
    
    L5 -->|Load| M
    L4 -->|Load & Merge| M
    L3 -->|Load & Override| M
    L2 -->|Load & Override| M
    L1 -->|Override| M
    
    M -->|Per-Layer| V
    V -->|Validated Data| FACADE
    APP -->|Request| FACADE
    FACADE -->|Resolve| L1
    FACADE -->|Fallback| L2
    FACADE -->|Fallback| L3
    FACADE -->|Fallback| L4
    FACADE -->|Fallback| L5

6. Usage Scenarios¶

6.1 Scenario: Local Development¶

Actors: Developer
Setup:

config/app/database.toml → defaults (versioned)
No env vars
No saved file

Behavior:

config.get("database.host")  # "localhost" (from TOML)
config.get("database.port")  # 5432 (from TOML)

6.2 Scenario: Staging Environment¶

Actors: CI/CD Pipeline, DevOps
Setup:

config/app/database.toml → defaults

ENV vars set by deployment:

STRUCTUM__DATABASE__HOST=staging-db.internal
STRUCTUM__DATABASE__PORT=5432

Behavior:

config.get("database.host")  # "staging-db.internal" (ENV wins)
config.get("database.port")  # 5432 (ENV, even if = default)

6.3 Scenario: Production with Secrets¶

Actors: SRE, Security Team
Setup:

config/app/database.toml → defaults
config/.secrets.toml → password (git-ignored)
ENV vars from K8s ConfigMap + Secrets

Behavior:

config.get("database.host")      # "prod-db.cluster.local" (from ENV)
config.get("database.auth.user") # "prod_user" (from ENV)
config.get("database.auth.password")  # "..." (from .secrets.toml)

6.4 Scenario: End User Customization¶

Actors: End User
Setup:

App installed with defaults
User changes UI theme
Preference saved in ~/.structum/ui_saved.json

Behavior:

# First modification
config.set("ui.theme", "dark")
config.save()  # Writes ~/.structum/ui_saved.json

# After restart
config.get("ui.theme")  # "dark" (from saved.json, not TOML)

7. Best Practices¶

7.1 What Goes in Each Layer¶

Layer	Content	Examples
Defaults (TOML)	Safe values, structure	`log_level = "INFO"`, `timeout = 30`
Secrets	Sensitive credentials	`password`, `api_key`, `private_key`
Environment	Infrastructure config	`host`, `port`, `replicas`
User Persistent	User preferences	`theme`, `language`, `last_opened`
Runtime	Temporary overrides	`debug = True`, `feature_flag_x = True`

7.2 Anti-Patterns to Avoid¶

❌ Secrets in versioned TOML

# config/app/database.toml
[default.auth]
password = "admin123"  # ❌ Exposed in Git!

❌ Hardcoded paths in code

# ❌ Coupling
with open("/etc/myapp/config.toml") as f:
    config = toml.load(f)

# ✅ Use the framework
config = get_config()

❌ Magic values without default

# ❌ Fails if key missing
db_host = config.get("database.host")  # KeyError!

# ✅ Always provide default
db_host = config.get("database.host", default="localhost")

7.3 Security Checklist¶

Production Deployment:

[ ] .secrets.toml in .gitignore
[ ] File permissions: chmod 600
[ ] Env vars for credentials (no files in prod)
[ ] Vault/K8s Secrets for production
[ ] Periodic password rotation
[ ] Audit logging of config changes
[ ] Pydantic validation enabled

7.4 Performance Tips¶

Aggressive Caching: The facade caches resolved values
Lazy Loading: Namespaces are loaded only when accessed
Hot Reload: Use with caution (watchdog overhead)
Validation: Validate on init, not on every get()

Changelog¶

v0.1.0 (Alpha) - 2026-01-13¶

🎉 Initial version
📝 Complete philosophy revision
✨ Clarified 5-layer model
📊 Added Mermaid and ASCII diagrams
🎯 Clear separation: Runtime vs User Persistent
✨ Introduced separate Secrets layer
🔒 Security best practices

References¶

Maintainer: Structum Core Team
Feedback: https://github.com/PythonWoods/structum/discussions