Structum Dynaconf (structum-dynaconf)

Documentation Source Code Python 3.11+ License: Apache-2.0

Structum Dynaconf integrates Dynaconf for advanced, multi-source, and type-safe configuration management.

Feature

Status

Version

Status

Alpha

0.1.0

Namespace

structum_lab.plugins.dynaconf

Core

dynaconf, pydantic


Index

  1. What is Dynaconf Plugin

  2. Core Concepts

  3. Quick Start (5 Minutes)

  4. Directory Structure & Conventions

  5. Auto-Discovery vs Manual Loading

  6. Validation with Pydantic

  7. Environment Variables Override

  8. Runtime Persistence

  9. Secrets Management

  10. Hot Reload

  11. Advanced: GenericConfigBuilder

  12. Troubleshooting

  13. API Reference


1. What is Dynaconf Plugin

structum-dynaconf is a configuration engine that brings enterprise capabilities to Structum:

1.1 The Problem

Before (Without Plugin):

# ❌ Hard-coded configuration
DB_HOST = "localhost"
DB_PORT = 5432

# ❌ No dev/prod separation
# ❌ Secrets in code
# ❌ Zero type validation

After (With Plugin):

# ✅ Multi-layer configuration
# Layer 1: config/app/database.toml (defaults)
# Layer 2: .secrets.toml (passwords)
# Layer 3: STRUCTUM_DATABASE__HOST env var (infra)
# Layer 4: ~/.structum/database_saved.json (user prefs)

config = get_config()
db_host = config.get("database.host")  # Resolves layers by priority

1.2 Key Features

Feature

Description

Multi-Layer Configuration

Defaults → Secrets → Env Vars → Runtime Overrides

Strong Typing

Automatic validation with Pydantic Models

Convention over Configuration

Auto-discovery based on directory structure

Hot Reload

Automatic reload on file change (watchdog)

Secrets Isolation

Separate .secrets.toml file (never versioned)

Environment Parity

Same config for dev/staging/prod with ENV overrides


2. Core Concepts

2.1 Namespace

A namespace is a logical container for related configuration.

Example:

# config/app/database.toml
[default]
host = "localhost"
port = 5432
pool_size = 10

The namespace here is "database". You use it like this:

config.get("database.host")      # "localhost"
config.get("database.port")      # 5432
config.get("database.pool_size") # 10

2.2 Configuration Layers (Priority)

When you request config.get("database.host"), the plugin searches in this order:

        graph TD
    classDef high fill:#ffcccc,stroke:#cc0000
    classDef med fill:#ffeedd,stroke:#eeaa00
    classDef low fill:#ccffcc,stroke:#006600

    subgraph "Resolution Order (Highest Wins)"
        L1["1. Runtime Layer"]:::high --> |Overrides| L2
        L2["2. Env Vars Layer"]:::high --> |Overrides| L3
        L3["3. Secrets Layer"]:::med --> |Overrides| L4
        L4["4. App Defaults"]:::low
    end

    subgraph "Examples"
        E1[~/.structum/database_saved.json] -.-> L1
        E2[STRUCTUM_DATABASE__HOST] -.-> L2
        E3[config/.secrets.toml] -.-> L3
        E4[config/app/database.toml] -.-> L4
    end
    

The first value found wins.

2.3 Configuration Builder

A Builder is an object that tells the plugin:

  • What to load: Which TOML files to read

  • How to name it: The namespace name

  • How to validate it: (Optional) The Pydantic schema

Three creation modes:

Mode

When to Use

Auto-Discovery

Standard setup with conventions (90% of cases)

Shortcut (load)

Single custom file outside conventions

Manual Builder

Complex logic (multi-file merge, custom validation)


3. Quick Start (5 Minutes)

Step 1: Installation

pip install structum-dynaconf

Step 2: Directory Structure

mkdir -p config/app config/models

Step 3: Create Configuration

File: config/app/database.toml

[default]
host = "localhost"
port = 5432
database = "myapp_dev"
pool_size = 5

[default.auth]
user = "dev_user"
# ⚠️ Do NOT put password here!

[production]
host = "db.prod.example.com"
database = "myapp_prod"
pool_size = 50

Step 4: Setup Plugin

File: src/main.py

from structum_lab.config import set_config_provider, get_config
from structum_lab.plugins.dynaconf import DynaconfConfigProvider

def setup_config():
    """
    Initialize configuration system.
    Call BEFORE any other code.
    """
    provider = DynaconfConfigProvider(
        root_path=".",  # Project root (where config/ is located)
        env_prefix="STRUCTUM",  # Prefix for env vars
        environments=True,  # Supports [default], [production], etc.
        current_env="development"  # Or from ENV: os.getenv("APP_ENV", "development")
    )
    
    # 🔍 Auto-discover: scans config/app/*.toml
    provider.auto_discover()
    
    # Register as global provider
    set_config_provider(provider)

if __name__ == "__main__":
    setup_config()
    
    # Now you can use config anywhere
    config = get_config()
    db_host = config.get("database.host")
    db_port = config.get("database.port")
    
    print(f"Connecting to {db_host}:{db_port}")

Step 5: Test

# Default (development)
python src/main.py
# Output: Connecting to localhost:5432

# Override with env var
export STRUCTUM_DATABASE__HOST=10.0.0.50
python src/main.py
# Output: Connecting to 10.0.0.50:5432

# Change environment
export APP_ENV=production
python src/main.py
# Output: Connecting to db.prod.example.com:5432

4. Directory Structure & Conventions

my_project/
├── config/
│   ├── app/                    # Application configuration (versioned)
│   │   ├── database.toml       # namespace: "database"
│   │   ├── api_client.toml     # namespace: "api_client"
│   │   └── features.toml       # namespace: "features"
│   │
│   ├── models/                 # Pydantic schemas (optional, versioned)
│   │   ├── database.py         # class DatabaseConfig(BaseModel)
│   │   └── api_client.py       # class ApiClientConfig(BaseModel)
│   │
│   └── .secrets.toml           # Secrets (Do NOT version, .gitignore)
│
├── ~/.structum/                # User runtime config (auto-created)
│   ├── database_saved.json     # Runtime modifications for "database"
│   └── api_client_saved.json   # Runtime modifications for "api_client"
│
└── src/
    └── main.py

4.2 Auto-Discovery Conventions

File

Detected Namespace

Description

config/app/database.toml

database

Filename (without .toml)

config/app/api_client.toml

api_client

Underscore preserved

config/app/api-client.toml

api_client

Dashes → underscores

config/models/database.py

database

Must match namespace name

⚠️ Important Rules:

  1. Relative paths: root_path in provider determines where search starts

  2. Namespace name: Derived from filename, normalized (-_, lowercase)

  3. Model matching: If config/models/{namespace}.py exists, it is used for validation

# Secrets (CRITICAL)
config/.secrets.toml
**/.secrets.toml

# User runtime config
.structum/
**/*_saved.json

# Environment-specific (if using local overrides)
config/local.toml

5. Auto-Discovery vs Manual Loading

When to use:

  • Standard setup with files in config/app/

  • New projects or refactoring towards conventions

  • You want to reduce boilerplate

How it works:

provider = DynaconfConfigProvider(root_path=".")
provider.auto_discover()

# After this call:
# ✅ All files in config/app/*.toml are registered
# ✅ Models in config/models/*.py are linked automatically
# ✅ .secrets.toml (if exists) is merged into every namespace

⚠️ Critical Note:
auto_discover() is explicit, not automatic. You must call it manually after creating the provider. This gives you control over when filesystem scanning happens (important for testing/mocking).

Example with error handling:

from structum_lab.plugins.dynaconf import DynaconfConfigProvider
from structum_lab.plugins.dynaconf.exceptions import (
    ConfigDiscoveryError,
    ConfigValidationError
)

provider = DynaconfConfigProvider(root_path=".")

try:
    discovered = provider.auto_discover()
    print(f"✅ Discovered {len(discovered)} namespaces:")
    for ns in discovered:
        print(f"  - {ns}")
except ConfigDiscoveryError as e:
    print(f"❌ Discovery error: {e}")
    # config/ directory missing or permission denied
except ConfigValidationError as e:
    print(f"❌ Validation failed: {e}")
    # Malformed TOML file or Pydantic schema violation

5.2 Manual Loading (Shortcut)

When to use:

  • Legacy files outside conventions

  • Temporary config for testing

  • Third-party files to load

Example:

provider = DynaconfConfigProvider(root_path=".")

# Load custom file
provider.load(
    namespace="legacy_system",
    config_file="external/old_config.toml"
)

# Now accessible as:
config = get_config()
value = config.get("legacy_system.some_key")

⚠️ Limitations:

  • Does not auto-match with Pydantic models

  • Does not support multi-file merge (use GenericConfigBuilder for that)


6. Validation with Pydantic

6.1 Why Validate?

Without validation:

# ❌ Hidden runtime errors
pool_size = config.get("database.pool_size")  # What does it return? str? int? None?
connections = [None] * pool_size  # 💥 TypeError if pool_size is string

With validation:

# ✅ Static guarantees
pool_size: int = config.get("database.pool_size")  # Type-safe
# If TOML contains pool_size = "invalid", fails immediately at init

6.2 Create a Pydantic Schema

File: config/models/database.py

from pydantic import BaseModel, Field, field_validator
from typing import Literal

class AuthConfig(BaseModel):
    """Database authentication."""
    user: str = Field(..., min_length=1, description="Username")
    password: str = Field(..., min_length=8, description="Password (from secrets)")

class DatabaseConfig(BaseModel):
    """Main database configuration."""
    
    host: str = Field(default="localhost", description="DB Hostname")
    port: int = Field(default=5432, ge=1, le=65535, description="DB Port")
    database: str = Field(..., min_length=1, description="Database Name")
    pool_size: int = Field(default=10, ge=1, le=1000, description="Connection Pool Size")
    
    auth: AuthConfig
    
    # Custom validators
    @field_validator('host')
    @classmethod
    def validate_production_host(cls, v, info):
        """In production, block localhost."""
        if info.context and info.context.get('env') == 'production':
            if v in ('localhost', '127.0.0.1'):
                raise ValueError("localhost not allowed in production")
        return v
    
    @field_validator('pool_size')
    @classmethod
    def validate_pool_size(cls, v, info):
        """Pool size must be proportional to environment."""
        env = info.context.get('env', 'development') if info.context else 'development'
        
        if env == 'production' and v < 20:
            raise ValueError(f"Pool size too low for production: {v} (min 20)")
        
        return v

6.3 Automatic Linking (Auto-Discovery)

If you follow the convention:

  • Config file: config/app/database.toml

  • Model file: config/models/database.py

  • Model class: DatabaseConfig (must be BaseModel)

The plugin automatically links them:

provider = DynaconfConfigProvider(root_path=".")
provider.auto_discover()  # Finds database.toml and DatabaseConfig automatically

# Validation is now active
config = get_config()
port = config.get("database.port")  # Guaranteed to be int between 1-65535

6.4 Manual Linking

If you don’t use auto-discovery or have custom structures:

from config.models.database import DatabaseConfig
from structum_lab.plugins.dynaconf import GenericConfigBuilder

builder = GenericConfigBuilder(
    name="database",
    files=["config/app/database.toml"],
    model=DatabaseConfig  # Explicit link
)

provider.register_builder("database", builder)

7. Environment Variables Override

7.1 Naming Convention

Formula:

{ENV_PREFIX}__{NAMESPACE}__{PATH__TO__KEY}

Components:

  • ENV_PREFIX: Default STRUCTUM, configurable in provider

  • NAMESPACE: Namespace name (e.g. DATABASE)

  • PATH__TO__KEY: Dot-notation path, with .__

⚠️ Important: Double underscore __ separates levels, single _ is part of the name.

7.2 Examples

Original Config:

# config/app/database.toml
[default]
host = "localhost"
port = 5432

[default.pool]
min_size = 5
max_size = 20

Override via ENV:

# Single value
export STRUCTUM_DATABASE__HOST=prod.db.com
export STRUCTUM_DATABASE__PORT=3306

# Nested value
export STRUCTUM_DATABASE__POOL__MAX_SIZE=100

# Namespace with underscore (api_client)
export STRUCTUM_API_CLIENT__TIMEOUT=60

7.3 Supported Types

Scalars:

export STRUCTUM_DATABASE__PORT=5432          # int
export STRUCTUM_DATABASE__ENABLED=true       # bool
export STRUCTUM_DATABASE__TIMEOUT=30.5       # float
export STRUCTUM_DATABASE__HOST=localhost     # str

Lists (separator: comma):

export STRUCTUM_DATABASE__HOSTS=db1.com,db2.com,db3.com

# In Python:
hosts = config.get("database.hosts")  # ["db1.com", "db2.com", "db3.com"]

Embedded JSON (for complex structures):

export STRUCTUM_DATABASE__AUTH='{"user":"admin","password":"secret"}'

# In Python:
auth = config.get("database.auth")  # {"user": "admin", "password": "secret"}

8. Runtime Persistence

8.1 How It Works

Whenconfig.set()`, the value is saved in the Runtime Layer:

config = get_config()

# In-memory modification
config.set("database.host", "10.0.0.100")

# Save to disk (optional but recommended)
config.save()

# Created file: ~/.structum/database_saved.json
# {
#   "host": "10.0.0.100"
# }

⚠️ Critical Note:

  • set() modifies only the in-memory cache

  • save() is required for persistence across restarts

  • Only keys modified via set() end up in the *_saved.json file

8.2 Runtime Files Location

Default:

~/.structum/
├── database_saved.json
├── api_client_saved.json
└── features_saved.json

Custom path:

provider = DynaconfConfigProvider(
    root_path=".",
    runtime_config_dir="/var/lib/myapp/config"
)

9. Secrets Management

9.1 The .secrets.toml File

Location:

config/.secrets.toml

⚠️ CRITICAL: Add to .gitignore:

config/.secrets.toml
**/.secrets.toml

Example Content:

# config/.secrets.toml

[database.auth]
password = "SuperSecretPassword123"

[api_client]
api_key = "sk-proj-abc123xyz"
api_secret = "shhh-very-secret"

9.2 Auto-Merge in Namespaces

The plugin automatically merges .secrets.toml with config/app/*.toml files:

Before merge:

# config/app/database.toml
[default.auth]
user = "admin"
# password missing

# config/.secrets.toml
[database.auth]
password = "secret123"

After merge (in-memory):

config.get("database.auth.password") # "secret123"

10. Hot Reload

To enable hot reload:

pip install dynaconf[watchdog]

It works automatically. If you modify config/app/database.toml, Dynaconf reloads the values.

Code Hooks:

from structum_lab.config import on_config_change

@on_config_change("database")
def reload_db_connection(new_config):
    print("Database config changed! Reconnecting...")
    db.reconnect(new_config.database.url)

Note: Pydantic validation runs again on reload. If the new file is invalid, the reload acts according to Dynaconf policies (usually logs error and keeps old values).


11. Advanced: GenericConfigBuilder

For complex scenarios where you need to manually assemble multiple sources.

from structum_lab.plugins.dynaconf import GenericConfigBuilder

builder = GenericConfigBuilder(
    name="complex_system",
    files=[
        "config/base.toml",
        "config/features.toml",
        "/etc/myapp/site.toml"
    ],
    includes=["config/extra/*.toml"],
    model=MyComplexSchema
)

provider.register_builder("complex_system", builder)

Benefits:

  • Merge arbitrary files

  • Custom validation schema

  • Specific namespace name independent of filenames


12. Troubleshooting

“Validation Error: field required”

You defined a Pydantic model with required fields, but you didn’t provide them in TOML or ENV.

Solution: Add default values in the model or ensure the value exists in [default].

“Config not found for namespace ‘xyz’”

You called config.get("xyz.key") but namespace xyz was not loaded.

Solution:

  1. Check if the file exists: config/app/xyz.toml

  2. Check if you called provider.auto_discover()

  3. Check permissions on the config directory

“Secrets are not loading”

Solution:

  1. Verify .secrets.toml is in config/ (or root_path)

  2. Verify the structure matches the namespace: [namespace.subkey]

  3. Ensure .secrets.toml is a valid TOML file


13. API Reference

13.1 DynaconfConfigProvider

class DynaconfConfigProvider:
    def __init__(
        self, 
        root_path: str, 
        env_prefix: str = "STRUCTUM",
        environments: bool = True,
        current_env: str = "development",
        runtime_config_dir: str = None
    ):
        ...

    def auto_discover(self) -> List[str]:
        """Scans for valid config files."""
        ...
        
    def load(self, namespace: str, config_file: str) -> None:
        """Load specific file."""
        ...