Structum Dynaconf (structum-dynaconf)¶
Structum Dynaconf integrates Dynaconf for advanced, multi-source, and type-safe configuration management.
Feature |
Status |
Version |
|---|---|---|
Status |
Alpha |
0.1.0 |
Namespace |
|
|
Core |
|
Index¶
1. What is Dynaconf Plugin¶
structum-dynaconf is a configuration engine that brings enterprise capabilities to Structum:
1.1 The Problem¶
Before (Without Plugin):
# ❌ Hard-coded configuration
DB_HOST = "localhost"
DB_PORT = 5432
# ❌ No dev/prod separation
# ❌ Secrets in code
# ❌ Zero type validation
After (With Plugin):
# ✅ Multi-layer configuration
# Layer 1: config/app/database.toml (defaults)
# Layer 2: .secrets.toml (passwords)
# Layer 3: STRUCTUM_DATABASE__HOST env var (infra)
# Layer 4: ~/.structum/database_saved.json (user prefs)
config = get_config()
db_host = config.get("database.host") # Resolves layers by priority
1.2 Key Features¶
Feature |
Description |
|---|---|
Multi-Layer Configuration |
Defaults → Secrets → Env Vars → Runtime Overrides |
Strong Typing |
Automatic validation with Pydantic Models |
Convention over Configuration |
Auto-discovery based on directory structure |
Hot Reload |
Automatic reload on file change (watchdog) |
Secrets Isolation |
Separate |
Environment Parity |
Same config for dev/staging/prod with ENV overrides |
2. Core Concepts¶
2.1 Namespace¶
A namespace is a logical container for related configuration.
Example:
# config/app/database.toml
[default]
host = "localhost"
port = 5432
pool_size = 10
The namespace here is "database". You use it like this:
config.get("database.host") # "localhost"
config.get("database.port") # 5432
config.get("database.pool_size") # 10
2.2 Configuration Layers (Priority)¶
When you request config.get("database.host"), the plugin searches in this order:
graph TD
classDef high fill:#ffcccc,stroke:#cc0000
classDef med fill:#ffeedd,stroke:#eeaa00
classDef low fill:#ccffcc,stroke:#006600
subgraph "Resolution Order (Highest Wins)"
L1["1. Runtime Layer"]:::high --> |Overrides| L2
L2["2. Env Vars Layer"]:::high --> |Overrides| L3
L3["3. Secrets Layer"]:::med --> |Overrides| L4
L4["4. App Defaults"]:::low
end
subgraph "Examples"
E1[~/.structum/database_saved.json] -.-> L1
E2[STRUCTUM_DATABASE__HOST] -.-> L2
E3[config/.secrets.toml] -.-> L3
E4[config/app/database.toml] -.-> L4
end
The first value found wins.
2.3 Configuration Builder¶
A Builder is an object that tells the plugin:
What to load: Which TOML files to read
How to name it: The namespace name
How to validate it: (Optional) The Pydantic schema
Three creation modes:
Mode |
When to Use |
|---|---|
Auto-Discovery |
Standard setup with conventions (90% of cases) |
Shortcut ( |
Single custom file outside conventions |
Manual Builder |
Complex logic (multi-file merge, custom validation) |
3. Quick Start (5 Minutes)¶
Step 1: Installation¶
pip install structum-dynaconf
Step 2: Directory Structure¶
mkdir -p config/app config/models
Step 3: Create Configuration¶
File: config/app/database.toml
[default]
host = "localhost"
port = 5432
database = "myapp_dev"
pool_size = 5
[default.auth]
user = "dev_user"
# ⚠️ Do NOT put password here!
[production]
host = "db.prod.example.com"
database = "myapp_prod"
pool_size = 50
Step 4: Setup Plugin¶
File: src/main.py
from structum_lab.config import set_config_provider, get_config
from structum_lab.plugins.dynaconf import DynaconfConfigProvider
def setup_config():
"""
Initialize configuration system.
Call BEFORE any other code.
"""
provider = DynaconfConfigProvider(
root_path=".", # Project root (where config/ is located)
env_prefix="STRUCTUM", # Prefix for env vars
environments=True, # Supports [default], [production], etc.
current_env="development" # Or from ENV: os.getenv("APP_ENV", "development")
)
# 🔍 Auto-discover: scans config/app/*.toml
provider.auto_discover()
# Register as global provider
set_config_provider(provider)
if __name__ == "__main__":
setup_config()
# Now you can use config anywhere
config = get_config()
db_host = config.get("database.host")
db_port = config.get("database.port")
print(f"Connecting to {db_host}:{db_port}")
Step 5: Test¶
# Default (development)
python src/main.py
# Output: Connecting to localhost:5432
# Override with env var
export STRUCTUM_DATABASE__HOST=10.0.0.50
python src/main.py
# Output: Connecting to 10.0.0.50:5432
# Change environment
export APP_ENV=production
python src/main.py
# Output: Connecting to db.prod.example.com:5432
4. Directory Structure & Conventions¶
4.1 Recommended Layout¶
my_project/
├── config/
│ ├── app/ # Application configuration (versioned)
│ │ ├── database.toml # namespace: "database"
│ │ ├── api_client.toml # namespace: "api_client"
│ │ └── features.toml # namespace: "features"
│ │
│ ├── models/ # Pydantic schemas (optional, versioned)
│ │ ├── database.py # class DatabaseConfig(BaseModel)
│ │ └── api_client.py # class ApiClientConfig(BaseModel)
│ │
│ └── .secrets.toml # Secrets (Do NOT version, .gitignore)
│
├── ~/.structum/ # User runtime config (auto-created)
│ ├── database_saved.json # Runtime modifications for "database"
│ └── api_client_saved.json # Runtime modifications for "api_client"
│
└── src/
└── main.py
4.2 Auto-Discovery Conventions¶
File |
Detected Namespace |
Description |
|---|---|---|
|
|
Filename (without |
|
|
Underscore preserved |
|
|
Dashes → underscores |
|
|
Must match namespace name |
⚠️ Important Rules:
Relative paths:
root_pathin provider determines where search startsNamespace name: Derived from filename, normalized (
-→_, lowercase)Model matching: If
config/models/{namespace}.pyexists, it is used for validation
4.3 Recommended .gitignore¶
# Secrets (CRITICAL)
config/.secrets.toml
**/.secrets.toml
# User runtime config
.structum/
**/*_saved.json
# Environment-specific (if using local overrides)
config/local.toml
5. Auto-Discovery vs Manual Loading¶
5.1 Auto-Discovery (Recommended Method)¶
When to use:
Standard setup with files in
config/app/New projects or refactoring towards conventions
You want to reduce boilerplate
How it works:
provider = DynaconfConfigProvider(root_path=".")
provider.auto_discover()
# After this call:
# ✅ All files in config/app/*.toml are registered
# ✅ Models in config/models/*.py are linked automatically
# ✅ .secrets.toml (if exists) is merged into every namespace
⚠️ Critical Note:
auto_discover() is explicit, not automatic. You must call it manually after creating the provider. This gives you control over when filesystem scanning happens (important for testing/mocking).
Example with error handling:
from structum_lab.plugins.dynaconf import DynaconfConfigProvider
from structum_lab.plugins.dynaconf.exceptions import (
ConfigDiscoveryError,
ConfigValidationError
)
provider = DynaconfConfigProvider(root_path=".")
try:
discovered = provider.auto_discover()
print(f"✅ Discovered {len(discovered)} namespaces:")
for ns in discovered:
print(f" - {ns}")
except ConfigDiscoveryError as e:
print(f"❌ Discovery error: {e}")
# config/ directory missing or permission denied
except ConfigValidationError as e:
print(f"❌ Validation failed: {e}")
# Malformed TOML file or Pydantic schema violation
5.2 Manual Loading (Shortcut)¶
When to use:
Legacy files outside conventions
Temporary config for testing
Third-party files to load
Example:
provider = DynaconfConfigProvider(root_path=".")
# Load custom file
provider.load(
namespace="legacy_system",
config_file="external/old_config.toml"
)
# Now accessible as:
config = get_config()
value = config.get("legacy_system.some_key")
⚠️ Limitations:
Does not auto-match with Pydantic models
Does not support multi-file merge (use
GenericConfigBuilderfor that)
6. Validation with Pydantic¶
6.1 Why Validate?¶
Without validation:
# ❌ Hidden runtime errors
pool_size = config.get("database.pool_size") # What does it return? str? int? None?
connections = [None] * pool_size # 💥 TypeError if pool_size is string
With validation:
# ✅ Static guarantees
pool_size: int = config.get("database.pool_size") # Type-safe
# If TOML contains pool_size = "invalid", fails immediately at init
6.2 Create a Pydantic Schema¶
File: config/models/database.py
from pydantic import BaseModel, Field, field_validator
from typing import Literal
class AuthConfig(BaseModel):
"""Database authentication."""
user: str = Field(..., min_length=1, description="Username")
password: str = Field(..., min_length=8, description="Password (from secrets)")
class DatabaseConfig(BaseModel):
"""Main database configuration."""
host: str = Field(default="localhost", description="DB Hostname")
port: int = Field(default=5432, ge=1, le=65535, description="DB Port")
database: str = Field(..., min_length=1, description="Database Name")
pool_size: int = Field(default=10, ge=1, le=1000, description="Connection Pool Size")
auth: AuthConfig
# Custom validators
@field_validator('host')
@classmethod
def validate_production_host(cls, v, info):
"""In production, block localhost."""
if info.context and info.context.get('env') == 'production':
if v in ('localhost', '127.0.0.1'):
raise ValueError("localhost not allowed in production")
return v
@field_validator('pool_size')
@classmethod
def validate_pool_size(cls, v, info):
"""Pool size must be proportional to environment."""
env = info.context.get('env', 'development') if info.context else 'development'
if env == 'production' and v < 20:
raise ValueError(f"Pool size too low for production: {v} (min 20)")
return v
6.3 Automatic Linking (Auto-Discovery)¶
If you follow the convention:
Config file:
config/app/database.tomlModel file:
config/models/database.pyModel class:
DatabaseConfig(must beBaseModel)
The plugin automatically links them:
provider = DynaconfConfigProvider(root_path=".")
provider.auto_discover() # Finds database.toml and DatabaseConfig automatically
# Validation is now active
config = get_config()
port = config.get("database.port") # Guaranteed to be int between 1-65535
6.4 Manual Linking¶
If you don’t use auto-discovery or have custom structures:
from config.models.database import DatabaseConfig
from structum_lab.plugins.dynaconf import GenericConfigBuilder
builder = GenericConfigBuilder(
name="database",
files=["config/app/database.toml"],
model=DatabaseConfig # Explicit link
)
provider.register_builder("database", builder)
7. Environment Variables Override¶
7.1 Naming Convention¶
Formula:
{ENV_PREFIX}__{NAMESPACE}__{PATH__TO__KEY}
Components:
ENV_PREFIX: DefaultSTRUCTUM, configurable in providerNAMESPACE: Namespace name (e.g.DATABASE)PATH__TO__KEY: Dot-notation path, with.→__
⚠️ Important: Double underscore __ separates levels, single _ is part of the name.
7.2 Examples¶
Original Config:
# config/app/database.toml
[default]
host = "localhost"
port = 5432
[default.pool]
min_size = 5
max_size = 20
Override via ENV:
# Single value
export STRUCTUM_DATABASE__HOST=prod.db.com
export STRUCTUM_DATABASE__PORT=3306
# Nested value
export STRUCTUM_DATABASE__POOL__MAX_SIZE=100
# Namespace with underscore (api_client)
export STRUCTUM_API_CLIENT__TIMEOUT=60
7.3 Supported Types¶
Scalars:
export STRUCTUM_DATABASE__PORT=5432 # int
export STRUCTUM_DATABASE__ENABLED=true # bool
export STRUCTUM_DATABASE__TIMEOUT=30.5 # float
export STRUCTUM_DATABASE__HOST=localhost # str
Lists (separator: comma):
export STRUCTUM_DATABASE__HOSTS=db1.com,db2.com,db3.com
# In Python:
hosts = config.get("database.hosts") # ["db1.com", "db2.com", "db3.com"]
Embedded JSON (for complex structures):
export STRUCTUM_DATABASE__AUTH='{"user":"admin","password":"secret"}'
# In Python:
auth = config.get("database.auth") # {"user": "admin", "password": "secret"}
8. Runtime Persistence¶
8.1 How It Works¶
Whenconfig.set()`, the value is saved in the Runtime Layer:
config = get_config()
# In-memory modification
config.set("database.host", "10.0.0.100")
# Save to disk (optional but recommended)
config.save()
# Created file: ~/.structum/database_saved.json
# {
# "host": "10.0.0.100"
# }
⚠️ Critical Note:
set()modifies only the in-memory cachesave()is required for persistence across restartsOnly keys modified via
set()end up in the*_saved.jsonfile
8.2 Runtime Files Location¶
Default:
~/.structum/
├── database_saved.json
├── api_client_saved.json
└── features_saved.json
Custom path:
provider = DynaconfConfigProvider(
root_path=".",
runtime_config_dir="/var/lib/myapp/config"
)
9. Secrets Management¶
9.1 The .secrets.toml File¶
Location:
config/.secrets.toml
⚠️ CRITICAL: Add to .gitignore:
config/.secrets.toml
**/.secrets.toml
Example Content:
# config/.secrets.toml
[database.auth]
password = "SuperSecretPassword123"
[api_client]
api_key = "sk-proj-abc123xyz"
api_secret = "shhh-very-secret"
9.2 Auto-Merge in Namespaces¶
The plugin automatically merges .secrets.toml with config/app/*.toml files:
Before merge:
# config/app/database.toml
[default.auth]
user = "admin"
# password missing
# config/.secrets.toml
[database.auth]
password = "secret123"
After merge (in-memory):
config.get("database.auth.password") # "secret123"
10. Hot Reload¶
To enable hot reload:
pip install dynaconf[watchdog]
It works automatically. If you modify config/app/database.toml, Dynaconf reloads the values.
Code Hooks:
from structum_lab.config import on_config_change
@on_config_change("database")
def reload_db_connection(new_config):
print("Database config changed! Reconnecting...")
db.reconnect(new_config.database.url)
Note: Pydantic validation runs again on reload. If the new file is invalid, the reload acts according to Dynaconf policies (usually logs error and keeps old values).
11. Advanced: GenericConfigBuilder¶
For complex scenarios where you need to manually assemble multiple sources.
from structum_lab.plugins.dynaconf import GenericConfigBuilder
builder = GenericConfigBuilder(
name="complex_system",
files=[
"config/base.toml",
"config/features.toml",
"/etc/myapp/site.toml"
],
includes=["config/extra/*.toml"],
model=MyComplexSchema
)
provider.register_builder("complex_system", builder)
Benefits:
Merge arbitrary files
Custom validation schema
Specific namespace name independent of filenames
12. Troubleshooting¶
“Validation Error: field required”¶
You defined a Pydantic model with required fields, but you didn’t provide them in TOML or ENV.
Solution: Add default values in the model or ensure the value exists in [default].
“Config not found for namespace ‘xyz’”¶
You called config.get("xyz.key") but namespace xyz was not loaded.
Solution:
Check if the file exists:
config/app/xyz.tomlCheck if you called
provider.auto_discover()Check permissions on the config directory
“Secrets are not loading”¶
Solution:
Verify
.secrets.tomlis inconfig/(orroot_path)Verify the structure matches the namespace:
[namespace.subkey]Ensure
.secrets.tomlis a valid TOML file
13. API Reference¶
13.1 DynaconfConfigProvider¶
class DynaconfConfigProvider:
def __init__(
self,
root_path: str,
env_prefix: str = "STRUCTUM",
environments: bool = True,
current_env: str = "development",
runtime_config_dir: str = None
):
...
def auto_discover(self) -> List[str]:
"""Scans for valid config files."""
...
def load(self, namespace: str, config_file: str) -> None:
"""Load specific file."""
...