Anyone who develops a project which involves multiple environments (e.g. DEV, QA, PROD) knows how painful it is to write code once which works everywhere, especially if it involves lots of env-specific tools (including cloud). I too have faced such problems & to solve it once-end for all I came up with a system called Environment Agnostic Config.

1. The Old School way

Using the config to control environments & settings is nothing new. Developers have been using it for ages, so why is it necessary to reinvent the wheel? I am not proposing to reinvent the wheel, but modernize it.

The most popular (& probably the most naïve way too 😕) is the use of file-based configs like JSON, YAML, TOML, etc. Apart from being naïve, they have two significant issues:

• they have to be hardcoded

• they cannot be generated dynamically.

This is a big deal breaker when dealing with multiple environments. If you have multiple sources to populate it (e.g. some secret vault, environment variables, hardcoded values, etc) then I would strongly recommend just dropping the idea of using a file-based config to save hardships in your life 😉.

2. Environment Agnostic Config: Generating config programmatically

To achieve this, I use Pydantic's Setting Management application. Anyone familiar with Pydantic will find this familiar & easy to use.

Usually, you must be using

• Something like a .env file to store all secrets & later read them. Or store secrets/configuration directly as environment variables & then read them.

• Other configurations/settings can be hardcoded and written into json/yaml/toml/ini files.

So with the help of Pydantic's BaseSettings we can combine both. Let's dive in to see it in action.

Pydantic's BaseSettings already have support to read from environment variable, .env file, etc (Follow the original docs to see in more details - Settings management - pydantic). So this way we can have some fields (or class attribute) as hardcoded feils & some populate from environment variables under one common Pydantic model. Let's see an example.

from pydantic import BaseSettings, Feild

class Config(BaseSettings):
    env_variable1: str = Feild(description="some description")
    env_variable: str = Feild(description="some description")
    hard_coded1: str = "some hardcoded value"
    hard_coded2: int = 999

From the above example, the first two fields will be automatically populated from matching environment variables, the next two are the hard coded variables.

Note: Since this is based on Pydantic, you can add all sorts of regular Pydantic validators. See the original docs (above) to see all the possibilities.

3. One Config to rule them all

Now coming to the most important part - How to use one config for all possible environments (e.g. DEV, QA, PROD, etc). Actually, it is quite easy, just create a respective Pydantic BaseSettings model for all environments.

from pydantic import BaseSettings, Feild

class LocalSettings(BaseSettings):
    env_variable1: str = Feild(description="some description")
    env_variable: str = Feild(description="some description")
    hard_coded1: str = "some hardcoded value"
    hard_coded2: int = 999

class DEVSettings(BaseSettings):
    env_variable1: str = Feild(description="some description")
    env_variable: str = Feild(description="some description")
    hard_coded1: str = "some hardcoded value"
    hard_coded2: int = 999

class PRODSettings(BaseSettings):
    env_variable1: str = Feild(description="some description")
    env_variable: str = Feild(description="some description")
    hard_coded1: str = "some hardcoded value"
    hard_coded2: int = 999

Since the underlying environment variables will be different for all environments, so will be the populated fields.

4. How to actually consume the config in code

Now we have a single source of config so moving to next part is to how actually consume it in some code. Again there is nothing novel here. What I do is create a function which takes the underlying environment as a parameter as input & return respective config. Also, I usually store all this in config.py

# logic in config.py
from pydantic import BaseSettings, Feild

class LocalSettings(BaseSettings):
# same as above
class DEVSettings(BaseSettings):
# same as above
class PRODSettings(BaseSettings):
# same as above

def get_config(environment: str):
    match environment:
        case "local":
            config = LocalSettings()
        case "dev":
            config = DEVSettings()
        case "prod":
            config = PRODSettings()
    return config


# in some other part of your code/library
from config import get_config
config = get_config("dev")
some_variable = config.env_variable

# NOTE - you don't need to even hardcode environment parameter. What I do is simply create a environment variable for environment it self & use to in function.
config = get_config(os.environ["environment"])
some_variable = config.env_variable
# Now this makes truly environment agnostic & excat same code will work everywhere.

What if you have multiple scopes of multiple config requirements? The pattern remians the same. Add as many as required configs as Pydantic BaseSettings model & return them.

Let me explain with a simple use case of mine. My application needs to support multiple languages. About 80% of the code is generic but there few logic which are language dependent & changes based on underlying language. So I just define them in their respective language Pydantic model. Let's see an example

from pydantic import BaseSettings, Feild

class EnglishConfig(BaseSettings):
    variable1: str = "some thing"
    variable: list = [1,2,3]
    hard_coded1: dict = {}
    hard_coded2: int = 999

class FrenchConfig(BaseSettings):
    variable1: str = "some thing"
    variable: list = [1,2,3]
    hard_coded1: dict = {}
    hard_coded2: int = 999

class HindiConfig(BaseSettings):
    variable1: str = "some thing"
    variable: list = [1,2,3]
    hard_coded1: dict = {}
    hard_coded2: int = 999

Now exactly similar to above logic for environment settings we can we make language dependant or language specific logic as language agnostic.

5. Bringing all things together

Finally, let me show you how my final config.py looks like

from typing import Any, Optional

from pydantic import BaseSettings

class EnglishConfig(BaseSettings):
    regex_pattern_alphanumeric: Optional[str] = "[^0-9a-z/s]"
    list_of_missing_must_include_words = ["Missing", "Must include"]
    list_of_name_prefixes = ["dr", "mr", "mrs", "jr", "sr"]

class SpanishConfig(BaseSettings):
    regex_pattern_alphanumeric: Optional[str] = "[^0-9a-záéíñóúü/s]"
    list_of_name_prefixes = ["sres", "señora"]
    list_of_missing_must_include_words = ["Falta", "Debe incluir lo siguiente"]

class FrenchConfig(BaseSettings):
    regex_pattern_alphanumeric: Optional[str] = "[^0-9a-z\u00C0-\u017F/s]"
    list_of_missing_must_include_words = ["Termes manquants", "Doit inclure"]
    list_of_name_prefixes = ["m", "madame"]

class LocalEnvironmentSettings(BaseSettings):
    common_root_folder: Optional[str] = "/tmp"
    logging_level: Optional[int] | Optional[tuple] = (10,10,10,)
    status_url: str = "https://some-url-dev.com"
    SOME_SECRET: str 
    CONNECTION_STRING: str

class DevEnvironmentSettings(BaseSettings):
    common_root_folder: Optional[str] = "/tmp"
    logging_level: Optional[int] | Optional[tuple] = (10,10,10,)
    status_url: str = "https://some-url-dev.com"
    SOME_SECRET: str 
    CONNECTION_STRING: str

class QAEnvironmentSettings(BaseSettings):
    common_root_folder: Optional[str] = "/tmp"
    logging_level: Optional[int] | Optional[tuple] = (10,10,20,)
    status_url: str = "https://some-url-qa.com"
    SOME_SECRET: str 
    CONNECTION_STRING: str
class PRODEnvironmentSettings(BaseSettings):
    common_root_folder: Optional[str] = "/tmp"
    logging_level: Optional[int] | Optional[tuple] = (10,10,20,)
    status_url: str = "https://some-url-prod.com"
    SOME_SECRET: str 
    CONNECTION_STRING: str
# NOTE - See how I have changed the `status_url` & `logging_level`for all environments & `regex_pattern_alphanumeric` for all languages.

def get_config(language: str, environment: str):
    # setting language based config
    match language:
        case "en":
            language_config = EnglishConfig()
        case "es":
            language_config = SpanishConfig()
        case "fr":
            language_config = FrenchConfig()
        case _:
            raise ValueError(f"given language: {language} must be either from en, es, pt,")
    # setting environment based config
    match environment:
        case "local":
            environment_settings = LocalEnvironmentSettings()
        case "dev":
            environment_settings = DevEnvironmentSettings()
        case "qa":
            environment_settings = QAEnvironmentSettings()
        case "pro":
            environment_settings = PRODEnvironmentSettings()

    class GlobalConfig(BaseSettings):
                global_language_config = language_config
                global_environment_settings = environment_settings

    return GlobalConfig()

Note: As my other blogs, this idea is not limited just to python but can be used anywhere. I have used python to explain the idea. Few modifications & same approach can be applied anywhere.