Anyone who develops a project which involves multiple environments (e.g. DEV, QA, PROD) knows how painful it is to write code once which works everywhere, especially if it involves lots of env-specific tools (including cloud). I too have faced such problems & to solve it once-end for all I came up with a system called Environment Agnostic Config.
1. The Old School way
Using the config to control environments & settings is nothing new. Developers have been using it for ages, so why is it necessary to reinvent the wheel? I am not proposing to reinvent the wheel, but modernize it.
The most popular (& probably the most naïve way too 😕) is the use of file-based configs like JSON, YAML, TOML, etc. Apart from being naïve, they have two significant issues:
This is a big deal breaker when dealing with multiple environments. If you have multiple sources to populate it (e.g. some secret vault, environment variables, hardcoded values, etc) then I would strongly recommend just dropping the idea of using a file-based config to save hardships in your life 😉.
2. Environment Agnostic Config: Generating config programmatically
To achieve this, I use Pydantic's Setting Management application. Anyone familiar with Pydantic will find this familiar & easy to use.
Usually, you must be using
Something like a .env file to store all secrets & later read them. Or store secrets/configuration directly as environment variables & then read them.
Other configurations/settings can be hardcoded and written into json/yaml/toml/ini files.
So with the help of Pydantic's BaseSettings we can combine both. Let's dive in to see it in action.
Pydantic's BaseSettings already have support to read from environment variable, .env file, etc (Follow the original docs to see in more details - Settings management - pydantic). So this way we can have some fields (or class attribute) as hardcoded feils & some populate from environment variables under one common Pydantic model. Let's see an example.
from pydantic import BaseSettings, Feild
class Config(BaseSettings):
env_variable1: str = Feild(description="some description")
env_variable: str = Feild(description="some description")
hard_coded1: str = "some hardcoded value"
hard_coded2: int = 999
From the above example, the first two fields will be automatically populated from matching environment variables, the next two are the hard coded variables.
Note: Since this is based on Pydantic, you can add all sorts of regular Pydantic validators. See the original docs (above) to see all the possibilities.
3. One Config to rule them all
Now coming to the most important part - How to use one config for all possible environments (e.g. DEV, QA, PROD, etc). Actually, it is quite easy, just create a respective Pydantic BaseSettings model for all environments.
from pydantic import BaseSettings, Feild
class LocalSettings(BaseSettings):
env_variable1: str = Feild(description="some description")
env_variable: str = Feild(description="some description")
hard_coded1: str = "some hardcoded value"
hard_coded2: int = 999
class DEVSettings(BaseSettings):
env_variable1: str = Feild(description="some description")
env_variable: str = Feild(description="some description")
hard_coded1: str = "some hardcoded value"
hard_coded2: int = 999
class PRODSettings(BaseSettings):
env_variable1: str = Feild(description="some description")
env_variable: str = Feild(description="some description")
hard_coded1: str = "some hardcoded value"
hard_coded2: int = 999
Since the underlying environment variables will be different for all environments, so will be the populated fields.
4. How to actually consume the config in code
Now we have a single source of config so moving to next part is to how actually consume it in some code. Again there is nothing novel here. What I do is create a function which takes the underlying environment as a parameter as input & return respective config. Also, I usually store all this in config.py
from pydantic import BaseSettings, Feild
class LocalSettings(BaseSettings):
class DEVSettings(BaseSettings):
class PRODSettings(BaseSettings):
def get_config(environment: str):
match environment:
case "local":
config = LocalSettings()
case "dev":
config = DEVSettings()
case "prod":
config = PRODSettings()
return config
from config import get_config
config = get_config("dev")
some_variable = config.env_variable
config = get_config(os.environ["environment"])
some_variable = config.env_variable
What if you have multiple scopes of multiple config requirements? The pattern remians the same. Add as many as required configs as Pydantic BaseSettings model & return them.
Let me explain with a simple use case of mine. My application needs to support multiple languages. About 80% of the code is generic but there few logic which are language dependent & changes based on underlying language. So I just define them in their respective language Pydantic model. Let's see an example
from pydantic import BaseSettings, Feild
class EnglishConfig(BaseSettings):
variable1: str = "some thing"
variable: list = [1,2,3]
hard_coded1: dict = {}
hard_coded2: int = 999
class FrenchConfig(BaseSettings):
variable1: str = "some thing"
variable: list = [1,2,3]
hard_coded1: dict = {}
hard_coded2: int = 999
class HindiConfig(BaseSettings):
variable1: str = "some thing"
variable: list = [1,2,3]
hard_coded1: dict = {}
hard_coded2: int = 999
Now exactly similar to above logic for environment settings we can we make language dependant or language specific logic as language agnostic.
5. Bringing all things together
Finally, let me show you how my final config.py looks like
from typing import Any, Optional
from pydantic import BaseSettings
class EnglishConfig(BaseSettings):
regex_pattern_alphanumeric: Optional[str] = "[^0-9a-z/s]"
list_of_missing_must_include_words = ["Missing", "Must include"]
list_of_name_prefixes = ["dr", "mr", "mrs", "jr", "sr"]
class SpanishConfig(BaseSettings):
regex_pattern_alphanumeric: Optional[str] = "[^0-9a-záéÃñóúü/s]"
list_of_name_prefixes = ["sres", "señora"]
list_of_missing_must_include_words = ["Falta", "Debe incluir lo siguiente"]
class FrenchConfig(BaseSettings):
regex_pattern_alphanumeric: Optional[str] = "[^0-9a-z\u00C0-\u017F/s]"
list_of_missing_must_include_words = ["Termes manquants", "Doit inclure"]
list_of_name_prefixes = ["m", "madame"]
class LocalEnvironmentSettings(BaseSettings):
common_root_folder: Optional[str] = "/tmp"
logging_level: Optional[int] | Optional[tuple] = (10,10,10,)
status_url: str = "https://some-url-dev.com"
SOME_SECRET: str
CONNECTION_STRING: str
class DevEnvironmentSettings(BaseSettings):
common_root_folder: Optional[str] = "/tmp"
logging_level: Optional[int] | Optional[tuple] = (10,10,10,)
status_url: str = "https://some-url-dev.com"
SOME_SECRET: str
CONNECTION_STRING: str
class QAEnvironmentSettings(BaseSettings):
common_root_folder: Optional[str] = "/tmp"
logging_level: Optional[int] | Optional[tuple] = (10,10,20,)
status_url: str = "https://some-url-qa.com"
SOME_SECRET: str
CONNECTION_STRING: str
class PRODEnvironmentSettings(BaseSettings):
common_root_folder: Optional[str] = "/tmp"
logging_level: Optional[int] | Optional[tuple] = (10,10,20,)
status_url: str = "https://some-url-prod.com"
SOME_SECRET: str
CONNECTION_STRING: str
def get_config(language: str, environment: str):
match language:
case "en":
language_config = EnglishConfig()
case "es":
language_config = SpanishConfig()
case "fr":
language_config = FrenchConfig()
case _:
raise ValueError(f"given language: {language} must be either from en, es, pt,")
match environment:
case "local":
environment_settings = LocalEnvironmentSettings()
case "dev":
environment_settings = DevEnvironmentSettings()
case "qa":
environment_settings = QAEnvironmentSettings()
case "pro":
environment_settings = PRODEnvironmentSettings()
class GlobalConfig(BaseSettings):
global_language_config = language_config
global_environment_settings = environment_settings
return GlobalConfig()
Note: As my other blogs, this idea is not limited just to python but can be used anywhere. I have used python to explain the idea. Few modifications & same approach can be applied anywhere.
