Config and Secret parser for your Python projects

Let me show you some code:

from gconfigs import envs, configs, secrets

HOME = envs('HOME', default='/')
DEBUG = configs.as_bool('DEBUG', default=False)
DATABASE_USER = configs('DATABASE_USER')
DATABASE_PASS = secrets('DATABASE_PASS')

>>> # envs, configs and secrets are iterables
>>> from gconfigs import envs
>>> for env in envs:
...     print(env)
...     print(env.key)
...     print(env.value)
...
EnvironmentVariable(key='ENV_TEST', value='env-test-1')
ENV_TEST
env-test-1
...

>>> 'ENV_TEST' in envs
True

>>> envs.json()
'{"ENV_TEST": "env-test-1", "HOME": "/root", ...}'

This is experimental, so you know, use at your own risk.

Table of Contents

• Features
• Installation
• Usage - Basics
  • Environment Variables
  • Local Mounted Configs and Secrets
    • configs
    • secrets
  • .env (dotenv) files
• Usage - Advanced
  • Get Your Config Value
    • use another key if the first one doesn't exist
    • default value
    • use_instead + default
    • typed value
    • casting (converting your strings to a specific type)
      • BOOLEAN - Converting to bool
      • LIST - Converting to list
      • DICT - Converting to dict
      • OTHER TYPES - Converting to int, float, tuple, str, set
  • Interators
  • Extra Goodies
• Beyond: from gconfigs import envs
  • Extending a Built-in Backend
  • Create Your Own Backend
• What's Next

Features

• Python 3.6
• No dependencies

Read configs from:

• Environment Variables
• Local Mounted Configs and Secrets
• .env (dotenv) files

Installation

To install gConfigs, run this command in your terminal:

$ pip install gconfigs
$ # or
$ pipenv install gconfigs

These are the preferred methods to install gConfigs.

If you don't have pip or pipenv installed, this Python installation guide can guide you through the process.

Usage - Basics

I will show you the basics with the built-in backends.

I'm still deciding about other backends. If you need a custom backend, it's easy to create. Check "Advanced" section for more.

Environment Variables

So, there are good reasons to not use environment variables for your configs, but if you want / need to use, please just not use for sensitive data, like: passwords, secret keys, private tokens and stuff like that.

>>> from gconfigs import envs
# contents from ``envs`` are just data from ``os.environ``
>>> envs
<GConfigs backend=LocalEnv>
>>> envs('HOME')
'/root'

Local Mounted Configs and Secrets

Configs and secrets can be mounted as text files, read-only and in a secure location if possible, and we can read its contents. Basically the file name will be like a var / key name and its contents will be the value.

configs

For configs, gConfigs will look for mouted files at /run/configs, for example:

File Absolute Path: /run/configs/LANGUAGE_CODE
File Name: LANGUAGE_CODE
File Contents: en-us

from gconfigs import configs
LANGUAGE_CODE = configs('LANGUAGE_CODE')
# ...translates into:
LANGUAGE_CODE = "en-us"

Of course you can change the path that gConfigs will look for your configs. Let's suppose your configs are mouted at /configs:

from gconfigs import configs
configs.root_dir = '/configs'
# will look for /configs/LANGUAGE_CODE
LANGUAGE_CODE = configs('LANGUAGE_CODE')

This is the simplest way to do it. Check section "Advanced" for more.

secrets

It follows the same flow as configs, so for more details go to configs.

For secrets, gConfigs will look for mouted files at /run/secrets.

from gconfigs import secrets
POSTGRES_PASSWORD = secrets('POSTGRES_PASSWORD')
# ...translates into:
POSTGRES_PASSWORD = "super-strong-password"
secrets.root_dir = '/secrets'
# will look for /secrets/POSTGRES_PASSWORD
POSTGRES_PASSWORD = secrets('POSTGRES_PASSWORD')

NOTE: If you don't know what tools follow these workflows for configurations and secrets, you could try with Docker. Check Docker Configs and / or Docker Secrets management with Docker.

.env (dotenv) files

.env files are present not only in Python projects, for that reason many developers are familiar with, it's just like a .ini file, but without the sections, you could say it's a key-value store in a file.

.env files could be a good solution depending on your stack. It's better than environment variables at least.

You could just put your configurations in a file called .env, (or whatever name you want), for example the contents of your file would be:

ROOT=/
PROJECT_NAME=gConfigs - Config and Secret parser
AUTH_MODULE=users.User

After that I'm going to save my .env file in /app/, then the full path will be /app/.env, now let's see how to load all it's contents in gConfigs:

from gconfigs import dotenvs
dotenvs.load_file('/app/.env')
# after that it's like using ``envs``, or ``configs``
ROOT = dotenvs('ROOT')
NAME = dotenvs('PROJECT_NAME')
AUTH = dotenvs('AUTH_MODULE')

NOTES:

• if it's a .ini syntax it will be parsed, but it will ignore sections
• duplicated keys will be overridden by the latest value
• inexistent keys will raise exception
• all values load as strings, use casting to convert them
• didn't like the name dotenvs? Just do: from gconfigs import dotenvs as configs

Usage - Advanced

With the basics, you are already running your projects just fine, but if you want the extra stuff of gConfigs, I'll show you.

I'll be using envs in the examples, but it should work for all built-in backends.

Get Your Config Value

use another key if the first one doesn't exist

Use another key in case the first doesn't exist. It's like a default value but instead of a value, you use the use_instead parameter to inform a key to be used when key is not found.

>>> from gconfigs import envs
>>> envs('NON-EXISTENT-ENV', use_instead='USE-THIS-ENV-INSTEAD')
'/'
>>> user_or_host = envs('USER', use_instead='HOSTNAME')

default value

You can provide a default value, in case the backend couldn't return the config.

>>> from gconfigs import envs
>>> envs('WHATEVER', default='/')
'/'

use_instead + default

It's simple, if both key and use_instead doesn't exist, the default value will be used.

>>> from gconfigs import envs
>>> envs('NON-EXISTENT-ENV', use_instead='NON-EXISTENT-ENV-2', default='hello')
'hello'

typed value

Generally backends will return key and value as strings, but you can return other types.

gconfigs.GConfigs.get won't try to cast your typed value.

For example when providing a default value you could set a int:

>>> from gconfigs import envs
>>> envs('WORKERS', default=1)
1

But you must know that if your backend, in that case it's just the LocalEnv backend, return a string value, you could create a bug in your configuration. Unless your software is prepared to deal with the number of WORKERS being a string and an integer, you could be in trouble.

What you want here is to cast your value, that you could achieve by simply converting what gConfigs return to the desired type or using some of the built-in casting methods.

casting (converting your strings to a specific type)

Most of the backends will return a string (str) as value. But sometimes you want to use a bool, int, list config.

NOTE: I choose to not do too much magic, so the cast methods implemented for gConfigs just loads the values with json.loads from the Python's built-in json module. Therefore, it must be a valid json value, I'll show you how:

BOOLEAN - Converting to bool

Let's say you want DEBUG as a boolean.

>>> from gconfigs import envs
>>> envs.as_bool('DEBUG')
True

I'm not doing any magic translation of "on" => True | "yes" => True. I don't want to introduce ambiguity, In my opinion, configurations must be straightforward and with limited variations.

LIST - Converting to list

Let's say you have a configuration value like this:

[1, 2.1, "string-value", true]

# if you want to try in your terminal:
export CONFIG_LIST='[1, 2.1, "string-value", true]'

The value must be just JSON-like, which is very close to a list in Python. And you will be able to get a list object by doing:

>>> from gconfigs import envs
>>> envs.as_list('CONFIG-LIST')
[1, 2.1, 'string-value', True]

DICT - Converting to dict

If you have a value that is basically a JSON valid object, you may already know you can turn into a dict using json.loads.

Here is an example, if your config value is:

{"endpoint": "/", "workers": 1, "debug": true}

# if you want to try in your terminal:
export CONFIG_DICT='{"endpoint": "/", "workers": 1, "debug": true}'

>>> from gconfigs import envs
>>> envs.as_list('CONFIG-LIST')
{'endpoint': '/', 'workers': 1, 'debug': True}

Again, nothing new, no surprises, boring, no magic... as intended.

OTHER TYPES - Converting to int, float, tuple, str, set

Well let's not reinvent the wheel, like I said before, most backends will return string by default, so if we have something like:

WORKERS="1"
WEIGHT="1.1"
MODULES='["auth", "session"]'

We could then do this:

>>> from gconfigs import envs
>>> int(envs('WORKERS'))
1
>>> float(envs('WEIGHT'))
1.1

If you want tuple or set, just get as list and then do whatever you want:

>>> from gconfigs import envs
>>> tuple(envs.as_list('MODULES'))
('auth', 'session')
>>> set(envs.as_list('MODULES'))
{'auth', 'session'}

What about strings? If you getting from your backend config values that aren't strings, and for some of them you need to convert to str, just use the Python built-in str():

>>> from gconfigs import envs
>>> envs('AN-INT-CONFIG')  # if this return an integer
1
>>> str(envs('AN-INT-CONFIG'))  # just use str
'1'

Interators

>>> from gconfigs import envs
>>> list(envs)  # envs is a iterator
[EnvironmentVariable(key='LANG', value='C.UTF-8'), ...]

>>> for env in envs:
...     print(env)
...     print(env.key)
...     print(env.value)
...
EnvironmentVariable(key='ENV_TEST', value='env-test-1')
ENV_TEST
env-test-1
...

If you use an iterator once, you can't iterate again, but if you want you can call .iterator() and get a new one:

>>> iter_envs = envs.iterator()
>>> for env in iter_envs:
...     print(env.key)
...
HOME
LANG

Extra Goodies

• How many configs with Python built-in len.
• Config key exists with Python built-in in.
• Output your key-value configs as JSON.

>>> from gconfigs import envs
>>> len(envs)
28
>>> 'HOME' in envs
True
>>> envs.json()
'{"HOME": "/root", ...}'

Beyond: from gconfigs import envs

Let's see some stuff you can do more than just import the ready for use configs and secrets.

We have GConfigs class which takes data from one of the backends gconfigs.backends and and add fancy stuff like casting and iterator behaviour.

A backend is simply a class implementing the methods:

• get(key: str): return a value given a key
• keys(): return all available keys

If you know some Python, just look the gconfigs.backends.LocalEnv and you'll see there's no secret.

Extending a Built-in Backend

Okay let's create a practical example of how to override the behaviour of one of our backends.

If you get your Configs and Secrets with gconfigs.configs and gconfigs.secrets, you are making use of gconfigs.LocalMountFile backend. That being said we could extend gconfigs.LocalMountFile and make it only get the configs if they are a mount point.

from gconfigs import GConfigs, LocalMountFile
import os

class MountPointConfigs(LocalMountFile):
    def get(self, key, **kwargs):
        file = self.root_dir / key
        if os.path.ismount(file):
            return super().get(key, **kwargs)

        raise Exception(f"The config {key} file must be a mount point.")

# :backend: can be a callable class or a instance
# :object_type_name: it's just the name of the namedtuple you get when you
# iterate over `configs`.
configs = GConfigs(
    backend=MountPointConfigs, object_type_name="MountPointConfig"
)

MY_CONFIG = configs('MY_CONFIG')

(if you use Docker Configs or Docker Secrets, you probably know that it does mount your configs / secrets in your container filesystem)

Create Your Own Backend

If you want to extend the usage of gConfigs with other backends, it's not a hard task.

Imagine my configs are stored in Redis (a key-value store), a backend for this would look like:

class RedisBackend:
    """Redis Backend for gConfigs
    NOTE: this is an example, so you probably would have to install the "redis"
    python package, then connect to Redis, then you would be able to implement
    ``get`` and ``keys`` methods.
    """
    def keys(self):
        # return a iterable of all keys available
        return available_keys

    def get(self, key: str):
        # this method receive a key (identifier of a config)
        # and return its respective value
        return value

gConfigs only expects you provide two methods:

get(key: str): return a value given a key

• connect to your backend
• based on the key get it's value
• return the value OR raise exception if it was not possible to get the config
• keep in mind that the return type it's up to you, str makes things kinda agnostic

keys(): return all available keys

• connect to your backend
• return an iterable (list, tuple, generator..) of all available keys if possible
• if you don't want or it's not possible to implement this, just raise a NotImplementedError or a more informative exception if you like

(Optional) load_file(filepath: str): parse file and just raise exception if fails

• IMPORTANT: the method name it has to be load_file, that way gConfigs will provide a load_file that just calls the backend to load the file, check gconfigs.GConfigs.__init__ for more
• read the file
• parse and get keys and values
• store the keys and values inside a dict if you want
• then implement get and keys as described above

You could also look at the module gconfigs.backends, so you can see how the built-in backends are implemented.

What's Next

• More backends, the really fun ones
• Don't know, you tell me on Issues