CLI Config

Build your experiment configurations for complex projects with robustness, flexibility and simplicity

CLI Config: Lightweight library that provides routines to merge nested configs and set parameters from command line. It is also provide processing functions that can change the whole configuration before and after each config merge, config saving, config loading and at the end of config building. It also contains many routines to manipulate the config as flatten or nested dictionaries.

The package is initially designed for machine learning experiments where the number of parameters is huge and a lot of them have to be set by the user between each experiment. If your project matches this description, this package is for you!

Documentation

Github project

Official badge:

Make default config yaml file(s) in your project (configs are merged from left to right):

# main.py
from cliconfig import make_config

config = make_config('default1.yaml', 'default2.yaml')

Then launch your script with additional config(s) file(s) and parameters by command line. The additional configs are merged on the default one’s then the parameters are set.

python main.py --config first.yaml,second.yaml --param1=1 --subconfig.param2='foo'

By default, these additional configs cannot add new parameters to the default config (for security and retro-compatibility reasons).

Now you can get your configuration parameters in your script:

# Nested config dict as a native python dict
config.dict
# Get a parameter value (you can also set it or delete it)
config.foo1.foo2.bar

You can also load and save configs with cliconfig.save_config and cliconfig.load_config.

With processing

The library provides powerful tools to modify the configuration called “processings”. One of the possibility they add is to merge multiple configurations, copy a parameter on another, enforce type and more. To do so, simply adding the corresponding tags to your parameter names (on config files or CLI parameters).

For instance with these config files:

# main.yaml
path_1@merge_add: sub1.yaml
path_2@merge_add: sub2.yaml
config3.selection@delete@select: config3.param1

# sub1.yaml
config1:
  param@copy@type:int: config2.param
  param2@type:float: 1  # wrong type -> converted to float

# sub2.yaml
config2.param: 2
config3:
  param1@def: "[(config1.param2 + config2.param) / 2] * 2 if config2.param else None"
  param2: 3
my_dict@dict:
  key1: 1
  key2: 2

Note that can also use YAML tags separated with “@” (like key: !tag@tag2 value) to add tags instead of putting them in the parameter name (like key@tag@tag2: value).

Here main.yaml will be interpreted like:

path_1: sub1.yaml
path_2: sub2.yaml
config1:
  param: 2  # the value of config2.param
  param2: 1.0  # converted to float
config2:
  param: 2
config3:
  param1: [1.5, 1.5]
  # param2 is deleted because it is not in the selection
my_dict: {key1: 1, key2: 2}  # (changing the whole dict further is allowed)

Then, all the parameters in config1 have enforced types, changing config2.param will also update config1.param accordingly (which is protected by direct update). Finally, changing config1.param2 or config2.param will update config3.param1 accordingly until a new value is set for config3.param1.

The default tags include:

@merge_add, @merge_before, and @merge_after: merge other yaml files indicating the path
@copy: copy an other parameter
@def: safely evaluate an expression to define the parameter value
@type:<my type>: try to convert and check the type
@select: target a sub-config and delete the sub-config in the parent config
@delete: delete the key from the config
@new: allow to add parameters that are not present in default config
@dict: create a dict object that is not a sub-config (see the Edge cases section)

See Quickstart section of the documentation for more details and Processing section for advanced usage.