CLI Config API

cliconfig.processing

cliconfig.base module

Base classes of Config object.

class Config(config_dict, process_list=None)

Bases: object

Class for configuration.

Config object contain the config dict and the processing list and no methods except __init__, __repr__, __eq__, __getattribute__, __setattr__ and __delattr__. The Config objects are mutable and not hashable.

Parameters:

config_dict (Dict[str, Any]) – The config dict.
process_list (Optional[List[Processing]], optional) – The list of Processing objects. If None, an empty list is used. The default is None.

cliconfig.config_routines

Functions to manipulate config as dict with yaml files and CLI.

make_config(*default_config_paths, process_list=None, add_default_processing=True, fallback='', no_cli=False)

Make a config from default config(s) and CLI argument(s) with processing.

The function uses the CLI Config routines parse_cli() to parse the CLI arguments and merge them with merge_flat_paths_processing(), applying the pre-merge and post-merge processing functions on each merge.

Parameters:

default_config_paths (Tuple[str]) – Paths to default configs. They are merged in order and new keys are allowed.
process_list (Optional[List[Processing]], optional) – The list of processing to apply during each merge. None for empty list. By default None.
add_default_processing (bool, optional) – If add_default_processing is True, the default processings (found on DefaultProcessings) are added to the list of processings. By default True.
fallback (str, optional) – Path of the configuration to use if no additional config is provided with --config. No fallback config if empty string (default), in that case, the config is the default configs plus the CLI arguments.
no_cli (bool, optional) – If True, the CLI arguments are not parsed and the config is only built from the default_config_paths in input and the fallback argument is ignored. By default False.

Raises:

ValueError – If additional configs have new keys that are not in default configs.

Returns:

config – The nested built config. Contains the config dict (config.dict) and the processing list (config.process_list) which can be used to apply further processing routines.

Return type:

Config

Note

Setting additional arguments from CLI that are not in default configs does NOT raise an error but only a warning. This ensures the compatibility with other CLI usage (e.g notebook, argparse, etc.)

Examples

# main.py
config = make_config('data.yaml', 'model.yaml', 'train.yaml')

$ python main.py -- config [bestmodel.yaml,mydata.yaml] \
      --architecture.layers.hidden_dim=64

load_config(path, default_config_paths=None, process_list=None, *, add_default_processing=True)

Load config from a file and merge into optional default configs.

First merge the default configs together (if any), then load the config from path, apply the post-load processing, and finally merge the loaded config.

Parameters:

path (str) – The path to the file to load the configuration.
default_config_paths (Optional[List[str]], optional) – Paths to default configs. They are merged in order, new keys are allowed. Then, the loaded config is merged into the result. None for no default configs. By default None.
process_list (Optional[List[Processing]]) – The list of processing to apply after loading and for the merges. If None, no processing is applied. By default None.
add_default_processing (bool, optional) – If add_default_processing is True, the default processings (found on DefaultProcessings) are added to the list of processings. By default True.

Returns:

config – The nested loaded config. Contains the config dict (config.dict) and the processing list (config.process_list) which can be used to apply further processing routines.

Return type:

Dict[str, Any]

Note

If default configs are provided, the function does not allow new keys for the loaded config. This is for helping the user to see how to adapt the config file if the default configs have changed.

save_config(config, path)

Save a config and apply pre-save processing before saving.

Alias for save_processing().

Parameters:

config (Dict[str, Any]) – The config to save.
path (str) – The path to the yaml file to save the dict.

Return type:

None

show_config(config)

Show the config dict in a pretty way.

The config dict is automatically unflattened before printing.

Parameters:: config (Config) – The config to show.
Return type:: None

flatten_config(config)

Flatten a config.

Parameters:: config (Config) – The config to flatten.
Returns:: confg – The config containing a flattened dict.
Return type:: Config

unflatten_config(config)

Unflatten a config.

Parameters:: config (Config) – The config to unflatten.
Returns:: config – The config containing an unflattened dict.
Return type:: Config

update_config(config, other, *, allow_new_keys=False)

Update a config with a new dict or config with processing triggering.

The pre-merge, post-merge and end-build processings will be triggered. The resulting config is unflattened.

Parameters:

config (Config) – The config to update.
other (Config | dict) – The config or dict to update the config with.
allow_new_keys (bool, optional) – If True, allow new keys in the other config. By default False.

Returns:

config – The updated config.

Return type:

Config

cliconfig.cli_parser

Parser for CLI commands.

parse_cli(sys_argv)

Parser for CLI commands.

Return list of config path(s) that are detected with --config followed by a space. If multiple paths are provided, they must be separated by a comma and no space around the comma. It also possible to provide a list of paths.

Return also a dictionary of parameters from CLI detected with --<key>=<value> (with “=” and without spaces around). If no value is provided, it is True by default (like for a flag).

Parameters:

sys_argv (List[str]) – List of arguments from sys.argv.

Raises:

Value Error – If the --config argument (with space) is used more than once.

Return type:

Tuple[List[str], Dict[str, Any]]

Returns:

config_paths (List[str]) – List of paths to config files to merge.
cli_params_dict (Dict[str, Any]) – Dictionary of parameters from CLI.

Examples

$ python my_script.py --config config.yaml --foo.bar.param=[1, 2, 3]

Will be parsed as config_paths=['config.yaml'] and cli_params={'foo.bar.param': [1, 2, 3]}. It is equivalent to: {'foo': {'bar': {'param': [1, 2, 3]}} for merge_flat() and make_config().

cliconfig.tag_routines

Routines to manipulate the tags on the keys of a dict.

Used by the processing functions.

clean_tag(flat_key, tag_name)

Clean a tag from a flat key.

It removes all occurrences of the tag with the exact name.

Parameters:

flat_key (str) – The flat key to clean.
tag_name (str) – The name of the tag to remove, with or without the ‘@’ prefix.

Returns:

flat_key – The cleaned flat key.

Return type:

str

Note

tag_name is supposed to be the exact name of the tag.

Examples

>>> clean_tag('abc@tag.def@tag_2.ghi@tag', 'tag')
abc.def@tag_2.ghi

clean_all_tags(flat_key)

Clean all tags from a flat key.

Parameters:: flat_key (str) – The flat key to clean.
Returns:: flat_key – The cleaned flat key.
Return type:: str

dict_clean_tags(flat_dict)

Clean a dict from all tags and return the list of keys with tags.

Parameters:

flat_dict (Dict[str, Any]) – The flat dict to clean.

Return type:

Tuple[Dict[str, Any], List[str]]

Returns:

clean_dict (Dict[str, Any]) – The cleaned flat dict without tags in the keys.
tagged_keys (List[str]) – The list of keys with tags that have been cleaned.

is_tag_in(flat_key, tag_name, *, full_key=False)

Check if a tag is in a flat key.

The tag name must be the exact name, with or without the “@”. It supports the case where there are other tags that are prefixes or suffixes of the considered tag.

Parameters:

flat_key (str) – The flat key to check.
tag_name (str) – The name of the tag to check, with or without the ‘@’ prefix.
full_key (bool, optional) – If True, check for the full key. If False, check for the last part of the flat key (after the last dot) that correspond to the parameter name. By default, False.

Returns:

True if the tag is in the flat key, False otherwise.

Return type:

bool

cliconfig.dict_routines

Routines to manipulate nested and flat dictionaries (and mix of both).

Used by process_routines and config_routines.

merge_flat(dict1, dict2, *, allow_new_keys=True)

Flatten then merge dict2 into dict1. The result is flat.

Work even if dict1 and dict2 have a mix of nested and flat dictionaries. For instance like this:

dict1 = {'a.b': 1, 'a': {'c': 2}, 'a.d': {'e.f': 3}}

Parameters:

dict1 (Dict[str, Any]) – The first dict. It can be nested, flat or a mix of both.
dict2 (Dict[str, Any]) – The second dict to merge into first dict.
allow_new_keys (bool, optional) – If True, new keys (that are not in dict1) are allowed in dict2. By default True.

Raises:

ValueError – If allow_new_keys is False and dict2 has new keys that are not in dict1.
ValueError – If there are conflicting keys when flatten one of the dicts. See last example. You may consider calling clean_pre_flat() on the input dicts in that case.

Returns:

flat_dict – The flat dict (all keys are at the root and separated by dots).

Return type:

Dict[str, Any]

Examples

>>> merge_dict({'a.b': 1, 'a': {'c': 2}},  {'c': 3}, allow_new_keys=True)
{'a.b': 1, 'a.c': 2, 'c': 3}
>>> merge_dict({'a.b': 1, 'a': {'c': 2}},  {'c': 3}, allow_new_keys=False)
ValueError: New parameter found 'c' that is not in the original dict.
>>> merge_dict({'a.b': 1, 'a': {'b': 1}},  {'c': 3}, allow_new_keys=True)
ValueError: duplicated key 'a.b'.
The above exception was the direct cause of the following exception:
ValueError: You may consider calling 'clean_pre_flat' on dict 1 before merging.

merge_flat_paths(dict_or_path1, dict_or_path2, *, allow_new_keys=True)

Flatten then merge two dict eventually loaded from yaml file paths.

Similar to merge_flat() but allow passing the paths of dicts as inputs. It merges the second dict into the first one.

Parameters:

dict_or_path1 (Union[str, Dict[str, Any]]) – The first dict or its path.
dict_or_path2 (Union[str, Dict[str, Any]]) – The second dict or its path, to merge into first dict.
allow_new_keys (bool, optional) – If True, new keys (that are not in dict1) are allowed in dict2. By default True.

Raises:

ValueError – If allow_new_keys is False and dict2 has new keys that are not in dict1.
ValueError – If there are conflicting keys when flatten one of the dicts. See last example. You may consider calling clean_pre_flat() on the input dicts in that case.

Returns:

flat_dict – The flat dict (all keys are at the root and separated by dots).

Return type:

Dict[str, Any]

flatten(in_dict)

Flatten dict then return it (flat keys are built with dots).

Work even if in_dict is a mix of nested and flat dictionaries. For instance like this:

>>> flatten({'a.b': {'c': 1}, 'a': {'b.d': 2}, 'a.e': {'f.g': 3}})
{'a.b.c': 1, 'a.b.d': 2, 'a.e.f.g': 3}

Parameters:: in_dict (Dict[str, Any]) – The dict to flatten. It can be nested, already flat or a mix of both.
Raises:: ValueError – If dict has some conflicting keys (like {'a.b': <x>, 'a': {'b': <y>}}).
Returns:: flat_dict – The flattened dict.
Return type:: Dict[str, Any]

Note

Nested empty dict are ignored even if they are conflicting (see last example).

Examples

>>> flatten({'a.b': 1, 'a': {'c': 2}, 'd': 3})
{'a.b': 1, 'a.c': 2, 'd': 3}
>>> flatten({'a.b': {'c': 1}, 'a': {'b.d': 2}, 'a.e': {'f.g': 3}})
{'a.b.c': 1, 'a.b.d': 2, 'a.e.f.g': 3}
>>> flatten({'a.b': 1, 'a': {'b': 1}})
ValueError: duplicated key 'a.b'
>>> flatten({'a.b': 1, 'a': {'c': {}}, 'a.c': 3})
{'a.b': 1, 'a.c': 3}

unflatten(flat_dict)

Unflatten a flat dict then return it.

Parameters:: flat_dict (Dict[str, Any]) – The dict to unflatten. Must be a fully flat dict (depth of 1 with keys separated by dots).
Raises:: ValueError – If flat_dict is not flat and then found conflicts.
Returns:: unflat_dict – The output nested dict.
Return type:: Dict[str, Any]

Examples

>>> unflatten({'a.b': 1, 'a.c': 2, 'c': 3})
{'a': {'b': 1, 'c': 2}, 'c': 3}
>>> unflatten({'a.b': 1, 'a': {'c': 2}})
ValueError: duplicated key 'a'
The dict must be flatten before calling unflatten function.

clean_pre_flat(in_dict, priority)

Remove keys in input dict that may cause conflicts when flattening.

Parameters:

in_dict (Dict[str, Any]) – The dict to clean. It must be the union of a fully flat dict (no nested dict i values) and a fully nested dict (without dots in keys). See warning section below.
priority (str) – One of ‘flat’ or ‘unflat’, ‘error’. If ‘flat’, keys with dots at the root like {'a.b': ...} (flat keys) have priority over nested keys like {'a': {'b': ...}} when there are conflicts. If ‘unflat’, nested keys have priority over flat keys when there are conflicts. If ‘error’, raise an error when there are conflicts.

Raises:

ValueError – If priority is not one of ‘flat’, ‘unflat’ or ‘error’.

Returns:

The cleansed dict.

Return type:

Dict[str, Any]

Warning

No flat key can contain a dict. Then, dicts like {'a.b': {'c': 1}} are not supported.
All the keys that contain dots (the flat keys) must be at the root. Then, dicts like {a: {'b.c': 1}} are not supported.
To summarize, the dict must contain only fully flat dicts and/or fully nested dicts.

Examples

>>> clean_pre_flat({'a.b': 1, 'a': {'b': 2}, 'c': 3}, priority='flat')
{'a.b': 1, 'c': 3}
>>> clean_pre_flat({'a.b': 1, 'a': {'b': 2}, 'c': 3}, priority='unflat')
{'a': {'b': 2}, 'c': 3}
>>> clean_pre_flat({'a.b': 1, 'a': {'b': 2}, 'c': 3}, priority='error')
ValueError: duplicated key 'a.b'

save_dict(in_dict, path)

Save a dict to a yaml file (with yaml.dump).

Parameters:

in_dict (Dict[str, Any]) – The dict to save.
path (str) – The path to the yaml file to save the dict.

Return type:

None

load_dict(path)

Load dict from a yaml file path.

Support multiple files in the same document and yaml tags.

Parameters:: path (str) – The path to the file to load the dict.
Returns:: out_dict – The nested (unflatten) loaded dict.
Return type:: Dict[str, Any]

Note

If multiple yaml files are in the same document, they are merged

from the first to the last. * To use multiple yaml tags, separate them with “@”. E.g. !tag1@tag2. * You can combine any number of yaml and cliconfig tags together.

show_dict(in_dict, start_indent=0)

Show the input dict in a pretty way.

The config dict is automatically unflattened before printing.

Parameters:

in_dict (Dict[str, Any]) – The dict to show.
start_indent (int, optional) – The number of starting tab indent (4 spaces), by default 0.

Return type:

None

cliconfig.process_routines

Routines to manipulate dictionaries with processing.

Used by config_routines.

merge_flat_processing(config1, config2, *, allow_new_keys=True, preprocess_first=True, preprocess_second=True, postprocess=True)

Flatten, merge config2 into config1 and apply pre and post processing.

Work even if the config dicts have a mix of nested and flat dictionaries. If both arguments are configs, the process lists are merged before applying the processing. The duplicate processings (with same internal variables) are removed.

Parameters:

config1 (Config) – The first config.
config2 (Config) – The second dict to merge into config1.
allow_new_keys (bool, optional) – If True, new keys (that are not in config1) are allowed in config2. Otherwise, it raises an error. By default True.
preprocess_first (bool, optional) – If True, apply pre-merge processing to config1. By default True.
preprocess_second (bool, optional) – If True, apply pre-merge processing to config2. By default True.
postprocess (bool, optional) – If True, apply post-merge processing to the merged config. By default True.

Raises:

ValueError – If allow_new_keys is False and config2 has new keys that are not in config1.
ValueError – If there are conflicting keys when flatten one of the dicts.

Returns:

flat_config – The merged flat config.

Return type:

Config

merge_flat_paths_processing(config_or_path1, config_or_path2, *, additional_process=None, allow_new_keys=True, preprocess_first=True, preprocess_second=True, postprocess=True)

Flatten, merge and apply processing to two configs or their yaml paths.

Similar to merge_flat_processing() but allows to pass configs or their yaml paths. Work even if the configs have a mix of nested and flat dicts. If both arguments are configs, the process lists are merged before applying the processing. The duplicate processings (with same internal variables) are removed.

Parameters:

config_or_path1 (Union[str, Config]) – The first config or its path.
config_or_path2 (Union[str, Config]) – The second config or its path, to merge into first config.
additional_process (Optional[List[Processing]], optional) – Additional processings to apply to the merged config. It can be useful to merge a config from its path while it has some specific processings.
allow_new_keys (bool, optional) – If True, new keys (that are not in config1) are allowed in config2. Otherwise, it raises an error. By default True.
preprocess_first (bool, optional) – If True, apply pre-merge processing to config1. By default True.
preprocess_second (bool, optional) – If True, apply pre-merge processing to config2. By default True.
postprocess (bool, optional) – If True, apply post-merge processing to the merged config. By default True.

Raises:

ValueError – If allow_new_keys is False and config2 has new keys that are not in config1.
ValueError – If there are conflicting keys when flatten one of the dicts.

Returns:

flat_config – The merged flat config.

Return type:

Config

save_processing(config, path)

Save a config and apply pre-save processing before saving.

Parameters:

config (Config) – The config to save.
path (str) – The path to the yaml file to save the config dict.

Return type:

None

load_processing(path, process_list)

Load a dict from yaml file path and apply post-load processing.

Parameters:

path (str) – The path to the file to load the dict.
process_list (List[Processing]) – The list of processing to apply after loading. Only post-load processing is applied. The order of the processing is given by the postload_order attribute of the processing.

Returns:

flat_config – The loaded flat config.

Return type:

Config

end_build_processing(flat_config)

Apply end-build processings to a flat config.

Parameters:: flat_config (Config) – The flat config to apply the end-build processings.
Returns:: flat_config – The flat config after applying the end-build processings.
Return type:: Config