cliconfig.processing

cliconfig.processing.base

Base class for processing.

Used to make configuration object and run the routines in process_routines and config_routines.

class Processing

Bases: object

Processing base class.

Each processing classes contains pre-merge, post-merge, pre-save and post-load processing. They are used with routines that apply processing in process_routines and config_routines.

That are applied in the order defined by the order attribute in case of multiple processing.

premerge(flat_config)

Pre-merge processing.

Function applied to the flat config to modify it before merging. It takes a flat config and returns a flat config.

Return type:: Config

postmerge(flat_config)

Post-merge processing.

Function applied to the flat config to modify it after merging . It takes a flat config and returns a flat config.

Return type:: Config

endbuild(flat_config)

End-build processing.

Function applied to the flat config to modify it at the end of a building process (typically make_config() or load_config()). It takes a flat config and returns a flat config.

Return type:: Config

presave(flat_config)

Pre-save processing.

Function applied to the flat config to modify it before saving. It takes a flat config and returns a flat config.

Return type:: Config

postload(flat_config)

Post-load processing.

Function applied to the flat config to modify it after loading. It takes a flat config and returns a flat config.

Return type:: Config

cliconfig.processing.builtin

Built-in processing classes.

Built-in classes to apply pre-merge, post-merge, pre-save and post-load modifications to dict with processing routines process_routines.

They are the default processing used by the config routines load_config() and make_config().

class ProcessMerge

Bases: Processing

Merge dicts just in time with @merge_after/_before/_add tags.

Tag your key with @merge_after, @merge_before or @merge_add to load the config corresponding to the value (which is a yaml path) and merge it just before or after the current config. The merged dicts will be processed with pre-merge but not post-merge to ensure that merged configurations are recursively processed with pre-merge before applying a post-merge processing. It allows the merged configs to make references to each other (typically for copy) even without containing the merge tags itself.

‘@merge_add’ merges the dict corresponding to the path by allowing ONLY new keys It is a security check when you want to add a dict completely new, the typical usage for a default config splitted in several files.
‘@merge_after’ merge the dict corresponding to the path on the current dict
‘@merge_before’ merge the current dict on the dict corresponding to the path

The processing is a pre-merge processing only and occurs before almost all of the other processings. Pre-merge order: -20.0

Examples

# config1.yaml
a:
  b: 1
  b_path@merge_after: dict2.yaml

# config2.yaml
a.b: 2
c_path@merge_add: config3.yaml

# config3.yaml
c: 3

Before merging, the config1 is interpreted as the dict:

{'a': {'b': 2, 'b_path': 'config2.yaml'}, 'c_path': 'config3.yaml', 'c': 3}

If you replace ‘@merge_after’ by ‘@merge_before’, it will be:

{'a': {'b': 1, 'b_path': 'config2.yaml'}, 'c_path': 'config3.yaml', 'c': 3}

Finally, if you replace @merge_after by @merge_add, it will raises an error because the key a.b already exists in the dict.

premerge(flat_config)

Pre-merge processing.

Return type:: Config

class ProcessCopy

Bases: Processing

Copy a value with @copy tag. The copy is protected from direct updates.

Tag your key with @copy and with value the name of the flat key to copy. The pre-merge processing removes the tag. The post-merge processing sets the value (if the copied key exists). The end-build processing checks that the key to copy exists and copy them. The pre-save processing restore the tag and the key to copy to keep the information on future loads. The post-merge and the end-build processings occur after most processings to allow the user to modify or add the copied key before the copy. Pre-merge order: 0.0 Post-merge order: 10.0 End-build order: 10.0 Pre-save order: 0.0

Examples

# config.yaml
a:
  b: 1
  c@copy: a.b

Before merging, the config is interpreted as the dict

{'a': {'b': 1, 'c': 1}}

Note

The copy key is protected against any modification and will raise an error if you try to modify it but will be updated if the copied key is updated.
If the key to copy does not exist in the config on post-merge, no error is raised to let the user the possibility to add the key later via merge. However, if the key still does not exist at the end of the build (and the key was never copied), an error is raised.

premerge(flat_config)

Pre-merge processing.

Return type:: Config

postmerge(flat_config)

Post-merge processing.

Return type:: Config

endbuild(flat_config)

End-build processing.

Return type:: Config

presave(flat_config)

Pre-save processing.

Return type:: Config

class ProcessDef

Bases: Processing

Dynamically define a value from math expression with @def tag.

The expression can contain any parameter name of the configuration, booleans, numbers as well as the following operators: +, -, , /, *, %, //, &, |, comparison operators (<, <=, >, >=, ==, !=), if/else statements and parentheses. The pre-merge processing removes the tag. The post-merge processing sets the value while the presave processing restore the tag and the expression. The post-merge processing occurs after most processings to allow the user to modify the used keys before the calculation. Pre-merge order: 0.0 Post-merge order: 10.0 Pre-save order: 0.0

Examples

# config.yaml
a:
  b: 1
  c: 2
d@def: "(a.b + a.c) * 2 > 5"

Before merging, the config is interpreted as the dict

{'a': {'b': 1, 'c': 2}, 'd': True}

Now the parameter d is automatically updated if a.b or a.c changes while also remaining editable by it-self.

Note

Unlike @copy processing you can change the value by setting an other value or an other definition with @def.
Unlike copy processing all the keys used in expression must be in the config at post-merge.
This processing does not use eval and is therefore safe from malicious code.

calc_func(expr, config)

Evaluate expression with ast.

Return type:: Any

premerge(flat_config)

Pre-merge processing.

Return type:: Config

postmerge(flat_config)

Post-merge processing.

Return type:: Config

presave(flat_config)

Pre-save processing.

Return type:: Config

class ProcessTyping

Bases: Processing

Force a type with @type:<mytype> tag. The type is then forced forever.

Allow basic types (none, any, bool, int, float, str, list, dict), nested lists, nested dicts, unions (with Union or the ‘|’ symbol) and Optional. The type description is lowercased and spaces are removed.

For instance: @type:None|List[Dict[str, int|float]] is valid and force the type to be None or a list containing dicts with str keys and int or float values.

The processing stores the type in pre-merge and check alls forced types on end-build. It restore the tag in pre-save to keep the information on future loads. The end-build processing occurs after almost all processings. Pre-merge order: 0.0 End-build order: 20.0 Pre-save order: 0.0

Note

The type is not checked on pre-merge or ost-merge to allow the parameter to be updated (by a copy or a merge for instance). The goal of this processing is to ensure the type at the end of the build.

Examples

in_dict = {"param@type:None|List[int|float]": None}
dict1 = {param: [0, 1, 2.0]}  # no error
dict2 = {param: [0, 1, 2.0, 'a']}  # error

Merging configs with dictionaries in_dict and dict1 raises no error and param is forced to be None or a list of int or float forever. Merging config with in_dict and dict3 raises an error on post-merge due to the ‘a’ value (which is a string).

Note that removing “None|” in the type description of param still doesn’t raise an error in the first case because the type checking is evaluated after the merge with dict2.

premerge(flat_config)

Pre-merge processing.

Return type:: Config

endbuild(flat_config)

End-build processing.

Return type:: Config

presave(flat_config)

Pre-save processing.

Return type:: Config

class ProcessSelect

Bases: Processing

Select a sub-config with @select and delete the rest of its parent config.

First look in pre-merge for a parameter tagged with @select containing a flat key corresponding to a sub-configurations to keep. The parent configuration is then deleted on post-merge, except the selected sub-configuration and eventually the tagged parameter (if it is in the same sub-configuration). It is also possible to select multiple keys of a same sub-configuration (meaning that the part before the last dot must be equal) by passing a list of flat keys. Pre-merge order: 0.0 Post-merge order: 0.0

Examples

models:
    model_names@select: [models.model1, models.model3]
    model1:
        param1: 1
        param2: 2
    model2:
        param1: 3
        param2: 4
    model3:
        submodel:
            param: 5
    model4:
        param: 6

Result in deleting models.model2 (param1 and param2) and models.model4.param, and keeping the rest.

Warning

For security reasons, this processing prevents from deleting the configuration at the root, which is the case when the selected key doesn’t contain a dot. It raises an error in this case.

premerge(flat_config)

Pre-merge processing.

Return type:: Config

postmerge(flat_config)

Post-merge processing.

Return type:: Config

presave(flat_config)

Pre-save processing.

Return type:: Config

class ProcessDelete

Bases: Processing

Delete the parameters tagged with @delete on pre-merge.

This processing is useful to activate a processing without adding an additional parameter in the default configuration to avoid the error on merge with allow_new_keys=False. This processing is applied very late on pre-merge to allow the others processing to be applied before deleting the parameters. Pre-merge order: 30.0

Examples

# main.yaml
1@select@delete: configs.config1
2@merge_add@delete: config1.yaml
3@merge_add@delete: config2.yaml

# config1.yaml
configs.config1.param: 1
configs.config1.param2: 2

# config2.yaml
configs.config2.param: 3
configs.config2.param: 4

Here we want to merge two config files and select one sub-config. We use the corresponding tags but we don’t have a good name for the keys and instead of adding a new parameter in the default configuration with random names like “1”, “2”, “3”, we use the @delete tag to delete the keys after the pre-merge processing.

Warning

The parameter is deleted on pre-merge. Therefore, if the parameter also exists on the other configuration during merge (without the tag), this parameter will be remain as it is. This processing is more used to delete parameter that is NOT present in the default configuration.

premerge(flat_config)

Pre-merge processing.

Return type:: Config

class ProcessNew

Bases: Processing

Allow new sub-config/parameter absent from default config with tag @new.

The tagged sub-config/parameter and its value is stored during pre-merge and is deleted to avoid error on merge due to new key. It is then restored on post-merge. Finally, the tag is restored on pre-save for further loading. This processing is applied very late on pre-merge to allow the others processing to be applied before deleting the parameters. The post-merge processing is applied very early to allow the other processing to use this new parameter. Pre-merge order: 30.0 Post-merge order: -20.0 Pre-save order: 0.0

Disclaimer: It is preferable to have an exhaustive default configuration instead of abusing this processing to improve the readability and to avoid typos errors in the name of the parameters or their sub-configs.

Examples

# default.yaml
param1: 1

# additional.yaml
param2@new: 2
subconfig@new.subsubconfig:
    param3: 3
    param4: 4

Use default.yaml as default configuration and add additional.yaml as additional configuration via CLI results on the configuration containing param1, param2 and the nested config containing param3 and param4. Without the @new tag, an error is raised because param2 is not present in the default configuration.

Note

Tag a subconfig by adding @new at the end of the key containing the sub-config dict in your yaml file.
When a parameter is added with this processing, it is possible to modify it later via config merge without the tag because the parameter is then present in the current configuration.
If the tagged parameter or sub-config is already present in the current configuration, no error are raised and the value is still updated on post-merge. It may no have influence in practice.

premerge(flat_config)

Pre-merge processing.

Return type:: Config

postmerge(flat_config)

Post-merge processing.

Return type:: Config

presave(flat_config)

Pre-save processing.

Return type:: Config

class ProcessCheckTags

Bases: Processing

Raise an error if a tag is present in a key after pre-merging processes.

This security processing is always applied after all pre-merge process and checks for ‘@’ in the keys. It raises an error if one is found.

premerge(flat_config)

Pre-merge processing.

Return type:: Config

class DefaultProcessings

Bases: object

Default list of built-in processings.

To add these processings to a Config instance, use:

config.process_list += DefaultProcessings().list

The current default processing list contains:

ProcessCheckTags: protect against ‘@’ in keys at the end of pre-merge)
ProcessMerge (@merge_all, @merge_before, @merge_after): merge multiple files into one.
ProcessCopy (@copy): persistently copy a value from one key to an other and protect it
ProcessTyping (@type:X): force the type of parameter to any type X.
ProcessSelect (@select): select sub-config(s) to keep and delete the other sub-configs in the same parent config.
ProcessDelete (@delete): delete the parameter tagged with @delete on pre-merge.

cliconfig.processing.create

Functions to create new processing quickly.

create_processing_value(func, processing_type='premerge', *, regex=None, tag_name=None, order=0.0, persistent=False)

Create a processing object that modifies a value in config using tag or regex.

The processing is applied on pre-merge. It triggers when the key matches the tag or the regex. The function apply flat_dict[key] = func(flat_dict[key]). You must only provide one of tag or regex. If tag is provided, the tag will be removed from the key during pre-merge.

It also possible to pass the flat config as a second argument of the function func. In this case, the function apply flat_dict[key] = func(flat_dict[key], flat_config).

Parameters:

func (Callable) – The function to apply to the value (and eventually the flat config) to make the new value so that: flat_dict[key] = func(flat_dict[key]) or func(flat_dict[key], flat_config)
processing_type (str, optional) – One of “premerge”, “postmerge”, “presave”, “postload” or “endbuild”. Timing to apply the value update. In all cases the tag is removed on pre-merge. By default “premerge”.
regex (Optional[str]) – The regex to match the key.
tag_name (Optional[str]) – The tag (without “@”) to match the key. The tag is removed on pre-merge.
order (int, optional) – The pre-merge order. By default 0.0.
persistent (bool, optional) – If True, the processing will be applied on all keys that have already matched the tag before. By nature, using regex make the processing always persistent. By default, False.

Raises:

ValueError – If both tag and regex are provided or if none of them are provided.
ValueError – If the processing type is not one of “premerge”, “postmerge”, “presave”, “postload” or “endbuild”.

Returns:

processing – The processing object with the pre-merge method.

Return type:

Processing

Examples

With the following config and 2 processings:

proc2 = create_processing_value(lambda val: -val, regex="neg_number.*",
    order=0.0)
proc1 = create_processing_value(lambda val: val + 1, tag_name="add1",
    order=1.0)

When config.yaml is merged with an other config, it will be considered before merging as:

{'number1': -1, 'number2': -1, 'number3': 0}

Using the config as a second argument of the function:

proc = create_processing_value(
    lambda val, config: eval(val, {'config': config}),
    processing_type='postmerge', tag_name='eval', persistent=False
)

After config.yaml is merged with another config, param2 will be evaluated as 2 (except if config.param1 has changed with a processing before).

create_processing_keep_property(func, regex=None, tag_name=None, premerge_order=0.0, postmerge_order=0.0, endbuild_order=0.0)

Create a processing object that keep a property from a value using tag or regex.

The pre-merge processing looks for keys that match the tag or the regex, apply the function func on the value and store the result (= the “property”): property = func(flat_dict[key]). The post-merge processing will check that the property is the same as the one stored during pre-merge. If not, it will raise a ValueError.

It also possible to pass the flat config as a second argument of the function func. In this case, the function apply property = func(flat_dict[key], flat_config).

Parameters:

func (Callable) – The function to apply to the value (and eventually the flat config) to define the property to keep. property = func(flat_dict[key]) or func(flat_dict[key], flat_config)
regex (Optional[str]) – The regex to match the key.
tag_name (Optional[str]) – The tag (without “@”) to match the key. The values are modified when triggering the pattern “.*@<tag_name>.*” and the tag is removed from the key.
premerge_order (float, optional) – The pre-merge order, by default 0.0
postmerge_order (float, optional) – The post-merge order, by default 0.0
endbuild_order (float, optional) – The end-build order, by default 0.0

Raises:

ValueError – If both tag and regex are provided or if none of them are provided.

Returns:

The processing object with the pre-merge and post-merge methods.

Return type:

Processing

Examples

A processing that enforce the types of all the parameters to be constant (equal to the type of the first value encountered):

create_processing_keep_property(type, regex=".*", premerge_order=15.0,
                                postmerge_order=15.0, endbuild_order=15.0)

A processing that protect parameters tagged with @protect from being changed:

create_processing_keep_property(lambda x: x, tag_name="protect",
                                premerge_order=15.0, postmerge_order=15.0)