Concepts¶

The pycnfg.Handler contains class to read and execute configuration.

The purpose of any configuration is to produced result (object) by combining resources (steps). Pycnfg offers unified patten to create arbitrary Python objects pipeline-wise. That naturally allows to control all parameters via single configuration.

Configuration is a python dictionary. It supports multiple sections. Each section specify set of sub-configurations. Each sub-configuration provide steps to construct an object, that can be utilize as argument in other sections. Whole configuration could be passed to pycnfg.run or to user-defined wrapper around pycnfg.Handler, that builds target sub-configurations objects one by one.

For each section there is common logic:

{'section_id':
    'configuration_id 1': {
        'init': Initial object state.
        'producer': Factory class, contained methods to run steps.
        'patch': Add custom methods to class.
        'steps': [
            ('method_id 1', {'kwarg_id': value, ..}),
            ('method_id 2', {'kwarg_id': value, ..}),
        ],
        'global': Shortcut to common parameters.
        'priority': Execution priority (integer).
    }
    'configuration_id 2':{
        ...
    }
}

The target for each sub-configuration is to create an object. init is the template for future object (for example empty dict). producer works as factory, it should contain run() method that: takes init and consecutive pass it and kwargs to steps methods and returns resulting object.

# pseudocode
def run(self, init, steps):
    obj = init
    for step in steps:
        obj = decorators(getattr(self, step_id))(obj, objects, **kwargs)
    return obj

That object will be store in objects and can be used as kwargs for any step in others sections.

objects['section_id__conf_id'] = obj

To specify the order in which sections handled, the priority key is available in each sub-configuration.

For flexibility, it is possible:

Specify default configuration for section(s).
Specify global value for common kwargs in steps via global key.
Create separate section for arbitrary parameter in steps.
Monkey-patch producer object with custom functions via patch key.
Set init as an instance, a class or a function
Set decorators for any step.

The whole configuration could be considered as sub-configuration in some higher level section. That potentially allows arbitrary levels of nesting.

Sub-configuration keys¶

initcallable or instance, optional (default={})

Initial state for constructing object. Will be passed consecutive in steps as argument. If set as callable ``init``(), auto called.

producerclass, optional (default=pycnfg.Producer)

The factory to construct an object: producer.run(init, steps). Class auto initialized: producer(objects, 'section_id__configuration_id', **kwargs), where objects is a dictionary with previously created objects {section_id__configuration_id: object}. If (‘__init__’, kwargs) step provided in steps, kwargs will be passed to initializer.

patchdict {‘method_id’function}, optional (default={})

Monkey-patching the producer object with custom functions.

stepslist of tuples, optional (default=[])

List of producer methods with kwargs to run consecutive. Each step should be a tuple: ('method_id', kwargs, decorators), where ‘method_id’ should match to producer functions’ names. In case of omitting any kwargs step executed with default, set in corresponding producer method.

kwargsdict, optional (default={})

Step arguments: {‘kwarg’: value, …}.

It is possible to create separate section for any argument. Set section_id__configuration_id as kwarg value, then it would be auto-filled with corresponding object from objects storage before step execution. List of section_id__configuration_id is also possible. To prevent auto substitution, set special ‘_id’ postfix kwarg_id.

It`is possible to turn on resolving None values for explicitly set kwargs in steps. See resolve_none argument in pycnfg. Handler.read() : If value is set to None, parser try to resolve it. First searches for value in sub-cconfiguration global. Then resolver looks up ‘kwarg’ in section names. If such section exist, there are two possibilities: if ‘kwarg’ name contains ‘_id’ postfix, resolver substitutes None with available section_id__configuration_id, otherwise with configuration object. If fails to find resolution, value is remained None. In case of resolution plurality, ValueError is raised.

decorators: list, optional (default=[])

Step decorators from most inner to outer: [decorator,]

prioritynon-negative integer, optional (default=1)

Priority of configuration execution. The more the higher priority. For two conf with same priority order is not guaranteed. If zero, not execute configuration.

globaldict , optional (default={})

Specify values to substitute for any kwargs: {'kwarg_name': value, }. This is convenient for example when the same kwarg used in all methods. It is possible to specify more targeted path with step_id in it: {'step_id__kwarg_name': value, }. If global contains multiple resolutions for some kwarg, targeted path has priority. By default global don`t replace explicitly set values in steps. See update_expl flag in pycnfg.Handler.read() to change this behavior. See below glabal usage in other levels of configuration dictionary.

**keysdict {‘kwarg_name’: value, …}, optional (default={})

All additional keys in configuration are moving to global automatically (rewrites already existed keys from the sub-conf level).

Notes

Any configuration key above could be set in the most outer configuration level or section level to specify common values either for all sections or all sub-sections in some section (Examples below). The more inner level has the higher priority, levels priority in ascending: cnfg level (0) => section level (1) => sub-conf level (2). Supported keys: global, init, producer, patch, steps, priority.

global keys from levels are merged according to level priorities.

Targeted path also possible, in priority:

section_id__conf_id__step_id__kwarg_name on level 0.

section_id__step_id__kwarg_name on level 0.

section_id__conf_id__kwarg_name on level 0.

section_id__kwarg_name on level 0.

conf_id__step_id__kwarg_name on levels 0,1.

conf_id__kwarg_name on levels 0,1.

step_id__kwarg_name on levels 0,1,2.

kwarg_name on levels 0,1,2.

On the same level “more” targeted path priority > non-targeted.

other keys` value are replaced in level priority.

section_id/configuration_id/step_id/kwargs_id should not contain double underscore ‘__’ (except magic methods in step_id).

Default configurations can be set in pycnfg.Handler.read() dcnfg argument. Arbitrary objects could be pre-accommodated objects argument in pycnfg.Handler.exec() , so no need to specify configurations for them. If any provided it has priority over sub-configuration with the same id.

To add functionality to producer use patch key or inheritance from pycnfg.Producer .

Examples

Patching producer with custom function.

def my_func(self, key, val):
    # ... custom logic ...
    return res

CNFG = {..{'patch': {'func': my_func,},}}

global on the most outer level to set kwargs in ‘func’ step and patch to add functionality. Section level init to set in both configuration simultaneously:

CNFG = {

        'global': {
            'section_1__set__val': 42,
            'val': 99,
            },
        'patch': {'func': my_func},
        'section_1': {
            'init': {'a': 7},
            'conf_1': {
                'steps': [
                    ('func', {'key': 'b', 'val': 24},),
                ],
            },
            'conf_2': {
                'steps': [
                    ('func', {'key': 'b', 'val': 24},),
                ],
            },
        },
}
# Result (update_expl=True):
{
    'section_1__conf_1': {'a': 7, 'b': 42},
    'section_1__conf_2': {'a': 7, 'b': 99},
}

There are two ways to set second level kwargs for some step, for example kwarg ‘c’ for step ‘func’:

def func(a, b=7, **kwargs):
    print(kwargs['c'])

# **kwargs could be set:
# Implicitly via 'global'.
CNFG = {..{
    'global': {'kwargs':{'c': 3}}
    'steps': [('func', {'a': 1)],
    }}
# Explicitly in 'steps' (only if no 'c' in the first level).
CNFG = {..{
    'steps': [('func', {'a': 1, 'c': 3)],
    }}
# In the last case, 'global' can address 'c' directly:
# 'global': {'c': 4}

See more detailed examples in Examples .