[MAGNOLIA-8126] Interpolation of magnolia properties and environment variables into YAML files Created: 01/Jul/21  Updated: 31/Jan/22  Resolved: 30/Aug/21

Status: Closed
Project: Magnolia
Component/s: configuration
Affects Version/s: 6.2
Fix Version/s: None

Type: Improvement Priority: Neutral
Reporter: Richard Gange Assignee: Unassigned
Resolution: Won't Do Votes: 1
Labels: yaml
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
relation
is related to MAGNOLIA-7883 Enable different configuration for di... Closed
Template:
Acceptance criteria:
Empty
Task DoD:
[ ]* Doc/release notes changes? Comment present?
[ ]* Downstream builds green?
[ ]* Solution information and context easily available?
[ ]* Tests
[ ]* FixVersion filled and not yet released
[ ]  Architecture Decision Record (ADR)
Date of First Response:

 Description   

Add the possibility to substitute or interpolate environment/magnolia properties. Specify placeholders in yaml definitions/decorations which will be parsed before loading them, so developers can refer to variables specified in e.g. magnolia.properties or environment variables. Configuration per environment/profile is very common and used in frameworks like spring.

Use case:
A rest-client definition has the same yaml definition, but only the baseUrl or a timeout setting is different.

Example configuration:

database:
  host: ${DB_HOST:-localhost}
  user: ${DB_USER:-defaultuser}
  password: ${DB_PASSWORD}

The DB_HOST will be replaced using the MagnoliaConfiguration class which will lookup the value from the layers of property sources (magnolia.properties, environment variables, etc..)

Workaround
Different configurations possible for different environments



 Comments   
Comment by Jordie Diepeveen [ 05/Jul/21 ]

Global replacement of anything ${VAR} can break some existing functionality. Rest clients definitions can have the same format (e.g. in jsonPaths) and thus values can be replaced if the key also exists in e.g. magnolia.properties.

 

(Some small note: The YamlReader is an injectable singleton, but in the constructor of YamlDefinitionDecorator, it's not injected but initialised, so hooking into the implementation of the YamlReader is not really easy.)

 

Sample of a custom yaml reader with support for replacing (multiple) variables for magnolia environment variables:

 

/**
 * Construct scalar for format ${VARIABLE} replacing the template with the value from MagnoliaConfigurationProperties.
 *
 * @see <a href="https://www.elastic.co/guide/en/beats/winlogbeat/current/using-environ-vars.html">Variable substitution</a>
 *
 * Support:
 * ${NAME}, ${NAME:default}, ${NAME?} or ${NAME:?You need to set the NAME environment variable}
 * @see <a href="https://regex101.com/r/PD39Rq/3">Examples on Regex101</a>
 */
public class MgnlEnvConstructor extends AbstractConstruct {

    public static final Tag ENV_TAG = new Tag("!ENV");
    public static final Pattern ENV_FORMAT = Pattern.compile("\\$\\{((?<name>[\\w.\\-]+)((?<separator>:?(:|:\\?))(?<value>[\\w\\s]+)?)?)\\}");

    private final MagnoliaConfigurationProperties configurationProperties;

    public MgnlEnvConstructor(final MagnoliaConfigurationProperties configurationProperties) {
        this.configurationProperties = configurationProperties;
    }

    @Override
    public Object construct(Node node) {
        var scalarNode = (ScalarNode) node;
        String val = scalarNode.getValue();

        var matcher = ENV_FORMAT.matcher(val);

        while (matcher.find()) {

            String name = matcher.group("name");
            String value = matcher.group("value");
            String separator = matcher.group("separator");

            var envValue = apply(name, separator, value != null ? value : "", getEnv(name));
            if (envValue == null) {
                envValue = "";
            } else {
                envValue = envValue.replace("\\", "\\\\");
            }

            var subexpr = Pattern.compile(Pattern.quote(matcher.group(0)));
            val = subexpr.matcher(val).replaceAll(envValue);
        }

        return val;
    }

    /**
     * Implement the logic for missing and unset variables
     *
     * @param name        - variable name in the template
     * @param separator   - separator in the template, can be :, :?, ?
     * @param value       - default value or the error in the template
     * @param environment - the value from environment for the provided variable
     * @return the value to apply in the template
     */
    public String apply(String name, String separator, String value, String environment) {
        if (environment != null && !environment.isEmpty()) return environment;
        // variable is either unset or empty
        if (separator != null) {
            //there is a default value or error
            if (separator.equals("?")) {
                if (environment == null)
                    throw new MissingEnvironmentVariableException("Missing mandatory variable " + name + ": " + value);
            }
            if (separator.equals(":?")) {
                if (environment == null)
                    throw new MissingEnvironmentVariableException("Missing mandatory variable " + name + ": " + value);
                if (environment.isEmpty())
                    throw new MissingEnvironmentVariableException("Empty mandatory variable " + name + ": " + value);
            }
            if (separator.startsWith(":")) {
                if (environment == null || environment.isEmpty())
                    return value;
            } else {
                if (environment == null)
                    return value;
            }
        }
        return "";
    }

    /**
     * Get value of the environment variable
     *
     * @param key - the name of the variable
     * @return value or null if not set
     */
    public String getEnv(String key) {
        return this.configurationProperties.getProperty(key);
    }
}
Generated at Mon Feb 12 04:29:53 CET 2024 using Jira 9.4.2#940002-sha1:46d1a51de284217efdcb32434eab47a99af2938b.