Expand your workflow horizons with some Jinja processing techniques you may not have tried!
About StackStorm
If you’re visiting our blog for the first time, StackStorm is a highly customizable and extensible event-driven If-this-then-that automation platform for DevOps.
A ‘trigger’ is generated, and StackStorm can process the information from that trigger and execute an appropriate action workflow in response. StackStorm action workflows are written in YAML, and allow for functions to be used from either YAQL or Jinja. In this blog, we’ll mostly focus on some aspects of writing Jinja2 in YAML.
About Jinja2
One of the best features of StackStorm is being able to mix and match both YAQL and Jinja when one is more convenient than the other, but one area where Jinja wins out almost every time is in scripting.
Jinja2 is a subset of python which means you have a good portion of the base python functions and features at your fingertips. From basic for-loops to some filtering, Jinja has most basic needs covered. When dealing with advanced processing you may want to reach out for a full python script, but before you do so, consider some of our techniques!
The Situation
Consider the following data, containing an array of hosts ['hostA','hostB','hostC'] each with various properties:
hosts:
- hostname: hostA
ip: 192.168.0.1
properties:
os: centos
version: 7
cpu: 2
ram: 4
- hostname: hostB
ip: 192.168.0.2
properties:
os: ubuntu
version: 18.08
cpu: 8
ram: 16
- hostname: hostC
ip: 192.168.0.3
properties:
os: centos
version: 8
cpu: 8
ram: 16We have an array of hosts, which is a list of only 3 servers, these hosts have several attributes, of which properties is an object itself.
Using For-If Loops
If you want to select only hosts which meet certain criteria and output their specs, most initial attempts look something like this:
{% for host in ctx().hosts %}
{% if 'centos' in host.properties["os"] %}
{% if loop.first %}
--- CentOS Hosts ---
{% endif %}
{{ host.hostname }} - {{ host.ip }}
Specs:
CPU: {{ host.properties["cpu"] }}
RAM: {{ host.properties["ram"] }}
{% endif %}
{% endfor %}Which works well enough for producing the following output:
--- CentOS Hosts ---
hostA - 192.168.0.1
Specs:
CPU:2
RAM: 4
hostC - 192.168.0.3
Specs:
CPU:8
RAM: 16However, you can make your code better and more readable!
You can combine the For and If statements into a single For-If loop, and drop the last endif tag:
{% for host in ctx().hosts if 'centos' in host.properties["os"] %}
{% if loop.first %}
--- CentOS Hosts ---
{% endif %}
{{ host.hostname }} - {{ host.ip }}
Specs:
CPU:{{ host.properties["cpu"] }}
RAM: {{ host.properties["ram"] }}
{% endfor %}Both are functional, but the second example is cleaner, making it more readable as to what the code is processing in the loop.
If the request is to create a report of multiple OS’s you may prefer the first version with a separate if such that you could create a second report heading for --- Ubuntu Hosts ---.
However, Jinja being a subset of Python, you should try to subscribe to the same tenants that Python focuses on when you can. Two of which are simplicity and readability and by combining some statements you can improve your code without losing any functionality.
Now that you have a grasp of For-If loops, let's start pushing Jinja in StackStorm to its limits.
Using set to Escape Processed Strings from Loops
Note: This example, while functional, is more designed as an experiment to see how you can push Jinja’s implementation and show off some Jinja features you may not have used. This level of processing is likely better suited for a dedicated python script where data types can be managed.
Starting with the same hosts file, what if instead of printing off some keys and information along with some basic formatting, you needed to re-process the data?
Consider the case of storing information in a database that other applications (Tableau, Power BI, a React App, etc.) will be using. As you store documents in the Database, you may want to process fields into a form more readable by one or more applications.
The Challenge
Take the properties from each host:
hosts:
- hostname: hostA
ip: 192.168.0.1
properties:
os: centos
version: 7
cpu: 2
ram: 4And create a customString based on the properties, and reattach it:
hosts:
- hostname: hostA
ip: 192.168.0.1
properties:
os: centos
version: 7
cpu: 2
ram: 4
customString: "os=centos;version=7;cpu=2;ram=4"then output a new format:
--- CentOS Hosts ---
hostA - 192.168.0.1
Specs:
CPU:2
RAM: 4
customString: "os=centos;version=7;cpu=2;ram=4"
hostC - 192.168.0.3
Specs:
CPU:8
RAM: 16
customString: "os=centos;version=7;cpu=8;ram=16"There’s a Better Way
What if we told you everything you just did above is almost completely doable within a single StackStorm task?
We say almost, as it’s difficult to preserve or re-create the object data typing because it will often be converted to a string. This issue can be mitigated using some of the Jinja JSON filters tojson, json_loads, json_parse, to_json_string, from_json_string, but it’s likely something you will end up butting heads with where you’ll understandably reach for a proper script.
Let’s look at your limits:
{% for host in ctx().hosts if (host.properties is defined) and ('centos' in host.properties["os"]) %}
{%- set customString = namespace(value='') -%}
{% for key,value in host.properties.items() %}
{% set customString.value = customString.value ~ key ~ '=' ~ value ~ ';' %}
{% endfor %}
{% if host.properties.update({'customString': customString.value}) %}{% endif %}
{% if loop.first %}
--- CentOS Hosts ---
{% endif %}
{{ host.hostname }} - {{ host.ip }}
Specs:
CPU:{{ host.properties["cpu"] }}
RAM: {{ host.properties["ram"] }}
customString: {{ host.properties["customString"] }}
{% endfor %}The above Jinja produces the output:
--- CentOS Hosts ---
hostA - 192.168.0.1
Specs:
CPU:2
RAM: 4
customString: os=centos;version=7;cpu=2;ram=4;
hostC - 192.168.0.3
Specs:
CPU:8
RAM: 16
customString: os=centos;version=8;cpu=8;ram=16;Output Deep-dive
Now, let’s dig into the Jinja line-by-line:
{% for host in ctx().hosts if (host.properties is defined) and ('centos' in host.properties["os"]) %}Start with a For-If loop and a check to make sure you have properties to iterate on and the OS is the correct target
{% set customString = namespace(value='') %}Set a new namespace variable and create a dict object customString with key value pair value: "". Namespaces are a method for defining the scope of variables you create in Jinja and are a way of building data outside of loops.
You’ll need to set this within the first for loop else it will continuously build elements and append all host’s properties to it.
{% for key,value in host.properties.items() %}Create a basic for loop on the list of properties. By calling items() you essentially feed properties as a list of tuples which you can call with {{ key }} and {{ value }}.
{% set customString.value = customString.value ~ key ~ '=' ~ value ~ ';' %}
{% endfor -%}Build your string and append each element while looping using Jinja’s fuzzy concatenation ~ which will attempt to concatenate and if one of the values is not a string, the non-string object or value will be turned into one. You close the loop once we’re done building the object.
{% if host.properties.update({'customString': customString.value}) %}{% endif %}Then you’ll need to use an interesting quirk of Jinja to modify the actual host.properties object. Expressions within Jinja if statements are executed prior to the evaluation of the if statement, and if the expressions happen to be modifying some other data, the if statement will return true as long as the operation is successful.
If you didn’t execute this as a template using {% and if, you would encounter a Jinja namespace/scoping error: "expected token 'end of print statement', got 'host'"
{% if loop.first %}
--- CentOS Hosts ---
{% endif %}
{{ host.hostname }} - {{ host.ip }}
Specs:
CPU:{{ host.properties["cpu"] }}
RAM: {{ host.properties["ram"] }}
customString: {{ host.properties["customString"] }}
{% endfor %}The rest looks like the rest of your good old For-If loop from above, only with your newly added value being called.
However, the workaround is kind of fake. You’re not truly updating your original hosts array—you’re allegedly reading your variable outputs.
In fact, if you created a follow-up core.echo task and called the original hosts array, you would find no trace of your customString value. This is due to loops and namespace scoping within the task.
There are ways around this, and as they say: “Where there’s a will there’s a way.” However, just because something can be done, doesn’t mean it necessarily should be done. Why put yourself through additional heartache when you can save yourself development time by just moving everything to a python script?
On the other hand, you might be surprised at just how much can be done within a single StackStorm task!
Where Do We Go From Here?
What if you wanted to actually replace the objects in our second loop challenge, how might you go about getting there? What more options could you try?
If you output {{ host }} with no formatting, you get something returned that looks like this:
"{'hostname': 'hostA', 'ip': '192.168.0.1', 'properties': {'os': 'centos', 'version': 7, 'cpu': 2, 'ram': 4, 'customString': 'os=centos;version=7;cpu=2;ram=4;'}}
{'hostname': 'hostC', 'ip': '192.168.0.3', 'properties': {'os': 'centos', 'version': 8, 'cpu': 8, 'ram': 16, 'customString': 'os=centos;version=8;cpu=8;ram=16;'}}
"It almost looks like it should work as an object, but note the double quotes " —those are not actually objects, but instead one long string. It’s not really an array as it's not comma , separated. There would still be some more tinkering needed to get this into object format.
Maybe instead of a customString you build a customStringList, which is an array of equal size to the hosts you're processing, and you append each customString to it as an element of the list. Then you publish that array as your variable and in the following task use YAQL’s zip() to zip them together in some fashion. That could be a viable option!
The more StackStorm-y way would be to have the task echo or output the values, and use <% task(build_customString).result.stdout %> to capture and re-append the customStrings to the proper hosts.
However, this becomes a bit of a guessing game as to what will work within certain confines of StackStorm. Using Jinja in Python you have almost complete freedom; using Jinja in a StackStorm task you are more confined to an action context in which you’re only allowed small adjustments.
Complex scripts and Jinja-fu like this all prevent readability which is un-Python-like and therefore un-StackStorm as ST2 was written in Python. Maintaining and updating also become more and more precarious as the script evolves.
Conclusion
Knowing when to step between a bit of a funky publish: and when something should be developed as a script is a skill. However, as mentioned, ST2 is written in Python, and Jinja itself is a subset of Python, so even if you do some wild experimentation and decide to change gears to a full script—you can reuse a lot of the exact same logic and code!
Not having fun StackStorming the castle?
Bitovi is here to help! We have expert StackStorm and DevOps consultants who are ready to save the day.
With StackStorm (IFTTT for ops) you can do custom DevOps automations, SRE auto-remediations, Security incident handling, CI/CD, Release, and Deployments that can integrate with more than 100 tools and services.
Bitovi is an official partner and contributor to the StackStorm Open Source project. If you need help with automation, training, or custom solutions on top of StackStorm - reach out to Bitovi experts. Schedule a free StackStorm consultation to get started.
stackstorm
Previous Post
