Improving Support For Sensitive Data in Mistral

At the recent OpenStack Project Teams Gathering in Atlanta, GA, one of the topics discussed by the Mistral developer community attendees is the security issue where data that might be sensitive could be exposed in Mistral log files, stored in the database or when transferred over the network.

This was one of my favorite sessions as every attendee in the room participated in the discussion and provided a valuable insight or opinion and the proposed idea for a solution was truly a collaborative effort. The idea agreed upon included adding a section to workflows called “secret” where a developer can identify data of a sensitive nature. All of the items in the secret section will be protected whenever they are persisted either to log files or the database. Actions can be executed independently from workflows, so a decorator will also be added to mistral-lib to provide a means for custom action developers to identify constructor arguments to be protected.

Proposed Workflows Syntax

Proposed Custom Actions Syntax

It is great to see so many developers in the Mistral community concerned about security.

Mistral Custom Actions API

The Mistral community has been discussing improving the developer experience for creating custom actions for nearly a year.  As my first experience with Mistral has been as a consumer via TripleO, the custom actions experience has been important to me and I’ve been fortunate enough to be a part of the planning and implementation.  Mistral allows developers to implement new actions and plug them into Mistral via a mechanism based on Stevedore.  The ability to create custom actions is important as some things are easier to implementor more easily readable in Python instead of YAML/YAQL.  Developers wanting to integrate with Mistral may already have code they want to reuse or integrate with other products and systems that Mistral doesn’t provide support for directly.

According to the spec, one of the primary motivations for the new custom actions effort is to make it clear what parts of Mistral are publicly callable.  This was originally a source of confusion when creating custom actions for TripleO.  We decided to use a utility class for getting information from Keystone and it wasn’t initially clear in the review process on whether or not we were accessing private internals or not.  unclear on what parts of the Mistral code base you can or should be importing into your own actions and what can be used without fear of change.  This was a good indicator that Mistral needs to give developers an explicit public interface.

As the discussion has continued, the efforts to improve custom action development have also spawned additional ideas that will likely have a positive impact on the overall Mistral project ecosystem.  At the recent OpenStack Project Teams Gathering in Atlanta, GA, the attendees from the Mistral developer community decided to move forward on the following changes:

All of the common code between the Mistral ecosystem of projects and general 3rd party integration code to include custom actions API and custom yaql functions API will be contained in the mistral-lib project.

All of the OpenStack specific integration code and examples will reside in the mistral-extra project.

There is already a patch up for review, but it will likely see some additional changes to land and I’ll write a more detailed example of creating custom actions once it lands.

 

Debugging a Custom Mistral Action

When I first started creating custom Mistral actions for TripleO, I quickly found the need to debug them interactively.  I wasn’t satisfied with dumping out information to logs and searching through them.  I started with creating scripts that used the same code so I could run them directly, but I soon found that approach inadequate.  I needed to be able to debug an action that used something internal to Mistral (e.g. mistral.context) or when an action is passing a unit test but fails within a workflow execution.   I found myself wanting a solution where you could create a Mistral execution and on any error have it open a pdb session in the shell.

To solve this problem, a coworker suggested remote debugging with pdb.  A quick search led to the Remote PDB package.  It turned out to be quick and simple to use and it worked immediately without a ton of effort.  To use remote-pdb on the undercloud, you need to perform the following steps:

# https://pypi.python.org/pypi/remote-pdb
pip install remote-pdb

# open port for remote-pdb access on undercloud
sudo iptables -A INPUT -p tcp -m multiport --dports 4445 -m comment --comment "remote pdb" -m state --state NEW -j ACCEPT

# add two lines to any action you want to debug
from remote_pdb import RemotePdb
RemotePdb('127.0.0.1', 4445).set_trace()

# run action or create exececution
mistral run-action tripleo.get_capabilities '{"container": "overcloud"}'

# open netcat in another terminal on undercloud
nc -C 127.0.0.1 4445