Assorted Thoughts on Agilism X Traditionalism - Part VI

Aerospace Engineering and Agilism
I feel really happy when I read something about complex environments which corroborates with our thoughts on how to produce software. A few years ago I read on IEEE Spectrum that the US Department of Defense (DoD) was using leaner acquisition processes. Together with the fact that the DoD is the biggest open source user in USA, it made me think on how some teams delude themselves with excuses that in the end be something like "Yes, we must rely on heavy processes and proprietary software, our organization is really more complex than the Department of Defense of the United States of America!"

Now I have just bumped on the part regarding Mission and Reliability of the classic The Logic of Microspace book. Starting on page 219, the author discuss on why the aerospace industry didn't follow the Lean Thinking principles decades ago, and, more important to this post, what are the consequences of this. Two of them consequences caught my attention: (i) Since manual testing of parts was known and automated testing was new, aerospace industry decided to keep on manual testing, with all its bureaucracy and human errors, and (ii) the qualification processes took so long that it is usual to rely on more than 10 year-old parts. The author also gives an advice: small teams, working on small time windows and using highly integrated and pre-tested components are the solution for a lot of mission complexity problems.

The moral behind this post is the same of many other posts in this blog: don't use the size of your working environment or the complexity of your task as excuses for not using open source software and agile methods.

Post Script 
On page 244 of the same book:
"Space qualified parts carry that label because they meet certain standards for fabrication, inspection, testing and documentation. Modern parts [those produced through automated processes] are likely not designed for the mostly manual steps prescribed by the space qualification requirements, which are often manually assembled and inspected, which lowers reliability. The documentation doesn't add anything to your product's reliability. And the additional people needed to handle all this documentation cut down on team cohesion, create new management layers, and hence lowers the focus of the team effort."
Do I really need to say anything more to convince you that manual testing and lots of documents don't increase your system's quality or do you think your inventory control information system is really more complex than a satellite?
Before I forget, the book is from 2000...

Enterprise Information Systems Patterns - Part XIII

Why I believe that the Relational Databases are the biggest barrier to make EIS really flexible
We are now making EIS Patterns Relational Database (RDB) -aware through Django framework. Just as I expected, problems would start to appear when it is needed to represent Python's dynamic nature in an RDB.

It is a fact that while the technology behind RDB is really mature and extremely efficient and safe, the relational "way of thinking" was created in the 1970s to solve 1970s' problems! In my humble opinion, although mathematically sound, the way the relational model represents all-part, one-to-many, and many-to-many relationships is weird. Think of representing books in a relational way. You store all books in one room, while all their pages are in another room. Retrieving a given book's pages is a question of entering the pages room and asking which pages belong to that book. The book by itself is not able to identify its pages... In other words, it doesn't matter how elegant or representative is your object model, it will transformed into a set of unnatural, strange relationships.

So let's check an interesting situation where the dynamic nature of Python is used in EIS Patterns. The Process class controls the way objects collaborate to perform a given business process, wrapping and logging objects' methods executions using the run_activity method. For this discussion the interesting part of this method is its return clause:

return {'actor':actor, 'arguments': execution_arguments,  

        'result':activity_result, 'start': activity_start,

        'end':activity_end}

This clause returns a dictionary containing:
a) The execution actor: a Node of which the method is wrapped;
b) The execution arguments: a list containing the parameters for the method;
c) The result: the result produced by the execution;
d) The start and end: date and time of the starting and conclusion of the execution.

Objects in (a) and (d) are of known types (Node and DateTime), while (b) and (d) are collections of objects of unknown types, including, potentially, other complex collections. So, how to represent it on a RDB? One solution is using descriptors to detail the dictionary, proving for each returned object or method argument, the following record:

movement_oid	return_or_argument	type	reference_or_value
Movement object identifier	Boolean for marking as return value or argument	Object's type. Basic types, such as numbers and strings, have their value declared, complex types are referenced	Stores basic types values as strings, and references for complex types.

Complex objects are properly stored, normalized, in tables or, in raw, as blobs. Retrieval involves developing specific algorithms for each type, which can become a serious problem when dealing with a lot of Decorators. Besides that, a serious problem would be do deal with multidimensional collections. In other words, mapping this logging scheme to RDB would turn the framework non-flexible and too costly to adapt, descriptors wasn't the solution.

The solution was to use JSON, but with some investigation on how to do it in a flexible way, which I will explain in the next post.

Enterprise Information Systems Patterns - Part XII

Step by Step Programming using EIS Patterns
This post will describe, step by step and using the example "bank system", how to program using the framework and Fluidity + Extreme Fluidity.

Step -1
You must know the following techniques and tools:
a) BDD using Lettuce.
b) The Decorator Design Pattern.
c) Should-dsl, to define expectations in both acceptance and unit tests.
d) Python Unittest Library.
e) How to use Python Decorators (@).

Step 0
Understand how and why things are done in the framework:
a) Start at Part IV, the previous parts focus on discussions and experimentations which are not used in the framework anymore.
b) Why Decorators are used to extend classes, explained in Part V.
c) Why should-dsl is also used, besides in tests, to define contracts between objects, explained in Part VI.
d) How things described in parts IV to VI are implemented in general is described in Part VII -  if you feel it too abstract, I suggest going to Part IX.
e) Part VIII only describes the status of the project by that time, you can ignore it, but it is necessary to accompany the change log to become aware of important changes in the code.
f) Part IX describes conceptually how to program using the framework. Part X describes in more detail the use of Fluidity, and Part XI explains the evolution of the concepts. This post does the same of Part IX, but showing the code instead.

Reading the posts will give you an idea of the modus facendi of the framework, however, it is necessary to start programming... probably it is necessary to go back and forth from code to these posts. The following steps are a kind of guide, but in no way it is meant to describe a complete development process, this is one thing that you developer should infer. Some details may be omitted to avoid too long descriptions.

*It is strongly recommended that you check the unitary tests to understand how the framework core classes work in detail*

Right now we are using BDD because we don't have a nice BLDD tool to work with. Therefore you are supposed to describe the business process in BDD terms, by interpreting a template workflow, and then incrementally define the features' steps.

The following steps were developed on top of a specific example, however, they can be generalized.

Step 1
Based on the bank system example, let's suppose that the code is part of a bigger system which have already core concepts such as Employee and Bank Account.
First the process is defined in Fluidity, them, incrementally, the scenarios are defined and finally their steps.  
Fluidity:
    state('requested')
    transition(from_='requested', event='create_loan_request', to='request_created')
    state('request_created')
Scenario:
Scenario Outline: Individual Customer asks for loan
    Given I am a registered Credit Analyst
    And an individual customer with account number <account number> asks for  a personal loan
    And the loan request is of <desired value>
    When I confirm the loan request
    Then a new loan request with the <account number> and <desired value> is created
    And the new loan request is associated to the Credit Analyst
Steps:
@step(u'Given I am a registered Credit Analyst')
def given_i_am_a_registered_credit_analyst(step):
     world.a_person = Person()
     an_employee_decorator = EmployeeDecorator()
     an_employee_decorator.decorate(world.a_person)
     world.credit_analyst = CreditAnalystDecorator('09876-5')
     world.credit_analyst.decorate(world.a_person)

Step 2:
By analyzing the code above it is identified the necessity of developing a credit analyst business decorator. You will do only the basic necessary to the first step pass: create a skeleton of CreditAnalystDecorator, and go back to next step afterwards.
At this point it is already important to define the Rule of Association for this decorator, which is "to be a credit analyst, someone must be a bank employee first". Before writing the rule code, a test for checking if the rule is being enforced and if the decoration is working properly must be prepared:
    def it_decorates_a_person(self):
        #should fail
        (self.a_credit_analyst_decorator.decorate, self.a_person) |should| throw(AssociationError)
        #should work
        an_employee_decorator = EmployeeDecorator()
        an_employee_decorator.decorate(self.a_person)
        self.a_credit_analyst_decorator.decorate(self.a_person)
        self.a_credit_analyst_decorator.decorated |should| be(self.a_person)
        self.a_credit_analyst_decorator.decorated |should| have(2).decorators

Now the rule's code can be written:
    @rule('association')
    def rule_should_contain_employee_decorator(self, decorated):
        ''' Decorated object should be already decorated by Employee '''
        decorated |should| be_decorated_by(EmployeeDecorator)

This rule will assert that only if the person is an employee then he/she can be a credit analyst, and it is implemented as a method of the credit analyst decorator class. The be_decorated_by matcher was created specifically for this framework.

Attention: as of versions 0.7.3 of the core and 0.1.3 of the example, a singleton Rule Manager object started to be used, in order to promote reuse of rules. This object is composed by a "rules" object, which in the core is called Core Rules. Checking the tests of the example it is possible to see that other rule base is used, in the case a subclass of the core rules, called Bank System Rule Base. It is important to note that some tests use the core rules, simply because they don't need to use rules specific to the bank system. In other words, if you need a new rule, it must be created into your rule base object - making it hard-coded, however it is envisioned a way of loading them from a file or other configuration object, maybe using an even higher DSL to define them.


Step 3:
Now that the basic credit analyst decorator is defined and tested, we go back to the acceptance level, in this example, we will jump straight to the "Then" step because the others are straightforward. At this point, the business process starts to be assembled.
@step(u'Then a new loan request with the (.+) and (.+) is created')
def then_a_new_loan_request_with_the_account_number_and_desired_value_is_created(step, account_number, desired_value):
     (...)
    Creates the business process object:
    world.an_individual_credit_operation = Process('Individual Customer Credit Operation')
    Defines the process' source and destination:
    world.an_individual_credit_operation.set_source(world.the_company)
    world.an_individual_credit_operation.set_destination(world.a_client)
    Configures the process using a template state machine:
    template = LoanProcess()
    Uses xFluidity's to get a configuration based on the template:
    configurator = StateMachineConfigurator(template)
    configurator.configure(world.an_individual_credit_operation)
    At this point, the process object (an_individual_credit_operation) has absorbed the template state machine
    Now it is time to configure the first movement - loan request creation
    The object the_movement will be a proxy for CreditAnalystDecorator.create_loan_request. This movement will be a Transformation, having credit_analyst.decorated (the associated person) as both source and destination, and an_individual_credit_operation.create_loan_request as the activity - that's an important point: create_loan_request is a method, symmetric to a workflow transition, which was injected into the process object by xFluidity.
    the_movement = world.an_individual_credit_operation.configure_activity_logger(world.credit_analyst.decorated, world.credit_analyst.decorated, world.an_individual_credit_operation.create_loan_request, CreditAnalystDecorator.create_loan_request)
    After this, the collection of movements of the process should contain the recently configured movement (found by the activity's name)
    world.an_individual_credit_operation.movements |should| contain(the_movement.activity.__name__)


Step 4:
At this point the CreditAnalystDecorator class has no create_loan_request method, thus it is time to go back to the unit level and write a test to this specific requirement:
    def it_creates_a_loan_request(self):
        an_employee_decorator = EmployeeDecorator()
        an_employee_decorator.decorate(self.a_person)
        self.a_credit_analyst_decorator.decorate(self.a_person)
        self.a_credit_analyst_decorator.create_loan_request(self.an_account, 10000) #self.an_account was created in the setUp()
        self.a_person.input_area |should| contain('1234567-8')

Now it is implemented as an @operation:

    @operation(category='business')
    def create_loan_request(self, account, value):
        ''' creates a loan request '''
        loan_request = LoanRequest(account, value, self)
        #Places the loan_request in the node's input area
        self.decorated.input_area[loan_request.account.number] = loan_request

Note that a LoanRequest class must be defined, which drive us to the next step.

Step 5:
The first thing to do is to create a unit test for LoanRequest and then create the class itself, which are trivial tasks. After running all the necessary tests, we can go back to the acceptance level, given that the loan request class now exists and it is operating properly.

Step 6:
    Time to run the loan request creation and store the execution context:
    the_movement.context = world.an_individual_credit_operation.run_activity(the_movement, world.credit_analyst, world.account, desired_value)
    Since the create loan request transition was fired, the process should be on state 'request_created'
    world.an_individual_credit_operation.current_state() |should| equal_to('request_created') 


Step 7:   
Now it is time to check the 'And' clause of this step, which is:

@step(u'And the new loan request is associated to the Credit Analyst')
def and_the_new_loan_request_is_associated_to_the_credit_analyst(step):
    #... this association was done during loan creation, just checking
    world.a_person.input_area[world.account.number].analyst |should| be(world.credit_analyst)

This step was easy, since the way the loan request creation was implemented already places it on the decorated (the Person) processing_area.


The next step would be go to the next process transition and implement it using a sequence similar to the one presented above.


In summary, developing in the framework involves:
1) Describing a user story guided by a business process template.
2) Developing the code for the story steps, which will identify new responsibilities, to be implemented into the decorators and resources.
3) Developing the code for the decorators, including defining their association rules and @operations.
4) Developing the code for resources, including defining their association rules.
5) Detailing the steps: use movements to wrap and associate @operations to each workflow activity, providing full control through the workflow engine while logging the execution context.

The use of the process template and the association of decorators' methods to business process activities provides the adherence of the application logic to the process' logic. Currently, BDD is in use, therefore the linking to the process logic is done manually. However, it is envisioned the creation of a tool to generate the steps skeletons in a way adherent to the process logic.

Enterprise Information Systems Patterns - Part XI

Evolving the interpretation of the concepts
As the framework evolves, and we play with it through the examples, new interpretations of the concepts appear. In fact, after playing with Fluidity, we realized that Transformation and Transportation were not necessarily implemented as classes, but categories of movements instead, given that they behave in the same way - it is only a question of explicitly saying if the work item is transformed into a Node (a transformation) or simply moved from one node to another (a transportation).

The history behind this is that, when developing the example, it was noticed that even a "simple" resource movement would need to be logged, as well as it could be associated to, or even generate, other resource(s). When we realized this, it was clear that transportations would potentially need the same machinery used by transformations, hence, they share a common behavior, which should be implemented into Movement class. Therefore, they were transformed into categories - although the Category class is not in use yet. Figure 1 shows how the framework's ontology is interpreted now.

Figure 1: EIS Patterns Ontology

Another important point is that the relationship "is_a" between Operation and Resource does not represent a classic object oriented inheritance relationship. In other words, Operation is not a class, instead, it is a Python Decorator which is applied to Person and Machine methods to mark them as Business Operations. Therefore, when it is said that "a Node performs an Operation", it means that some of the Node's method is decorated as an operation - or, this node is able of performing this operation. Finally, the relationship "operates_on" between Operation and Resource means that operations are always transforming or transporting resources, through movements which encapsulates them (the operations).

In summary:
a) An operation is an abstract representation of a production operation.
b) A node's method is decorated by an operation to inform that the original abstract concept is now represented concretely by the method.
c) Inside a business process, a movement is used to encapsulate and log the execution context of the concrete operation defined in (b).
d) Running a process consists of executing and logging the movements defined in (c).

Part IX of this series explains in less abstract terms how to deal with operations and movements. Part X explains how to do this using Fluidity and Extreme Fluidity. As expected, in order to understand how the whole thing is implemented, it is necessary to check the EIS Patterns Examples project, in special, how BDD steps are implemented.

Enterprise Information Systems Patterns - Part X

EIS Patterns, Business Processes, and Workflow Engines
Every EIS is based on business processes, and it can be said that workflows are the computational realization of these processes. Therefore it is necessary to provide some workflow engine to EIS Patterns.

Basically, there are two types of workflow engines:
a) Associated to an application development platform: such as the ones found in Web application development environments, like Zope, or associated to databases, such as Oracle, or as part of Groupware environments. In this case, applications are developed "inside" the environment, using the engine's API.
b) Implemented as Libraries: using some programming language, a library of objects representing the workflow features is implemented, such as the State Machine as Python Decorators, or Ruby's AASM. In this case, you must use the libraries' objects to implement your engine.

Enters Fluidity & Extreme Fluidity
Fluidity was developed as a type (b) workflow engine, however, Extreme Fluidity turns it into a third type of workflow engine:
c) Inoculated & Expelled: using the dynamic nature of the underlying programming language, provide a way of making objects workflow-aware, and symmetrically, a way of turning these objects back to their initial structure when desired.
Why this? Because type (a) engines forces you to use a given environment to develop your applications, while type (b) forces you to use specific objects to implement workflows, most of times creating a mixed code with application specific and workflow specific statements.
With Extreme Fluidity (or xFluidity for short) it is possible to insert the code necessary to make your workflow run inside your business objects. In other words, they become workflow-aware objects, while keeping your programming style, standards, and patterns. In order to understand how this happens, let's check how Fluidity can be used when xFluidity is associated to it (the complete source code can be found here).
First, it is necessary to declare a template State Machine:

    class LoanProcess(StateMachine):
        state('requested')
        state('request_created')
        state('request_analyzed')
        state('refusal_letter_sent')
        state('loan_created')
        state('value_transfered')
        initial_state = 'requested'
        transition(from_='requested', event='create_loan_request', to='request_created')
        transition(from_='request_created', event='analyst_select_request', to='request_analyzed')
        transition(from_='request_analyzed', event='loan_refused', to='refusal_letter_sent')
        transition(from_='request_analyzed', event='loan_accepted', to='loan_created')
        transition(from_='loan_created', event='time_to_transfer_value', to='value_transfered')

Using xFluidity, the template is injected in another object - in this case a Process:

    def setUp(self):
        self.process = Process()
        template = LoanProcess()
        configurator = StateMachineConfigurator(template)
        configurator.configure(self.process)

Magically, the process object starts to respond to the state machine's events:

    def it_makes_the_process_respond_to_the_example_state_machine_events(self):
        self.process |should| respond_to('create_loan_request')
        self.process |should| respond_to('analyst_select_request')
        self.process |should| respond_to('loan_refused')
        self.process |should| respond_to('loan_accepted')
        self.process |should| respond_to('time_to_transfer_value')

As expected, the engine avoids the firing of transitions at wrong situations:

    def it_runs_the_example_acceptance_path(self):
        self.process.create_loan_request()
        self.process.current_state() |should| equal_to('request_created')
        # the line below means that if the process tries to fire loan_refused
        # when at 'request_created' state, an exception is generated
        self.process.loan_refused |should| throw(InvalidTransition)
        self.process.analyst_select_request()
        self.process.current_state() |should| equal_to('request_analyzed')
        #loan accepted
        self.process.loan_accepted()
        self.process.current_state() |should| equal_to('loan_created')
        self.process.time_to_transfer_value()
        self.process.current_state() |should| equal_to('value_transfered')

In that way, it is possible to use a workflow engine in your own development environment and using your own models. Fluidity also provides guards and state actions, not shown here for the sake of simplicity.

A Note on Workflow Notations
Fluidity is a state-based engine, which on those days of fancy business process notations may appear a bit old-fashioned. However, state machines represent a well known and easy to understand notation. EIS Patterns' Process and Movements objects are being developed in a way that they don't need to know which notation is in use, at least at the unit level - meaning that some code specific to a given notation may be used to "glue" elements at the application level.

Conclusion
Currently Fluidity and xFluidity are works in progress, including the expelling capabilities and some details on implementation of guards. When these features become fully functional, a Petri Net based engine will be developed. A clear limitation of Fluidity is that it only works for Python objects, but the necessity of developing in a given programming language is a limitation of all workflow engines, although some provide XML-like APIs.