Enterprise Information Systems Patterns - Part X

EIS Patterns, Business Processes, and Workflow Engines
Every EIS is based on business processes, and it can be said that workflows are the computational realization of these processes. Therefore it is necessary to provide some workflow engine to EIS Patterns.

Basically, there are two types of workflow engines:
a) Associated to an application development platform: such as the ones found in Web application development environments, like Zope, or associated to databases, such as Oracle, or as part of Groupware environments. In this case, applications are developed "inside" the environment, using the engine's API.
b) Implemented as Libraries: using some programming language, a library of objects representing the workflow features is implemented, such as the State Machine as Python Decorators, or Ruby's AASM. In this case, you must use the libraries' objects to implement your engine.

Enters Fluidity & Extreme Fluidity
Fluidity was developed as a type (b) workflow engine, however, Extreme Fluidity turns it into a third type of workflow engine:
c) Inoculated & Expelled: using the dynamic nature of the underlying programming language, provide a way of making objects workflow-aware, and symmetrically, a way of turning these objects back to their initial structure when desired.
Why this? Because type (a) engines forces you to use a given environment to develop your applications, while type (b) forces you to use specific objects to implement workflows, most of times creating a mixed code with application specific and workflow specific statements.
With Extreme Fluidity (or xFluidity for short) it is possible to insert the code necessary to make your workflow run inside your business objects. In other words, they become workflow-aware objects, while keeping your programming style, standards, and patterns. In order to understand how this happens, let's check how Fluidity can be used when xFluidity is associated to it (the complete source code can be found here).
First, it is necessary to declare a template State Machine:

    class LoanProcess(StateMachine):
        state('requested')
        state('request_created')
        state('request_analyzed')
        state('refusal_letter_sent')
        state('loan_created')
        state('value_transfered')
        initial_state = 'requested'
        transition(from_='requested', event='create_loan_request', to='request_created')
        transition(from_='request_created', event='analyst_select_request', to='request_analyzed')
        transition(from_='request_analyzed', event='loan_refused', to='refusal_letter_sent')
        transition(from_='request_analyzed', event='loan_accepted', to='loan_created')
        transition(from_='loan_created', event='time_to_transfer_value', to='value_transfered')

Using xFluidity, the template is injected in another object - in this case a Process:

    def setUp(self):
        self.process = Process()
        template = LoanProcess()
        configurator = StateMachineConfigurator(template)
        configurator.configure(self.process)

Magically, the process object starts to respond to the state machine's events:

    def it_makes_the_process_respond_to_the_example_state_machine_events(self):
        self.process |should| respond_to('create_loan_request')
        self.process |should| respond_to('analyst_select_request')
        self.process |should| respond_to('loan_refused')
        self.process |should| respond_to('loan_accepted')
        self.process |should| respond_to('time_to_transfer_value')

As expected, the engine avoids the firing of transitions at wrong situations:

    def it_runs_the_example_acceptance_path(self):
        self.process.create_loan_request()
        self.process.current_state() |should| equal_to('request_created')
        # the line below means that if the process tries to fire loan_refused
        # when at 'request_created' state, an exception is generated
        self.process.loan_refused |should| throw(InvalidTransition)
        self.process.analyst_select_request()
        self.process.current_state() |should| equal_to('request_analyzed')
        #loan accepted
        self.process.loan_accepted()
        self.process.current_state() |should| equal_to('loan_created')
        self.process.time_to_transfer_value()
        self.process.current_state() |should| equal_to('value_transfered')

In that way, it is possible to use a workflow engine in your own development environment and using your own models. Fluidity also provides guards and state actions, not shown here for the sake of simplicity.

A Note on Workflow Notations
Fluidity is a state-based engine, which on those days of fancy business process notations may appear a bit old-fashioned. However, state machines represent a well known and easy to understand notation. EIS Patterns' Process and Movements objects are being developed in a way that they don't need to know which notation is in use, at least at the unit level - meaning that some code specific to a given notation may be used to "glue" elements at the application level.

Conclusion
Currently Fluidity and xFluidity are works in progress, including the expelling capabilities and some details on implementation of guards. When these features become fully functional, a Petri Net based engine will be developed. A clear limitation of Fluidity is that it only works for Python objects, but the necessity of developing in a given programming language is a limitation of all workflow engines, although some provide XML-like APIs.

Enterprise Information Systems Patterns - Part IX

How to use EIS Patterns?
When eispatterns started to be developed, the original goal was to build a set of basic concepts that could be configured to represent various business entities. However, this seemed to be very hard to accomplish, since as new behavior appears, new code is necessary, making the economy of using masks to represent new entities being not enough to justify the use of the framework.
Therefore, the framework's focus shifted to on how to represent real world entities that dynamically acquire new responsibilities. Then, a new path was followed, by defining:
-Essential abstract concepts in the framework: Node, Resource, and Movement.
-The roles for each of the abstract concepts: Node: active, responsible for executing production operations; Resource: passive, responsible for storing  production resources data; and Movements: coordinative, responsible for managing the nodes while they work on resources.
-How to extend each of the concepts: Node: through Decorator Pattern; Resource: through subclassing; Movements: through configuration (better explained here).

Programming in EIS Patterns
In the project repository a super-simplified "bank system" can be found, with the intention of demonstrating how the framework can and should be used. The framework is process-centric, meaning that the development should evolve around the underlying business processes, by following these steps:

1) Represent the business process using a Process object. Ideally, this object should "absorb" the attributes and methods necessary to run as a workflow. For instance, if a state machine is used to represent business processes, the process object should have states (attributes) and transitions (methods), as well respond to specific events. Moreover, if it is decided to represent the same process using other notation, such as BPMN, this object must be transformed accordingly. We implemented Fluidity, a state machine for Python, which responds to these demands*. Process objects use simpler Movement objects to connect business operations to business processes and log their execution.

2) Define business decorators to adorn Node objects. Nodes are persons or machines which gain new responsibilities through business decorators, which in turn manipulate Nodes' input, processing and output areas. Business decorators make Nodes become the active part of the business process. Decorators must implement a decorate() method, where rules of association are defined using should-dsl. These rules ensure that a given object can be adorned by the decorator.

3) Each process activity (using a state machine: a transition or state action) must be realized by a business decorator method. These methods are called (business) operations, can be categorized, and are encapsulated by Movements**, which in turn are linked to the Process object. In that way, business decorators' methods take the responsibility of making the process work***. The same template can be used to configure different processes, using different operations to implement them.

4) Define the operations by decorating them with @operation. These operations should be able to ask for the transference of resources from/to the different nodes' areas using node's transfer() method. @operations implement the logic for each business process activity.

5) Define WorkItems, which are, in general, data bags or passive entities****. The reason for that is that in a real-world, physical production system, humans and machines transform or transport the work items using their abilities (@operations). A report is passive, being written by a human; a truck hood is passively spot-welded by a robot in a factory; a motor is passively transported from one manufacturing cell to another.

These steps are not necessarily executed in order, nor the business process is necessarily fully defined beforehand. It is envisioned a development process guided by the incrementally testing/building of the business processes. The use of templates is also a form of defining the acceptance criteria for the tests.

It is important to note that every class and every method must have a docstring, so that keyword searches can be performed by the Documenter module, which will be (soon) extended by Natural Language Processing facilities, so that requirements can be parsed and used to try to find candidate reusable elements in a given library of business decorators and work items. Complimentary, the Rule Checker is a module that runs the association rules of the business decorators found by the search, so that users can verify which of them can be used for adorning a given object. A new module will be developed to find and run  acceptance tests, so that the "live system" can be used to check its fitness to a given set of requirements.

*Note 1: In comparison with workflow engines and workflow libraries, Fluidity (in conjunction with Extreme Fluidity) is innovative, because while engines force the development inside their environment and libraries force the use of their objects to implement the process, Fluidity allows the definition of template machines that can be absorbed by any Python object. In other words, you can still use your own classes, your own naming convention, your own programming style, and your own architecture, in your own environment. The "state-based behavior" is inoculated into your objects dynamically. Moreover, if you don't want to use Fluidity anymore, your objects can expel the state machine at any time.

**Note 2: Movements are of the type Transformation - which transforms a given resource in a node's processing area, and Transportations - which transports a given resource from a node to another. These concepts became conceptual classifications of movements, only Movement objects do the job.

***Note 3: Implementing business process is a two step task: first a movement is configured to log an @operation execution, and then this operation is executed.
Using a state machine notation, the property movement.activity is set to a transition or a state action, and the  property movement.activity_associated_method is set to @operations. It would be easier if transitions and actions would be directly associated to @operations, however, Movements are necessary to store execution information (parameters, result, date and time) as well as to indicate the source and destination nodes. Source and destination nodes allow the mapping of the process to physical movements in the production plant. After configuration, a movement can be executed. These steps are implemented into Process objects, because they coordinate movements. This layered method improves reuse and configuration of processes and @operations.

**** Note 4: If workitems are physically aggregated to form a final product, this set of workitems must be promoted into a Kit, which can be "active" in some contexts.

Enterprise Information Systems Patterns - Part VIII

The current status of the project

The code in commits of April 9 presents the first version of Rule Checker, a class able of finding Business Decorators' rules implemented in should-dsl and run them against the object to be decorated. In April 11 the initial version of Documenter was committed. Documenter simply returns the documentation of Resources and Business Decorators. Rule Checker and Documenter form the basis for the following configuration process:

a) Requirement input phase
    1) User enters the keywords that represent the main concepts of the requirement.
    In the future:
    2) User enters the requirement's text.
    3) A Natural Language processor will identify the main concepts of the requirements.
b) Concepts search phase
    1) By simply matching keywords, the framework suggests candidate business decorators and resources which can be reused to configure the new business process.
    In the future:
    2) Based on documentation and rules of association, an ontology is built to enhance searching of candidate decorators and resources.
c) Concept compliance testing
    1) To better understand how candidate decorators work, run automated tests, so that the user can see the decorator in action.
    2) To check if the decorator can be used to decorate a selected Node object  (Person or Machine), test which decorators can be applied to it by checking (running) association rules.
d) Configure Business Process
    1) Using a domain specific language (DSL), describe the logic of the business process and generate the appropriate code for the business process.
    2) With the information obtained in (c), configure the business process. If new and/or changed functionality is necessary, enter the process of developing it, using the partially configured business process as a basis for acceptance tests. In that way the business process is constructed incrementally. At this point, the use of BLDD is envisioned.

The core idea of the process above is to provide ways of creating new systems in a dynamic and flexible way, through configuration . By dynamic I mean allowing the user to manipulate live objects which can demonstrate their capabilities - going far beyond documentation reading; by flexible I mean the use of Decorator Pattern to create new business entities on the fly. In the cases where changes are needed, the use of BLDD can provide a safer and dynamic way of extending the framework with new or improved business decorators and resources.

The code necessary to implement this process is in the following state:
-a.1: create a nice GUI, planned to medium term (end of the year).
-a.2 & a.3: work in progress based on NLTK, planned to medium term.
-b.1: done.
-b.2: planned to medium term.
-c.1: planned to short term.
-c.2: done.
-d.1: dsl for state diagrams is a work in progress.
-d.2: planned BLDD use for the medium term, full BLDD to long term.

Enterprise Information Systems Patterns - Part VII

Understanding how concepts are implemented
In Part V of this series, I used an UML Class Diagram to show the relationships among the framework's concepts. Although this diagram can help understanding these relationships, being it a class diagram, it also "says" that all concepts are "classes", which is not true. A better way for describing these relationships is using an ontology, as shown in Figure 1.

Figure 1: Ontology representing the relationships among framework's concepts

Operations, since they are immaterial resources, need to be realized by some active entity, in this case, by Nodes. Therefore, they are implemented as "functions" (in fact "methods"), instead of instances of a given class. Although in Python functions are also objects, one cannot define them as objects of a given business class. However, it is necessary to typify Operations somehow so that we can identify among some object methods which represent business operations. The solution for this is using Python Decorators, which are not exactly the same thing as the Decorator Pattern used in this framework. A Python decorator is a specific change to the language syntax that allows to more conveniently alter functions, methods, and classes.

Thus, it is possible to create a @operation(a_category) pythonic decorator which will store into method objects an attribute named category, with value a_category. In that way it is possible to mark methods as of Operation type, which is important to query a given class for its business skills. For instance, in a bank information system, it is possible to query a Credit_Analyst class to check which business operations objects of this type can perform. Or the contrary, using a keyword search discover, identify which methods of which classes may perform a given business operation. Moreover, I can create an ontology and navigate through it to check relationships among resources, nodes, and movements. With this ontology I can even suggest to a business analyst specific configurations for the system, given appropriate search terms.

Processes are sets of transformations and transportations, which in turn are used to encapsulate operations. Encapsulate means to forward calls to the appropriate Node subclasses methods - in fact to their Business Decorators methods - and log all information related to these calls, such as date and time, parameters and associated objects. Thus, Process objects implement the logic which coordinates how Nodes work. In other words, Processes implement workflows, which means, in turn, that the framework is process centric, or workflow centric.

Decorating, Decorating, and Subclassing
To make things more clear, let's call "decorators" the extension to Python syntax, and "business decorators", the classes created to decorate the framework concepts, as explained in Parts IV and V. Said that, let's check how the framework is implemented and extended:
a) Resources:
-Operations: methods, implemented into business decorators and marked with @operation decorator (yes, decorators for decorators).
-Work Items (formerly known as Materials): "classical" objects, for each new type of material, a new subclass of material is created.
-Kits: "classical" objects, for each new type of kit, a new subclass of kit is created. Given that kits can be composed by Operations, which are in fact methods, this composition is implemented through a list of references to these methods.
b) Nodes:
-All three subclasses are extend through business decorators. For instance, a person can be decorated by Developer (business) decorator to represent a software developer, or a Credit Analyst (business) decorator to represent a bank's credit analyst. No subclasses used. They can have methods related to @operations, to @rules of association (check Part VI), as well as methods representing internal machinery, in this last case, typically private.
c) Movements**:
-Transformations and Transportations are used as is, no subclassing, no pythonic decoration, no business decoration. They are used to build processes and store information related to operations calls, by encapsulating @operations during their configuration process.
-Processes are extended by mimicking workflow templates (current work), or by using workflow engines to make them work. A way of implementing processes is using Business Language Driven Development implemented through State Machine using decorators*.

For the reasons presented in Parts IV and V, subclassing is avoided at maximum - it is openly suggested only in the case of Work Items (Materials), which are considered passive "data bag" classes.

*Post-publishing note #1: we decided to develop our own State Machine code, and make it independent of this framework, check the Fluidity project.

**Post-publishing note #2: Transformations and Transportations are not classes anymore, as explained in Part XI.

Enterprise Information Systems Patterns - Part III

After my last post, the code in Github changed many times, since I was experimenting with both the basic concepts and the testing stack we are using. We decided not to write features (BDD's Given-When-Then constructs) for the configuration of Resources, Nodes, and Movements, since it would be overkill (as you can see in the sketch branch, from December 12, 2010): by reading the specs anyone can understand and check the functioning of configuration.

The comments below are based on the last implementation (January 09, 2011), which doesn't implement Path nor the parsers for the configurators. We are considering writing features for Path configurations, because Paths represent business processes, and therefore, it make sense to have high level descriptions of requirements.

Currently, the proposal is based on two base classes, from which all others derive. The first one, Configurable (previously known as Maskable), is the base for building the abstract concepts, and the second one, Configurator, is used for providing a "friend" class for each concept which is responsible for managing the configuration of the concepts' instances.

Besides the basic attributes, Configurable abstract class holds a "To Do" method, which is simply a placeholder to remember that a method for doing "basic things" with instances should be implemented by subclasses. Configurators on the other hand must be able to parse, persist, and retrieve configurations to be applied to concept's instances. They are multitons, in other words, for a given key, composed by configuration's mask and version attributes, only one configurator object may exist. Therefore, every concept class, such as Movement, has a configurator class, such as Movement_Configurator.

Configurations supply the following attributes for instances (spec files provide complete configuration examples):
-Mask:  defines a specific type for the instance. Example: payment (a payment movement).
-Version: holds the version of the mask, multiple versions can be in use at the same moment. Example: monthly (monthly payment).
-Category: associates a category to a specific concept. Example: financial (monthly payment is a movement of category financial).
-Description: provides a description for the key mask+version. Example: ' a monthly payment from a organization to a person'. Note that, in this example, organization and person can also be categories.

Right now the implementation is in a very simple state, the next step is to implement Path, which is expected to reveal more requirements for making this framework more usable. For testing, we are going to use some real-world workflow implemented in ERP5, applying the mappings presented in the technical report "Mapping Business Process Modeling constructs to Behavior Driven Development Ubiquitous Language" to describe the workflow using Given-When-Then.

Enterprise Information Systems Patterns - Part II

The project in Github reached an initial implementation, which I separated in a branch named sketch, given that it is mixing configuration and instantiation. The master branch now is in accordance with the proposal, which I continue to discuss below.

Remember that the core idea is to use the concepts as Lego parts. This mean avoiding using subclasses in general, but masking the abstract concepts through configuration. After configuration, I need to implement specific functionalities for the concepts.  These functionalities will be implemented in two places:
a) At instances of concrete concepts, by defining specific behavior for
different contexts (paths).
b) At path objects, in the form of coordination code, which will make the path''s
movements collaborate to realize a business process.

In that way, we are going to have a two-phased development/customization process:
a) Configuration: defines descriptors, which represent concrete uses of the
abstract concepts. Descriptors list the types used to transform the abstract
concepts into concrete ones. Configuration is done through a Domain Specific
Language (DSL), having each concept a template text to be used for its
configuration. In the future, we expect to define a proper grammar for this DSL.
b) Implementation: uses descriptors to make the concrete concepts instantiable
and implement the specific code related to their concrete use. Each concept has
a callable object with a proper name, which is defined during the implementation
of user stories.

Thus, in a first moment, a domain specialist will configure concrete concepts
using the DSL. Configurations are reused during the implementation, when
user stories instantiate and define the specific behavior of the concrete
concepts.

Right now, we have implemented Resource and Node, as well as the Maskable superclass, which will serve all concepts. I can say that the project is in the configuration phase yet, meaning that we haven't implemented any code for "using" the concepts in some business process.

Regarding the DSL used for configuration, we have to investigate its relation with our BLDD proposal. Remember that this proposal is used for implementing business processes, while the current DSL is used for configuring objects, which occurs before implementation of the processes. Maybe an execution DSL which uses the same terms used for configuration will grow from this investigation.

Enterprise Information Systems Patterns - Part I

This post aims to start the discussion on design patterns for EIS. We have just started a Github project, which right now is just a rough sketch, however, it already shows some basic ideas, implemented in Python with BDD (specloud, should-dsl, and Lettuce). By checking the features (*.feature files) you can understand the patterns requirements, and by checking the specs (*_spec.py files) you can understand how the code implements them.

We decided to start by revisiting ERP5 concepts, separating them from ERP5's underlying platform, Zope, and, in some cases, even reinterpreting them. The concepts are the following:

Concept 1: Resource
A resource is anything that is used for production. It can be a material, money,
machine time, or even human skills.
Concept 2: Node
Is something that transforms resources. For instance, a machine, a factory, a
bank account.
Concept 3: Movement is a movement of any Resource from one Node to another.
Concept 4: Path*
It describes how Movements are used to implement business processes.

ERP5 also has the concept of Item, an instance of a Resource. Until this point, we
decided not to use this, otherwise it would be necessary to create classes for
instancing the other three abstract concepts.

The core idea is to use the concepts as Lego parts. This mean not using
subclasses in general, but masking the abstract concepts through
configuration. For instance, a Movement is firstly configured as a concrete
movement, such as transferring goods from supplier to customer. After that, it
can be instantiated. Of course, at some point extra coding is necessary, and we believe that methods for implementing specific algorithms will emerge
on the coordinator element, the Path instances.

We therefore define two new "patterns": Mask, which means to define an abstract concept and turn it into a concrete concept by a proper configuration, and Coordinator, which is an object (also maskable) that coordinates others to realize a given business process. Maybe these patterns already exist with others names and/or similar purposes, however, we will check this more carefully as the project goes forward (we are really iterative & incremental).

The motivation for this project is that people like to say that frameworks are like Lego, but when you need to subclass their concepts, instead of using them as is, I think this is not the "Lego way" of doing things. We only use masks to provide useful labels for users, but the intention is not to extend the original concepts.  When I mask a Movement as a Sale, I am only setting its mask attribute to 'sale'.

The only concept that will be extended is Path (nothing is perfect!), since it will be used to implement the specific algorithms necessary to appropriately integrate movements - realizing the business process.

We probably won't supply something really useful for programming, because we are not considering platform issues, therefore, if you need a ready to use ERP framework in Python, check ERP5. The intention is to discuss how to build highly configurable frameworks, instead of proving a real framework. After we get the basic configuration machinery done, we are going to use BLDD to implement didactic cases: from the business process modeling to coding and testing.

(The Change Log maps this series of posts to commits in GitHub)

Proof of Concept for the BLDD Tool - Part VI

We have released a compilation of the posts with some more explanations on our proposal for the Business Language Driven Development (BLDD) draft, the technical report can be found here.

There is a lot of work to do on top of BLDD, such as:
- Defining in more detail the use of a companion textual Domain Specific Language (DSL). Probably with support of a Rule Engine.
- Implement an usable tool, we are investigating an extension of Drools for this purpose. Drools seems to be a good choice because it is BPMN compatible, implements a Rule Engine, and an Eclipse plugin. And, of course, it is open source.
- Implement BLDD on top of an ERP, our choice is ERP5, for the reasons listed on the first post of this series. This case study should show the applicability of the proposal.

You can find more "future work" in the report. We are open to collaboration and discussion.

Proof of Concept for the BLDD Tool - Part V

Last week I bumped into what I think is one of the firsts academic papers on BDD to be published in a scientific journal. We have published three technical reports in arxiv.org, but they weren't peer reviewed: Filling the Gap between Business Process Modeling and Behavior Driven Development, Mapping Business Process Modeling constructs to Behavior Driven Development Ubiquitous Language and  A Tool Stack for BDD in Python.

This paper is on applying Model Driven Development (MDD) techniques to BDD, following the principles presented in the Agile for MDD white paper. The authors use Foundational UML (fUML), which defines a basic virtual machine for UML, to create bUML, a tool to support BDD activities that automatically updates the project status.

Our approach holds some differences in relation to theirs:
1) Although the examples in the paper are too "recursive" (given-when-then is used to describe the creation of stories and scenarios) to check details of their proposal, it is obvious that they follow MDD, in fact, at some point they say "The generated code intends to be complete, with no placeholders for the developer to fill out." We do the opposite: we leave placeholders to be filled out.
However, I still need to see some more practical example to affirm this.
2) Our proposed tool is meant to run step by step, I saw no reference to this kind of feature in bUML's description.
3) They use OCL as the basis for textual modeling, we propose to use a more natural should language. We envision the use of (some) should-dsl to insert and describe conditions (logical, acceptance) for the tests. OCL is a standard, however, should-dsl is closer to the user's language.

I need some more time to check bUML - I couldn't find the software or real world examples.  What I can say now is that they follow MDD, we follow Agile, so our proposals differ in the way we intend to implement and use the tools.

BLDD, BPEL, and BPMN - a first opinion

My good colleague Charles Moller asked me about in what BLDD would differ from BPEL and BPMN.

I will start by the easier part: BPMN. According to Wikipedia (from where I copied all phrases between quotes), it is "a standard for business process modeling, and provides a graphical notation for specifying business processes in a Business Process Diagram (BPD)." Since BLDD's proposal is to use any BPM notation to write specifications (possibly complemented by a textual language), BPMN is an option to BLDD.

Going further, "The primary goal of BPMN is to provide a standard notation that is readily understandable by all business stakeholders. These business stakeholders include the business analysts who create and refine the processes, the technical developers responsible for implementing the processes, and the business managers who monitor and manage the processes. Consequently, BPMN is intended to serve as common language to bridge the communication gap that frequently occurs between business process design and implementation."
In other words, as expected, BPMN can be used as an Ubiquitous Language (or a Shared Language, as I dare to call this concept).

Moreover, BPMN also provides a mapping between the graphics of the notation to the underlying constructs of execution languages, particularly Business Process Execution Language (BPEL). BPEL is an orchestration language, in other words, it is used to specify "an executable process that involves message exchanges with other systems, such that the message exchange sequences are controlled by the orchestration designer."  Now, it is interesting to read about the relation between BPMN and BPEL:

"
Relationship of BPEL to BPMN
There is no standard graphical notation for BPEL, as the OASIS technical committee decided this was out of scope. Some vendors have invented their own notations.
(...)
Others have proposed to use a substantially different business process modeling language, namely Business Process Modeling Notation (BPMN), as a graphical front-end to capture BPEL process descriptions.
(...)
However, the development of [tools that map from BPMN to BPEL] has exposed fundamental differences between BPMN and BPEL, which make it very difficult, and in some cases impossible, to generate human-readable BPEL code from BPMN models. Even more difficult is the problem of BPMN-to-BPEL round-trip engineering: generating BPEL code from BPMN diagrams and maintaining the original BPMN model and the generated BPEL code synchronized, in the sense that any modification to one is propagated to the other.
"
This final paragraph gave me some hope that BLDD is not an in vain proposal!

BLDD is based and inspired in BDD, which in turn is influenced by Domain Driven Design (DDD). In other words, for all reasons exposed by these techniques, having all the artifacts (models, requirements, source code, tests, documentation...) readable by humans is a must have feature. Also, round-trip is totally necessary. Besides following these principles, BLDD is simpler and independent of any execution engine - although we are investigating it in environments with workflow engines. I am not saying that BPEL isn't good, I am sure that some really complex systems can take a lot of advantage of it, however, we want to have a more generic proposal, which can be implemented in any language.

I know of BPEL Script, however, it seems to be a language to describe processes, it think is not a full fledged programming language. An idea to investigate is the use of automated tests with BPELScripts and make it another proposal for BLDD (something like BPELScripts + TDD or even BDD). However, what we propose now with BLDD is to use generic languages, such as Python, Ruby, and Java, to implement all the stuff, and more, in a way readable by humans.

So, in other words, I think that BLDD and BPEL has the same goal of implementing systems based on business processes. However:
a) BPEL (i) needs a heavier machinery, (ii) already provides a series of artifacts for dealing with highly complex systems, and (iii) is well supported by industry.
b) BLDD (i) is simpler in general, (ii) is heavily based in TDD, with all its advantages, and (iii) it works with human-readable artifacts from end to end.

Which one to use in your case? I think it depends on your demands, technical knowledge, and development culture. I don't have a final answer, mainly because we need to advance on BLDD proposal and also because I prefer to use the Keep it as Simple as Possible (KISP) principle. I think there is room for BPELScript + BLDD, however, more time is needed for analyzing this proposal (any volunteers?).

Anyway, after reading this quite interesting two-year old article (some of the problems expressed in it maybe are solved right now), I have a final remark: sometimes "enterprise class solutions" become so complex that programmers abandon them in the end, such as CORBA (and J2EE at some extent).

Proof of Concept for the BLDD Tool - Part IV

One way of making a tool prove its utility is using it in scenarios where things go wrong. Figure 1 shows what happens if the code implemented is not in compliance with the requirements. The user will see a message saying that the scenario didn't pass.

Figure 1: Error generated by code not compliant to requirements

In the message in Figure 1, the error was generated because the last step was not implemented correctly: the message "9 sales awaiting to be sent" was not generated by the system, therefore, the developer must check the code and correct it. In that way it is possible to immediately identify where the system must be improved to work as expected, which is very useful when it is still in development, or when changes are needed.

Imagine that the "9 sales awaiting to be sent" message represents an improvement to this business process. I insert it on my specifications and run the simulation, so that I can identify exactly the point where the system must be modified to become compliant to the requirements. This is exactly what BDD proposes, the difference here is that we are joining a graphical representation to it and giving the user the option of running the business process step by step and see how the system is behaving.

In other words, the user can check a live process and the live system that answers to this given process. If anything goes wrong, the tool will identify what and where, making corrections easier.

It is important to note that this is a proof of concept, my idea was to launch it quickly so that more people can discuss and contribute to both the method and the tool. Besides Woped, we used Cucumber, which in turn uses BDD's Ubiquitous Language (Given-When-Then), however, as I said in this thread's previous post, we may define an UL for every BP representation and make the underlying text mirror the representation. That's our next step.

Proof of Concept for the BLDD Tool - Part III

If you are following this post thread, you may noticed that it is necessary to know BDD to understand our proposal. In order to make you spare your time and decide now if you want to follow our method, I will stop a bit and summarize the proposal.

Initially I will explain BDD very, very quickly, highlighting the points that are interesting to understand SLDD/BLDD. BDD is similar to Acceptance TDD (ATDD), and relies heavily in TDD. It bases the declaration of requirements in the triad Given-When-Then (GWT), using the generic form

Given some initial context,
When an event occurs,
Then ensure some outcome.

Some BDD tool will get the scenarios described as above and create a file with a structure similar to

@Given('some initial context')
# you write test code here that excites the code that treats the initial context
@When ('an event occurs')
# you write test code here that excites the code that treats the event
@Then('ensure some outcome')
# you write test code here that excites the code that treats the outcome

When you run your specification, if you forgot to implement some step, the tool will say that the tests didn't pass. If you implemented code that returns outcomes different of the expected, the tool will say that the tests didn't pass.

Simply, isn't it? But it works!

Your users can read the specifications and you write tests for everything, assuring that the system will produce the correct outcomes. If anything goes wrong, the tool will tell you. If changes need to be implemented, simply change the text and run the specifications, forcing errors to appear, thus indicating where the code need to be changed, facilitating your work and the effort estimation for implementing the changes.

The first time I saw BDD working (for a system developed by an associate), some two years ago, I felt impressed with its simplicity and efficiency. However, since that first time I felt uncomfortable with its text declarations... as an EIS researcher I couldn't imagine specifying a lot of complex business processes using pure text.

Our first decision at NSI was to develop a tool stack for BDD in Python, so that we could use BDD in our projects. Only in the beginning of this year we could find time to solve my discomfort with BDD's text declarations. The core idea is to provide executable documentation to EIS, in other words, the same way we "run" specifications in GWT, I would like to "run" a business process represented by a diagram. Moreover, given that a BP can have many different paths, I would like to run it step by step, with the user choosing which path to follow.

This proposal holds some similarities with Models@RunTime, however I think it is simpler, since the second seems to focus on creating self-adaptable systems, while our proposal here is directed to a "simpler" problem: making EIS development safer, simpler, faster, and cheaper (I am modest). Also we want to make our proposal usable by the common developer, without experience in Formal Methods and alike (thanks God this is a blog, can you imagine someone writing this in a scientific paper?).

When we started to discuss about this proposal, another discomfort of mine was related to the term Ubiquitous Language. I prefer to use the term Shared Language, or a language that may be a composition of other existent languages and results of an agreement between all project stakeholders, or at least between users and developers. For instance, the shared language can be a combination of some BP notation and Math notation for complex requirements, and BDD's GWT for the simpler ones.

For this specific proposal of using BPM, we call it Business Language Driven Development (BLDD). In order to avoid repetition, if you want to understand better this name discussion, please follow the posts with the Shared Language Driven Development (SLDD) labels. The original ideas are also described on our reports Filling the Gap between Business Process Modeling and Behavior Driven Development and Mapping Business Process Modeling constructs to Behavior Driven Development Ubiquitous Language.

For instance, a SLDD/BLDD tool that is based in Statechart Diagrams would generate entries more or less like this (we are still discussing how to do this exactly)

@SourceState('current state')
# excite the code that treats the exit of the current state
@Event ('event')
# excite the code that treats the event
@Transition('produces some outcome')
# excite the code that produces the expected outcome
@DestinationState('new state')
#excite the code that treats the entrance into a new state

With that in mind we can go further with the example we used in the previous posts. The next post in this thread will treat errors in the implementation of the business process.

Proof of Concept for the BLDD Tool - Part II

One thing in BDD that always bothered me is the fact that things like parallel threads are not clearly represented, you simply use ANDs in the Then clauses and that's it. This kind of situation became even more clear to me when we mapped UML Statechart Diagrams' constructs to BDD's Given-When-Then, or GWT for short (you can find this report here).

Some agilists will complain that I am trying to introduce unnecessary complexity into BDD, however, EIS are complex in general, and many times they need to implement concurrency and other sophisticated constructs. Therefore, our tests MUST verify these conditions. Given that instead of filling forms we want to use executable documentation, we must prepare our testing toolset to deal with the intrinsic system complexity, of course, having the Keep It as Simple as Possible (KISP) principle in mind.

Continuing the example that was started on the previous post, we now reached an AND-Split, making the tests automatically fill the form with the Credit Card data, and driving the validation process to a stop at the AND-Split, waiting for the user to click on it. When the user clicks on it, we see the screen on Figure 1, which shows the equivalent GWT construct for our AND-Split, being the "Wait for confirmation..." part equivalent to the upper flow of the split.

Figure 1: The equivalent GWT construct for the AND-Split

As the user clicks OK, two subsets of tests excite in parallel two different parts of the system: the one that tests the sending of an email (the lower flow of the split, not shown here since there is no user interaction in it) and the inventory-checking procedure, of which equivalent in GWT is presented by Figure 2. After the user clicks OK, the process will go forward, reaching an OR-Join, as shown in Figure 3.

Figure 2: Excitation of inventory-checking procedure

Figure 3: An OR-Join is reached

Since we are using BDD tools for implementing our proof of concept, we were forced to provide hand-written GWT constructs, however, we envision the definition of specific languages for each BP representation.  For instance, for Statechart Diagrams we can use State-Event-Action tags to map from the graphical representation to tests.

It is important to note that we don't propose to substitute GWT constructs, but to complement them, because simpler requirements can be described without the need of BP models. In other words, we envision tools that can deal with different languages, that's why the name for our general proposal is not BLDD, but SLDD - Shared Language Driven Development.

Proof of Concept for the BLDD Tool - Part I

Executable Documentation is a vital element for achieving end to end testing and thus real code quality (instead of "quality by annotation"). Understanding that BPM is an important tool for EIS development, I proposed Business Language Driven Development (BLDD) as an extension to BDD (to understand this better, including a discussion on names for this technique, please follow the links of my previous posts).

The core idea is to use some Business Process (BP) Modeling tool to run, step by step, a given process representation and, at each step, excite the associated automated tests, making the system to pop-up accordingly as the user clicks on the representation. It is the same thing we see when we use tools such as Selenium, however, in that case, we would run the tests step by step and, instead of using textual descriptions of the requirements (like in BDD), we use a BP Model.

We know that there isn't a real standard for Business Process Modeling, in fact there are various, such as UML Models, Petri Nets, BPML, derivations of these etc. We also know that many EIS are based on Workflow Engines, and each of these engines has particularities. Therefore I decided to open two fronts:

1) Use a generic BP representation: the choice was Petri Nets, because I can derive others representations from them, such as Statechart and Activity Diagrams, and also there is the "Token Game", which is the simulation of the BP execution.
2) Use some workflow engine to work on top of it. The obvious choice for us was working with ERP5, an open source ERP (FOS-ERP), based on Zope, which in turn uses a state-based Wf engine, DCWorkflow. ERP5 was an obvious choice because (i) we are partners of this project since 2002, (ii) it is open, so we have access to all its code, (iii) we like state-based workflows.


I will focus now on (1) and will be back to (2) some weeks from now.

The first step for achieving (1) was to implement a proof of concept, using some Petri Nets open source tool that would be able to run Token Games, so that we could animate our model. During last August, Rodrigo Manhães, one of my associated researchers, prepared a proof of concept based on Woped.

Using an example PN, he created a very basic application, without taking care of the application logic or GUI - the goal is to show our proposal, not discuss BPM or GUI. The screenshot in Figure 1 shows the moment when the user reaches an OR-Split and must choose which path to follow in the workflow.

Figure 1: An user reaches an OR-Split in the example process

The OR-Split in Figure 1 is implemented as a list box. Appropriate tests excite this code, as shown in Figure 2.

Figure 2: The equivalent test run for the OR-Split

Remember that tests "are by example", therefore, the user doesn't interact with the real system, instead, he/she is a spectator of the tests, launching them by choosing which path to follow in the PN. After checking if the system is implemented the way he/she expected, the user closes the GUI and then choose the path equivalent to Credit Card, making the test code to run this option, which, in turn, makes the form in Figure 3 to pop. Meanwhile in Woped, the user can see a message that shows which scenario was chosen, including the Given-When-Then equivalent instructions (Figure 4).

 Figure 3: Credit Card data form

 Figure 4: Message with equivalent GWT.

When the user clicks in OK, the flow goes to the next step, an AND-Split (Figure 5) and the tests responsible for filling the form will run, as shown in Figure 6.

 Figure 5: The process enters an AND-Split

Figure 6: Running the test for filling Credit Card data

Although we had to implement specific logic for each of the PN structures (I will start discussing this on the next post of this series), the next steps for verifying the consistency between the BP model and the system are similar.

The goal of this post was to show our basic idea: make the user run the business process and see immediately how the system is behaving. If there isn't correspondence between requirements and code (originated by process change or code wrongly implemented), errors will appear, automatically indicating where the system is not compliant to the requirements described by the model. Error situations will be shown in the next posts.

Deciding on the names for BDD extentions

On a previous post I talked about our proposal of extending Behavior Driven Development (BDD) to use other languages than then "classic" Given-When-Then (GWT), in special, but not limited to, Business Process Modeling (BPM) artifacts, such as Petri Nets, Statechart or Activity Diagrams. Today we decided at NSI to use two terms to refer to our proposal:

a) When referring to the idea of using a composition of languages, previously agreed by all stakeholders, we are going to call it Shared Language Driven Development (SLDD).
b) The specific case of using BPM + domain specific language (DSL) as an input for the BDD machinery is called Business Language Driven Development (BLDD).

In other words, SLDD is the general proposal and BLDD is the proposal for EIS. You can also interpret like this: SLDD is a generalization of BDD, and BLDD is an extension to BDD that uses BPM.

BPDD | BLDD| CLDD | SLDD, Dynamic Business Process Composition, The Future of ERP, REST, and Here We go!

The economic dynamism has been pushing a lot enterprise systems development. Some business processes must be altered and/or re-composed frequently to meet changing business conditions. Dynamic Business Process (or Service) Composition represents a set of techniques that try to provide the basis for achieving this dynamism.
Also, some say the future of ERP is to become process integration frameworks. Instead of integrated modules, ERPs would provide the means for quickly integrating disparate business processes living in different systems - in fact, this is an old Supply Chain Management need, far away from be quickly (and cheaply) fulfilled in real world applications.

Many proposals are based on the use of Intelligent Agents, Ontologies and other automated decision making supportive methods. Agents and ontologies can be very, very useful for filtrating, among hundreds or thousands of options, the few dozen that may interest to a given organization. Ontologies are very, very useful for helping finding the points of integration for these process. However, I still think that in the end, it will be a human-decision-maker and a human-programmer who will choose the best combination of processes and implement their integration respectively.

Experience says that even if the leading-edge organizations can reach the Nirvana of totally automated dynamic composition of business process, we are going to find most of smaller organizations striving to do this "manually".

Therefore, I think there is still room (and a lot of it) for developing supportive techniques for "semi-automated" Business Process Composition (BPC). And I believe our proposal of having a Business Process extension of BDD (we are deciding which name to use - BPDD, BLDD, CLDD, or SLDD - see my previous post) associated to tools like Restfulie may offer a powerful method to implement executable documentation for business applications that demand integration of disparate applications and their business processes.

I can envision a business analyst building a graphical representation of the macro-process, using a shared development language - understandable by all stakeholders (see my previous post),  and, after implemented the automated tests, checking the live application running as he/she follows (clicks) on the different paths of a workflow representation. As the macro-process changes, the tool will visually identify where and what to change accordingly so that the system becomes business compliant again. That would be a way of building a "next-generation ERP" or "next-generation Enterprise Information Systems" through a combination of visually manipulating BP representations and Test Driven Development (TDD).

Well, that was one more teaser to the matters I want to discuss here, I will go back to it latter.