How to use EIS Patterns?
When eispatterns started to be developed, the original goal was to build a set of basic concepts that could be configured to represent various business entities. However, this seemed to be very hard to accomplish, since as new behavior appears, new code is necessary, making the economy of using masks to represent new entities being not enough to justify the use of the framework.
Therefore, the framework's focus shifted to on how to represent real world entities that dynamically acquire new responsibilities. Then, a new path was followed, by defining:
-Essential abstract concepts in the framework: Node, Resource, and Movement.
-The roles for each of the abstract concepts: Node: active, responsible for executing production operations; Resource: passive, responsible for storing production resources data; and Movements: coordinative, responsible for managing the nodes while they work on resources.
-How to extend each of the concepts: Node: through Decorator Pattern; Resource: through subclassing; Movements: through configuration (better explained here).
Programming in EIS Patterns
In the project repository a super-simplified "bank system" can be found, with the intention of demonstrating how the framework can and should be used. The framework is process-centric, meaning that the development should evolve around the underlying business processes, by following these steps:
1) Represent the business process using a Process object. Ideally, this object should "absorb" the attributes and methods necessary to run as a workflow. For instance, if a state machine is used to represent business processes, the process object should have states (attributes) and transitions (methods), as well respond to specific events. Moreover, if it is decided to represent the same process using other notation, such as BPMN, this object must be transformed accordingly. We implemented Fluidity, a state machine for Python, which responds to these demands*. Process objects use simpler Movement objects to connect business operations to business processes and log their execution.
2) Define business decorators to adorn Node objects. Nodes are persons or machines which gain new responsibilities through business decorators, which in turn manipulate Nodes' input, processing and output areas. Business decorators make Nodes become the active part of the business process. Decorators must implement a decorate() method, where rules of association are defined using should-dsl. These rules ensure that a given object can be adorned by the decorator.
3) Each process activity (using a state machine: a transition or state action) must be realized by a business decorator method. These methods are called (business) operations, can be categorized, and are encapsulated by Movements**, which in turn are linked to the Process object. In that way, business decorators' methods take the responsibility of making the process work***. The same template can be used to configure different processes, using different operations to implement them.
4) Define the operations by decorating them with @operation. These operations should be able to ask for the transference of resources from/to the different nodes' areas using node's transfer() method. @operations implement the logic for each business process activity.
5) Define WorkItems, which are, in general, data bags or passive entities****. The reason for that is that in a real-world, physical production system, humans and machines transform or transport the work items using their abilities (@operations). A report is passive, being written by a human; a truck hood is passively spot-welded by a robot in a factory; a motor is passively transported from one manufacturing cell to another.
These steps are not necessarily executed in order, nor the business process is necessarily fully defined beforehand. It is envisioned a development process guided by the incrementally testing/building of the business processes. The use of templates is also a form of defining the acceptance criteria for the tests.
It is important to note that every class and every method must have a docstring, so that keyword searches can be performed by the Documenter module, which will be (soon) extended by Natural Language Processing facilities, so that requirements can be parsed and used to try to find candidate reusable elements in a given library of business decorators and work items. Complimentary, the Rule Checker is a module that runs the association rules of the business decorators found by the search, so that users can verify which of them can be used for adorning a given object. A new module will be developed to find and run acceptance tests, so that the "live system" can be used to check its fitness to a given set of requirements.
*Note 1: In comparison with workflow engines and workflow libraries, Fluidity (in conjunction with Extreme Fluidity) is innovative, because while engines force the development inside their environment and libraries force the use of their objects to implement the process, Fluidity allows the definition of template machines that can be absorbed by any Python object. In other words, you can still use your own classes, your own naming convention, your own programming style, and your own architecture, in your own environment. The "state-based behavior" is inoculated into your objects dynamically. Moreover, if you don't want to use Fluidity anymore, your objects can expel the state machine at any time.
**Note 2: Movements are of the type Transformation - which transforms a given resource in a node's processing area, and Transportations - which transports a given resource from a node to another. These concepts became conceptual classifications of movements, only Movement objects do the job.
***Note 3: Implementing business process is a two step task: first a movement is configured to log an @operation execution, and then this operation is executed.
Using a state machine notation, the property movement.activity is set to a transition or a state action, and the property movement.activity_associated_method is set to @operations. It would be easier if transitions and actions would be directly associated to @operations, however, Movements are necessary to store execution information (parameters, result, date and time) as well as to indicate the source and destination nodes. Source and destination nodes allow the mapping of the process to physical movements in the production plant. After configuration, a movement can be executed. These steps are implemented into Process objects, because they coordinate movements. This layered method improves reuse and configuration of processes and @operations.
**** Note 4: If workitems are physically aggregated to form a final product, this set of workitems must be promoted into a Kit, which can be "active" in some contexts.
Showing posts with label BDD. Show all posts
Showing posts with label BDD. Show all posts
Sunday, May 29, 2011
Sunday, February 13, 2011
Enterprise Information Systems Patterns - Part III
After my last post, the code in Github changed many times, since I was experimenting with both the basic concepts and the testing stack we are using. We decided not to write features (BDD's Given-When-Then constructs) for the configuration of Resources, Nodes, and Movements, since it would be overkill (as you can see in the sketch branch, from December 12, 2010): by reading the specs anyone can understand and check the functioning of configuration.
The comments below are based on the last implementation (January 09, 2011), which doesn't implement Path nor the parsers for the configurators. We are considering writing features for Path configurations, because Paths represent business processes, and therefore, it make sense to have high level descriptions of requirements.
Currently, the proposal is based on two base classes, from which all others derive. The first one, Configurable (previously known as Maskable), is the base for building the abstract concepts, and the second one, Configurator, is used for providing a "friend" class for each concept which is responsible for managing the configuration of the concepts' instances.
Besides the basic attributes, Configurable abstract class holds a "To Do" method, which is simply a placeholder to remember that a method for doing "basic things" with instances should be implemented by subclasses. Configurators on the other hand must be able to parse, persist, and retrieve configurations to be applied to concept's instances. They are multitons, in other words, for a given key, composed by configuration's mask and version attributes, only one configurator object may exist. Therefore, every concept class, such as Movement, has a configurator class, such as Movement_Configurator.
Configurations supply the following attributes for instances (spec files provide complete configuration examples):
-Mask: defines a specific type for the instance. Example: payment (a payment movement).
-Version: holds the version of the mask, multiple versions can be in use at the same moment. Example: monthly (monthly payment).
-Category: associates a category to a specific concept. Example: financial (monthly payment is a movement of category financial).
-Description: provides a description for the key mask+version. Example: ' a monthly payment from a organization to a person'. Note that, in this example, organization and person can also be categories.
Right now the implementation is in a very simple state, the next step is to implement Path, which is expected to reveal more requirements for making this framework more usable. For testing, we are going to use some real-world workflow implemented in ERP5, applying the mappings presented in the technical report "Mapping Business Process Modeling constructs to Behavior Driven Development Ubiquitous Language" to describe the workflow using Given-When-Then.
The comments below are based on the last implementation (January 09, 2011), which doesn't implement Path nor the parsers for the configurators. We are considering writing features for Path configurations, because Paths represent business processes, and therefore, it make sense to have high level descriptions of requirements.
Currently, the proposal is based on two base classes, from which all others derive. The first one, Configurable (previously known as Maskable), is the base for building the abstract concepts, and the second one, Configurator, is used for providing a "friend" class for each concept which is responsible for managing the configuration of the concepts' instances.
Besides the basic attributes, Configurable abstract class holds a "To Do" method, which is simply a placeholder to remember that a method for doing "basic things" with instances should be implemented by subclasses. Configurators on the other hand must be able to parse, persist, and retrieve configurations to be applied to concept's instances. They are multitons, in other words, for a given key, composed by configuration's mask and version attributes, only one configurator object may exist. Therefore, every concept class, such as Movement, has a configurator class, such as Movement_Configurator.
Configurations supply the following attributes for instances (spec files provide complete configuration examples):
-Mask: defines a specific type for the instance. Example: payment (a payment movement).
-Version: holds the version of the mask, multiple versions can be in use at the same moment. Example: monthly (monthly payment).
-Category: associates a category to a specific concept. Example: financial (monthly payment is a movement of category financial).
-Description: provides a description for the key mask+version. Example: ' a monthly payment from a organization to a person'. Note that, in this example, organization and person can also be categories.
Right now the implementation is in a very simple state, the next step is to implement Path, which is expected to reveal more requirements for making this framework more usable. For testing, we are going to use some real-world workflow implemented in ERP5, applying the mappings presented in the technical report "Mapping Business Process Modeling constructs to Behavior Driven Development Ubiquitous Language" to describe the workflow using Given-When-Then.
Wednesday, December 15, 2010
Enterprise Information Systems Patterns - Part II
The project in Github reached an initial implementation, which I separated in a branch named sketch, given that it is mixing configuration and instantiation. The master branch now is in accordance with the proposal, which I continue to discuss below.
Remember that the core idea is to use the concepts as Lego parts. This mean avoiding using subclasses in general, but masking the abstract concepts through configuration. After configuration, I need to implement specific functionalities for the concepts. These functionalities will be implemented in two places:
a) At instances of concrete concepts, by defining specific behavior for
different contexts (paths).
b) At path objects, in the form of coordination code, which will make the path''s
movements collaborate to realize a business process.
In that way, we are going to have a two-phased development/customization process:
a) Configuration: defines descriptors, which represent concrete uses of the
abstract concepts. Descriptors list the types used to transform the abstract
concepts into concrete ones. Configuration is done through a Domain Specific
Language (DSL), having each concept a template text to be used for its
configuration. In the future, we expect to define a proper grammar for this DSL.
b) Implementation: uses descriptors to make the concrete concepts instantiable
and implement the specific code related to their concrete use. Each concept has
a callable object with a proper name, which is defined during the implementation
of user stories.
Thus, in a first moment, a domain specialist will configure concrete concepts
using the DSL. Configurations are reused during the implementation, when
user stories instantiate and define the specific behavior of the concrete
concepts.
Right now, we have implemented Resource and Node, as well as the Maskable superclass, which will serve all concepts. I can say that the project is in the configuration phase yet, meaning that we haven't implemented any code for "using" the concepts in some business process.
Regarding the DSL used for configuration, we have to investigate its relation with our BLDD proposal. Remember that this proposal is used for implementing business processes, while the current DSL is used for configuring objects, which occurs before implementation of the processes. Maybe an execution DSL which uses the same terms used for configuration will grow from this investigation.
Remember that the core idea is to use the concepts as Lego parts. This mean avoiding using subclasses in general, but masking the abstract concepts through configuration. After configuration, I need to implement specific functionalities for the concepts. These functionalities will be implemented in two places:
a) At instances of concrete concepts, by defining specific behavior for
different contexts (paths).
b) At path objects, in the form of coordination code, which will make the path''s
movements collaborate to realize a business process.
In that way, we are going to have a two-phased development/customization process:
a) Configuration: defines descriptors, which represent concrete uses of the
abstract concepts. Descriptors list the types used to transform the abstract
concepts into concrete ones. Configuration is done through a Domain Specific
Language (DSL), having each concept a template text to be used for its
configuration. In the future, we expect to define a proper grammar for this DSL.
b) Implementation: uses descriptors to make the concrete concepts instantiable
and implement the specific code related to their concrete use. Each concept has
a callable object with a proper name, which is defined during the implementation
of user stories.
Thus, in a first moment, a domain specialist will configure concrete concepts
using the DSL. Configurations are reused during the implementation, when
user stories instantiate and define the specific behavior of the concrete
concepts.
Right now, we have implemented Resource and Node, as well as the Maskable superclass, which will serve all concepts. I can say that the project is in the configuration phase yet, meaning that we haven't implemented any code for "using" the concepts in some business process.
Regarding the DSL used for configuration, we have to investigate its relation with our BLDD proposal. Remember that this proposal is used for implementing business processes, while the current DSL is used for configuring objects, which occurs before implementation of the processes. Maybe an execution DSL which uses the same terms used for configuration will grow from this investigation.
Wednesday, December 1, 2010
Enterprise Information Systems Patterns - Part I
This post aims to start the discussion on design patterns for EIS. We have just started a Github project, which right now is just a rough sketch, however, it already shows some basic ideas, implemented in Python with BDD (specloud, should-dsl, and Lettuce). By checking the features (*.feature files) you can understand the patterns requirements, and by checking the specs (*_spec.py files) you can understand how the code implements them.
We decided to start by revisiting ERP5 concepts, separating them from ERP5's underlying platform, Zope, and, in some cases, even reinterpreting them. The concepts are the following:
Concept 1: Resource
A resource is anything that is used for production. It can be a material, money,
machine time, or even human skills.
Concept 2: Node
Is something that transforms resources. For instance, a machine, a factory, a
bank account.
Concept 3: Movement is a movement of any Resource from one Node to another.
Concept 4: Path*
It describes how Movements are used to implement business processes.
ERP5 also has the concept of Item, an instance of a Resource. Until this point, we
decided not to use this, otherwise it would be necessary to create classes for
instancing the other three abstract concepts.
The core idea is to use the concepts as Lego parts. This mean not using
subclasses in general, but masking the abstract concepts through
configuration. For instance, a Movement is firstly configured as a concrete
movement, such as transferring goods from supplier to customer. After that, it
can be instantiated. Of course, at some point extra coding is necessary, and we believe that methods for implementing specific algorithms will emerge
on the coordinator element, the Path instances.
We therefore define two new "patterns": Mask, which means to define an abstract concept and turn it into a concrete concept by a proper configuration, and Coordinator, which is an object (also maskable) that coordinates others to realize a given business process. Maybe these patterns already exist with others names and/or similar purposes, however, we will check this more carefully as the project goes forward (we are really iterative & incremental).
The motivation for this project is that people like to say that frameworks are like Lego, but when you need to subclass their concepts, instead of using them as is, I think this is not the "Lego way" of doing things. We only use masks to provide useful labels for users, but the intention is not to extend the original concepts. When I mask a Movement as a Sale, I am only setting its mask attribute to 'sale'.
The only concept that will be extended is Path (nothing is perfect!), since it will be used to implement the specific algorithms necessary to appropriately integrate movements - realizing the business process.
We probably won't supply something really useful for programming, because we are not considering platform issues, therefore, if you need a ready to use ERP framework in Python, check ERP5. The intention is to discuss how to build highly configurable frameworks, instead of proving a real framework. After we get the basic configuration machinery done, we are going to use BLDD to implement didactic cases: from the business process modeling to coding and testing.
(The Change Log maps this series of posts to commits in GitHub)
We decided to start by revisiting ERP5 concepts, separating them from ERP5's underlying platform, Zope, and, in some cases, even reinterpreting them. The concepts are the following:
Concept 1: Resource
A resource is anything that is used for production. It can be a material, money,
machine time, or even human skills.
Concept 2: Node
Is something that transforms resources. For instance, a machine, a factory, a
bank account.
Concept 3: Movement is a movement of any Resource from one Node to another.
Concept 4: Path*
It describes how Movements are used to implement business processes.
ERP5 also has the concept of Item, an instance of a Resource. Until this point, we
decided not to use this, otherwise it would be necessary to create classes for
instancing the other three abstract concepts.
The core idea is to use the concepts as Lego parts. This mean not using
subclasses in general, but masking the abstract concepts through
configuration. For instance, a Movement is firstly configured as a concrete
movement, such as transferring goods from supplier to customer. After that, it
can be instantiated. Of course, at some point extra coding is necessary, and we believe that methods for implementing specific algorithms will emerge
on the coordinator element, the Path instances.
We therefore define two new "patterns": Mask, which means to define an abstract concept and turn it into a concrete concept by a proper configuration, and Coordinator, which is an object (also maskable) that coordinates others to realize a given business process. Maybe these patterns already exist with others names and/or similar purposes, however, we will check this more carefully as the project goes forward (we are really iterative & incremental).
The motivation for this project is that people like to say that frameworks are like Lego, but when you need to subclass their concepts, instead of using them as is, I think this is not the "Lego way" of doing things. We only use masks to provide useful labels for users, but the intention is not to extend the original concepts. When I mask a Movement as a Sale, I am only setting its mask attribute to 'sale'.
The only concept that will be extended is Path (nothing is perfect!), since it will be used to implement the specific algorithms necessary to appropriately integrate movements - realizing the business process.
We probably won't supply something really useful for programming, because we are not considering platform issues, therefore, if you need a ready to use ERP framework in Python, check ERP5. The intention is to discuss how to build highly configurable frameworks, instead of proving a real framework. After we get the basic configuration machinery done, we are going to use BLDD to implement didactic cases: from the business process modeling to coding and testing.
(The Change Log maps this series of posts to commits in GitHub)
Saturday, November 27, 2010
Proof of Concept for the BLDD Tool - Part VI
We have released a compilation of the posts with some more explanations on our proposal for the Business Language Driven Development (BLDD) draft, the technical report can be found here.
There is a lot of work to do on top of BLDD, such as:
- Defining in more detail the use of a companion textual Domain Specific Language (DSL). Probably with support of a Rule Engine.
- Implement an usable tool, we are investigating an extension of Drools for this purpose. Drools seems to be a good choice because it is BPMN compatible, implements a Rule Engine, and an Eclipse plugin. And, of course, it is open source.
- Implement BLDD on top of an ERP, our choice is ERP5, for the reasons listed on the first post of this series. This case study should show the applicability of the proposal.
You can find more "future work" in the report. We are open to collaboration and discussion.
There is a lot of work to do on top of BLDD, such as:
- Defining in more detail the use of a companion textual Domain Specific Language (DSL). Probably with support of a Rule Engine.
- Implement an usable tool, we are investigating an extension of Drools for this purpose. Drools seems to be a good choice because it is BPMN compatible, implements a Rule Engine, and an Eclipse plugin. And, of course, it is open source.
- Implement BLDD on top of an ERP, our choice is ERP5, for the reasons listed on the first post of this series. This case study should show the applicability of the proposal.
You can find more "future work" in the report. We are open to collaboration and discussion.
Wednesday, November 24, 2010
Assorted Thoughts on Agilism X Traditionalism - Part III
This blog focuses on Free/Open Source (FOSS) and Agile/Lean (AL) for Enterprise Information Systems (EIS). NSI has been using Free/Open Source since its creation, in 2002 (I use since 1998), however, AL is relatively new to us. I think it is interesting to tell how we "realized" AL.
Our Turning Point
In a broader sense, we started using automated testing in 2005, when we were developing a system for vibration analysis in LabView, which uses G, a visual language for Virtual Instrumentation. For us to test it, we had to simulate working gas turbines, which means to generate matrices with thousands of lines and dozens of columns, representing the vibration signals produced by dozens of sensors. In fact, we had to develop a complete simulation module that excited the analysis algorithms. Curiously, parts of this module were created before the algorithms, because we needed to understand better the behavior of a real equipment, in a kind of "proto" test-first way. Also, we filled vectors and matrices with trustful results obtained from laboratory's hardware and software, and automatically compared with our results, to check our algorithms' accuracy.
In the realm of Enterprise Information Systems, our use of automated testing was initiated only in 2007, when we started two important Enterprise Content Management (ECM) projects in parallel. We started to automate the "black box" tests - because we had no patience to simulate dozens of time the same users actions - and load tests, for obvious reasons. However, our unit tests weren't disciplined and sometimes were hampered by the platform we used. Obviously, the acceptance tests started to find wrong behavior in the software, given the lack of complete unit testing. By that time (with results published in 2007/2008), and in parallel, we developed a hybrid development process for the ERP5 system that used MDD (including a nice set of supportive tools) with automated testing.
The year of 2008 represented our turning point, because of three factors:
(i) We realized that our MDD-based process for ERP5 was sound, however, our industrial partners needed something leaner, and MDD didn't made much sense when developing on top of a highly reusable framework. Moreover, refactoring made modeling a waste in general. We realized in practice that a good design can be done on top of code, instead of models.
(ii) I managed to bring to the team an associate researcher who was (is) a very competent developer, and was (and still is) experiencing very good results by using Agile Methods. He made me remember of my Master Thesis (1997), which proposed a hybrid push-pull production planning and control method: instead of using complicated production programming algorithms, which needed very accurate (and usually inexistent) informations, Kanbans were used to smooth the production flow.
(iii) I read the book Ghost Force - The Secret History of SAS, written by Ken Connor, and realized that trying to use conventional methods for managing unconventional projects and teams was stupid. After exposing my impressions on some team coordination techniques narrated by the book, we immediately adopted two principles*: a) one man, one vote (the Chinese Parliament) and b) the one that holds more knowledge in the specific mission, leads it, doesn't matter his/her rank. Of course, we already knew that the most valuable asset of any team is its human resources.
*strangely my team didn't accept things like parashooting training, as well as other really interesting techniques presented in the book ;-)
So, I can say that we woke up for agile techniques as a whole in 2007/2008. In 2008 we started to use a Scrumban process, a mixture of Scrum for planning with kanbans for programming the production. Things weren't enforced, we realized that disciplined automated testing and Scrumban was superior for our environment, and that's it. When I say we, I am referring to everyone, from vocational to master students and researchers (one man, one vote, doesn't matter the rank...). In 2009 we started to use BDD, actually, we developed a tool stack for BDD in Python to serve to our projects.
In other words, although we are a small organization (circa 30 developers in different projects, divided into development cells, plus some 10 aggregated people), we came from a traditional way of thinking (but we never were bureaucrats) that in fact never worked well for us. In other words, we use AL because we realized in practice that it is better for us, not because it is fashionable. An interesting point is that our last paper on MDD + Automated Testing was cited by a very well known Software Engineering researcher as one of the references for modeling EIS.
We studied and tested the techniques by ourselves, using the Chinese Parliament and The Best in the Mission is The Leader principles to put them in practice. We did right and wrong things, adapted methods, mixed techniques, created tools for BDD, for improving programming productivity, for automated testing, for kanban control, and many others, only to cite a few, and we also used other people's tools. Basically, we discussed a lot. In other words, instead of following some manual, or standard, or course, or pursuing some certification, we created project management knowledge! A good result of this is that all five students that left our research group this year are working for innovative organizations, with salaries higher than the average. Moreover, they got their positions in highly competitive markets, including Europe. They were all hired in the same way: staying some days in the organizations, programming with the teams and taking part of meetings.
An intimate commentary: now I feel totally comfortable to teach project management and tell my daily experiences. Before that I had to teach techniques that didn't work for us, but I had to insist on teaching them because the books said that they worked - while feeling as an incompetent project manager. Now I can see that it doesn't work because this type of process cannot deal well with uncertainty in general (maybe really I am not a good project manager, however, now thing work well for us anyway).
A last commentary: although we are part of a teaching & research organization, we've been developing serious projects with companies and government agencies for years. When I say serious, I mean projects with deadlines, fixed budgets, lots of users, need of continuous support and evolution, and that are expected to aggregate value to the users' organizations. Every single software we developed since 2002 is/was either used internally - in our development process - or by our industrial and government partners. Moreover, since we develop free software, our products are also used by other people and organizations besides our formal partners.
Our Turning Point
In a broader sense, we started using automated testing in 2005, when we were developing a system for vibration analysis in LabView, which uses G, a visual language for Virtual Instrumentation. For us to test it, we had to simulate working gas turbines, which means to generate matrices with thousands of lines and dozens of columns, representing the vibration signals produced by dozens of sensors. In fact, we had to develop a complete simulation module that excited the analysis algorithms. Curiously, parts of this module were created before the algorithms, because we needed to understand better the behavior of a real equipment, in a kind of "proto" test-first way. Also, we filled vectors and matrices with trustful results obtained from laboratory's hardware and software, and automatically compared with our results, to check our algorithms' accuracy.
In the realm of Enterprise Information Systems, our use of automated testing was initiated only in 2007, when we started two important Enterprise Content Management (ECM) projects in parallel. We started to automate the "black box" tests - because we had no patience to simulate dozens of time the same users actions - and load tests, for obvious reasons. However, our unit tests weren't disciplined and sometimes were hampered by the platform we used. Obviously, the acceptance tests started to find wrong behavior in the software, given the lack of complete unit testing. By that time (with results published in 2007/2008), and in parallel, we developed a hybrid development process for the ERP5 system that used MDD (including a nice set of supportive tools) with automated testing.
The year of 2008 represented our turning point, because of three factors:
(i) We realized that our MDD-based process for ERP5 was sound, however, our industrial partners needed something leaner, and MDD didn't made much sense when developing on top of a highly reusable framework. Moreover, refactoring made modeling a waste in general. We realized in practice that a good design can be done on top of code, instead of models.
(ii) I managed to bring to the team an associate researcher who was (is) a very competent developer, and was (and still is) experiencing very good results by using Agile Methods. He made me remember of my Master Thesis (1997), which proposed a hybrid push-pull production planning and control method: instead of using complicated production programming algorithms, which needed very accurate (and usually inexistent) informations, Kanbans were used to smooth the production flow.
(iii) I read the book Ghost Force - The Secret History of SAS, written by Ken Connor, and realized that trying to use conventional methods for managing unconventional projects and teams was stupid. After exposing my impressions on some team coordination techniques narrated by the book, we immediately adopted two principles*: a) one man, one vote (the Chinese Parliament) and b) the one that holds more knowledge in the specific mission, leads it, doesn't matter his/her rank. Of course, we already knew that the most valuable asset of any team is its human resources.
*strangely my team didn't accept things like parashooting training, as well as other really interesting techniques presented in the book ;-)
So, I can say that we woke up for agile techniques as a whole in 2007/2008. In 2008 we started to use a Scrumban process, a mixture of Scrum for planning with kanbans for programming the production. Things weren't enforced, we realized that disciplined automated testing and Scrumban was superior for our environment, and that's it. When I say we, I am referring to everyone, from vocational to master students and researchers (one man, one vote, doesn't matter the rank...). In 2009 we started to use BDD, actually, we developed a tool stack for BDD in Python to serve to our projects.
In other words, although we are a small organization (circa 30 developers in different projects, divided into development cells, plus some 10 aggregated people), we came from a traditional way of thinking (but we never were bureaucrats) that in fact never worked well for us. In other words, we use AL because we realized in practice that it is better for us, not because it is fashionable. An interesting point is that our last paper on MDD + Automated Testing was cited by a very well known Software Engineering researcher as one of the references for modeling EIS.
We studied and tested the techniques by ourselves, using the Chinese Parliament and The Best in the Mission is The Leader principles to put them in practice. We did right and wrong things, adapted methods, mixed techniques, created tools for BDD, for improving programming productivity, for automated testing, for kanban control, and many others, only to cite a few, and we also used other people's tools. Basically, we discussed a lot. In other words, instead of following some manual, or standard, or course, or pursuing some certification, we created project management knowledge! A good result of this is that all five students that left our research group this year are working for innovative organizations, with salaries higher than the average. Moreover, they got their positions in highly competitive markets, including Europe. They were all hired in the same way: staying some days in the organizations, programming with the teams and taking part of meetings.
An intimate commentary: now I feel totally comfortable to teach project management and tell my daily experiences. Before that I had to teach techniques that didn't work for us, but I had to insist on teaching them because the books said that they worked - while feeling as an incompetent project manager. Now I can see that it doesn't work because this type of process cannot deal well with uncertainty in general (maybe really I am not a good project manager, however, now thing work well for us anyway).
A last commentary: although we are part of a teaching & research organization, we've been developing serious projects with companies and government agencies for years. When I say serious, I mean projects with deadlines, fixed budgets, lots of users, need of continuous support and evolution, and that are expected to aggregate value to the users' organizations. Every single software we developed since 2002 is/was either used internally - in our development process - or by our industrial and government partners. Moreover, since we develop free software, our products are also used by other people and organizations besides our formal partners.
Saturday, October 9, 2010
Proof of Concept for the BLDD Tool - Part V
Last week I bumped into what I think is one of the firsts academic papers on BDD to be published in a scientific journal. We have published three technical reports in arxiv.org, but they weren't peer reviewed: Filling the Gap between Business Process Modeling and Behavior Driven Development, Mapping Business Process Modeling constructs to Behavior Driven Development Ubiquitous Language and A Tool Stack for BDD in Python.
This paper is on applying Model Driven Development (MDD) techniques to BDD, following the principles presented in the Agile for MDD white paper. The authors use Foundational UML (fUML), which defines a basic virtual machine for UML, to create bUML, a tool to support BDD activities that automatically updates the project status.
Our approach holds some differences in relation to theirs:
1) Although the examples in the paper are too "recursive" (given-when-then is used to describe the creation of stories and scenarios) to check details of their proposal, it is obvious that they follow MDD, in fact, at some point they say "The generated code intends to be complete, with no placeholders for the developer to fill out." We do the opposite: we leave placeholders to be filled out.
However, I still need to see some more practical example to affirm this.
2) Our proposed tool is meant to run step by step, I saw no reference to this kind of feature in bUML's description.
3) They use OCL as the basis for textual modeling, we propose to use a more natural should language. We envision the use of (some) should-dsl to insert and describe conditions (logical, acceptance) for the tests. OCL is a standard, however, should-dsl is closer to the user's language.
I need some more time to check bUML - I couldn't find the software or real world examples. What I can say now is that they follow MDD, we follow Agile, so our proposals differ in the way we intend to implement and use the tools.
This paper is on applying Model Driven Development (MDD) techniques to BDD, following the principles presented in the Agile for MDD white paper. The authors use Foundational UML (fUML), which defines a basic virtual machine for UML, to create bUML, a tool to support BDD activities that automatically updates the project status.
Our approach holds some differences in relation to theirs:
1) Although the examples in the paper are too "recursive" (given-when-then is used to describe the creation of stories and scenarios) to check details of their proposal, it is obvious that they follow MDD, in fact, at some point they say "The generated code intends to be complete, with no placeholders for the developer to fill out." We do the opposite: we leave placeholders to be filled out.
However, I still need to see some more practical example to affirm this.
2) Our proposed tool is meant to run step by step, I saw no reference to this kind of feature in bUML's description.
3) They use OCL as the basis for textual modeling, we propose to use a more natural should language. We envision the use of (some) should-dsl to insert and describe conditions (logical, acceptance) for the tests. OCL is a standard, however, should-dsl is closer to the user's language.
I need some more time to check bUML - I couldn't find the software or real world examples. What I can say now is that they follow MDD, we follow Agile, so our proposals differ in the way we intend to implement and use the tools.
Tuesday, September 21, 2010
Proof of Concept for the BLDD Tool - Part IV
One way of making a tool prove its utility is using it in scenarios where things go wrong. Figure 1 shows what happens if the code implemented is not in compliance with the requirements. The user will see a message saying that the scenario didn't pass.
In the message in Figure 1, the error was generated because the last step was not implemented correctly: the message "9 sales awaiting to be sent" was not generated by the system, therefore, the developer must check the code and correct it. In that way it is possible to immediately identify where the system must be improved to work as expected, which is very useful when it is still in development, or when changes are needed.
Imagine that the "9 sales awaiting to be sent" message represents an improvement to this business process. I insert it on my specifications and run the simulation, so that I can identify exactly the point where the system must be modified to become compliant to the requirements. This is exactly what BDD proposes, the difference here is that we are joining a graphical representation to it and giving the user the option of running the business process step by step and see how the system is behaving.
In other words, the user can check a live process and the live system that answers to this given process. If anything goes wrong, the tool will identify what and where, making corrections easier.
It is important to note that this is a proof of concept, my idea was to launch it quickly so that more people can discuss and contribute to both the method and the tool. Besides Woped, we used Cucumber, which in turn uses BDD's Ubiquitous Language (Given-When-Then), however, as I said in this thread's previous post, we may define an UL for every BP representation and make the underlying text mirror the representation. That's our next step.
Figure 1: Error generated by code not compliant to requirements
Imagine that the "9 sales awaiting to be sent" message represents an improvement to this business process. I insert it on my specifications and run the simulation, so that I can identify exactly the point where the system must be modified to become compliant to the requirements. This is exactly what BDD proposes, the difference here is that we are joining a graphical representation to it and giving the user the option of running the business process step by step and see how the system is behaving.
In other words, the user can check a live process and the live system that answers to this given process. If anything goes wrong, the tool will identify what and where, making corrections easier.
It is important to note that this is a proof of concept, my idea was to launch it quickly so that more people can discuss and contribute to both the method and the tool. Besides Woped, we used Cucumber, which in turn uses BDD's Ubiquitous Language (Given-When-Then), however, as I said in this thread's previous post, we may define an UL for every BP representation and make the underlying text mirror the representation. That's our next step.
Wednesday, September 15, 2010
Proof of Concept for the BLDD Tool - Part III
If you are following this post thread, you may noticed that it is necessary to know BDD to understand our proposal. In order to make you spare your time and decide now if you want to follow our method, I will stop a bit and summarize the proposal.
Initially I will explain BDD very, very quickly, highlighting the points that are interesting to understand SLDD/BLDD. BDD is similar to Acceptance TDD (ATDD), and relies heavily in TDD. It bases the declaration of requirements in the triad Given-When-Then (GWT), using the generic form
Given some initial context,
When an event occurs,
Then ensure some outcome.
Some BDD tool will get the scenarios described as above and create a file with a structure similar to
@Given('some initial context')
# you write test code here that excites the code that treats the initial context
@When ('an event occurs')
# you write test code here that excites the code that treats the event
@Then('ensure some outcome')
# you write test code here that excites the code that treats the outcome
When you run your specification, if you forgot to implement some step, the tool will say that the tests didn't pass. If you implemented code that returns outcomes different of the expected, the tool will say that the tests didn't pass.
Simply, isn't it? But it works!
Your users can read the specifications and you write tests for everything, assuring that the system will produce the correct outcomes. If anything goes wrong, the tool will tell you. If changes need to be implemented, simply change the text and run the specifications, forcing errors to appear, thus indicating where the code need to be changed, facilitating your work and the effort estimation for implementing the changes.
The first time I saw BDD working (for a system developed by an associate), some two years ago, I felt impressed with its simplicity and efficiency. However, since that first time I felt uncomfortable with its text declarations... as an EIS researcher I couldn't imagine specifying a lot of complex business processes using pure text.
Our first decision at NSI was to develop a tool stack for BDD in Python, so that we could use BDD in our projects. Only in the beginning of this year we could find time to solve my discomfort with BDD's text declarations. The core idea is to provide executable documentation to EIS, in other words, the same way we "run" specifications in GWT, I would like to "run" a business process represented by a diagram. Moreover, given that a BP can have many different paths, I would like to run it step by step, with the user choosing which path to follow.
This proposal holds some similarities with Models@RunTime, however I think it is simpler, since the second seems to focus on creating self-adaptable systems, while our proposal here is directed to a "simpler" problem: making EIS development safer, simpler, faster, and cheaper (I am modest). Also we want to make our proposal usable by the common developer, without experience in Formal Methods and alike (thanks God this is a blog, can you imagine someone writing this in a scientific paper?).
When we started to discuss about this proposal, another discomfort of mine was related to the term Ubiquitous Language. I prefer to use the term Shared Language, or a language that may be a composition of other existent languages and results of an agreement between all project stakeholders, or at least between users and developers. For instance, the shared language can be a combination of some BP notation and Math notation for complex requirements, and BDD's GWT for the simpler ones.
For this specific proposal of using BPM, we call it Business Language Driven Development (BLDD). In order to avoid repetition, if you want to understand better this name discussion, please follow the posts with the Shared Language Driven Development (SLDD) labels. The original ideas are also described on our reports Filling the Gap between Business Process Modeling and Behavior Driven Development and Mapping Business Process Modeling constructs to Behavior Driven Development Ubiquitous Language.
For instance, a SLDD/BLDD tool that is based in Statechart Diagrams would generate entries more or less like this (we are still discussing how to do this exactly)
@SourceState('current state')
# excite the code that treats the exit of the current state
@Event ('event')
# excite the code that treats the event
@Transition('produces some outcome')
# excite the code that produces the expected outcome
@DestinationState('new state')
#excite the code that treats the entrance into a new state
With that in mind we can go further with the example we used in the previous posts. The next post in this thread will treat errors in the implementation of the business process.
Initially I will explain BDD very, very quickly, highlighting the points that are interesting to understand SLDD/BLDD. BDD is similar to Acceptance TDD (ATDD), and relies heavily in TDD. It bases the declaration of requirements in the triad Given-When-Then (GWT), using the generic form
Given some initial context,
When an event occurs,
Then ensure some outcome.
Some BDD tool will get the scenarios described as above and create a file with a structure similar to
@Given('some initial context')
# you write test code here that excites the code that treats the initial context
@When ('an event occurs')
# you write test code here that excites the code that treats the event
@Then('ensure some outcome')
# you write test code here that excites the code that treats the outcome
When you run your specification, if you forgot to implement some step, the tool will say that the tests didn't pass. If you implemented code that returns outcomes different of the expected, the tool will say that the tests didn't pass.
Simply, isn't it? But it works!
Your users can read the specifications and you write tests for everything, assuring that the system will produce the correct outcomes. If anything goes wrong, the tool will tell you. If changes need to be implemented, simply change the text and run the specifications, forcing errors to appear, thus indicating where the code need to be changed, facilitating your work and the effort estimation for implementing the changes.
The first time I saw BDD working (for a system developed by an associate), some two years ago, I felt impressed with its simplicity and efficiency. However, since that first time I felt uncomfortable with its text declarations... as an EIS researcher I couldn't imagine specifying a lot of complex business processes using pure text.
Our first decision at NSI was to develop a tool stack for BDD in Python, so that we could use BDD in our projects. Only in the beginning of this year we could find time to solve my discomfort with BDD's text declarations. The core idea is to provide executable documentation to EIS, in other words, the same way we "run" specifications in GWT, I would like to "run" a business process represented by a diagram. Moreover, given that a BP can have many different paths, I would like to run it step by step, with the user choosing which path to follow.
This proposal holds some similarities with Models@RunTime, however I think it is simpler, since the second seems to focus on creating self-adaptable systems, while our proposal here is directed to a "simpler" problem: making EIS development safer, simpler, faster, and cheaper (I am modest). Also we want to make our proposal usable by the common developer, without experience in Formal Methods and alike (thanks God this is a blog, can you imagine someone writing this in a scientific paper?).
When we started to discuss about this proposal, another discomfort of mine was related to the term Ubiquitous Language. I prefer to use the term Shared Language, or a language that may be a composition of other existent languages and results of an agreement between all project stakeholders, or at least between users and developers. For instance, the shared language can be a combination of some BP notation and Math notation for complex requirements, and BDD's GWT for the simpler ones.
For this specific proposal of using BPM, we call it Business Language Driven Development (BLDD). In order to avoid repetition, if you want to understand better this name discussion, please follow the posts with the Shared Language Driven Development (SLDD) labels. The original ideas are also described on our reports Filling the Gap between Business Process Modeling and Behavior Driven Development and Mapping Business Process Modeling constructs to Behavior Driven Development Ubiquitous Language.
For instance, a SLDD/BLDD tool that is based in Statechart Diagrams would generate entries more or less like this (we are still discussing how to do this exactly)
@SourceState('current state')
# excite the code that treats the exit of the current state
@Event ('event')
# excite the code that treats the event
@Transition('produces some outcome')
# excite the code that produces the expected outcome
@DestinationState('new state')
#excite the code that treats the entrance into a new state
With that in mind we can go further with the example we used in the previous posts. The next post in this thread will treat errors in the implementation of the business process.
Proof of Concept for the BLDD Tool - Part II
One thing in BDD that always bothered me is the fact that things like parallel threads are not clearly represented, you simply use ANDs in the Then clauses and that's it. This kind of situation became even more clear to me when we mapped UML Statechart Diagrams' constructs to BDD's Given-When-Then, or GWT for short (you can find this report here).
Some agilists will complain that I am trying to introduce unnecessary complexity into BDD, however, EIS are complex in general, and many times they need to implement concurrency and other sophisticated constructs. Therefore, our tests MUST verify these conditions. Given that instead of filling forms we want to use executable documentation, we must prepare our testing toolset to deal with the intrinsic system complexity, of course, having the Keep It as Simple as Possible (KISP) principle in mind.
Continuing the example that was started on the previous post, we now reached an AND-Split, making the tests automatically fill the form with the Credit Card data, and driving the validation process to a stop at the AND-Split, waiting for the user to click on it. When the user clicks on it, we see the screen on Figure 1, which shows the equivalent GWT construct for our AND-Split, being the "Wait for confirmation..." part equivalent to the upper flow of the split.
Some agilists will complain that I am trying to introduce unnecessary complexity into BDD, however, EIS are complex in general, and many times they need to implement concurrency and other sophisticated constructs. Therefore, our tests MUST verify these conditions. Given that instead of filling forms we want to use executable documentation, we must prepare our testing toolset to deal with the intrinsic system complexity, of course, having the Keep It as Simple as Possible (KISP) principle in mind.
Continuing the example that was started on the previous post, we now reached an AND-Split, making the tests automatically fill the form with the Credit Card data, and driving the validation process to a stop at the AND-Split, waiting for the user to click on it. When the user clicks on it, we see the screen on Figure 1, which shows the equivalent GWT construct for our AND-Split, being the "Wait for confirmation..." part equivalent to the upper flow of the split.
Figure 1: The equivalent GWT construct for the AND-Split
As the user clicks OK, two subsets of tests excite in parallel two different parts of the system: the one that tests the sending of an email (the lower flow of the split, not shown here since there is no user interaction in it) and the inventory-checking procedure, of which equivalent in GWT is presented by Figure 2. After the user clicks OK, the process will go forward, reaching an OR-Join, as shown in Figure 3.
Figure 2: Excitation of inventory-checking procedure
Figure 3: An OR-Join is reached
Since we are using BDD tools for implementing our proof of concept, we were forced to provide hand-written GWT constructs, however, we envision the definition of specific languages for each BP representation. For instance, for Statechart Diagrams we can use State-Event-Action tags to map from the graphical representation to tests.
It is important to note that we don't propose to substitute GWT constructs, but to complement them, because simpler requirements can be described without the need of BP models. In other words, we envision tools that can deal with different languages, that's why the name for our general proposal is not BLDD, but SLDD - Shared Language Driven Development.
Tuesday, September 14, 2010
Proof of Concept for the BLDD Tool - Part I
Executable Documentation is a vital element for achieving end to end testing and thus real code quality (instead of "quality by annotation"). Understanding that BPM is an important tool for EIS development, I proposed Business Language Driven Development (BLDD) as an extension to BDD (to understand this better, including a discussion on names for this technique, please follow the links of my previous posts).
The core idea is to use some Business Process (BP) Modeling tool to run, step by step, a given process representation and, at each step, excite the associated automated tests, making the system to pop-up accordingly as the user clicks on the representation. It is the same thing we see when we use tools such as Selenium, however, in that case, we would run the tests step by step and, instead of using textual descriptions of the requirements (like in BDD), we use a BP Model.
We know that there isn't a real standard for Business Process Modeling, in fact there are various, such as UML Models, Petri Nets, BPML, derivations of these etc. We also know that many EIS are based on Workflow Engines, and each of these engines has particularities. Therefore I decided to open two fronts:
1) Use a generic BP representation: the choice was Petri Nets, because I can derive others representations from them, such as Statechart and Activity Diagrams, and also there is the "Token Game", which is the simulation of the BP execution.
2) Use some workflow engine to work on top of it. The obvious choice for us was working with ERP5, an open source ERP (FOS-ERP), based on Zope, which in turn uses a state-based Wf engine, DCWorkflow. ERP5 was an obvious choice because (i) we are partners of this project since 2002, (ii) it is open, so we have access to all its code, (iii) we like state-based workflows.
I will focus now on (1) and will be back to (2) some weeks from now.
The first step for achieving (1) was to implement a proof of concept, using some Petri Nets open source tool that would be able to run Token Games, so that we could animate our model. During last August, Rodrigo Manhães, one of my associated researchers, prepared a proof of concept based on Woped.
Using an example PN, he created a very basic application, without taking care of the application logic or GUI - the goal is to show our proposal, not discuss BPM or GUI. The screenshot in Figure 1 shows the moment when the user reaches an OR-Split and must choose which path to follow in the workflow.
Remember that tests "are by example", therefore, the user doesn't interact with the real system, instead, he/she is a spectator of the tests, launching them by choosing which path to follow in the PN. After checking if the system is implemented the way he/she expected, the user closes the GUI and then choose the path equivalent to Credit Card, making the test code to run this option, which, in turn, makes the form in Figure 3 to pop. Meanwhile in Woped, the user can see a message that shows which scenario was chosen, including the Given-When-Then equivalent instructions (Figure 4).
When the user clicks in OK, the flow goes to the next step, an AND-Split (Figure 5) and the tests responsible for filling the form will run, as shown in Figure 6.
Although we had to implement specific logic for each of the PN structures (I will start discussing this on the next post of this series), the next steps for verifying the consistency between the BP model and the system are similar.
The goal of this post was to show our basic idea: make the user run the business process and see immediately how the system is behaving. If there isn't correspondence between requirements and code (originated by process change or code wrongly implemented), errors will appear, automatically indicating where the system is not compliant to the requirements described by the model. Error situations will be shown in the next posts.
The core idea is to use some Business Process (BP) Modeling tool to run, step by step, a given process representation and, at each step, excite the associated automated tests, making the system to pop-up accordingly as the user clicks on the representation. It is the same thing we see when we use tools such as Selenium, however, in that case, we would run the tests step by step and, instead of using textual descriptions of the requirements (like in BDD), we use a BP Model.
We know that there isn't a real standard for Business Process Modeling, in fact there are various, such as UML Models, Petri Nets, BPML, derivations of these etc. We also know that many EIS are based on Workflow Engines, and each of these engines has particularities. Therefore I decided to open two fronts:
1) Use a generic BP representation: the choice was Petri Nets, because I can derive others representations from them, such as Statechart and Activity Diagrams, and also there is the "Token Game", which is the simulation of the BP execution.
2) Use some workflow engine to work on top of it. The obvious choice for us was working with ERP5, an open source ERP (FOS-ERP), based on Zope, which in turn uses a state-based Wf engine, DCWorkflow. ERP5 was an obvious choice because (i) we are partners of this project since 2002, (ii) it is open, so we have access to all its code, (iii) we like state-based workflows.
I will focus now on (1) and will be back to (2) some weeks from now.
The first step for achieving (1) was to implement a proof of concept, using some Petri Nets open source tool that would be able to run Token Games, so that we could animate our model. During last August, Rodrigo Manhães, one of my associated researchers, prepared a proof of concept based on Woped.
Using an example PN, he created a very basic application, without taking care of the application logic or GUI - the goal is to show our proposal, not discuss BPM or GUI. The screenshot in Figure 1 shows the moment when the user reaches an OR-Split and must choose which path to follow in the workflow.
Figure 1: An user reaches an OR-Split in the example process
The OR-Split in Figure 1 is implemented as a list box. Appropriate tests excite this code, as shown in Figure 2.
Figure 2: The equivalent test run for the OR-Split
Remember that tests "are by example", therefore, the user doesn't interact with the real system, instead, he/she is a spectator of the tests, launching them by choosing which path to follow in the PN. After checking if the system is implemented the way he/she expected, the user closes the GUI and then choose the path equivalent to Credit Card, making the test code to run this option, which, in turn, makes the form in Figure 3 to pop. Meanwhile in Woped, the user can see a message that shows which scenario was chosen, including the Given-When-Then equivalent instructions (Figure 4).
Figure 3: Credit Card data form
Figure 4: Message with equivalent GWT.
When the user clicks in OK, the flow goes to the next step, an AND-Split (Figure 5) and the tests responsible for filling the form will run, as shown in Figure 6.
Figure 5: The process enters an AND-Split
Figure 6: Running the test for filling Credit Card data
Although we had to implement specific logic for each of the PN structures (I will start discussing this on the next post of this series), the next steps for verifying the consistency between the BP model and the system are similar.
The goal of this post was to show our basic idea: make the user run the business process and see immediately how the system is behaving. If there isn't correspondence between requirements and code (originated by process change or code wrongly implemented), errors will appear, automatically indicating where the system is not compliant to the requirements described by the model. Error situations will be shown in the next posts.
Deciding on the names for BDD extentions
On a previous post I talked about our proposal of extending Behavior Driven Development (BDD) to use other languages than then "classic" Given-When-Then (GWT), in special, but not limited to, Business Process Modeling (BPM) artifacts, such as Petri Nets, Statechart or Activity Diagrams. Today we decided at NSI to use two terms to refer to our proposal:
a) When referring to the idea of using a composition of languages, previously agreed by all stakeholders, we are going to call it Shared Language Driven Development (SLDD).
b) The specific case of using BPM + domain specific language (DSL) as an input for the BDD machinery is called Business Language Driven Development (BLDD).
In other words, SLDD is the general proposal and BLDD is the proposal for EIS. You can also interpret like this: SLDD is a generalization of BDD, and BLDD is an extension to BDD that uses BPM.
a) When referring to the idea of using a composition of languages, previously agreed by all stakeholders, we are going to call it Shared Language Driven Development (SLDD).
b) The specific case of using BPM + domain specific language (DSL) as an input for the BDD machinery is called Business Language Driven Development (BLDD).
In other words, SLDD is the general proposal and BLDD is the proposal for EIS. You can also interpret like this: SLDD is a generalization of BDD, and BLDD is an extension to BDD that uses BPM.
Saturday, September 11, 2010
Pyramid: BDD in Python
NSI started developing in March 2009 a set of tools for using BDD in Python language. They are briefly described in the technical report A tool stack for implementing Behaviour-Driven Development in Python Language, which needs to be improved and updated, however, it gives a summary of this work. This project started as an desire of using BDD in a real-world Python application: the Federal EPCT Network Digital Library (BD-EPCT) project. By the time we started, with should-dsl, Pyramid was the only set of integrated tools for BDD in Python.
Thursday, September 9, 2010
Abstraction and Test By Example in the EIS Case
Another point to consider in BDD for EIS is the question of Abstraction during testing. The same way we expect to describe requirements through BPM, we probably would like to write tests not using examples, but parameters instead. I mean, instead of declaring things like
Given that a client buys a jumper
And I have an inventory of three jumpers
When the client returns a jumper
Then I will have four jumpers in inventory
I would have something like
Given that a client buys a given product
When the client returns this product
Then I will add this returned product to my inventory
Since tests must deal with examples, parameters must be marked for parsing and when we run tests we must supply these parameters. Many automated tests tools already do that.
I think parameterizing is a must in the case of EIS development and customization. Think in terms of an ERP and its different adopters, probably they will demand different test sets during customization or even system evaluation. Actually, during deployment, you are going to import a lot of legacy data, thus it is interesting to use this data for testing.
See? Parameterization is a must for Agile EIS testing.
Given that a client buys a jumper
And I have an inventory of three jumpers
When the client returns a jumper
Then I will have four jumpers in inventory
I would have something like
Given that a client buys a given product
When the client returns this product
Then I will add this returned product to my inventory
Since tests must deal with examples, parameters must be marked for parsing and when we run tests we must supply these parameters. Many automated tests tools already do that.
I think parameterizing is a must in the case of EIS development and customization. Think in terms of an ERP and its different adopters, probably they will demand different test sets during customization or even system evaluation. Actually, during deployment, you are going to import a lot of legacy data, thus it is interesting to use this data for testing.
See? Parameterization is a must for Agile EIS testing.
Ubiquitous Language or a Composition of Languages?
According to Wikipedia, "BDD focuses on obtaining a clear understanding of desired software behavior through discussion with stakeholders. It extends TDD by writing test cases in a natural language that non-programmers can read." Also, Dan North says that "BDD provides an Ubiquitous Language for Analysis" and then presents the Given-When-Then (GWT) triad.
Thinking on these definitions, I proposed the use of Business Process Modeling (BPM) artifacts for describing requirements in BDD. Things like Petri Nets and Statechart or Activity Diagrams are a "natural language" for many users, in special for those that interact with EIS.
I understand that GWT is an UL for System Requirements Elicitation. In fact, I think that ULs should behave as a composition of languages, something like (some language that can describe system requirements) + (the user's business language).
For EIS, I envision something like some BP notation with a textual language that is simultaneously understandable by users, parseable, and that we can write code that is the closest to the user's way of writing about their business, using expressions such as "Vectorial Calculus is_a_pre-requisite_of Linear Algebra" (Yes, that's what people call DSL - Domain Specific Language).
Ideally this textual language should go from the requirements to the code, letting the program's structures be described through the BPM notation.
I wrote all this text to say only this:
-UMLers would say "Activity Diagrams + OCL" is the answer, and I say, yes, that's an option, but OCL is not the right answer, it isn't natural!
So, we need to go further...
Thinking on these definitions, I proposed the use of Business Process Modeling (BPM) artifacts for describing requirements in BDD. Things like Petri Nets and Statechart or Activity Diagrams are a "natural language" for many users, in special for those that interact with EIS.
I understand that GWT is an UL for System Requirements Elicitation. In fact, I think that ULs should behave as a composition of languages, something like (some language that can describe system requirements) + (the user's business language).
For EIS, I envision something like some BP notation with a textual language that is simultaneously understandable by users, parseable, and that we can write code that is the closest to the user's way of writing about their business, using expressions such as "Vectorial Calculus is_a_pre-requisite_of Linear Algebra" (Yes, that's what people call DSL - Domain Specific Language).
Ideally this textual language should go from the requirements to the code, letting the program's structures be described through the BPM notation.
I wrote all this text to say only this:
-UMLers would say "Activity Diagrams + OCL" is the answer, and I say, yes, that's an option, but OCL is not the right answer, it isn't natural!
So, we need to go further...
Monday, September 6, 2010
BPDD | BLDD| CLDD | SLDD, Dynamic Business Process Composition, The Future of ERP, REST, and Here We go!
The economic dynamism has been pushing a lot enterprise systems development. Some business processes must be altered and/or re-composed frequently to meet changing business conditions. Dynamic Business Process (or Service) Composition represents a set of techniques that try to provide the basis for achieving this dynamism.
Also, some say the future of ERP is to become process integration frameworks. Instead of integrated modules, ERPs would provide the means for quickly integrating disparate business processes living in different systems - in fact, this is an old Supply Chain Management need, far away from be quickly (and cheaply) fulfilled in real world applications.
Many proposals are based on the use of Intelligent Agents, Ontologies and other automated decision making supportive methods. Agents and ontologies can be very, very useful for filtrating, among hundreds or thousands of options, the few dozen that may interest to a given organization. Ontologies are very, very useful for helping finding the points of integration for these process. However, I still think that in the end, it will be a human-decision-maker and a human-programmer who will choose the best combination of processes and implement their integration respectively.
Experience says that even if the leading-edge organizations can reach the Nirvana of totally automated dynamic composition of business process, we are going to find most of smaller organizations striving to do this "manually".
Therefore, I think there is still room (and a lot of it) for developing supportive techniques for "semi-automated" Business Process Composition (BPC). And I believe our proposal of having a Business Process extension of BDD (we are deciding which name to use - BPDD, BLDD, CLDD, or SLDD - see my previous post) associated to tools like Restfulie may offer a powerful method to implement executable documentation for business applications that demand integration of disparate applications and their business processes.
I can envision a business analyst building a graphical representation of the macro-process, using a shared development language - understandable by all stakeholders (see my previous post), and, after implemented the automated tests, checking the live application running as he/she follows (clicks) on the different paths of a workflow representation. As the macro-process changes, the tool will visually identify where and what to change accordingly so that the system becomes business compliant again. That would be a way of building a "next-generation ERP" or "next-generation Enterprise Information Systems" through a combination of visually manipulating BP representations and Test Driven Development (TDD).
Well, that was one more teaser to the matters I want to discuss here, I will go back to it latter.
Also, some say the future of ERP is to become process integration frameworks. Instead of integrated modules, ERPs would provide the means for quickly integrating disparate business processes living in different systems - in fact, this is an old Supply Chain Management need, far away from be quickly (and cheaply) fulfilled in real world applications.
Many proposals are based on the use of Intelligent Agents, Ontologies and other automated decision making supportive methods. Agents and ontologies can be very, very useful for filtrating, among hundreds or thousands of options, the few dozen that may interest to a given organization. Ontologies are very, very useful for helping finding the points of integration for these process. However, I still think that in the end, it will be a human-decision-maker and a human-programmer who will choose the best combination of processes and implement their integration respectively.
Experience says that even if the leading-edge organizations can reach the Nirvana of totally automated dynamic composition of business process, we are going to find most of smaller organizations striving to do this "manually".
Therefore, I think there is still room (and a lot of it) for developing supportive techniques for "semi-automated" Business Process Composition (BPC). And I believe our proposal of having a Business Process extension of BDD (we are deciding which name to use - BPDD, BLDD, CLDD, or SLDD - see my previous post) associated to tools like Restfulie may offer a powerful method to implement executable documentation for business applications that demand integration of disparate applications and their business processes.
I can envision a business analyst building a graphical representation of the macro-process, using a shared development language - understandable by all stakeholders (see my previous post), and, after implemented the automated tests, checking the live application running as he/she follows (clicks) on the different paths of a workflow representation. As the macro-process changes, the tool will visually identify where and what to change accordingly so that the system becomes business compliant again. That would be a way of building a "next-generation ERP" or "next-generation Enterprise Information Systems" through a combination of visually manipulating BP representations and Test Driven Development (TDD).
Well, that was one more teaser to the matters I want to discuss here, I will go back to it latter.
Business Process Driven Development - BPDD
At NSI we are working on adding Business Process (BP) representations to Behavior Driven Development (BDD). I mean, substituting the "classic" Given-When-Then (GWT) by some BP graphical representation, such as Petri Nets or UML Statechart Diagrams.
The core idea, already working as a proof of concept using Woped, is to allow a final user to run a business process in a step by step way and then see the living system pop up as he/she clicks on the activity elements, such as edges in a Petri Net. In that way we provide executable documentation using BP graphical representations, delivering BDD to the realm of Enterprise Information Systems.
Initially, we thought of calling this extension to BDD "Business Process Driven Development" (BPDD), however, we realized that a lot of related but different methods use this name with many different meanings. Then I thought of other names, and leave to my readers the choice of the best one:
a)Common Language Driven Development (CLDD)
The use of common language for users and developers (yes, another name for UL...)
b)Business Language Driven Development (BLDD)
The use of the user's business language as a common language.
c)Shared Language Driven Development (SLDD)
The use of a shared language, resulting of an agreement between users and developers.
(b) is interesting because keep our focus on business systems. However, I think we can use the concept of UL for scientific systems, for instance. So "business" may not map well for these systems.
(c) is interesting because it shows that users and developers agreed to use a shared language.
Yes, I know that UL's definition may cover all that, but I don't want to call ULDD!
We prepared two technical reports and posted them at Arxiv.org, please check them and help us to find a nice name to this BDD extension:
a) Filling the Gap between Business Process Modeling and Behavior Driven Development
b) Mapping Business Process Modeling constructs to Behavior Driven Development Ubiquitous Language
PS: in fact I believe that the Ubiquitous Language (UL) is not a single language, but a composition of languages. For instance, we can mix mathematical formulas and plain text into a process representation so that this representation is the closest possible to the "real thing". In other words, since there are languages such as should-dsl, we could also have things like math-dsl, or logistics-dsl, used externally and internally.
I will go back to this later on...
The core idea, already working as a proof of concept using Woped, is to allow a final user to run a business process in a step by step way and then see the living system pop up as he/she clicks on the activity elements, such as edges in a Petri Net. In that way we provide executable documentation using BP graphical representations, delivering BDD to the realm of Enterprise Information Systems.
Initially, we thought of calling this extension to BDD "Business Process Driven Development" (BPDD), however, we realized that a lot of related but different methods use this name with many different meanings. Then I thought of other names, and leave to my readers the choice of the best one:
a)Common Language Driven Development (CLDD)
The use of common language for users and developers (yes, another name for UL...)
b)Business Language Driven Development (BLDD)
The use of the user's business language as a common language.
c)Shared Language Driven Development (SLDD)
The use of a shared language, resulting of an agreement between users and developers.
(b) is interesting because keep our focus on business systems. However, I think we can use the concept of UL for scientific systems, for instance. So "business" may not map well for these systems.
(c) is interesting because it shows that users and developers agreed to use a shared language.
Yes, I know that UL's definition may cover all that, but I don't want to call ULDD!
We prepared two technical reports and posted them at Arxiv.org, please check them and help us to find a nice name to this BDD extension:
a) Filling the Gap between Business Process Modeling and Behavior Driven Development
b) Mapping Business Process Modeling constructs to Behavior Driven Development Ubiquitous Language
PS: in fact I believe that the Ubiquitous Language (UL) is not a single language, but a composition of languages. For instance, we can mix mathematical formulas and plain text into a process representation so that this representation is the closest possible to the "real thing". In other words, since there are languages such as should-dsl, we could also have things like math-dsl, or logistics-dsl, used externally and internally.
I will go back to this later on...
Subscribe to:
Comments (Atom)