tag:blogger.com,1999:blog-71502355659098106042024-03-13T12:40:36.557-03:00Enterprise Information Systems DevelopmentA blog devoted to the discussion of practical techniques for developing Enterprise Information Systems (EIS).Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.comBlogger55125tag:blogger.com,1999:blog-7150235565909810604.post-7479206119369785512012-08-28T21:30:00.001-03:002012-08-29T15:30:38.102-03:00Assorted Thoughts on Agilism X Traditionalism - Part VI<b>Aerospace Engineering and Agilism</b><br />
I feel really happy when I read something about complex environments which corroborates with our thoughts on how to produce software. A few years ago I read on <a href="http://spectrum.ieee.org/" target="_blank">IEEE Spectrum</a> that the US Department of Defense (DoD) was using leaner acquisition processes. Together with the fact that the DoD is the <a href="http://eis-development.blogspot.com.br/2011/03/why-open-source-for-eis.html" target="_blank">biggest open source user in USA</a>, it made me think on how some teams delude themselves with excuses that in the end be something like "Yes, we must rely on heavy processes and proprietary software, our organization is really more complex than the Department of Defense of the United States of America!"<br />
<br />
Now I have just bumped on the part regarding Mission and Reliability of the classic <a href="http://books.google.com.br/books?id=MLuaZxSg8nQC&printsec=frontcover&source=gbs_ge_summary_r&cad=0#v=onepage&q&f=false" target="_blank">The Logic of Microspace</a> book. Starting on page 219, the author discuss on why the aerospace industry didn't follow the Lean Thinking principles decades ago, and, more important to this post, what are the consequences of this. Two of them consequences caught my attention: (i) Since manual testing of parts was known and automated testing was new, aerospace industry decided to keep on manual testing, with all its bureaucracy and human errors, and (ii) the qualification processes took so long that it is usual to rely on more than 10 year-old parts. The author also gives an advice: small teams, working on small time windows and using highly integrated and pre-tested components are the solution for a lot of mission complexity problems.<br />
<br />
The moral behind this post is the same of many other posts in this blog: don't use the size of your working environment or the complexity of your task as excuses for not using open source software and agile methods.<br />
<br />
<b>Post Script </b><br />
On page 244 of the same book:<br />
"Space qualified parts carry that label because they meet certain standards for fabrication, inspection, testing and documentation. Modern parts [those produced through automated processes] are likely not designed for the <b>mostly manual steps</b> prescribed by the space qualification requirements, which are often manually assembled and inspected, which lowers reliability. <b>The documentation doesn't add anything to your product's reliability</b>. And the additional people needed to handle all this documentation cut down on team cohesion, create new management layers, and hence lowers the focus of the team effort."<br />
Do I really need to say anything more to convince you that manual testing and lots of documents don't increase your system's quality or do you think your inventory control information system is really more complex than a satellite?<br />
Before I forget, the book is from 2000...<br />
<br />
Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com9tag:blogger.com,1999:blog-7150235565909810604.post-86496358705409412372012-08-23T20:48:00.000-03:002012-08-23T20:48:03.149-03:00Enterprise Information Systems Patterns - Part XIII<b>Why I believe that the Relational Databases are the biggest barrier to make EIS really flexible</b><br />
We are now making EIS Patterns Relational Database (RDB) -aware through <a href="https://www.djangoproject.com/">Django</a> framework. Just as I expected, problems would start to appear when it is needed to represent Python's dynamic nature in an RDB.<br />
<br />
It is a fact that while the technology behind RDB is really mature and extremely efficient and safe, the relational "way of thinking" was created in the 1970s to solve 1970s' problems! In my humble opinion, although mathematically sound, the way the relational model represents all-part, one-to-many, and many-to-many relationships is weird. Think of representing books in a relational way. You store all books in one room, while all their pages are in another room. Retrieving a given book's pages is a question of entering the pages room and asking which pages belong to that book. The book by itself is not able to identify its pages... In other words, it doesn't matter how elegant or representative is your object model, it will transformed into a set of unnatural, strange relationships.<br />
<br />
So let's check an interesting situation where the dynamic nature of Python is used in EIS Patterns. The <a href="https://github.com/ratem/eispatterns/blob/master/domain/movement/process.py">Process</a> class controls the way objects collaborate to perform a given business process, wrapping and logging objects' methods executions using the<span class="nf"> <i>run_activity</i></span><span class="p"></span> method. For this discussion the interesting part of this method is its <i>return</i> clause:<br />
<pre><span class="k">return</span> <span class="p">{</span><span class="s">'actor'</span><span class="p">:</span><span class="n">actor</span><span class="p">,</span> <span class="s">'arguments'</span><span class="p">:</span> <span class="n">execution_arguments</span><span class="p">,</span> <span class="s"> </span></pre><pre><span class="s"> 'result'</span><span class="p">:</span><span class="n">activity_result</span><span class="p">,</span> <span class="s">'start'</span><span class="p">:</span> <span class="n">activity_start</span><span class="p">,</span></pre><pre><span class="p"> </span> <span class="s">'end'</span><span class="p">:</span><span class="n">activity_end</span><span class="p">}</span></pre><br />
This clause returns a dictionary containing:<br />
a) The execution <i>actor</i>: a Node of which the method is wrapped;<br />
b) The <i>execution arguments</i>: a list containing the parameters for the method;<br />
c) The <i>result</i>: the result produced by the execution;<br />
d) The <i>start</i> and <i>end</i>: date and time of the starting and conclusion of the execution.<br />
<br />
Objects in (a) and (d) are of known types (Node and DateTime), while (b) and (d) are collections of objects of unknown types, including, potentially, other complex collections. So, how to represent it on a RDB? One solution is using descriptors to detail the dictionary, proving for each returned object or method argument, the following record:<br />
<style type="text/css">
<!--
@page { margin: 2cm }
P { margin-bottom: 0.21cm }
-->
</style> <br />
<div class="western" style="margin-bottom: 0cm;"><br />
</div><table border="3" cellpadding="4" cellspacing="0"><colgroup><col width="64*"></col> <col width="64*"></col> <col width="64*"></col> <col width="64*"></col> </colgroup><tbody>
<tr valign="TOP"> <td width="25%"><div align="CENTER" class="western" style="text-decoration: none;"><span style="text-decoration: none;">movement_oid</span></div></td> <td width="25%"><div align="CENTER" class="western" style="text-decoration: none;"><span style="text-decoration: none;">return_or_argument</span></div></td> <td width="25%"><div align="CENTER" class="western" style="text-decoration: none;"><span style="text-decoration: none;">type</span></div></td> <td width="25%"><div align="CENTER" class="western" style="text-decoration: none;"><span style="text-decoration: none;">reference_or_value</span></div></td> </tr>
<tr></tr>
<tr valign="TOP"> <td width="25%"><div class="western">Movement object identifier</div></td> <td width="25%"><div class="western">Boolean for marking as return value or argument</div></td> <td width="25%"><div class="western">Object's type. Basic types, such as numbers and strings, have their value declared, complex types are referenced</div></td> <td width="25%"><div class="western">Stores basic types values as strings, and references for complex types.</div></td> </tr>
</tbody></table><div class="western" style="margin-bottom: 0cm;"><br />
</div><div class="western" style="margin-bottom: 0cm;">Complex objects are properly stored, normalized, in tables or, in raw, as blobs. Retrieval involves developing specific algorithms for each type, which can become a serious problem when dealing with a lot of <a href="http://eis-development.blogspot.com.br/2011/03/enterprise-information-systems-patterns_22.html">Decorators</a>. Besides that, a serious problem would be do deal with multidimensional collections. In other words, mapping this logging scheme to RDB would turn the framework non-flexible and too costly to adapt, descriptors wasn't the solution.<br />
<br />
The solution was to use <a href="http://en.wikipedia.org/wiki/JSON" target="_blank">JSON</a>, but with some investigation on how to do it in a flexible way, which I will explain in the next post.</div><div class="western" style="margin-bottom: 0cm;"><br />
</div><style type="text/css">
<!--
@page { margin: 2cm }
P { margin-bottom: 0.21cm }
-->
</style>Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com1tag:blogger.com,1999:blog-7150235565909810604.post-90064391934985981752011-08-18T16:46:00.002-03:002011-09-01T18:25:59.821-03:00Enterprise Information Systems Patterns - Part XII<b>Step by Step Programming using EIS Patterns</b><br />
This post will describe, step by step and using the example "<a href="https://github.com/ratem/eispatterns-examples">bank system</a>", how to program using the framework and <a href="https://github.com/nsi-iff/fluidity">Fluidity</a> + <a href="https://github.com/nsi-iff/extreme_fluidity">Extreme Fluidity</a>.<br />
<br />
<b>Step -1</b><br />
You must know the following techniques and tools:<br />
a) <a href="http://en.wikipedia.org/wiki/Behavior_Driven_Development">BDD</a> using <a href="http://lettuce.it/">Lettuce</a>.<br />
b) The <a href="http://en.wikipedia.org/wiki/Decorator_design_pattern">Decorator Design Pattern</a>. <br />
c) <a href="http://www.should-dsl.info/">Should-dsl</a>, to define expectations in both acceptance and unit tests. <br />
d) <a href="http://pyunit.sourceforge.net/pyunit.html">Python Unittest Library</a>.<br />
e) How to use <a href="http://wiki.python.org/moin/PythonDecorators">Python Decorators</a> (@). <br />
<br />
<b>Step 0</b><br />
Understand how and why things are done in the framework:<br />
a) Start at <a href="http://eis-development.blogspot.com/2011/03/enterprise-information-systems-patterns.html">Part IV</a>, the previous parts focus on discussions and experimentations which are not used in the framework anymore.<br />
b) Why Decorators are used to extend classes, explained in <a href="http://eis-development.blogspot.com/2011/03/enterprise-information-systems-patterns_09.html">Part V</a>.<br />
c) Why should-dsl is also used, besides in tests, to define contracts between objects, explained in <a href="http://eis-development.blogspot.com/2011/03/enterprise-information-systems-patterns_11.html">Part VI</a>.<br />
d) How things described in parts IV to VI are implemented in general is described in <a href="http://eis-development.blogspot.com/2011/03/enterprise-information-systems-patterns_22.html">Part VII</a> - if you feel it too abstract, I suggest going to <a href="http://eis-development.blogspot.com/2011/05/enterprise-information-systems-patterns.html">Part IX</a>.<br />
e) <a href="http://eis-development.blogspot.com/2011/04/enterprise-information-systems-patterns.html">Part VIII</a> only describes the status of the project by that time, you can ignore it, but it is necessary to accompany the <a href="http://eis-development.blogspot.com/p/eis-patterns-change-log.html">change log</a> to become aware of important changes in the code.<br />
f) <a href="http://eis-development.blogspot.com/2011/05/enterprise-information-systems-patterns.html">Part IX</a> describes conceptually how to program using the framework. <a href="http://eis-development.blogspot.com/2011/06/enterprise-information-systems-patterns.html">Part X</a> describes in more detail the use of <a href="https://github.com/nsi-iff/fluidity">Fluidity</a>, and <a href="http://eis-development.blogspot.com/2011/08/enterprise-information-systems-patterns.html">Part XI</a> explains the evolution of the concepts. This post does the same of Part IX, but showing the code instead.<br />
<br />
Reading the posts will give you an idea of the <i>modus facendi</i> of the framework, however, it is necessary to start programming... probably it is necessary to go back and forth from code to these posts. The following steps are a kind of guide, but in no way it is meant to describe a complete development process, this is one thing that you developer should infer. Some details may be omitted to avoid too long descriptions.<br />
<br />
*<b>It is strongly recommended that you check the unitary tests to understand how the framework core classes work in detail</b>*<br />
<br />
Right now we are using BDD because we don't have a nice <a href="http://eis-development.blogspot.com/2010/09/business-process-driven-development.html">BLDD</a> <a href="http://eis-development.blogspot.com/2010/09/proof-of-concept-for-bldd-tool-part-i.html">tool</a> to work with. Therefore you are supposed to describe the business process in <a href="https://github.com/ratem/eispatterns-examples/commit/102d6bcb5de5704ddd0e1f0f8906fb8181175be2#diff-8">BDD terms</a>, by interpreting a template workflow, and then incrementally define the <a href="https://github.com/ratem/eispatterns-examples/blob/fa20a23c593d20e57c9891f9c6d1301164f5fae9/bank_system/features/step_definitions/individual_credit_operation_steps.py">features' steps</a>.<br />
<br />
The following steps were developed on top of a specific example, however, they can be generalized.<br />
<br />
<b>Step 1</b><br />
Based on the <a href="https://github.com/ratem/eispatterns-examples/">bank system example</a>, let's suppose that the code is part of a bigger system which have already core concepts such as Employee and Bank Account. <br />
First the process is defined in Fluidity, them, incrementally, the scenarios are defined and finally their steps. <span style="color: red;"> </span><br />
<span style="color: red;">Fluidity:</span><br />
state('requested')<br />
transition(from_='requested', event='create_loan_request', to='request_created')<br />
state('request_created')<br />
<span style="color: red;"> Scenario:</span><br />
<b>Scenario Outline:</b> Individual Customer asks for loan<br />
Given I am a registered Credit Analyst<br />
And an individual customer with account number <account number> asks for a personal loan<br />
And the loan request is of <desired value><br />
When I confirm the loan request<br />
Then a new loan request with the <account number> and <desired value> is created<br />
And the new loan request is associated to the Credit Analyst<br />
<span style="color: red;"> Steps:</span><br />
<b>@step</b>(u'Given I am a registered Credit Analyst')<br />
<b>def</b> given_i_am_a_registered_credit_analyst(step):<br />
world.a_person = Person()<br />
an_employee_decorator = EmployeeDecorator()<br />
an_employee_decorator.decorate(world.a_person)<br />
world.credit_analyst = CreditAnalystDecorator('09876-5')<br />
world.credit_analyst.decorate(world.a_person)<br />
<br />
<b>Step 2:</b> <br />
By analyzing the code above it is identified the necessity of developing a credit analyst business decorator. You will do only the basic necessary to the first step pass: create a skeleton of CreditAnalystDecorator, and go back to next step afterwards.<br />
At this point it is already important to define the Rule of Association for this decorator, which is "to be a credit analyst, someone must be a bank employee first". Before writing the rule code, a test for checking if the rule is being enforced and if the decoration is working properly must be prepared:<br />
<b>def</b> it_decorates_a_person(self):<br />
#should fail<br />
(self.a_credit_analyst_decorator.decorate, self.a_person) |should| throw(AssociationError)<br />
#should work<br />
an_employee_decorator = EmployeeDecorator()<br />
an_employee_decorator.decorate(self.a_person)<br />
self.a_credit_analyst_decorator.decorate(self.a_person)<br />
self.a_credit_analyst_decorator.decorated |should| be(self.a_person)<br />
self.a_credit_analyst_decorator.decorated |should| have(2).decorators<br />
<br />
Now the rule's code can be written:<br />
<b>@rule('association')</b><br />
<b>def</b> rule_should_contain_employee_decorator(self, decorated):<br />
''' Decorated object should be already decorated by Employee '''<br />
decorated |should| be_decorated_by(EmployeeDecorator)<br />
<br />
This rule will assert that only if the person is an employee then he/she can be a credit analyst, and it is implemented as a method of the credit analyst decorator class. The be_decorated_by matcher was created specifically for this framework.<br />
<b></b><br />
<b>Attention</b>: as of versions 0.7.3 of the core and 0.1.3 of the example, a singleton Rule Manager object started to be used, in order to promote reuse of rules. This object is composed by a "rules" object, which in the core is called Core Rules. Checking the tests of the example it is possible to see that other rule base is used, in the case a subclass of the core rules, called Bank System Rule Base. It is important to note that some tests use the core rules, simply because they don't need to use rules specific to the bank system. In other words, if you need a new rule, it must be created into your rule base object - making it hard-coded, however it is envisioned a way of loading them from a file or other configuration object, maybe using an even higher DSL to define them.<br />
<br />
<br />
<b>Step 3:</b><br />
Now that the basic credit analyst decorator is defined and tested, we go back to the acceptance level, in this example, we will jump straight to the "Then" step because the others are straightforward. At this point, the business process starts to be assembled.<br />
<b>@step</b>(u'Then a new loan request with the (.+) and (.+) is created')<br />
<b>def</b> then_a_new_loan_request_with_the_account_number_and_desired_value_is_created(step, account_number, desired_value):<br />
(...)<br />
<span style="color: purple;">Creates the business process object:</span><br />
world.an_individual_credit_operation = Process('Individual Customer Credit Operation')<br />
<span style="color: purple;">Defines the process' source and destination:</span><br />
world.an_individual_credit_operation.set_source(world.the_company)<br />
world.an_individual_credit_operation.set_destination(world.a_client)<br />
<span style="color: purple;">Configures the process using a template state machine:</span><br />
template = LoanProcess()<br />
<span style="color: purple;">Uses xFluidity's to get a configuration based on the template:</span><br />
configurator = StateMachineConfigurator(template)<br />
configurator.configure(world.an_individual_credit_operation)<br />
<span style="color: purple;"> At this point, the process object (an_individual_credit_operation) has absorbed the template state machine</span><br />
<span style="color: purple;">Now it is time to configure the first movement - loan request creation<br />
The object <i>the_movement</i> will be a proxy for CreditAnalystDecorator</span>.<span style="color: purple;">create_loan_request. This movement will be a Transformation, having credit_analyst.decorated (the associated person) as both source and destination, and an_individual_credit_operation.create_loan_request as the activity - that's an important point: create_loan_request is a method, symmetric to a workflow transition, which was injected into the process object by xFluidity.</span><br />
the_movement = world.an_individual_credit_operation.configure_activity_logger(world.credit_analyst.decorated, world.credit_analyst.decorated, world.an_individual_credit_operation.create_loan_request, CreditAnalystDecorator.create_loan_request)<br />
<span style="color: blue;"><span style="color: purple;"> After this, the collection of movements of the process should contain the recently configured movement (found by the activity's name</span></span>)<br />
world.an_individual_credit_operation.movements |should| contain(the_movement.activity.__name__)<br />
<br />
<br />
<b>Step 4:</b> <br />
At this point the CreditAnalystDecorator class has no <i>create_loan_request</i> method, thus it is time to go back to the unit level and write a test to this specific requirement:<br />
<b>def</b> it_creates_a_loan_request(self):<br />
an_employee_decorator = EmployeeDecorator()<br />
an_employee_decorator.decorate(self.a_person)<br />
self.a_credit_analyst_decorator.decorate(self.a_person)<br />
self.a_credit_analyst_decorator.create_loan_request(self.an_account, 10000) #self.an_account was created in the setUp()<br />
self.a_person.input_area |should| contain('1234567-8')<br />
<br />
Now it is implemented as an @operation:<br />
<br />
@operation(category='business')<br />
<span style="font-weight: bold;">def</span> create_loan_request(self, account, value):<br />
''' creates a loan request '''<br />
loan_request = LoanRequest(account, value, self)<br />
#Places the loan_request in the node's input area<br />
self.decorated.input_area[loan_request.account.number] = loan_request<br />
<br />
Note that a LoanRequest class must be defined, which drive us to the next step.<br />
<br />
<b>Step 5:</b><br />
The first thing to do is to create a unit test for LoanRequest and then create the class itself, which are trivial tasks. After running all the necessary tests, we can go back to the acceptance level, given that the loan request class now exists and it is operating properly.<br />
<br />
<b>Step 6:</b><br />
<span style="color: blue;"> <span style="color: purple;">Time to run the loan request creation and store the execution context:</span></span><br />
the_movement.context = world.an_individual_credit_operation.run_activity(the_movement, world.credit_analyst, world.account, desired_value)<br />
<span style="color: blue;"><span style="color: purple;"> Since the create loan request transition was fired, the process should be on state 'request_created'</span></span><br />
world.an_individual_credit_operation.current_state() |should| equal_to('request_created')<span style="font-weight: bold;"> </span><br />
<span style="font-weight: bold;"><br />
</span><br />
<span style="font-weight: bold;">Step 7:</span> <span style="color: blue;"> <span style="color: purple;"></span></span><br />
Now it is time to check the 'And' clause of this step, which is: <br />
<span style="font-weight: bold;"></span><br />
<span style="font-weight: bold;">@step</span>(u'And the new loan request is associated to the Credit Analyst')<br />
<span style="font-weight: bold;">def</span> and_the_new_loan_request_is_associated_to_the_credit_analyst(step):<br />
#... this association was done during loan creation, just checking<br />
world.a_person.input_area[world.account.number].analyst |should| be(world.credit_analyst)<br />
<br />
This step was easy, since the way the loan request creation was implemented already places it on the decorated (the Person) processing_area.<br />
<br />
<br />
The next step would be go to the next process transition and implement it using a sequence similar to the one presented above.<br />
<br />
<br />
In summary, developing in the framework involves:<br />
1) Describing a user story guided by a business process template.<br />
2) Developing the code for the story steps, which will identify new responsibilities, to be implemented into the decorators and resources.<br />
3) Developing the code for the decorators, including defining their association rules and @operations.<br />
4) Developing the code for resources, including defining their association rules.<br />
5) Detailing the steps: use movements to wrap and associate @operations to each workflow activity, providing full control through the workflow engine while logging the execution context.<br />
<br />
The use of the process template and the association of decorators' methods to business process activities provides the adherence of the application logic to the process' logic. Currently, BDD is in use, therefore the linking to the process logic is done manually. However, it is envisioned the creation of a tool to generate the steps skeletons in a way adherent to the process logic.Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com3tag:blogger.com,1999:blog-7150235565909810604.post-89583049620207119042011-08-11T17:05:00.024-03:002011-08-16T19:25:50.726-03:00Enterprise Information Systems Patterns - Part XI<b>Evolving the interpretation of the concepts</b><br />
As the framework evolves, and we play with it through the <a href="https://github.com/ratem/eispatterns-examples">examples</a>, new interpretations of the concepts appear. In fact, after <a href="http://eis-development.blogspot.com/2011/06/enterprise-information-systems-patterns.html">playing with Fluidity</a>, we realized that Transformation and Transportation were not necessarily implemented as classes, but categories of movements instead, given that they behave in the same way - it is only a question of explicitly saying if the work item is transformed into a Node (a transformation) or simply moved from one node to another (a transportation).<br />
<br />
The history behind this is that, when developing the example, it was noticed that even a "simple" resource movement would need to be logged, as well as it could be associated to, or even generate, other resource(s). When we realized this, it was clear that transportations would potentially need the same machinery used by transformations, hence, they share a common behavior, which should be implemented into Movement class. Therefore, they were transformed into categories - although the Category class is not in use yet. Figure 1 shows how the framework's ontology is interpreted now.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-5d5LVTAfCI0/TkQ0h51-ZLI/AAAAAAAAADk/fzQRWQHUJ9E/s1600/EISPatternsOntology_v070.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="282" src="http://4.bp.blogspot.com/-5d5LVTAfCI0/TkQ0h51-ZLI/AAAAAAAAADk/fzQRWQHUJ9E/s400/EISPatternsOntology_v070.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 1: EIS Patterns Ontology</td></tr>
</tbody></table><br />
Another important point is that the relationship "is_a" between Operation and Resource does not represent a classic object oriented inheritance relationship. In other words, Operation is not a class, instead, it is a Python Decorator which is applied to Person and Machine methods to mark them as Business Operations. Therefore, when it is said that "a Node performs an Operation", it means that some of the Node's method is decorated as an operation - or, this node is able of performing this operation. Finally, the relationship "operates_on" between Operation and Resource means that operations are always transforming or transporting resources, through movements which encapsulates them (the operations).<br />
<br />
In summary:<br />
a) An operation is an abstract representation of a production operation.<br />
b) A node's method is decorated by an operation to inform that the original abstract concept is now represented concretely by the method.<br />
c) Inside a business process, a movement is used to encapsulate and log the execution context of the concrete operation defined in (b).<br />
d) Running a process consists of executing and logging the movements defined in (c).<br />
<br />
<a href="http://eis-development.blogspot.com/2011/05/enterprise-information-systems-patterns.html">Part IX</a> of this series explains in less abstract terms how to deal with operations and movements. <a href="http://eis-development.blogspot.com/2011/06/enterprise-information-systems-patterns.html">Part X</a> explains how to do this using <a href="https://github.com/nsi-iff/fluidity">Fluidity</a> and <a href="https://github.com/nsi-iff/extreme_fluidity">Extreme Fluidity</a>. As expected, in order to understand how the whole thing is implemented, it is necessary to check the <a href="https://github.com/ratem/eispatterns-examples">EIS Patterns Examples</a> project, in special, how <a href="http://en.wikipedia.org/wiki/Behavior_Driven_Development">BDD</a> steps are implemented.Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com0tag:blogger.com,1999:blog-7150235565909810604.post-7395678541801648442011-06-09T16:43:00.001-03:002011-06-09T19:32:02.029-03:00Enterprise Information Systems Patterns - Part X<b>EIS Patterns, Business Processes, and Workflow Engines</b><br />
Every EIS is based on business processes, and it can be said that workflows are the computational realization of these processes. Therefore it is necessary to provide some workflow engine to EIS Patterns.<br />
<br />
Basically, there are two types of workflow engines:<br />
a) Associated to an application development platform: such as the ones found in Web application development environments, like <a href="http://www.zope.org/">Zope</a>, or associated to databases, such as <a href="http://wiki.oracle.com/page/Oracle+Workflow">Oracle</a>, or as part of Groupware environments. In this case, applications are developed "inside" the environment, using the engine's <a href="http://en.wikipedia.org/wiki/Application_programming_interface">API</a>.<br />
b) Implemented as Libraries: using some programming language, a library of objects representing the workflow features is implemented, such as the <a href="http://wiki.python.org/moin/State%20Machine%20via%20Decorators">State Machine as Python Decorators</a>, or Ruby's <a href="https://github.com/rubyist/aasm">AASM</a>. In this case, you must use the libraries' objects to implement your engine.<br />
<br />
<b>Enters Fluidity & Extreme Fluidity</b><br />
<a href="https://github.com/nsi-iff/fluidity">Fluidity</a> was developed as a type (b) workflow engine, however, <a href="https://github.com/nsi-iff/extreme_fluidity">Extreme Fluidity</a> turns it into a third type of workflow engine:<br />
c) Inoculated & Expelled: using the dynamic nature of the underlying programming language, provide a way of making objects workflow-aware, and symmetrically, a way of turning these objects back to their initial structure when desired. <br />
Why this? Because type (a) engines forces you to use a given environment to develop your applications, while type (b) forces you to use specific objects to implement workflows, most of times creating a mixed code with application specific and workflow specific statements.<br />
With Extreme Fluidity (or xFluidity for short) it is possible to insert the code necessary to make your workflow run inside your business objects. In other words, they become workflow-aware objects, while keeping your programming style, standards, and patterns. In order to understand how this happens, let's check how Fluidity can be used when xFluidity is associated to it (the complete source code can be found <a href="https://github.com/ratem/eispatterns/blob/master/spec/fluidity_process_spec.py">here</a>).<br />
First, it is necessary to declare a template State Machine:<br />
<br />
<pre><div class="line" id="LC17" style="background-color: transparent;"><span style="font-size: x-small;"><span class="k"> class</span> <span class="nc">LoanProcess</span><span class="p">(</span><span class="n">StateMachine</span><span class="p">):</span></span></div><div class="line" id="LC18" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="n">state</span><span class="p">(</span><span class="s">'requested'</span><span class="p">)</span></span></div><div class="line" id="LC19" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="n"> state</span><span class="p">(</span><span class="s">'request_created'</span><span class="p">)</span></span></div><div class="line" id="LC20" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="n"> state</span><span class="p">(</span><span class="s">'request_analyzed'</span><span class="p">)</span></span></div><div class="line" id="LC21" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="n"> state</span><span class="p">(</span><span class="s">'refusal_letter_sent'</span><span class="p">)</span></span></div><div class="line" id="LC22" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="n"> state</span><span class="p">(</span><span class="s">'loan_created'</span><span class="p">)</span></span></div><div class="line" id="LC23" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="n"> state</span><span class="p">(</span><span class="s">'value_transfered'</span><span class="p">)</span></span></div><div class="line" id="LC24" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="n"> initial_state</span> <span class="o">=</span> <span class="s">'requested'</span></span></div><div class="line" id="LC25" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="n"> transition</span><span class="p">(</span><span class="n">from_</span><span class="o">=</span><span class="s">'requested'</span><span class="p">,</span> <span class="n">event</span><span class="o">=</span><span class="s">'create_loan_request'</span><span class="p">,</span> <span class="n">to</span><span class="o">=</span><span class="s">'request_created'</span><span class="p">)</span></span></div><div class="line" id="LC26" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="n"> transition</span><span class="p">(</span><span class="n">from_</span><span class="o">=</span><span class="s">'request_created'</span><span class="p">,</span> <span class="n">event</span><span class="o">=</span><span class="s">'analyst_select_request'</span><span class="p">,</span> <span class="n">to</span><span class="o">=</span><span class="s">'request_analyzed'</span><span class="p">)</span></span></div><div class="line" id="LC27" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="n"> transition</span><span class="p">(</span><span class="n">from_</span><span class="o">=</span><span class="s">'request_analyzed'</span><span class="p">,</span> <span class="n">event</span><span class="o">=</span><span class="s">'loan_refused'</span><span class="p">,</span> <span class="n">to</span><span class="o">=</span><span class="s">'refusal_letter_sent'</span><span class="p">)</span></span></div><div class="line" id="LC28" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="n"> transition</span><span class="p">(</span><span class="n">from_</span><span class="o">=</span><span class="s">'request_analyzed'</span><span class="p">,</span> <span class="n">event</span><span class="o">=</span><span class="s">'loan_accepted'</span><span class="p">,</span> <span class="n">to</span><span class="o">=</span><span class="s">'loan_created'</span><span class="p">)</span></span></div><div class="line" id="LC29" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="n"> transition</span><span class="p">(</span><span class="n">from_</span><span class="o">=</span><span class="s">'loan_created'</span><span class="p">,</span> <span class="n">event</span><span class="o">=</span><span class="s">'time_to_transfer_value'</span><span class="p">,</span> <span class="n">to</span><span class="o">=</span><span class="s">'value_transfered'</span><span class="p">)</span></span></div></pre><span style="font-size: x-small;"></span><br />
Using xFluidity, the template is injected in another object - in this case a Process:<br />
<br />
<pre><div class="line" id="LC33" style="background-color: transparent;"> <span style="font-size: x-small;"> <span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span></span></div><div class="line" id="LC34" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="bp">self</span><span class="o">.</span><span class="n">process</span> <span class="o">=</span> <span class="n">Process</span><span class="p">()</span></span></div><div class="line" id="LC35" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="n">template</span> <span class="o">=</span> <span class="n">LoanProcess</span><span class="p">()</span></span></div><div class="line" id="LC36" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="n">configurator</span> <span class="o">=</span> <span class="n">StateMachineConfigurator</span><span class="p">(</span><span class="n">template</span><span class="p">)</span></span></div><div class="line" id="LC37" style="background-color: transparent; color: #cc0000;"><span style="font-size: x-small;"> <span class="n">configurator</span><span class="o">.</span><span class="n">configure</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">process</span><span class="p">)</span></span></div></pre><br />
<br />
Magically, the process object starts to respond to the state machine's events:<br />
<br />
<pre><div class="line" id="LC39" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="k">def</span> <span class="nf">it_makes_the_process_respond_to_the_example_state_machine_events</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span></span></div><div class="line" id="LC40" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="bp">self</span><span class="o">.</span><span class="n">process</span> <span class="o">|</span><span class="n">should</span><span class="o">|</span> <span class="n">respond_to</span><span class="p">(</span><span class="s">'create_loan_request'</span><span class="p">)</span></span></div><div class="line" id="LC41" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="bp">self</span><span class="o">.</span><span class="n">process</span> <span class="o">|</span><span class="n">should</span><span class="o">|</span> <span class="n">respond_to</span><span class="p">(</span><span class="s">'analyst_select_request'</span><span class="p">)</span></span></div><div class="line" id="LC42" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="bp">self</span><span class="o">.</span><span class="n">process</span> <span class="o">|</span><span class="n">should</span><span class="o">|</span> <span class="n">respond_to</span><span class="p">(</span><span class="s">'loan_refused'</span><span class="p">)</span></span></div><div class="line" id="LC43" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="bp">self</span><span class="o">.</span><span class="n">process</span> <span class="o">|</span><span class="n">should</span><span class="o">|</span> <span class="n">respond_to</span><span class="p">(</span><span class="s">'loan_accepted'</span><span class="p">)</span></span></div><div class="line" id="LC44" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="bp">self</span><span class="o">.</span><span class="n">process</span> <span class="o">|</span><span class="n">should</span><span class="o">|</span> <span class="n">respond_to</span><span class="p">(</span><span class="s">'time_to_transfer_value'</span><span class="p">)</span></span></div><div class="line" id="LC44" style="background-color: transparent;"></div><div class="line" id="LC44" style="background-color: transparent;"><span class="p"></span></div><div class="line" id="LC44" style="background-color: transparent;"></div></pre><br />
As expected, the engine avoids the firing of transitions at wrong situations:<b><br />
</b><br />
<pre><div class="line" id="LC56" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="k">def</span> <span class="nf">it_runs_the_example_acceptance_path</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span></span></div><div class="line" id="LC58" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="bp">self</span><span class="o">.</span><span class="n">process</span><span class="o">.</span><span class="n">create_loan_request</span><span class="p">()</span></span></div><div class="line" id="LC59" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="bp">self</span><span class="o">.</span><span class="n">process</span><span class="o">.</span><span class="n">current_state</span><span class="p">()</span> <span class="o">|</span><span class="n">should</span><span class="o">|</span> <span class="n">equal_to</span><span class="p">(</span><span class="s">'request_created'</span><span class="p">)</span></span></div><div class="line" id="LC59" style="background-color: transparent; color: #cc0000;"><b><span style="font-size: x-small;"><span class="p"> # the line below means that if the process tries to fire loan_refused</span></span></b></div><div class="line" id="LC59" style="background-color: transparent; color: #cc0000;"><b><span style="font-size: x-small;"><span class="p"> # when at 'request_created' state, an exception is generated</span></span></b></div><div class="line" id="LC60" style="background-color: transparent; color: #cc0000;"><span style="font-size: x-small;"> <b><span class="bp">self</span><span class="o">.</span><span class="n">process</span><span class="o">.</span><span class="n">loan_refused</span> <span class="o">|</span><span class="n">should</span><span class="o">|</span> <span class="n">throw</span><span class="p">(</span><span class="n">InvalidTransition</span><span class="p">)</span></b></span></div><div class="line" id="LC61" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="bp">self</span><span class="o">.</span><span class="n">process</span><span class="o">.</span><span class="n">analyst_select_request</span><span class="p">()</span></span></div><div class="line" id="LC62" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="bp">self</span><span class="o">.</span><span class="n">process</span><span class="o">.</span><span class="n">current_state</span><span class="p">()</span> <span class="o">|</span><span class="n">should</span><span class="o">|</span> <span class="n">equal_to</span><span class="p">(</span><span class="s">'request_analyzed'</span><span class="p">)</span></span></div><div class="line" id="LC63" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="c">#loan accepted</span></span></div><div class="line" id="LC64" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="bp">self</span><span class="o">.</span><span class="n">process</span><span class="o">.</span><span class="n">loan_accepted</span><span class="p">()</span></span></div><div class="line" id="LC65" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="bp">self</span><span class="o">.</span><span class="n">process</span><span class="o">.</span><span class="n">current_state</span><span class="p">()</span> <span class="o">|</span><span class="n">should</span><span class="o">|</span> <span class="n">equal_to</span><span class="p">(</span><span class="s">'loan_created'</span><span class="p">)</span></span></div><div class="line" id="LC66" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="bp">self</span><span class="o">.</span><span class="n">process</span><span class="o">.</span><span class="n">time_to_transfer_value</span><span class="p">()</span></span></div><div class="line" id="LC67" style="background-color: transparent;"><span style="font-size: x-small;"> <span class="bp">self</span><span class="o">.</span><span class="n">process</span><span class="o">.</span><span class="n">current_state</span><span class="p">()</span> <span class="o">|</span><span class="n">should</span><span class="o">|</span> <span class="n">equal_to</span><span class="p">(</span><span class="s">'value_transfered'</span><span class="p">)</span></span></div></pre>In that way, it is possible to use a workflow engine in your own development environment and using your own models. Fluidity also provides guards and state actions, not shown here for the sake of simplicity.<br />
<br />
<b>A Note on Workflow Notations</b><br />
Fluidity is a state-based engine, which on those days of fancy business process notations may appear a bit old-fashioned. However, state machines represent a well known and easy to understand notation. EIS Patterns' Process and Movements objects are being developed in a way that they don't need to know which notation is in use, at least at the unit level - meaning that some code specific to a given notation <i>may</i> be used to "glue" elements at the application level.<br />
<br />
<b>Conclusion</b><br />
Currently Fluidity and xFluidity are works in progress, including the expelling capabilities and some details on implementation of guards. When these features become fully functional, a Petri Net based engine will be developed. A clear limitation of Fluidity is that it only works for Python objects, but the necessity of developing in a given programming language is a limitation of all workflow engines, although some provide XML-like APIs.Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com8tag:blogger.com,1999:blog-7150235565909810604.post-22222292791524980892011-05-29T23:04:00.006-03:002011-08-16T19:15:17.520-03:00Enterprise Information Systems Patterns - Part IX<b>How to use EIS Patterns?</b><br />
When <a href="http://github.com/ratem/eispatterns">eispatterns</a> started to be developed, the original goal was to build a set of basic concepts that could be configured to represent various business entities. However, this seemed to be very hard to accomplish, since as new behavior appears, new code is necessary, making the economy of using masks to represent new entities being not enough to justify the use of the framework.<br />
Therefore, the framework's focus shifted to on how to represent real world entities that dynamically acquire new responsibilities. Then, a new path was followed, by defining:<br />
-Essential abstract concepts in the framework: Node, Resource, and Movement. <br />
-The roles for each of the abstract concepts: Node: active, responsible for executing production operations; Resource: passive, responsible for storing production resources data; and Movements: coordinative, responsible for managing the nodes while they work on resources.<br />
-How to extend each of the concepts: Node: through Decorator Pattern; Resource: through subclassing; Movements: through configuration (better explained <a href="http://eis-development.blogspot.com/2011/03/enterprise-information-systems-patterns_22.html">here</a>).<br />
<br />
<b>Programming in EIS Patterns</b><br />
In the <a href="https://github.com/ratem/eispatterns">project repository</a> a super-simplified "<a href="https://github.com/ratem/eispatterns/tree/master/bank_system">bank system</a>" can be found, with the intention of demonstrating how the framework can and <u>should</u> be used. The framework is process-centric, meaning that the development should evolve around the underlying business processes, by following these steps:<br />
<br />
1) Represent the business process using a Process object. Ideally, this object should "absorb" the attributes and methods necessary to run as a workflow. For instance, if a state machine is used to represent business processes, the process object should have states (attributes) and transitions (methods), as well respond to specific events. Moreover, if it is decided to represent the same process using other notation, such as BPMN, this object must be transformed accordingly. <a href="http://nsi.iff.edu.br/">We</a> implemented <a href="https://github.com/nsi-iff/fluidity">Fluidity</a>, a state machine for Python, which responds to these demands*. Process objects use simpler Movement objects to connect business operations to business processes and log their execution.<br />
<br />
2) Define business decorators to adorn Node objects. Nodes are persons or machines which gain new responsibilities through <i>business decorators</i>, which in turn manipulate Nodes' input, processing and output areas. Business decorators make Nodes become the active part of the business process. Decorators must implement a decorate() method, where <a href="http://eis-development.blogspot.com/2011/03/enterprise-information-systems-patterns_11.html">rules of association</a> are defined using <a href="http://www.should-dsl.info/">should-dsl</a>. These rules ensure that a given object can be adorned by the decorator.<br />
<br />
3) Each process activity (using a state machine: a transition or state action) must be realized by a business decorator method. These methods are called (business) operations, can be categorized, and are encapsulated by Movements**, which in turn are linked to the Process object. In that way, business decorators' methods take the responsibility of making the process work***. The same template can be used to configure different processes, using different operations to implement them.<br />
<br />
4) Define the operations by decorating them with @operation. These operations should be able to ask for the transference of resources from/to the different nodes' areas using node's transfer() method. @operations implement the logic for each business process activity.<br />
<br />
5) Define WorkItems, which are, in general, data bags or passive entities****. The reason for that is that in a real-world, physical production system, humans and machines transform or transport the work items using their abilities (@operations). A report is passive, being written by a human; a truck hood is passively spot-welded by a robot in a factory; a motor is passively transported from one manufacturing cell to another.<br />
<br />
These steps are not necessarily executed in order, nor the business process is necessarily fully defined beforehand. It is envisioned a development process guided by the incrementally testing/building of the business processes. The use of templates is also a form of defining the acceptance criteria for the tests.<br />
<br />
It is important to note that every class and every method must have a <i>docstring</i>, so that keyword searches can be performed by the Documenter module, which will be (soon) extended by Natural Language Processing facilities, so that requirements can be parsed and used to try to find candidate reusable elements in a given library of business decorators and work items. Complimentary, the Rule Checker is a module that runs the association rules of the business decorators found by the search, so that users can verify which of them can be used for adorning a given object. A new module will be developed to find and run acceptance tests, so that the "live system" can be used to check its fitness to a given set of requirements.<br />
<br />
*Note 1: In comparison with workflow engines and workflow libraries, <a href="https://github.com/nsi-iff/fluidity">Fluidity</a> (in conjunction with <a href="https://github.com/nsi-iff/extreme_fluidity">Extreme Fluidity</a>) is innovative, because while engines force the development inside their environment and libraries force the use of their objects to implement the process, Fluidity allows the definition of template machines that can be absorbed by <u>any</u> Python object. In other words, you can still use your own classes, your own naming convention, your own programming style, and your own architecture, in your own environment. The "state-based behavior" is inoculated into your objects dynamically. Moreover, if you don't want to use Fluidity anymore, your objects can expel the state machine at any time.<br />
<br />
**Note 2: Movements are of the type Transformation - which transforms a given resource in a node's processing area, and Transportations - which transports a given resource from a node to another. These concepts became conceptual classifications of movements, only Movement objects do the job.<br />
<br />
***Note 3: Implementing business process is a two step task: first a movement is configured to log an @operation execution, and then this operation is executed.<br />
Using a state machine notation, the property movement.activity is set to a transition or a state action, and the property movement.activity_associated_method is set to @operations. It would be easier if transitions and actions would be directly associated to @operations, however, Movements are necessary to store execution information (parameters, result, date and time) as well as to indicate the source and destination nodes. Source and destination nodes allow the mapping of the process to physical movements in the production plant. After configuration, a movement can be executed. These steps are implemented into Process objects, because they coordinate movements. This layered method improves reuse and configuration of processes and @operations.<br />
<br />
**** Note 4: If workitems are physically aggregated to form a final product, this set of workitems must be promoted into a Kit, which can be "active" in some contexts.Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com0tag:blogger.com,1999:blog-7150235565909810604.post-39191866622562373302011-04-14T16:29:00.002-03:002011-04-21T14:55:29.779-03:00Enterprise Information Systems Patterns - Part VIII<b>The current status of the project</b><br />
<br />
The code in commits of <a href="https://github.com/ratem/eispatterns/commit/357c2712549af06c200127a80d0b999ee5ee196f">April 9</a> presents the first version of Rule Checker, a class able of finding Business Decorators' <a href="http://eis-development.blogspot.com/2011/03/enterprise-information-systems-patterns_11.html">rules implemented in should-dsl</a> and run them against the object to be decorated. In <a href="https://github.com/ratem/eispatterns/commit/c2932efefaa35612bb099dc9e45fb10d2034b70a">April 11</a> the initial version of Documenter was committed. Documenter simply returns the documentation of Resources and Business Decorators. Rule Checker and Documenter form the basis for the following configuration process:<br />
<br />
a) Requirement input phase<br />
1) User enters the keywords that represent the main concepts of the requirement.<br />
In the future:<br />
2) User enters the requirement's text.<br />
3) A Natural Language processor will identify the main concepts of the requirements.<br />
b) Concepts search phase<br />
1) By simply matching keywords, the framework suggests candidate business decorators and resources which can be reused to configure the new business process.<br />
In the future:<br />
2) Based on documentation and rules of association, an ontology is built to enhance searching of candidate decorators and resources.<br />
c) Concept compliance testing <br />
1) To better understand how candidate decorators work, run automated tests, so that the user can see the decorator in action.<br />
2) To check if the decorator can be used to decorate a selected Node object (Person or Machine), test which decorators can be applied to it by checking (running) association rules.<br />
d) Configure Business Process<br />
1) Using a domain specific language (DSL), describe the logic of the business process and generate the appropriate code for the business process.<br />
2) With the information obtained in (c), configure the business process. If new and/or changed functionality is necessary, enter the process of developing it, using the partially configured business process as a basis for acceptance tests. In that way the business process is constructed incrementally. At this point, the use of <a href="http://eis-development.blogspot.com/2010/09/proof-of-concept-for-bldd-tool-part-i.html">BLDD</a> is envisioned.<br />
<br />
The core idea of the process above is to provide ways of creating new systems in a dynamic and flexible way, through configuration . By dynamic I mean allowing the user to manipulate live objects which can demonstrate their capabilities - going far beyond documentation reading; by flexible I mean the use of Decorator Pattern to create new business entities on the fly. In the cases where changes are needed, the use of BLDD can provide a safer and dynamic way of extending the framework with new or improved business decorators and resources. <br />
<br />
The code necessary to implement this process is in the following state:<br />
-a.1: create a nice GUI, planned to medium term (end of the year).<br />
-a.2 & a.3: work in progress based on <a href="http://www.nltk.org/">NLTK</a>, planned to medium term. <br />
-b.1: done.<br />
-b.2: planned to medium term.<br />
-c.1: planned to short term.<br />
-c.2: done.<br />
-d.1: dsl for state diagrams is a work in progress.<br />
-d.2: planned BLDD use for the medium term, full BLDD to long term.Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com0tag:blogger.com,1999:blog-7150235565909810604.post-743816706018567962011-03-28T16:20:00.003-03:002011-07-29T13:25:48.590-03:00Why Open Source for EIS?<b>Short Answer</b><br />
It is safer. And cheaper. And <a href="http://eis-development.blogspot.com/2011/02/obvious-but-yet-ill-applied-relation-of.html">auditable</a>...<br />
<br />
<b>A bit longer answer</b><br />
I could stop here because every IT professional heard this at least once in life, but companies still rely a lot on Windows in both client and server sides. Today I read from a big Brazilian newspaper that USA's Department of Defense estimates that 55,000 new computer viruses appear a day. Guess which operational system these viruses aim for?<br />
<br />
If you are not convinced, I will summarize some figures I found <a href="http://bit.ly/gdh5qq">here</a> (as of June 2010):<br />
• Windows:<br />
-1 to 1.2 billion MS-Windows virus signatures<br />
-Only 1 million signatures are checked by an antivirus => 0.1% can be scanned<br />
• Others:<br />
-270 Apple Mac (OS-9 or older) viruses<br />
-No known Apple Mac OS-X viruses as of June 2010<br />
-60 Linux viruses (of older distributions)<br />
-No know Linux viruses as of June 2010<br />
• And more:<br />
-55,000 new Microsoft Windows malware per day (as of March 2010)<br />
-Compare to 22,000 daily average in 2008 and 40,000 daily average in 2009.<br />
<br />
Why people still use Windows and MS-Office in companies? The answer relies on legacy. Legacy technologies, and a terrible asset found in many enterprises: IT people with legacy minds.<br />
<br />
Sorry for being so direct and a bit rude, but I challenge anyone to prove, unless recent heavy investments justifies, that Windows and Office are not replaceable by Linux and Libre Office on most enterprise areas. Libre Office does 95% of what Office does, and the other 5% probably you will find few users in your organization who really need it. Some other softwares, in special the graphical tools, are not Linux-friendly in general, however, even for companies with design bureaus - people that uses CAD and such - again you will find many users that don't need Windows. And still, why use MS-Office?<br />
<br />
<b>The IT Legacy Minds</b><br />
I think that with the figures above, it is very unlikely to justify "scientifically" the use of Windows in most organizations, even when we talk about previous investments. Think of the money spent on Windows, Office, and anti-viruses licenses. Sum up the cost of repairing infected machines, the loss of information, the resulting productivity losses, the private data disclosure, and you see that the cost of training people to use Linux and Open Office is much, much lower.<br />
<br />
A real problem is with legacy systems, written to run in fat Windows clients. In the case of business systems, if they are more than 10 years old, it is justifiable, if not, it is strange that your developers or your software supplier haven't developed them for the Web, given that one of the things that were clear after the Y2K problem was that fat-client information systems were much more expensive to maintain. In other words, unless some clear necessity of using fat clients justifies, IT people made a wrong decision.<br />
<br />
In some specific areas, such as industrial automation, fat clients are really necessary, however, even in these cases, you can develop in Linux. I myself took part of a team which developed, deployed, and supported, from 2005 to 2009, an industrial application made in <a href="http://www.ni.com/labview/">LabView</a> for Linux. In fact, we were one of the firsts to put in production <a href="http://sine.ni.com/nips/cds/view/p/lang/en/nid/10314">PXI-SCXI</a> hardware running Linux in the world, and we did it for an offshore oil producing plant - a quite severe environment.<br />
<br />
Therefore, I feel free to conclude that the main barrier is really the lack of IT professionals to deal with Open Source solutions, and this problem is caused by the number of IT professionals that simply don't want to study new things. It seems paradoxical, however, in my opinion, IT people are quite conservative!<br />
<br />
There are also personal interests involved: if the organization adopts an IT structure that is safer and bring much less security problems, maybe some people will loose their jobs (you can find a fable on this type of employee behavior <a href="http://eis-development.blogspot.com/2010/09/on-certifications-and-tribal-culture_26.html">here</a>).<br />
<br />
<b>Conclusion</b><br />
Finally, I would like to mention a post from the Agile Scot blog: <a href="http://agilescout.com/department-of-defense-dod-is-1-using-open-source-for-govt">Department of Defense (DoD) is #1 using Open Source for Government</a>. As the title suggests, the DoD uses a lot of Open Source, thus, even if your organization is bigger than the USA's DoD, do you think you have more tentatives of invading your systems than them?Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com3tag:blogger.com,1999:blog-7150235565909810604.post-58181870387744556012011-03-22T14:29:00.003-03:002011-08-17T15:06:42.532-03:00Enterprise Information Systems Patterns - Part VII<b>Understanding how concepts are implemented</b><br />
In Part <a href="http://eis-development.blogspot.com/2011/03/enterprise-information-systems-patterns_09.html">V</a> of this series, I used an UML Class Diagram to show the relationships among the framework's concepts. Although this diagram can help understanding these relationships, being it a class diagram, it also "says" that all concepts are "classes", which is not true. A better way for describing these relationships is using an ontology, as shown in Figure 1.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-8cPQM6yNqqs/TdGkTBYwWaI/AAAAAAAAADQ/K54s_aEKIcQ/s1600/Rogerio+Ontology+Graph+v02.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="226" src="http://3.bp.blogspot.com/-8cPQM6yNqqs/TdGkTBYwWaI/AAAAAAAAADQ/K54s_aEKIcQ/s320/Rogerio+Ontology+Graph+v02.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 1: Ontology representing the relationships among framework's concepts</td></tr>
</tbody></table><div class="separator" style="clear: both; text-align: center;"></div><br />
Operations, since they are immaterial resources, need to be realized by some active entity, in this case, by Nodes. Therefore, they are implemented as "functions" (in fact "methods"), instead of instances of a given class. Although in Python functions are also objects, one cannot define them as objects of a given business class. However, it is necessary to typify Operations somehow so that we can identify among some object methods which represent business operations. The solution for this is using Python Decorators, which are not exactly the same thing as the <a href="http://en.wikipedia.org/wiki/Decorator_pattern">Decorator Pattern</a> used in this framework. A <a href="http://wiki.python.org/moin/PythonDecorators">Python decorator</a> is a specific change to the language syntax that allows to more conveniently alter functions, methods, and classes.<br />
<br />
Thus, it is possible to create a @operation(a_category) <i>pythonic</i> decorator which will store into method objects an attribute named category, with value a_category. In that way it is possible to mark methods as of Operation type, which is important to query a given class for its business skills. For instance, in a bank information system, it is possible to query a Credit_Analyst class to check which business operations objects of this type can perform. Or the contrary, using a keyword search discover, identify which methods of which classes may perform a given business operation. Moreover, I can create an ontology and navigate through it to check relationships among resources, nodes, and movements. With this ontology I can even suggest to a business analyst specific configurations for the system, given appropriate search terms.<br />
<br />
Processes are sets of transformations and transportations, which in turn are used to encapsulate operations. Encapsulate means to forward calls to the appropriate Node subclasses methods - in fact to their Business Decorators methods - and log all information related to these calls, such as date and time, parameters and associated objects. Thus, Process objects implement the logic which coordinates how Nodes work. In other words, Processes implement workflows, which means, in turn, that the framework is process centric, or workflow centric.<br />
<br />
<b>Decorating, Decorating, and Subclassing</b><br />
To make things more clear, let's call "decorators" the extension to Python syntax, and "business decorators", the classes created to decorate the framework concepts, as explained in Parts <a href="http://eis-development.blogspot.com/2011/03/enterprise-information-systems-patterns.html">IV</a> and <a href="http://eis-development.blogspot.com/2011/03/enterprise-information-systems-patterns_09.html">V</a>. Said that, let's check how the framework is implemented and extended:<br />
a) Resources:<br />
-Operations: methods, implemented into business decorators and marked with @operation decorator (yes, decorators for decorators).<br />
-Work Items (formerly known as Materials): "classical" objects, for each new type of material, a new subclass of material is created.<br />
-Kits: "classical" objects, for each new type of kit, a new subclass of kit is created. Given that kits can be composed by Operations, which are in fact methods, this composition is implemented through a list of references to these methods.<br />
b) Nodes:<br />
-All three subclasses are extend through business decorators. For instance, a person can be decorated by Developer (business) decorator to represent a software developer, or a Credit Analyst (business) decorator to represent a bank's credit analyst. No subclasses used. They can have methods related to @operations, to @rules of association (check Part <a href="http://eis-development.blogspot.com/2011/03/enterprise-information-systems-patterns_11.html">VI</a>), as well as methods representing internal machinery, in this last case, typically private.<br />
c) Movements**:<br />
-Transformations and Transportations are used as is, no subclassing, no <i>pythonic</i> decoration, no business decoration. They are used to build processes and store information related to operations calls, by encapsulating @operations during their configuration process.<br />
-Processes are extended by mimicking workflow templates (current work), or by using workflow engines to make them work. A way of implementing processes is using <a href="http://arxiv.org/abs/1011.2238">Business Language Driven Development</a> implemented through <a href="http://wiki.python.org/moin/PythonDecoratorLibrary#State_Machine_Implementaion">State Machine using decorators</a>*.<br />
<br />
For the reasons presented in Parts <a href="http://eis-development.blogspot.com/2011/03/enterprise-information-systems-patterns.html">IV</a> and <a href="http://eis-development.blogspot.com/2011/03/enterprise-information-systems-patterns_09.html">V</a>, subclassing is avoided at maximum - it is openly suggested only in the case of Work Items (Materials), which are considered passive "data bag" classes.<br />
<br />
*Post-publishing note #1: we decided to develop our own State Machine code, and make it independent of this framework, check the <a href="https://github.com/nsi-iff/fluidity">Fluidity</a> project.<br />
<br />
**Post-publishing note #2: Transformations and Transportations are not classes anymore, as explained in <a href="http://eis-development.blogspot.com/2011/08/enterprise-information-systems-patterns.html">Part XI</a>.Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com3tag:blogger.com,1999:blog-7150235565909810604.post-6973795508889742562011-03-11T10:04:00.001-03:002011-03-14T17:15:41.341-03:00Enterprise Information Systems Patterns - Part VI<b>A DSL for Rules of Association</b><br />
The use of rules of association may facilitate configuration: by querying them it is possible to check, for a given concept, which responsibilities can be assigned to it. The definition of a Domain Specific Language (DSL) for creating rules of association would be interesting, because it would allow the construction of more readable rules. In fact, a branch of <a href="http://www.should-dsl.info/">should-dsl</a> can be used. Should-DSL is a DSL for assertions, which are commonly used by coding techniques based on <a href="http://en.wikipedia.org/wiki/Design_by_contract"><b>design by contract</b></a>. <br />
<br />
In fact, should-dsl's Predicate Matchers and some of the already available "ordinary" matchers such as respond_to, which checks if an object has a given attribute or method, and be_instance_of, which verifies if an object is of a given type, can be used for formatting rules. For instance, let's suppose we have a decorator that is supposed to work only with persons:<br />
<br />
class A_person_decorator:<br />
<br />
def __init__(decorated):<br />
...<br />
#checks if decorated is compliant to the rules<br />
self.check_rules_of_association(decorated)<br />
#if it is compliant, sets a pointer to it<br />
self.decorated = decorated<br />
...<br />
<br />
def check_rules_of_association(decorated):<br />
try:<br />
#uses be_instance_of matcher to check if decorated is compliant<br />
decorated |should| be_instance_of(Person)<br />
except:<br />
raise RuleOfAssociation('Person instance expected, instead %s passed' % type(decorated)) <br />
<br />
...<br />
<br />
In the example above, type checking is necessary given the nature of Python language, however, the focus is in creating more complex rules related to the <b>business rules</b>. <br />
<br />
Another interesting possibility is rule querying:<br />
(i) For a given decorator, what are its rules of association (list its rules)?<br />
(ii) For a given decorated, which decorators it can use (list which responsibilities a given class can assume)?<br />
Query results could be presented in both human and machine readable ways. In other words, it would be some mechanism for querying rules, for both automatically checking associable objects in a object base, as well as informing people what is needed for configuring a concept to receive a given type of decoration.<br />
<br />
A funny situation is that should-dsl matchers are used in unit tests to check the correct functioning of the rules... written in should-dsl.<br />
<br />
The next post will describe some more details that decorators need to implement.Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com0tag:blogger.com,1999:blog-7150235565909810604.post-9699974840041200232011-03-09T21:54:00.002-03:002011-03-14T17:11:57.419-03:00Enterprise Information Systems Patterns - Part V<b>Decorators X Subclasses</b><br />
In the <a href="http://eis-development.blogspot.com/2011/03/enterprise-information-systems-patterns.html">previous</a> post, I redefined the level of abstraction for the framework, and now it is time to discuss a bit about Decorators. In order to facilitate the communication, a simple UML Class Diagram is provided by Figure 1, representing the structure after this redefinition.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://lh4.googleusercontent.com/-kqrdf2OXqjo/TX52GVFIYoI/AAAAAAAAADE/bBtfO3dPprI/s1600/eis3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="122" src="https://lh4.googleusercontent.com/-kqrdf2OXqjo/TX52GVFIYoI/AAAAAAAAADE/bBtfO3dPprI/s400/eis3.png" width="400" /></a></div><div class="separator" style="clear: both; text-align: center;"></div><div style="text-align: center;">Figure 1: UML Class Diagram of the new structure</div><div style="text-align: left;"><br />
</div><div style="text-align: left;">In Figure 1, each of the (now) three abstract concepts (resource, node, movement) has two "opposite" subclasses (operation X material, person X machine, transformation X transportation) and one aggregator subclass (kit, organization, process). It is important to note that aggregators are <a href="http://en.wikipedia.org/wiki/Composite_pattern">Composite</a> objects.</div><div style="text-align: left;"><br />
</div><div style="text-align: left;">By (re)checking the <a href="http://en.wikipedia.org/wiki/Decorator_pattern">Decorator</a> pattern documentation, both on Wikipedia and on the classic <a href="http://www.amazon.com/Design-Patterns-Elements-Reusable-Object-Oriented/dp/0201633612/ref=cm_cr_pr_product_top/179-8915616-2305209">book</a> from Gamma and colleagues, we can find that:</div><div style="text-align: left;">- While subclassing adds behavior to all instances of the original class, decorating can provide new behavior, at runtime, for individual objects. At runtime means that decoration is a "pay-as-you-go" approach to adding responsibilities.</div><div style="text-align: left;">- Using decorators allows mix-and-matching of responsibilities.</div><div style="text-align: left;">- Decorator classes are free to add operations for specific functionalities.</div><div style="text-align: left;">- Using decorators facilitates system configuration, however, typically, it is necessary to deal with lots of small objects.</div><div style="text-align: left;"><br />
</div><div style="text-align: left;">Therefore, by using decorators it is possible to, during a business process realization, create an object, associate and/or dissociate different responsibilities to it - in accordance to the process logic, and log all this. In that way, I have two main benefits:</div><div style="text-align: left;">i) The same object, with the same identifier, is used during the whole business process, there is no need for creating different objects of different classes. </div><div style="text-align: left;">ii) Given (i), auditing is facilitated, since it is not necessary to follow different objects, instead, the decoration of the same object is logged. Moreover, it is possible to follow a single object during all its life-cycle, including through different business process: after an object is created and validated - meaning that it reflects a real-world business entity - it will keep its identity forever.</div><div style="text-align: left;"><br />
</div><div style="text-align: left;"><b>Summarizing... </b><br />
Thus, the benefits of using Decorators are:</div><div style="text-align: left;">i) More dynamic and flexible enterprise systems, through the use of configuration and pay-as-you-go features.</div><div style="text-align: left;">ii) Easier auditing, given the fact that objects keep their class and identification while get new responsibilities.</div><div style="text-align: left;"><br />
<b>An example</b><br />
To better understand this, I will use a simple example: a teacher & researcher career. Let's suppose in our institution we have two kinds of teachers, the 20-hour and the 40-hour. While the first one is supposed only to teach, the second is also a researcher, and therefore holds more responsibilities. Given that there is vacancy, a teacher can change from one to another category. Since both types are "teachers", the ordinary object oriented solution would be to create a basic class named Teacher, which would hold the common features of the teaching career, and two subclasses, named 20-hour Teacher and 40-hour Teacher. With this architecture, I can see no simple solution than creating objects of the different classes and copying attributes back and forth every time someone changed his/her option of working 20 or 40 hours.<br />
<br />
Moreover, imagine that teachers can also be assigned to administrative positions, such as department dean or university president, with a lot of new responsibilities associated, while still teaching. Keeping track of all these assignments and withdraws of responsibilities would be complex and error-prone. Also, I like to think that being a dean is getting a new responsibility, instead of becoming a different type of employee, in special when we think that this is a temporary assignment.<br />
<br />
Now imagine that we have Persons and I decorate them as they are assigned to new responsibilities. In that way one can use the same object to register all career assignments: 20-hour, 40-hour, dean who still teaches, and even represent some administrative employee who teaches in a specific course - without loosing his/her attributions in the university's management. Teaching, researching, and administering would be decorators that could be associated and dissociated to objects as the business processes require (<b>pay-as-you-go</b>).<br />
<br />
Another point is that each decorator must keep a set of <b>rules of association</b>, which is responsible for allowing or prohibiting objects to be assigned to responsibilities. Each "decorated" object is also responsible for keeping a record of its responsibilities (bi-directional relationship). If a given object respects the rules of association of a given decorator, it can be decorated by it, allowing a very flexible way of creating new "types" (<b>mix-and-match</b>).<br />
<br />
This reasoning is also valid for other concepts, for instance, a given manufacturing cell (Machine), can run new operations as new tools are attached to it. The same is valid for organizations, as new machines and persons are associated to it. <br />
<br />
An interesting point on using decorators is when we think of business entities ruled by laws and regulations, such as contracts, government agencies, or even governmental economic measures: as regulation or environment changes, things can be added and/or deleted from the entities' scope. Rule/Law decorators can be programmed to function during certain period of time, detaching themselves from the decorated objects when this period is over. As an example of this last case, let's think of a tax refunding mechanism that is valid only during recession periods and for specific types of sales.<br />
<br />
Of course, a problem is that we have to deal with a potentially big set of decorators. However, given the mix-an-match possibilities of decoration, this number is smaller than the number of classes that would be created to map all possible combinations. Using an extreme example, if I have three concepts that can, each one, be assigned to four responsibilities, we would have 12 classes - and possibly by the use of multiple inheritance. By using decorators we would have 3 classes and 4 decorators, or 7 entities to manage (in 12 possible combinations, of course).<br />
<br />
Said that, the next step is to discuss the rules of association and details for implementing decorators. <br />
<br />
</div>Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com0tag:blogger.com,1999:blog-7150235565909810604.post-18063479994915169442011-03-03T20:23:00.009-03:002011-08-17T15:14:51.553-03:00Enterprise Information Systems Patterns - Part IV<b>Changing the abstraction level</b><br />
As expected, the high abstraction level brought by the use of only 4 concepts to try to define a whole system would bring side effects*. The basic one is that, if on one hand abstract concepts are highly reusable, on the other, they need to be extended a lot in order to become useful for representing concrete concepts, even the basic ones, such as Person or Product. I mean, a Node can be the abstract concept behind a Person object as well as a Factory object. However, you will need to provide a lot of extra code to make these concepts work. Given that I don't want to create lots of subclasses, the alternative is to use Decorators. However, in the same way, it is not good to abuse of a given technique, even if this technique is a well known design pattern. What I was going to do was create a lot of decorators instead of creating a lot of subclasses. In other words, I would be going against my main goal, which is to create a framework extensible by configuration, instead of programming.<br />
<br />
Therefore, I stopped a bit to rethink the framework's abstraction level. One thing I realized is that it would be very hard to configure too abstract concepts, thus, by creating one more class level, I could find less generic but easier to reuse concepts. A side effect is that I can get rid of one more of the original concepts - Path. In other words, I shortened the width and broadened the length of the class hierarchy. The idea now is to use three classes: Resource, representing the production resources; Node, representing the entities that transform and transport these resources; and Movements, which represent transformations and transportations of resources. Each of these abstract concepts has three subclasses, representing two "opposite" concepts and an aggregator of these first two. Thus, the structure will be like this:<br />
<br />
1)Resource()<br />
-Material(Resource): product, component, tool, document, raw material...<br />
-Operation(Resource): human operation and machine operation, as well as their derivatives. <br />
-Kit(Resource): a collective of material and/or immaterial resources. Ex.: bundled services and components for manufacturing.<br />
Comments:<br />
a)Alternative terms such as Object (material) and Action (immaterial) can also be considered, however, the term Object can bring a lot of trouble in programming.<br />
b)Another point is <i>information</i>, which is not material, neither it is something like operation. <i>Document</i> is used as the physical representation of information. c)Structural and Behavioral would be alternatives, however, they seem to be too academic to be used.<br />
<br />
2)Node()<br />
-Person(Node): employee, supplier's contact person, free lancer...<br />
-Machine(Node): hardware, software, drill machine...<br />
-Organization(Node): a collective of machines and/or persons. Ex.: manufacturing cell, department, company, government.<br />
<br />
3)Movement()<br />
-Transformation(Movement): an "internal" movement. Ex: transforming raw material, writing a report<br />
-Transportation(Movement): a movement of resources between two nodes. Ex: moving a component from one workstation to another, moving a document from one department to another<br />
-Process(Movement): a collective of transformations and/or transportations, in other words, a business process.<br />
Comments<br />
a) There maybe some confusion between Transformations and Transportations and Operations, but they represent different things. Movements use operations to transform or to transport resources. For instance, an industrial transformation uses a given machine operation during a given period of time to transform a given quantity of raw material into a component.<br />
<br />
The next step is to define how to extend the basic concepts to make them work properly in concrete business processes representations, which will be discussed in the next post of this series.<br />
<br />
* As I said in <a href="http://eis-development.blogspot.com/2010/12/enterprise-information-systems-patterns.html">my first post</a> of this series, if you need a "real-world" and flexible Python framework, give <a href="http://www.erp5.com/">ERP5</a> - from which the basic ideas for this framework were taken - a try. Remember that this framework is a didactic one, therefore, some assumptions are simplified.<br />
<br />
(The <a href="http://eis-development.blogspot.com/p/eis-patterns-change-log.html">Change Log</a> maps this series of posts to commits in GitHub)Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com0tag:blogger.com,1999:blog-7150235565909810604.post-48493380633226361282011-02-25T17:57:00.001-03:002011-02-25T18:04:52.807-03:00The obvious, but yet ill applied, relation of Open Source and AuditabilityIn this post I comment two blog entries related to the use of open source software for auditing purposes in the banking area.<br />
<br />
<br />
In the first one, <a class="contentpagetitle " href="http://www.computeruser.com/articles/open-source-banking-for-financial-stability.html">Open Source Banking for Financial Stability</a>, Jean Paul Smets affirms that some serious problems presented by financial products not compliant to the regulations are caused by software systems which "are not subject to any kind of regulatory control and, therefore, may be used to circumvent regulations either through functional inconsistencies or through temporal inconsistencies." Therefore, he defends the idea of using a reference open source software, without getting rid of the software in which banks invested for decades. This reference, besides offering an auditable software representation of the regulations rules, would be used "to provide to regulation authorities a very accurate picture of [banks] activity, much more precise than what accounting can provide." - by importing the data handled by existing bank's software. <br />
<br />
<br />
In the second post, <a href="http://www.infoq.com/news/2010/04/Python-ABS">Python Could Become the Language of Finance</a>, Jonathan Allen informs that <a href="http://www.sec.gov/">SEC</a> is proposing that Asset Backed Securities should include a “program that gives effect to the flow of funds, or “waterfall,” provisions of the transaction.” Waterfall in this case "refers to how some bonds are broken into levels, where those who bought into the higher level must be paid off before the lower level sees any money." The interesting part is that SEC is planning to mandate <a href="http://www.python.org/">Python</a> as the language for building this reporting tool, because it is open source, uses a standalone interpreter, and is supported by many platforms, including proprietary software such as .NET.<br />
<br />
Although the first post is an opinion, the second isn't, and both present good arguments to motivate regulatory organs to use more open source for auditing tasks in the financial area, and, why not say, in other areas. The use of open source reference softwares would (i) help people compare their implementations of regulatory rules with a standard implementation, (ii) provide cheaper software that could be customized for specific adopters, while following the regulation, (iii) provide automated tools for regulatory agencies to audit data. In that way, maybe the number of financial scandals could be reduced, as well as regulatory work in other areas, such as environment, could be simplified and streamlined.Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com0tag:blogger.com,1999:blog-7150235565909810604.post-12924188716001056902011-02-13T16:10:00.000-02:002011-02-13T16:10:53.149-02:00Enterprise Information Systems Patterns - Part IIIAfter my last post, the code in <a href="https://github.com/ratem/eispatterns">Github</a> changed many times, since I was experimenting with both the basic concepts and the testing stack we are using. We decided not to write features (BDD's Given-When-Then constructs) for the configuration of Resources, Nodes, and Movements, since it would be overkill (as you can see in the sketch branch, from December 12, 2010): by reading the specs anyone can understand and check the functioning of configuration.<br />
<br />
The comments below are based on the last implementation (January 09, 2011), which doesn't implement Path nor the parsers for the configurators. We are considering writing features for Path configurations, because Paths represent business processes, and therefore, it make sense to have high level descriptions of requirements.<br />
<br />
Currently, the proposal is based on two base classes, from which all others derive. The first one, Configurable (previously known as Maskable), is the base for building the abstract concepts, and the second one, Configurator, is used for providing a "friend" class for each concept which is responsible for managing the configuration of the concepts' instances.<br />
<br />
Besides the basic attributes, Configurable abstract class holds a "To Do" method, which is simply a placeholder to remember that a method for doing "basic things" with instances should be implemented by subclasses. Configurators on the other hand must be able to parse, persist, and retrieve configurations to be applied to concept's instances. They are multitons, in other words, for a given key, composed by configuration's mask and version attributes, only one configurator object may exist. Therefore, every concept class, such as Movement, has a configurator class, such as Movement_Configurator.<br />
<br />
Configurations supply the following attributes for instances (spec files provide complete configuration examples):<br />
-Mask: defines a specific type for the instance. Example: payment (a payment movement).<br />
-Version: holds the version of the mask, multiple versions can be in use at the same moment. Example: monthly (monthly payment).<br />
-Category: associates a category to a specific concept. Example: financial (monthly payment is a movement of category financial). <br />
-Description: provides a description for the key mask+version. Example: ' a monthly payment from a organization to a person'. Note that, in this example, organization and person can also be categories.<br />
<br />
Right now the implementation is in a very simple state, the next step is to implement Path, which is expected to reveal more requirements for making this framework more usable. For testing, we are going to use some real-world workflow implemented in <a href="http://www.erp5.com/">ERP5</a>, applying the mappings presented in the technical report "<a href="http://arxiv.org/abs/1006.4892">Mapping Business Process Modeling constructs to Behavior Driven Development Ubiquitous Language</a>" to describe the workflow using Given-When-Then.Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com0tag:blogger.com,1999:blog-7150235565909810604.post-35711317626668242342010-12-15T21:15:00.003-02:002010-12-23T14:49:13.410-02:00Enterprise Information Systems Patterns - Part IIThe project in <a href="https://github.com/ratem/eispatterns">Github</a> reached an initial implementation, which I separated in a branch named <i>sketch</i>, given that it is mixing configuration and instantiation. The master branch now is in accordance with the proposal, which I continue to discuss below.<br />
<br />
Remember that the core idea is to use the concepts as Lego parts. This mean avoiding using subclasses in general, but masking the abstract concepts through configuration. After configuration, I need to implement specific functionalities for the concepts. These functionalities will be implemented in two places:<br />
a) At instances of concrete concepts, by defining specific behavior for<br />
different contexts (paths).<br />
b) At path objects, in the form of coordination code, which will make the path''s<br />
movements collaborate to realize a business process.<br />
<br />
In that way, we are going to have a two-phased development/customization process:<br />
a) Configuration: defines descriptors, which represent concrete uses of the<br />
abstract concepts. Descriptors list the types used to transform the abstract<br />
concepts into concrete ones. Configuration is done through a Domain Specific<br />
Language (DSL), having each concept a template text to be used for its<br />
configuration. In the future, we expect to define a proper grammar for this DSL.<br />
b) Implementation: uses descriptors to make the concrete concepts instantiable<br />
and implement the specific code related to their concrete use. Each concept has<br />
a callable object with a proper name, which is defined during the implementation<br />
of user stories.<br />
<br />
Thus, in a first moment, a domain specialist will configure concrete concepts<br />
using the DSL. Configurations are reused during the implementation, when<br />
user stories instantiate and define the specific behavior of the concrete<br />
concepts.<br />
<br />
Right now, we have implemented Resource and Node, as well as the Maskable superclass, which will serve all concepts. I can say that the project is in the configuration phase yet, meaning that we haven't implemented any code for "using" the concepts in some business process.<br />
<br />
Regarding the DSL used for configuration, we have to investigate its relation with our <a href="http://eis-development.blogspot.com/2010/09/proof-of-concept-for-bldd-tool-part-i.html">BLDD proposal</a>. Remember that this proposal is used for implementing business processes, while the current DSL is used for configuring objects, which occurs before implementation of the processes. Maybe an execution DSL which uses the same terms used for configuration will grow from this investigation.Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com6tag:blogger.com,1999:blog-7150235565909810604.post-78153874547622888732010-12-09T15:49:00.001-02:002010-12-10T15:53:06.519-02:00Assorted Thoughts on Agilism X Traditionalism - Part V<b>Am I against UP, </b><b>PMBoK, </b><b>UML, and Certification?</b><br />
(a post that summarizes some opinions of mine)<br />
<br />
<b>No</b>, I am against the way these techniques were transformed by some people. I call it the Plague of 40-hours-courses: people hear about some "new" and fashionable technique, and rush to attend to some expensive one-week course on this technique. They don't try to learn from other sources, at most they buy some book to glance sometimes. Usually, they buy expensive tools that "make your process adherent to something-fashionable"*, believing that if they use the tool correctly, everything will work fine.<br />
<br />
If you don't practice, if you don't try things, if you don't drink from different fountains, you won't have a nice process, because organizations differs from each other, and there isn't something like the <a href="http://en.wikipedia.org/wiki/Philosopher%27s_stone"><i>Lapis Philosophorum</i></a>, which will turn any development process into gold.<br />
<br />
The typical managers' answer to the advice above is " we don't have time for experimentation", and then they buy packaged courses and tools, which won't work well many times, making managers go to the next phase of the project for process improvement, which is either blaming the team, either blaming the technique. Others start doing bad things, using the new technique to justify them. <br />
<br />
Managers shouldn't waste time controlling small things, but studying better ways of making things self-controlled, on making their teams work more productively and with more quality, or, in other words, becoming the facilitator of good practices. Things like checking if everyone is filling their daily working time in some "project management" tool does not aggregates value to clients. In fact, doesn't aggregate value to you and your team. What all this has to do with the theme of this post? The answer is that if managers used their times to understand and adapt techniques to their process, instead of acting as controllers, they would use these techniques in better ways.<br />
<br />
The way people transform good ideas into waste-generation machines is amazing! So, people transformed<br />
(i) UP phases into waterfall cycle.<br />
(ii) PMBoK into a heavy and expensive annotation effort.<br />
(iii) Modeling into the main software development activity.<br />
(iv) Certification into a reason for not evolving processes.<br />
<br />
Regarding these techniques, I think that<br />
(i) UP is a nice list of "things to worry about during product construction". But you don't have to use all of its workflows exactly how they are presented. UP is a framework, therefore, it exists to be adapted.<br />
(ii) Similar to UP, PMBoK is a nice list of "things to worry about your process management", but you don't need to use all of its documents. PMBoK is a framework, therefore, it exists to be adapted.<br />
(iii) Modeling is for understanding the domain area and also for communication. The users' demands are the focus, and users need running code, not models. Check out Martin Fowler's <a href="http://martinfowler.com/bliki/UmlAsSketch.html">UML as Sketch</a> principles.<br />
(iv) Certification is a moment to review and document your practices, but you should never forget of continuous improvement. More important than saying to others that you work well is actually working well! IMHO, certification would be a process of checking the quality of products and of asking the clients about your service. Snapshots of the documents used in your process don't provide any real guarantee of quality in the long, or even the medium, term.<br />
<br />
*In fact, worse than the Plague of 40-hours-courses is the Plague-of-Tools-That-Make-Your-Process-Adherent-to-Some-Fashionable-Technique. Some years ago I heard a happy colleague saying that "could finally understand object orientation", after attending a course on an UML case tool...Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com3tag:blogger.com,1999:blog-7150235565909810604.post-77016925288862712832010-12-05T22:06:00.007-02:002010-12-07T16:05:02.410-02:00Assorted Thoughts on Agilism X Traditionalism - Part IVSome days ago I bumped again into an "old" internal discussion by email on the <a href="http://agile2009.agilealliance.org/">Agile 2009</a>'s <a href="http://www.math.tau.ac.il/%7Eamiramy/SoftwareSeminar/agile09.ppt">talk</a> "Lets stop calling it 'agile' ", which summary I copy below:<br />
"<br />
Agile development has grown a lot since its rebeleous 2001 start. In fact, it has grown to be the mainstream way of developing software. The time has come to drop the word "agile." Agile development is just modern practices in software development. There is no need to explicitly mark practices as Agile. There is no need anymore for opposing camps. Keeping the word Agile and things like "the Agile conference" is holding the development of modern SW development practices back. This session will be in debate form to discuss the above mentioned motion.<br />
"<br />
Some can find this radical. When I first read it one year ago, I also thought the same. However, reading it now I can clear understand the proposal, and it made me remember of my way through software processes.<br />
<br />
During my undergraduation course I had contact with Yourdon's <a href="http://en.wikipedia.org/wiki/Structured_analysis">Structured Analysis</a> as well as Object Oriented Analysis & Design, by that time (1990-1994), the first was mainstream and the second a new proposal which needed to be improved. However, the second one attracted me more, given that I wanted to use Object Oriented concepts the most, since my first contacts with C++ in 1991 I was already convinced that OO was a better way of programming.<br />
<br />
<b>First Experiences</b><br />
Outside the classroom, my first professional contact with "analysis" occurred in 1993, when I took part of a Management Information System (MIS) development in Clipper and Structured Analysis and Design. On those days of waterfall cycles, someone did the analysis and design and I implemented the system. As expected, after deploying it, users asked for changes almost every week. Firstly I implemented them and once a week updated the modeling artifacts, for maintenance purposes . Yes, first I changed the programs, because my users were eager to get the software running with the changes, and them, when had time, updated the models. Looking back I see how natural is the idea of using models only to understand the problem, and updating them when needed.<br />
<br />
In early 1994 I inverted positions, acting as the analyst for another MIS, using Structured Analysis and a classical waterfall cycle. The implementation was successful until at least its deployment, later in the year. After that, I lost contact with it.<br />
<br />
In the second semester of 1994 I entered a bigger Enterprise Information System project, with circa 15 developers and two consulting companies working on it. It was very uncommon environment, using <a href="http://en.wikipedia.org/wiki/IDEF0">IDEF</a> artifacts for modeling, C++ for programming, and Sybase. Transformation from functional models to C++ methods were done during interesting <a href="http://en.wikipedia.org/wiki/Crc_cards">CRC Cards</a> sessions, and class diagrams were derived from <a href="http://en.wikipedia.org/wiki/Entity-relationship_diagrams">Entity-Relationship Diagrams</a>. Object-relational mapping was done according a set of very basic rules, created internally by the team. The only problem we had was the fact that some process models were prepared almost one year before we started to touch them. However, we had a very good business analyst (he was an Industrial Engineer) who re-checked all business process with the clients before they enter the programming phase. Therefore I can say that the process was interesting, however, the "waterfall way" of doing things created a lot of rework.<br />
<br />
Before I forget, effort estimating in all cases was done by "expert opinion", people didn't believe in <a href="http://en.wikipedia.org/wiki/Cocomo">Cocomo</a>'s use of constants and in <a href="http://en.wikipedia.org/wiki/Function_points">Function Points</a> counting - which needs a lot of things already defined to have a chance of being "accurate". I could only remember my Software Engineering classes and started to see them as something closer to Alice in Wonderland than to an Engineering discipline.<br />
<br />
<b>Enters Object Modeling Technique (OMT)</b><br />
However, it was only in 1995 that I had the first chance of using professionally an OO method in the whole process, specifically, I used the <a href="http://en.wikipedia.org/wiki/Object-modeling_technique">OMT method</a>. OMT was interesting because it used a mixture of class and state diagrams with Data Flow Diagrams (DFD's) to describe its Functional Model. OMT was a product of its time: OO languages were becoming mainstream, but Structured Analysis was still the mainstream process paradigm.<br />
<br />
Our Environment in 1995<br />
In this year I moved back to my hometown, aiming to take part of the creation of the first big and public university in the region. We were a two men team, in partial time, working at the Math Laboratory - from where the Computer Science Laboratory would emerge years after. The mission was to develop a library management system to be used by the three campus' libraries. The university was new and many, many information systems were necessary. Library Management Systems where really expensive by that time, thus the university's president asked us to create the system. <br />
<br />
<br />
Our Process in 1995<br />
We decided to use a mixture of Prototyping Life Cycle with OMT's artifacts to build the system, since we had very few knowledge in the problem area (libraries). For more complex operations, we would use CRC cards to define how classes would interact. We decided to use Delphi 1.0, which allowed us to create nice GUI as well as use Pascal, instead of Visual Basic (which I hated). Moreover, my colleague was developing an object-relational mapping library in Pascal for his Master's Thesis, which gave us a cleaner and powerful way of doing Object-Relational Mapping (ORM). We resisted the temptation to use Delphi's components for direct access to the database for the sake of really using object orientation for coding business logic.<br />
<br />
<br />
Firstly we drafted a list of high level requirements - separated in modules, and then we asked the librarian to prioritize them. Then we started to follow a process that we built on top of our experiences in previous systems: <br />
1) Get the next priority.<br />
2) Detail its requirements.<br />
3) Create a prototypical GUI and a draft database to reflect the requirements.<br />
4) Re-check requirements during GUI presentation to the user.<br />
5) Model stuff in OMT.<br />
6) When appropriate, use CRC cards to define classes' responsibilities and collaborators.<br />
7) Write code ignoring the database*.<br />
8) Write ORM and database code.<br />
9) Present the implementation to the user.<br />
10) If the user was happy with the results:<br />
a) we deployed the new module and went back to 1)<br />
else<br />
b) we looped back to 5) if modeling was needed, 6) if changes in classes were likely to occur, even without new modeling, or 7) if only changes in algorithms were needed.<br />
<b><br />
</b><br />
* We used an ancestor technique of what is called today "test doubles". In our case, collections of objects with typical values were created "by hand", so that we could test the algorithms. When we put the algorithms to work properly, we then implemented the persistence code, and finally we took way these "fake objects". Sometimes I developed algorithms and my colleague put things to work with the database, since he was the ORM library developer. We didn't want to waste time with persistence problems while we were trying to solve users' problems. Persistence was a pure technical question, not a main development concern.<br />
<br />
Effort estimation was done before each of these "proto"-iterations, by experience, because I kept the same opinion regarding effort metrics. The project was really successful, we both left the university in the end of 1996, however it was used for years, until the university bought a new system, with external support from a development company.<br />
<br />
I used this process more or less unchanged until 2000, when I had access to my first UML Case Tool, and started reading about the <a href="http://en.wikipedia.org/wiki/Unified_process">Unified Process</a> and <a href="http://portal.acm.org/citation.cfm?id=552474">Eriksson-Penker's</a> business modeling extensions for UML. I knew UML since 1997, however, by that time, and until 1999, I had only one project, which used OMT already, and I had no reasons to change from it to UML. I used UP variations until <a href="http://eis-development.blogspot.com/2010/11/assorted-thoughts-on-agilism-x_24.html">2006/2007</a>, sometimes without the expected output. And all times not fully implemented, since I thought many things were too bureaucratic to be used.<br />
<br />
<br />
The main thing I learned from these days was that you must mix techniques and even create new ones to make the development process work for your environment, and not the <a href="http://eis-development.blogspot.com/2010/09/on-certifications-and-tribal-culture.html">opposite</a>. Hardly you will find a process that fits exactly to your environment, team, culture, and problem. I've seen a lot of people eager to follow some guru or certification, and when things go wrong they blame themselves for not being able to do exactly "what must be done." They should blame themselves for not being able to adapt the process to their environment. <br />
<br />
<br />
Another thing I can clear identify now is that some recent Agile techniques are modern versions of old programming tricks, those kind of hints shared during the coffee break, and that never appeared in the "software engineering" books.<br />
<b><br />
</b><br />
<b>Going back to the "drop the word agile"</b><br />
I think my intimate history in dealing with development process is the same of a lot of others developers. In early 1990's, there were a lot of different Object Oriented notations and processes, all seeking to solve the same problems, in slightly different ways. Hybrid methods such as OMT (and its DFDs) were told to be the ideal solution for a world in transition. And for me they seemed to be.<br />
<br />
Then came UML and UP, both bringing some standardization to the software development world. Finally, <a href="http://en.wikipedia.org/wiki/Model_driven_development">Model Driven Engineering</a> (MDE) and Model Driven Development (MDD) could realize their full potential. And they did, given that I think their potential has finished. At least, the potential to be innovative in research terms, and to answer to constant changes in practice terms.<br />
<br />
I agree with the phrase "Agile development is just modern practices in software development." Yes, one should try to list a single innovation that came from the MDD world in the last few years. Every "new" MDD technique that I saw recently, either was repetition, either was "extreme cases", hardly useful.<br />
<br />
Many people still resist to <a href="http://en.wikipedia.org/wiki/Test_Driven_Development">Test Driven Development</a> (TDD), for instance, because they can't understand how tests can be used for design. There is a common misunderstanding that design is related only to blueprints, thus, design = modeling in software development. What about <a href="http://en.wikipedia.org/wiki/Stress_analysis">Stress Analysis</a>, mostly based on Math and Physics, instead of drawings... so design is not only about blueprints, and, actually, if you can rely on Math to do the design, it is more likely that your design will work. <a href="http://eis-development.blogspot.com/2010/11/assorted-thoughts-on-agilism-x.html">As I said before</a>, coding is closer to Math than modeling.<br />
<br />
For a developer to understand how TDD can drive design, he/she must practice it. You can draft a model to understand the problem in the beginning, however, given that you are going to interact with tests - which are code - and the code that answer to them, you are going to evolve the design as the need for testing evolves. An tests evolve as the implementation of requirements evolves. The problem is that many decision makers are those that "don't program anymore", so will "never" be convinced, unless some programmer show to them how things work. <br />
<br />
There is also an economic and cultural reason for people not recognizing agilism as the "modern" software engineering: agile techniques mostly were - and still are - nurtured by individuals and communities of developers, in opposition to big consultancy companies. Because of this, Agile techniques will take longer to enter the EIS world. People will repeat to the exhaustion "Why big companies like A or B don't use a lot of Agile if it is so good?", and will answer to themselves "because it doesn't scale!". Isn't because you don't need expensive tools and related consultancy to adopt them? Isn't because of culture? Isn't because a lot of people who stopped programming after becoming project managers are afraid of seeing part of their daily activities labeled "waste", thus endangering their jobs?<br />
<br />
I witnessed similar reactions in the early 1990's regarding Object Orientation, and if the "legacy guys" were right, I would be saying stuff like "Structured Analysis rules!".<br />
<br />
So, drop the word Agile!Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com9tag:blogger.com,1999:blog-7150235565909810604.post-62729024593516396112010-12-01T19:59:00.004-02:002011-03-14T17:20:08.967-03:00Enterprise Information Systems Patterns - Part IThis post aims to start the discussion on design patterns for EIS. We have just started a <a href="https://github.com/ratem/eispatterns">Github project</a>, which right now is just a <b>rough sketch</b>, however, it already shows some basic ideas, implemented in Python with BDD (<a href="https://github.com/hugobr/specloud">specloud</a>, <a href="http://www.should-dsl.info/">should-dsl</a>, and <a href="http://lettuce.it/">Lettuce</a>). By checking the features (*.feature files) you can understand the patterns requirements, and by checking the specs (*_spec.py files) you can understand how the code implements them.<br />
<br />
We decided to start by revisiting <a href="http://www.erp5.com/">ERP5</a> <a href="http://ieeexplore.ieee.org/Xplore/login.jsp?url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F6294%2F27356%2F01216231.pdf%3Farnumber%3D1216231&authDecision=-203">concepts</a>, separating them from ERP5's underlying platform, <a href="http://www.zope.org/">Zope,</a> and, in some cases, even reinterpreting them. The concepts are the following:<br />
<br />
Concept 1: Resource<br />
A resource is anything that is used for production. It can be a material, money,<br />
machine time, or even human skills.<br />
Concept 2: Node<br />
Is something that transforms resources. For instance, a machine, a factory, a<br />
bank account.<br />
Concept 3: Movement is a movement of any Resource from one Node to another.<br />
Concept 4: Path*<br />
It describes how Movements are used to implement business processes.<br />
<br />
ERP5 also has the concept of Item, an instance of a Resource. Until this point, we<br />
decided not to use this, otherwise it would be necessary to create classes for<br />
instancing the other three abstract concepts.<br />
<br />
The core idea is to use the concepts as Lego parts. This mean not using<br />
subclasses in general, but masking the abstract concepts through<br />
configuration. For instance, a Movement is firstly configured as a concrete<br />
movement, such as transferring goods from supplier to customer. After that, it<br />
can be instantiated. Of course, at some point extra coding is necessary, and we believe that methods for implementing specific algorithms will emerge <br />
on the coordinator element, the Path instances.<br />
<br />
We therefore define two new "patterns": Mask, which means to define an abstract concept and turn it into a concrete concept by a proper configuration, and Coordinator, which is an object (also maskable) that coordinates others to realize a given business process. <b>Maybe</b> these patterns already exist with others names and/or similar purposes, however, we will check this more carefully as the project goes forward (we are really iterative & incremental).<br />
<br />
The motivation for this project is that people like to say that frameworks are like Lego, but when you need to subclass their concepts, instead of using them <i>as is</i>, I think this is not the "Lego way" of doing things. We only use masks to provide useful labels for users, but the intention is not to extend the original concepts. When I mask a Movement as a Sale, I am only setting its mask attribute to 'sale'.<br />
<br />
The only concept that will be extended is Path (nothing is perfect!), since it will be used to implement the specific algorithms necessary to appropriately integrate movements - realizing the business process.<br />
<br />
We probably won't supply something really useful for programming, because we are not considering platform issues, therefore, if you need a ready to use ERP framework in Python, check ERP5. The intention is to discuss how to build highly configurable frameworks, instead of proving a real framework. After we get the basic configuration machinery done, we are going to use BLDD to implement didactic cases: from the business process modeling to coding and testing.<br />
<br />
(The <a href="http://eis-development.blogspot.com/p/eis-patterns-change-log.html">Change Log</a> maps this series of posts to commits in GitHub)Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com2tag:blogger.com,1999:blog-7150235565909810604.post-65989357554604156412010-11-27T17:53:00.001-02:002010-11-27T18:05:57.103-02:00Proof of Concept for the BLDD Tool - Part VIWe have released a compilation of the posts with some more explanations on our proposal for the Business Language Driven Development (BLDD) <b>draft</b>, the technical report can be found <a href="http://arxiv.org/abs/1011.2238">here</a>.<br />
<br />
There is a lot of work to do on top of BLDD, such as:<br />
- Defining in more detail the use of a companion textual Domain Specific Language (DSL). Probably with support of a Rule Engine.<br />
- Implement an usable tool, we are investigating an extension of <a href="http://jboss.org/drools">Drools</a> for this purpose. Drools seems to be a good choice because it is BPMN compatible, implements a Rule Engine, and an Eclipse plugin. And, of course, it is open source. <br />
- Implement BLDD on top of an ERP, our choice is <a href="http://www.erp5.com/">ERP5</a>, for the reasons listed on the first post of this series. This case study should show the applicability of the proposal.<br />
<br />
You can find more "future work" in the report. We are open to collaboration and discussion.Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com0tag:blogger.com,1999:blog-7150235565909810604.post-83977552220069420402010-11-24T22:04:00.006-02:002010-11-25T16:55:48.335-02:00Assorted Thoughts on Agilism X Traditionalism - Part IIIThis blog focuses on Free/Open Source (FOSS) and Agile/Lean (AL) for Enterprise Information Systems (EIS). NSI has been using Free/Open Source since its creation, in 2002 (I use since 1998), however, AL is relatively new to us. I think it is interesting to tell how we "realized" AL.<br />
<br />
<b>Our Turning Point</b><br />
<br />
In a broader sense, <a href="http://nsi.iff.edu.br/">we</a> started using automated testing in 2005, when we were developing a system for vibration analysis in LabView, which uses G, a visual language for Virtual Instrumentation. For us to test it, we had to simulate working gas turbines, which means to generate matrices with thousands of lines and dozens of columns, representing the vibration signals produced by dozens of sensors. In fact, we had to develop a complete simulation module that excited the analysis algorithms. Curiously, parts of this module were created before the algorithms, because we needed to understand better the behavior of a real equipment, in a kind of "proto" test-first way. Also, we filled vectors and matrices with trustful results obtained from laboratory's hardware and software, and automatically compared with our results, to check our algorithms' accuracy.<br />
<br />
In the realm of Enterprise Information Systems, our use of automated testing was initiated only in 2007, when we started two important Enterprise Content Management (ECM) projects in parallel. We started to automate the "black box" tests - because we had no patience to simulate dozens of time the same users actions - and load tests, for obvious reasons. However, our unit tests weren't disciplined and sometimes were hampered by the platform we used. Obviously, the acceptance tests started to find wrong behavior in the software, given the lack of complete unit testing. By that time (with results published in 2007/2008), and in parallel, we developed a hybrid development process for the <a href="http://www.erp5.com/">ERP5</a> system that used MDD (including <a href="http://www.computer.org/portal/web/csdl/doi/10.1109/MITP.2008.100">a nice set of supportive tools</a>) with automated testing.<br />
<br />
The year of 2008 represented our turning point, because of three factors:<br />
(i) We realized that our MDD-based process for ERP5 was sound, however, our industrial partners needed something leaner, and MDD didn't made much sense when developing on top of a highly reusable framework. Moreover, refactoring made modeling a waste in general. We realized in practice that a good design can be done on top of code, instead of models.<br />
<br />
(ii) I managed to bring to the team an <a href="https://github.com/rodrigomanhaes">associate researcher</a> who was (is) a very competent developer, and was (and still is) experiencing very good results by using Agile Methods. He made me remember of my Master Thesis (1997), which proposed a hybrid push-pull production planning and control method: instead of using complicated production programming algorithms, which needed very accurate (and usually inexistent) informations, Kanbans were used to smooth the production flow.<br />
<br />
(iii) I read the book <a href="http://www.amazon.co.uk/Ghost-Force-History-Military-Paperbacks/dp/0304363677">Ghost Force - The Secret History of SAS, written by Ken Connor</a>, and realized that trying to use conventional methods for managing unconventional projects and teams was stupid. After exposing my impressions on some team coordination techniques narrated by the book, we immediately adopted two principles*: a) one man, one vote (the Chinese Parliament) and b) the one that holds more knowledge in the specific mission, leads it, doesn't matter his/her rank. Of course, we already knew that the most valuable asset of any team is its human resources.<br />
<br />
*strangely my team didn't accept things like parashooting training, as well as other really interesting techniques presented in the book ;-)<br />
<br />
So, I can say that we woke up for agile techniques as a whole in 2007/2008. In 2008 we started to use a <a href="http://leansoftwareengineering.com/ksse/scrum-ban/">Scrumban</a> process, a mixture of Scrum for planning with kanbans for programming the production. Things weren't enforced, we <b>realized</b> that disciplined automated testing and Scrumban was superior for our environment, and that's it. When I say we, I am referring to everyone, from vocational to master students and researchers (one man, one vote, doesn't matter the rank...). In 2009 we started to use BDD, actually, we developed <a href="http://arxiv.org/abs/1007.1722">a tool stack for BDD in Python</a> to serve to our projects. <br />
<br />
In other words, although <a href="http://nsi.iff.edu.br/">we</a> are a small organization (circa 30 developers in different projects, divided into development cells, plus some 10 aggregated people), we came from a traditional way of thinking (but we never were bureaucrats) that in fact never worked well for us. In other words, we use AL because we realized in practice that it is better for us, not because it is fashionable. An interesting point is that our last paper on MDD + Automated Testing was cited by a very well known Software Engineering researcher as one of the references for modeling EIS. <br />
<br />
We studied and tested the techniques by ourselves, using the Chinese Parliament and The Best in the Mission is The Leader principles to put them in practice. We did right and wrong things, adapted methods, mixed techniques, created <a href="https://github.com/rodrigomanhaes/should-dsl">tools for BDD</a>, for <a href="https://github.com/hugomaiavieira/batraquio">improving programming productivity</a>, <a href="https://github.com/nsigustavo/ludibrio">for automated testing</a>, for <a href="https://github.com/hugomaiavieira/kanban-roots">kanban control</a>, and many others, only to cite a few, and we also used other people's tools. Basically, <b>we discussed a lot</b>. In other words, instead of following some manual, or standard, or course, or pursuing some certification, we created project management knowledge! A good result of this is that all five students that left our research group this year are working for innovative organizations, with salaries higher than the average. Moreover, they got their positions in highly competitive markets, including Europe. They were all hired in the same way: staying some days in the organizations, programming with the teams and taking part of meetings.<br />
<br />
An intimate commentary: now I feel totally comfortable to teach project management and tell my daily experiences. Before that I had to teach techniques that didn't work for us, but I had to insist on teaching them because the books said that they worked - while feeling as an incompetent project manager. Now I can see that it doesn't work because this type of process cannot deal well with uncertainty in general (maybe really I am not a good project manager, however, now thing work well for us anyway).<br />
<br />
A last commentary: although we are part of a teaching & research organization, we've been developing serious projects with companies and government agencies for years. When I say serious, I mean projects with deadlines, fixed budgets, lots of users, need of continuous support and evolution, and that are expected to aggregate value to the users' organizations. Every single software we developed since 2002 is/was either used internally - in our development process - or by our industrial and government partners. Moreover, since we develop free software, our products are also used by other people and organizations besides our formal partners.Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com0tag:blogger.com,1999:blog-7150235565909810604.post-74456398680153971682010-11-14T22:24:00.001-02:002010-11-15T21:42:02.614-02:00Assorted Thoughts on Agilism X Traditionalism - Part II<b>Back to Basics</b><br />
<br />
Management has three basic functions: Planning, Execution, and Control. We can summarize the basic differences between traditional methods and agilism/lean in a table:<br />
<br />
<style type="text/css">
td p { margin-bottom: 0cm; }p { margin-bottom: 0.21cm; }
</style> <br />
<table border="1" cellpadding="4" cellspacing="0"><col width="200*"></col> <col width="64*"></col> <col width="64*"></col> <col width="64*"></col> <tbody>
<tr valign="TOP"> <td width="25%"><div align="CENTER" class="western">Philosophy</div></td> <td width="25%"><div align="CENTER" class="western">Planning</div></td> <td width="25%"><div align="CENTER" class="western">Execution</div></td> <td width="25%"><div align="CENTER" class="western">Control</div></td> </tr>
<tr valign="TOP"> <td width="25%"><div align="LEFT" class="western">Traditional</div></td> <td width="25%"><div align="LEFT" class="western">-Based on effort metrics to plan the whole project<br />
<br />
-Change is a “risk”</div></td> <td width="25%"><div align="LEFT" class="western">-Design = models<br />
-Try to automate programming <br />
-Centralized decision-making<br />
-Quality a posteriori, by annotation, and responsibility of a specific team</div></td> <td width="25%"><div align="LEFT" class="western">-Centralized<br />
-Try to be static (follow the original plan)</div></td> </tr>
<tr valign="TOP"> <td width="25%"><div align="LEFT" class="western">Agile/Lean</div></td> <td width="25%"><div align="LEFT" class="western">-Based on collective evaluation and on measuring the current development velocity<br />
-Plan for the next period<br />
-Embrace changes</div></td> <td width="25%"><div align="LEFT" class="western">-Design = code<br />
-Automate repetitive tasks <br />
-Decentralized decision-making<br />
-Quality a priori, automated, responsibility of producers</div></td> <td width="25%"><div align="LEFT" class="western">-Decentralized<br />
-Try to be dynamic (follow the demand)</div></td></tr>
</tbody></table><br />
In my opinion, the difference is basically one: Agile and Lean seeks to be dynamic, Traditional seeks to be static.Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com0tag:blogger.com,1999:blog-7150235565909810604.post-19216669587652167942010-11-12T20:07:00.005-02:002010-11-14T18:52:25.040-02:00Assorted Thoughts on Agilism X Traditionalism - Part ISome days ago I took part on a debate on Agilism X Traditionalism for Software Quality Assurance. Firstly I accepted to take part of it after its organizers changed the original name: "Agilism X Software Quality", as if Agilism were against Quality! Yes, there's still people that thinks that Agilism means non-organized, low-quality development, simply because agilists don't like to take a lot of notes or of using a lot of models. It reminds me of the early days of Free/Open Source, when most people that were against it never took part of a FOSS project, or even "touched" any open source software.<br />
<br />
I read somewhere that Agilism (like Lean Thinking) means a change in the mindset. That's why, IMHO, that many people cannot understand how Agilism works. The bureaucratic mindset cannot understand how a technique that doesn't use a lot of annotations, long meetings, change management boards, hierarchical planning etc can work.<br />
<br />
That's the same mindset found on USA's automobile industry in the 60's and 70's that was defeated by Japanese car builders. USA industry couldn't understand how a plant controlled by paper cards could work better than the enormous management structure they had, which was matured on top of decades of experience in building car - after all, they had Ford! <br />
<br />
But the Japanese did. <br />
<br />
And they still do, in spite of the current Toyota's quality crisis, its followers still reign in the automobilistic industry. I see people from the traditional methods in software industry with the same arguments:<br />
<br />
-Agile doesn't scale = JIT doesn't scale.<br />
-You need to plan in detail beforehand* = JIT has no planning tools.<br />
-You need to take notes of everything you do otherwise you will loose control = for JIT taking notes is considered waste most of times, do things in a way you don't need to take notes.<br />
-How can quality be controlled by developers, instead of a separate group = how can workers control quality, instead of Quality Control Department.<br />
-etc<br />
<br />
*(<a href="http://eis-development.blogspot.com/2010/10/effort-metrics-sophisma-part-i.html">although detailed and accurate plans for software development are unfeasible in real word</a>)<br />
<br />
The truth is that things like automated testing, continuous integration, refactoring, and all other modern development techniques were born in the Agile cradle. These are Product related tools. Traditionalists still play with Process related tools, repeating the "if the process is good the product will be good" mantra. Yes, we need a good process, sure, however, you must try to avoid become <a href="http://eis-development.blogspot.com/2010/09/on-certifications-and-tribal-culture_26.html">mesmerized by the beauty of complex project management processes</a>. Instead of putting a lot of people to take notes of a lot of stuff and correct your planning all the time, try to avoid problems, instead of trying to correct them. The focus must be on the product, therefore, product related tools will bring much value to your production chain than process related tools.<br />
<br />
I believe that the only really scientific way of producing software is through Formal Methods (FM), because it represents the use of Mathematical principles for building software. If someone can use the term Software Engineering, are the "FM guys". The problem with FM is that they are still quite hard to understand and apply, and there are only a few people really capable of using them to produce software. While FM doesn't produce things like the tables that engineers use to simplify their calculations, FM won't reach the mainstream. FM methods are still dependent on developing "ad-hoc" mathematical models, and the ordinary developer doesn't have the skills for that - like the ordinary engineer, who doesn't develop mathematical models, instead, make use of them. Maybe when FM become usable by the ordinary developer we will be able to (really) do Software Engineering.<br />
<br />
Therefore, we must, at least, be the closest to FM we can. Now, tell me what's more formal:<br />
-Running code.<br />
-Drawing models and using manual annotations to connect them to code*?<br />
<br />
*(code generation is mostly limited to skeletons, someone will write algorithms anyway, therefore you will need to connect things by hand in the end).<br />
<br />
You must agree with me that running code is much more formal than models and annotations, simply because the first is the thing of what software is done. So, please don't tell me that MDD is "more scientific than Agile because has the same principles of the other Engineering disciplines"... Running code means a functional machine, and that's the basic principle of... Engineering!<br />
<br />
ps.: Like in the (hard) industry, Agilism automates repetitive tasks, such as testing and building, and this automation provides safer software verification. The other "end", Validation, is provided by BDD.Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com0tag:blogger.com,1999:blog-7150235565909810604.post-73010709650874598282010-11-10T15:37:00.001-02:002010-11-11T19:54:08.911-02:00Effort Metrics Sophisma - Part IV<div class="post-header"></div>This a summarization and translation to English of a <a href="http://eis-development.blogspot.com/2010/09/effort-metrics-sophisma-in-portuguese.html">presentation</a> prepared by me and an <a href="http://github.com/rodrigomanhaes">associated researcher</a> in 2008. It was used to support our arguments against detailed one-year plans, a demand from the management of a big and interesting project we've been taking part since 2006.<br />
<br />
<br />
<b>So, what's the point?</b><br />
The point is that the starting point is wrong! When people from the software development area sought for references in planning in the industry they thought it was obvious to check the techniques used by the custom production industry - where each product represents a different production project, and therefore, design is done product by product - not from the repetitive production industry, after all, software development is custom production, instead of reproduction of the same design in large quantities.<br />
<br />
However, the reality is not that simple. While custom production runs under a low uncertainty environment (given that the product design was done beforehand), repetitive production usually runs under uncertainty - demand uncertainty.<br />
<br />
Demand uncertainty is what happens to software. I not saying uncertainty in quantity, but uncertainty in building <b>effort</b>. Therefore, although we are talking here about a custom product, the techniques that apply to its production planning activities are the ones related to repetitive production.<br />
<br />
In my opinion, those related to Lean Thinking... I am going to talk about this in future posts, now let's go to our third and last logical chaining.<br />
<br />
<b>Third Logical Chaining</b><br />
From the <a href="http://eis-development.blogspot.com/2010/11/effort-metrics-sophisma-part-iii.html">second chaining</a>:<br />
Knowledge based production AND intangible good → there is no physical metrics <br />
Non physical metrics → inaccurate estimates<br />
We have:<br />
Inaccurate estimates → high uncertainty in the production process<br />
AND<br />
Custom Production → low uncertainty in the production process is expected<br />
Then:<br />
Although software development is custom production, it is not reasonable to use industry's custom production planning methods because they expect low uncertainty in the production process, but what we have is high uncertainty !!!<br />
<br />
<br />
Now that I have shown, with some formality, that the current effort metrics don't work for software, I feel more comfortable to go forward and explain how to use Agile and Lean Thinking methods for software development planning.Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com0tag:blogger.com,1999:blog-7150235565909810604.post-65367440179545643352010-11-05T15:58:00.007-02:002010-11-11T19:53:48.882-02:00Effort Metrics Sophisma - Part IIIThis a summarization and translation to English of a <a href="http://eis-development.blogspot.com/2010/09/effort-metrics-sophisma-in-portuguese.html">presentation</a> prepared by me and an <a href="http://github.com/rodrigomanhaes">associated researcher</a> in 2008. It was used to support our arguments against detailed one-year plans, a demand from the management of a big and interesting project we've been taking part since 2006.<br />
<br />
<br />
<b>The (Tough) Software Reality </b><br />
Unfortunately, software is an intangible good, produced by knowledge workers. An intangible product means that it is not possible to measure its physical features, such as weight or diameter. Production based on knowledge is also hard to measure, because it isn't repetitive: there isn't something like the industry's "medium operation leadtime". Moreover, software development is highly dependent of people*, with all their specificities, problems, way of thinking etc.<br />
<br />
<br />
(*<span class="long_text" id="result_box" lang="en"><span title="">If MDD were really able to drastically reduce the need for programming, organizations would no longer be hiring lots of programmers ...</span></span>)<br />
<br />
<br />
In other words, uncertainty in software development comes from:<br />
- Non-repetitive and creativity-based work.<br />
- Constant changes in technology.<br />
- Some tasks considered as simple behave as complex.<br />
- Frequent changes in requirements. <br />
- Necessity to learn about the domain area during the development<br />
(the above arguments are also valid for defending iterative & incremental cycles...)<br />
<br />
<b>Second Logical Chaining </b><br />
Software → intangible good, non-physical<br />
Software → knowledge based production, knowledge is a non-physical resource<br />
Knowledge based production AND intangible good → there is no physical metrics (by definition)<br />
Non physical metrics → inaccurate estimates<br />
<a href="http://eis-development.blogspot.com/2010/10/effort-metrics-sophisma-part-i.html">From the first chaining</a>:<br />
Detailed planning needs → detailed estimates <br />
However<br />
Software → inaccurate estimates<br />
Inaccurate estimates → non detailed estimates<br />
Then<br />
Non detailed estimates → <b>Non</b> detailed planning<br />
<br />
<span style="color: red;">In other words, it is not feasible to produce detailed planning for software, even when using waterfall cycles !!!!!</span><br />
<br />
The central point for reaching this sad conclusion is<br />
Non physical metrics → inaccurate estimates<br />
<br />
I leave this hook here so that people can try to dismantle our arguments: if you prove that you can have accurate estimates without having physical metrics, the argument here is broken.<br />
<br />
<b>More Conclusions</b><br />
Detailed planning works in the hard industry, however some people try to transplant it to the software development industry in carelessly ways. Many organizations insist in this broken model, bringing high costs to the control and (re)planning functions, and in the end the goal is not to deliver a good quality product ontime anymore, but to create a false notion of control and of showing that managers are able of "thinking ahead" (although this "ahead" is unknown in the real word...).<br />
<br />
For business people, not being able to create a detailed plan for months ahead is something like signing up a "proof of incompetency", many prefer to create plans that need strict control and even so, their baselines change all the time. At this point, the tail starts to wag the dog...Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com1tag:blogger.com,1999:blog-7150235565909810604.post-42440919980519227162010-11-02T18:04:00.003-02:002010-11-11T19:53:32.858-02:00Effort Metrics Sophisma - Part IIThis a summarization and translation to English of a <a href="http://eis-development.blogspot.com/2010/09/effort-metrics-sophisma-in-portuguese.html">presentation</a> prepared by me and an <a href="http://github.com/rodrigomanhaes">associated researcher</a> in 2008. It was used to support our arguments against detailed one-year plans, a demand from the management of a big and interesting project we've been taking part since 2006.<br />
<br />
<br />
<b>Interruption</b><br />
<br />
<br />
I will interrupt a bit this series because I have just bumped into an "old" article on software metrics, about which I would like to comment briefly: Jones, C. Software Metrics: Good, Bad and Missing, <span class="abs-title">Computer</span>, vol. 27, no. 9, pp. 98-100, Sept. 1994. <br />
Its abstract says<br />
"<br />
<div class="abs-articlesummary">The software industry is an embarrassment when it comes to measurement and metrics. Many software managers and practitioners, including tenured academics in software engineering and computer science, seem to know little or nothing about these topics. <b>Many of the measurements found in the software literature are not used with enough precision to replicate the author's findings-a canon of scientific writing in other fields.</b> Several of the most widely used software metrics have been proved unworkable, yet they continue to show up in books, encyclopedias, and refereed journals.<br />
" <br />
<br />
Although it was published 16 years ago, I think the problems listed on this paper are still valid, and the paragraph above manifests my opinion on the use of metrics for planning in software: they only work under certain conditions and may have worked for a given sample of projects, but may not work for my project. In my humble opinion, the current metrics don't present a sufficient level of determinism so that we can use them seriously for planning.<br />
<br />
As I said in this series' first post, if I produce a brick here in Brazil, it will present the same size and weight in China, for instance. If I use the correct Engineering units and calculations, someone will reproduce "my experiment" of using these bricks anywhere in the Globe*. But the same doesn't occur with software requirements and thus with source code. I can send the same requirements for different people and they will implement them using different languages, paradigms, algorithms...<br />
<br />
*(before someone say that bricks done in Brazil can present different plastic behavior in other regions of the Globe, I say that Materials Science has already developed methods to deal with this, while Software "Engineering" is still far away from doing the same for software.)<br />
<br />
It is amazing how people use those constants found in some metrics (such as the one found in Use Case Points, which still gives 47,000 hits in Google...) without thinking that they maybe be not applicable to their specific project!<br />
<br />
Therefore, if "So long as these invalid metrics are used carelessly, there can be no true software engineering, only a kind of amateurish craft that uses rough approximations instead of precise measurement.", I prefer to use experience and short iterations for planning.<br />
<br />
ps. 1: the cited paper defends the use of Function Points, which I think is not applicable to Iterative & Incremental cycles.<br />
<br />
ps. 2: I will repost a <a href="http://eis-development.blogspot.com/2010/09/why-agile-for-eis-part-ii.html">previous remark</a> on UCP: Tell me how can you use it in an (really) Iterative & Incremental project (remember that real I&I cycles only details Use Cases when needed, so how are you going to count the number of steps for every use case and then get a "precise" effort estimation?). In spite of this, many teachers still teach UCP and I&I cycles in the same package...</div>Rogerio Atem de Carvalhohttp://www.blogger.com/profile/02803759826148716394noreply@blogger.com0