Ideas on Enterprise Information Systems Development

This blog is devoted to ideas on Enterprise Information Systems (EIS) development. It focuses on Lean Thinking, Agile Methods, and Free/Open Source Software, as means of improving EIS development and evolution, under a more practical than academical view. You may find here a lot of "thinking aloud" material, sometimes without scientific treatment... don't worry, this is a blog!
Every post is marked with at least one of Product or Process labels, meaning that they are related to execution techniques (programming and testing) or management techniques (planning and monitoring), respectively.

Thursday, September 30, 2010

On Certifications and Tribal Culture - Part III

In the two previous posts of this series I told a case and a tale. Although the first one is real and the second is a fantasy, they hold a similarity: both are stories about how people loose the focus on the product they should deliver and start worrying more about the execution of a given process, which may have worked in the past or in some specific situations, but is not necessarily good right now. In both cases we could see a process that was formalized, is "scientific", and reached its goals - however, with a lot of resource waste.

So what's this all about?

It is about instead of trying to solve problems, you must try to avoid they happen. This is quite obvious, however, many, many people waste resources by solving problems that could be avoided, because they are stuck to assumptions that are no longer valid, to techniques that are obsolete, and mainly, because they have difficulties in promoting changes to their process, given political and cultural resistances.

Is is also about keeping the focus on the product, I mean, to subordinate the process to the product, not the opposite. In the isolated tribe tale, do you remember when one of the specialists says "He has no experience in burning forests"? What's your focus, burning forest or having roasted pork? Again the answer is obvious, however many, many companies are proudly burning forests around the Globe to have roasted pork...

Why keeping a document representing a Requirements Traceability Matrix if you can automatically and safely connect requirements to code? Because I know how to do a traceability matrix, because I don't know automate tests, because it is so hard to change our process... Sorry, if you used one of these answers, you are burning forests! You are now solving problems caused by your own solutions...

Ok, but what certification has to do with all that? I believe that, in the end, certification is about following a given process, making people loose the focus on the final product. Unfortunately, all certified teams that I know use expensive and slow processes - this doesn't mean they are bad teams, they are slow. And I also know some very good teams without a single certification. Thus, certification doesn't prove that your team is good in the medium term. The only proof of quality is a list of satisfied customers...

You must review your process all the time, and make it flexible enough to be changed in relatively cheap ways. Check which activities aggregate value to the product and make you reach the client's goals faster, safer and cheaper. If an activity doesn't aggregate value to the user and isn't enforced by Law, Environmental questions or Moral, it is waste. If it is vital to your process, maybe your process is wrong. It may have worked in the past, but it needs to be reviewed.

Every activity that is performed to correct errors is waste in general. On the other hand, avoiding errors is to promote process improvement. Yes, welcome to  Lean Thinking!

Keep your eyes on product quality and costs, and in process leadtime and responsiveness to the demand. Make everything run, even documentation - and requirements in special, because your users won't interact with traceability matrices and stuff like that, even indirectly.

Ah, and one more thing: try to automate the repetitive tasks, not the creative ones! Software is about knowledge!

Sunday, September 26, 2010

On Certifications and Tribal Culture - Part II

Now it is time to tell a tale, which I heard from an associated who in turn read it from somewhere I can't remember, so sorry for not referencing the original and for changing its last 20%.

Once upon a time an isolated tribe that lived in an island and loved pork, although they had it raw, since they were a bit primitive regarding the cooking process in general. One day, during a storm, a lightning struck a tree and started a fire in the forest where the pigs lived. After the fire, they found a lot of burned pigs. Since they were hungry, they decided to have the pigs anyway, and realized how good was roasted pork!

Realizing that the next storm would take too long to come, they decided to reproduce the roasted pork process, by setting fire to the forest. Their first tentative wasn't successful, because the wind was blowing in another direction. The chief, with all his wisdom, decided to appoint a very intelligent man to study the winds and determine when the wind would change to the necessary direction. After months of careful observation and data analysis, this man determined the best moment to set fire. And he was right!

All adults in the tribe stopped what they were doing and ran to the forest to set fire to it. They got some pigs, however in a smaller quantity than expected. They needed to improve their productivity in fire setting! The chief than appointed a commission to discover what was happening and they, after doing a lot of data analysis and long meetings, realized that grass was easier to set fire than trees, and, even better, grew faster!

The next step was to put the trees down and let the grass grow. However, putting trees down is a hard task, therefore, the chief decided to appoint a team for doing that, and, of course, a sub-chief to coordinate the team. After putting a lot of tress down, they realized that the grass didn't grew as expected. Again, the chief then appointed some specialist who created a system for wetting the grass, and, of course, allocated people to his team.

Now their process was really beautiful and well organized: they had a council in forest firing (not in pork roasting, but firing the forest, which in turn would give them roasted pork!), specialists in wind direction, and specialists, workers, and managers for putting trees down, moving the logs, wetting the grass etc, etc!

One day a castaway arrived at the island. The chief, with all his wisdom, realized that the man was good and invited him to check his marvelous process on obtaining roasted pork, with many specialized teams and a nice management, based on scientific methods. Aiming to show a different way of doing things, the man asked their best hunter to accompany him, and although the hunter was a bit out of shape (he was now working on log transportation - a vital activity to the process!), he followed the man to the jungle. After some time, they arrived with a dead pig. The man chopped some log, set fire to it, and in one hour they had roasted pork!

The man told the chief that they didn't need to have all that people and all that resource expenditure involved to have roasted pork!

Suspicious, the chief summoned the council, formed by a set of specialists in fire setting, log transportation and such, to ask what they think. They presented a lot of good reasons to not trust the man:
-He has no experience in burning forests!
-It works for one pig, I would like to see this working with a lot of pigs!
-We don't have enough skilled hunters! (they were doing other stuff right now)
-Our process is mature and works, in the end we do have roasted pigs!
-And our scientific methods, they are not lying!
-Why going after pigs when we can set fire to the jungle and kill them without running?
-Can you imagine a bunch of hunters running in the middle of the forest without us to guide them?

The chief then decided to put that crazy man in a canoe and send him back to where he came. Peace was back to the tribe, and all people were guaranteed in their roles, in special, the many managers and specialists.

But a serious problem was happening... since they devastated their environment, the roasted pork production wasn't supplying the demand, because pigs were not reproducing in the same pace anymore. Moreover, with all men involved in the fire setting process, huts were going down even with small rains.
Many adults were dedicated to the process and couldn't do other stuff, so their hut were destroyed by a storm.

When elders, women and children started to complain, the "process people" said: "can't you understand? that's the best way of having roasted pork, we evolved this process during years!"

However, after some time, finally the pigs disappeared, and even the "process teams" started to starve - and sleep outdoors. The chief then summoned the council and they said:
-Fire setting is like this, we cannot change its nature. Now we have to move to another island!

The chief, happy for having such a team of specialists at his side, appointed a new council, formed by the same people - after all, they had shown their great management skills in making such a complex process (the fire setting) work!

But a serious problem was happening... since they devastated their environment, the roasted pork production wasn't supplying the demand, because pigs were not reproducing in the same pace anymore. Moreover, with all men involved in the fire setting process, huts were going down even with small rains.

Many adults were dedicated to the process and couldn't do other stuff, so their hut were destroyed by a storm.

When elders, women and children started to complain, the "process people" said: "can't you understand? that's the best way of having roasted pork, we evolved this process during years!"

However, after some time, finally the pigs disappeared, and even the "process teams" started to starve - and sleep outdoors. The chief then summoned the council and they said:
-Fire setting is like this, we cannot change its nature. Now we have to move to another island!

The chief, happy for having such a team of specialists at his side, appointed a new council, formed by the same people - after all, they had shown their great management skills in making such a complex process (the fire setting) work!


+++++++++++++++++++++++++++++++++++++


What's the moral behind this tale?


The moral is that after some time, you loose the focus on the user demands and in the value chain. You start to follow the process, even when it is becoming expensive, complicated, and slow, mesmerized by its apparent perfection.

I call it the "The Beauty of the Beast Phenomenon in Project Management".

After lapidating your process you think it is beautiful and perfect. The problem is not in trying to improve your process, the problem is (i) not identifying waste
and (ii) not mapping activities to the value chain - things that aggregate value to the user.

A process that works doesn't mean it is a good process. IT cannot be a burden to Production, doesn't matter the certification. As Goldratt would say, the Goal is to make money - not getting certified... No problem if you think that certification is in the path for making money, but beware the beauty of slow and heavy processes.

Saturday, September 25, 2010

On Certifications and Tribal Culture - Part I

This post talks a bit about process and certifications, as this previous one did.

Some time ago two colleagues were faced with a curious situation. After developing and testing in semi-critical environments, they finally would put in operation an industrial decision support system in a critical environment.

The first funny thing was the meeting with the company's IT Department: two developers, one user, and... six persons from IT! A DBA, a network manager, an information security specialist, a "requirements specialist" - who had absolutely no idea of the application domain, and two i-have-no-idea-what-they-were-doing-there specialists in something-that-doesn't-aggregate-any-value-to-the-client-but-is-vital-to-IT-department.

The three first guys quickly got to the point, took their notes, supplied a nice and short schedule and were eager to leave the meeting and do their stuff. The other three started to do very basic questions (one of them did the same questions done by herself in two previous meetings) and in the end provided a three month schedule to "certify the solution". The second funny detail was that both the hardware and development tools were bought by them - and they took almost one year to do this. Thus, the certification of the application, would take one month, while the re-certification of the platform would take three months.

When our user asked "why all that?", they said "it's our process."

Now I go straight to the end: while the first three took one month to do the really necessary stuff, the "process guys" took SIX months to do their job. The fact was that any, I say, any, problem that they could have found (and they didn't) wouldn't cause more (or even 10% of) damage that a six month delay caused in terms of production throughput problems and related costs. However, of course, they had a certified process to follow.

This IT Department is Cobit'ed, ITIL'ed, PMP'ed, and even Function Point'ed.

However, their users hate them.

Almost every single time we met our users, they complained about how IT was slow and bureaucratic, and worse than that, how it caused delays in production and consequently financial loss. And every time users complained, IT repeated their mantra "it is the process, if we don't follow it, something wrong can happen." Even when our user tried to assume all the risks, they again repeated their other mantra "the process doesn't allow, only higher level decision makers can change this."

One can see two weird things in this story: first, doesn't matter if you are causing problems to the production department, you must follow the rules of the supportive department. Second, if changes to the process are needed, they are going to take long to happen.

Have you ever heard of the tail wagging the dog?

I know that all that certifications promise alignment of business and IT. And I DO believe that in most cases when a company get them, it really has alignment between business and IT.

However, certifications have three problems: i) they are based on documents, and documents are not software or hardware. Or, in other words, users don't use documents. In special those filled by IT! ii) They represent a snapshot of the environment, and this environment changes, however iii) how to be responsive to changes if the process imposes a lot of bureaucracy to implement them?

What a company need to stay competitive is to be responsive to changes. The only way of being bureaucratic and responsive at the same time is having a lot of people to manage change. The problem is that a lot of people costs... a lot! Therefore, bureaucratic processes are slow or expensive. Sorry for saying that, but in real world, things are even worse: these process become slow AND expensive. Of course, neither certifiers, nor certified will tell you this. Just ask users if they are satisfied.

So, what's the solution? Continuous improvement. I would exaggerate: Quick Continuous Improvement. If your process, certified or not, cannot provide this, change it. Quickly.


And the tribal culture part of this post? I will tell a tale in my next post.

Tuesday, September 21, 2010

Proof of Concept for the BLDD Tool - Part IV

One way of making a tool prove its utility is using it in scenarios where things go wrong. Figure 1 shows what happens if the code implemented is not in compliance with the requirements. The user will see a message saying that the scenario didn't pass.

Figure 1: Error generated by code not compliant to requirements

In the message in Figure 1, the error was generated because the last step was not implemented correctly: the message "9 sales awaiting to be sent" was not generated by the system, therefore, the developer must check the code and correct it. In that way it is possible to immediately identify where the system must be improved to work as expected, which is very useful when it is still in development, or when changes are needed.

Imagine that the "9 sales awaiting to be sent" message represents an improvement to this business process. I insert it on my specifications and run the simulation, so that I can identify exactly the point where the system must be modified to become compliant to the requirements. This is exactly what BDD proposes, the difference here is that we are joining a graphical representation to it and giving the user the option of running the business process step by step and see how the system is behaving.

In other words, the user can check a live process and the live system that answers to this given process. If anything goes wrong, the tool will identify what and where, making corrections easier.

It is important to note that this is a proof of concept, my idea was to launch it quickly so that more people can discuss and contribute to both the method and the tool. Besides Woped, we used Cucumber, which in turn uses BDD's Ubiquitous Language (Given-When-Then), however, as I said in this thread's previous post, we may define an UL for every BP representation and make the underlying text mirror the representation. That's our next step.

Thursday, September 16, 2010

Is Java the better choice for developing Enterprise Information Systems?

EIS present a complex nature, and, by definition, need to accompany the dynamics of their underlying business processes. Developers have to apply from Design Patterns to Service Oriented Architecture (SOA) to make systems flexible enough to cope with an ever growing change rate. New research areas, such as Dynamic Business Process Integration/Management also came to help.

However, it seems that the very basic element of every system stack - the programming language - was forgotten: is a static language such as Java the better choice for making flexible systems? I don't think so. I really believe that dynamic languages such as Ruby and Python are the "future" of EIS development, only flexible languages can give real flexibility to systems. Although many of the techniques presented in this blog, such as Domain Specific Languages (DSL), can be implemented in Java and other static languages, it is much easier to implement them using dynamic languages.

Side note: my opinion regarding ERP5 can be biased - we've been contributing to this project since 2002, but given that SAP-supported folks openly refer to R3, I feel comfortable to do the same. Having said this, now I can go ahead.

Maybe one of the first EIS projects to take advantage of dynamic languages was ERP5. It is implemented in Python, and from this six year-old story, I copied some interesting parts:

"Two key factors led to the choice of Python. First, was the need to make extensive use of metaprogramming. Second, for simplicity's sake, ERP5 needed to be implemented in a single language from core architecture to scripting.
"

(...)

"Python supports efficient metaprogramming through powerful introspection features that allow programs to inspect and modify code at runtime. Java's introspection are by contrast quite poor and inflexible. Metaprogramming in Java requires writing preprocessors, which would have added prohibitively to the cost and complexity of implementing the ERP5 system.
"

(...)

"Another advantage that Python had over Java was that it could be used at all levels of the system, from core implementation to scripting. Most ERP systems, while written in one language, use another scripting language to allow flexible configuration at run time by ERP administrators. Python is equally well suited both for scripting and core development, reducing complexity and increasing the flexibility of the system.
"

This example shows some advantages of using dynamic languages for implementing EIS. Now you can ask me: "ok, but what about all that reliable Java code around?" The answer is threefold: (i) SOA is here to integrate different platforms, including legacy code, (ii) we have to pay a price for being innovative, and (iii) if we were think like that all the time, we would be programming in Cobol...

Wednesday, September 15, 2010

Proof of Concept for the BLDD Tool - Part III

If you are following this post thread, you may noticed that it is necessary to know BDD to understand our proposal. In order to make you spare your time and decide now if you want to follow our method, I will stop a bit and summarize the proposal.

Initially I will explain BDD very, very quickly, highlighting the points that are interesting to understand SLDD/BLDD. BDD is similar to Acceptance TDD (ATDD), and relies heavily in TDD. It bases the declaration of requirements in the triad Given-When-Then (GWT), using the generic form

Given some initial context,
When an event occurs,
Then ensure some outcome.

Some BDD tool will get the scenarios described as above and create a file with a structure similar to

@Given('some initial context')
# you write test code here that excites the code that treats the initial context
@When ('an event occurs')
# you write test code here that excites the code that treats the event
@Then('ensure some outcome')
# you write test code here that excites the code that treats the outcome

When you run your specification, if you forgot to implement some step, the tool will say that the tests didn't pass. If you implemented code that returns outcomes different of the expected, the tool will say that the tests didn't pass.

Simply, isn't it? But it works!

Your users can read the specifications and you write tests for everything, assuring that the system will produce the correct outcomes. If anything goes wrong, the tool will tell you. If changes need to be implemented, simply change the text and run the specifications, forcing errors to appear, thus indicating where the code need to be changed, facilitating your work and the effort estimation for implementing the changes.

The first time I saw BDD working (for a system developed by an associate), some two years ago, I felt impressed with its simplicity and efficiency. However, since that first time I felt uncomfortable with its text declarations... as an EIS researcher I couldn't imagine specifying a lot of complex business processes using pure text.

Our first decision at NSI was to develop a tool stack for BDD in Python, so that we could use BDD in our projects. Only in the beginning of this year we could find time to solve my discomfort with BDD's text declarations. The core idea is to provide executable documentation to EIS, in other words, the same way we "run" specifications in GWT, I would like to "run" a business process represented by a diagram. Moreover, given that a BP can have many different paths, I would like to run it step by step, with the user choosing which path to follow.

This proposal holds some similarities with Models@RunTime, however I think it is simpler, since the second seems to focus on creating self-adaptable systems, while our proposal here is directed to a "simpler" problem: making EIS development safer, simpler, faster, and cheaper (I am modest). Also we want to make our proposal usable by the common developer, without experience in Formal Methods and alike (thanks God this is a blog, can you imagine someone writing this in a scientific paper?).

When we started to discuss about this proposal, another discomfort of mine was related to the term Ubiquitous Language. I prefer to use the term Shared Language, or a language that may be a composition of other existent languages and results of an agreement between all project stakeholders, or at least between users and developers. For instance, the shared language can be a combination of some BP notation and Math notation for complex requirements, and BDD's GWT for the simpler ones.

For this specific proposal of using BPM, we call it Business Language Driven Development (BLDD). In order to avoid repetition, if you want to understand better this name discussion, please follow the posts with the Shared Language Driven Development (SLDD) labels. The original ideas are also described on our reports Filling the Gap between Business Process Modeling and Behavior Driven Development and Mapping Business Process Modeling constructs to Behavior Driven Development Ubiquitous Language.

For instance, a SLDD/BLDD tool that is based in Statechart Diagrams would generate entries more or less like this (we are still discussing how to do this exactly)

@SourceState('current state')
# excite the code that treats the exit of the current state
@Event ('event')
# excite the code that treats the event
@Transition('produces some outcome')
# excite the code that produces the expected outcome
@DestinationState('new state')
#excite the code that treats the entrance into a new state

With that in mind we can go further with the example we used in the previous posts. The next post in this thread will treat errors in the implementation of the business process.

Proof of Concept for the BLDD Tool - Part II

One thing in BDD that always bothered me is the fact that things like parallel threads are not clearly represented, you simply use ANDs in the Then clauses and that's it. This kind of situation became even more clear to me when we mapped UML Statechart Diagrams' constructs to BDD's Given-When-Then, or GWT for short (you can find this report here).

Some agilists will complain that I am trying to introduce unnecessary complexity into BDD, however, EIS are complex in general, and many times they need to implement concurrency and other sophisticated constructs. Therefore, our tests MUST verify these conditions. Given that instead of filling forms we want to use executable documentation, we must prepare our testing toolset to deal with the intrinsic system complexity, of course, having the Keep It as Simple as Possible (KISP) principle in mind.

Continuing the example that was started on the previous post, we now reached an AND-Split, making the tests automatically fill the form with the Credit Card data, and driving the validation process to a stop at the AND-Split, waiting for the user to click on it. When the user clicks on it, we see the screen on Figure 1, which shows the equivalent GWT construct for our AND-Split, being the "Wait for confirmation..." part equivalent to the upper flow of the split.


Figure 1: The equivalent GWT construct for the AND-Split

As the user clicks OK, two subsets of tests excite in parallel two different parts of the system: the one that tests the sending of an email (the lower flow of the split, not shown here since there is no user interaction in it) and the inventory-checking procedure, of which equivalent in GWT is presented by Figure 2. After the user clicks OK, the process will go forward, reaching an OR-Join, as shown in Figure 3.
Figure 2: Excitation of inventory-checking procedure

Figure 3: An OR-Join is reached

Since we are using BDD tools for implementing our proof of concept, we were forced to provide hand-written GWT constructs, however, we envision the definition of specific languages for each BP representation.  For instance, for Statechart Diagrams we can use State-Event-Action tags to map from the graphical representation to tests.

It is important to note that we don't propose to substitute GWT constructs, but to complement them, because simpler requirements can be described without the need of BP models. In other words, we envision tools that can deal with different languages, that's why the name for our general proposal is not BLDD, but SLDD - Shared Language Driven Development.

Tuesday, September 14, 2010

Proof of Concept for the BLDD Tool - Part I

Executable Documentation is a vital element for achieving end to end testing and thus real code quality (instead of "quality by annotation"). Understanding that BPM is an important tool for EIS development, I proposed Business Language Driven Development (BLDD) as an extension to BDD (to understand this better, including a discussion on names for this technique, please follow the links of my previous posts).

The core idea is to use some Business Process (BP) Modeling tool to run, step by step, a given process representation and, at each step, excite the associated automated tests, making the system to pop-up accordingly as the user clicks on the representation. It is the same thing we see when we use tools such as Selenium, however, in that case, we would run the tests step by step and, instead of using textual descriptions of the requirements (like in BDD), we use a BP Model.

We know that there isn't a real standard for Business Process Modeling, in fact there are various, such as UML Models, Petri Nets, BPML, derivations of these etc. We also know that many EIS are based on Workflow Engines, and each of these engines has particularities. Therefore I decided to open two fronts:

1) Use a generic BP representation: the choice was Petri Nets, because I can derive others representations from them, such as Statechart and Activity Diagrams, and also there is the "Token Game", which is the simulation of the BP execution.
2) Use some workflow engine to work on top of it. The obvious choice for us was working with ERP5, an open source ERP (FOS-ERP), based on Zope, which in turn uses a state-based Wf engine, DCWorkflow. ERP5 was an obvious choice because (i) we are partners of this project since 2002, (ii) it is open, so we have access to all its code, (iii) we like state-based workflows.


I will focus now on (1) and will be back to (2) some weeks from now.

The first step for achieving (1) was to implement a proof of concept, using some Petri Nets open source tool that would be able to run Token Games, so that we could animate our model. During last August, Rodrigo Manhães, one of my associated researchers, prepared a proof of concept based on Woped.

Using an example PN, he created a very basic application, without taking care of the application logic or GUI - the goal is to show our proposal, not discuss BPM or GUI. The screenshot in Figure 1 shows the moment when the user reaches an OR-Split and must choose which path to follow in the workflow.

Figure 1: An user reaches an OR-Split in the example process
The OR-Split in Figure 1 is implemented as a list box. Appropriate tests excite this code, as shown in Figure 2.

Figure 2: The equivalent test run for the OR-Split

Remember that tests "are by example", therefore, the user doesn't interact with the real system, instead, he/she is a spectator of the tests, launching them by choosing which path to follow in the PN. After checking if the system is implemented the way he/she expected, the user closes the GUI and then choose the path equivalent to Credit Card, making the test code to run this option, which, in turn, makes the form in Figure 3 to pop. Meanwhile in Woped, the user can see a message that shows which scenario was chosen, including the Given-When-Then equivalent instructions (Figure 4).

 Figure 3: Credit Card data form
 Figure 4: Message with equivalent GWT.


When the user clicks in OK, the flow goes to the next step, an AND-Split (Figure 5) and the tests responsible for filling the form will run, as shown in Figure 6.

 Figure 5: The process enters an AND-Split

Figure 6: Running the test for filling Credit Card data

Although we had to implement specific logic for each of the PN structures (I will start discussing this on the next post of this series), the next steps for verifying the consistency between the BP model and the system are similar.

The goal of this post was to show our basic idea: make the user run the business process and see immediately how the system is behaving. If there isn't correspondence between requirements and code (originated by process change or code wrongly implemented), errors will appear, automatically indicating where the system is not compliant to the requirements described by the model. Error situations will be shown in the next posts.

Deciding on the names for BDD extentions

On a previous post I talked about our proposal of extending Behavior Driven Development (BDD) to use other languages than then "classic" Given-When-Then (GWT), in special, but not limited to, Business Process Modeling (BPM) artifacts, such as Petri Nets, Statechart or Activity Diagrams. Today we decided at NSI to use two terms to refer to our proposal:

a) When referring to the idea of using a composition of languages, previously agreed by all stakeholders, we are going to call it Shared Language Driven Development (SLDD).
b) The specific case of using BPM + domain specific language (DSL) as an input for the BDD machinery is called Business Language Driven Development (BLDD).

In other words, SLDD is the general proposal and BLDD is the proposal for EIS. You can also interpret like this: SLDD is a generalization of BDD, and BLDD is an extension to BDD that uses BPM.

Monday, September 13, 2010

ERP Licensing perspectives - Confenis 2010

Paper presented at Confenis 2010, compares, in general, licensing perspectives for Free/Open Source ERP (FOS-ERP) and Proprietary ERP (P-ERP). The main findings are that Small companies are different of Medium companies in terms of adoption and that Community Sponsored is different of Commercially Sponsored FOS-ERP.

Process Evaluation X Product Evaluation, How Companies Hire, and Parnassianism

It is usual to companies to post recruitment announces asking for people from given educational institutions (at least here in Brazil): "we seek professionals who graduated from X, Y, or Z...". Do you think companies check these institutions pedagogical material to choose which institutions list? I am talking about the dozens of documents we teachers must fill to show that we are following a giving teaching process. No, companies check the product: the professional that graduated from these institutions. It is something like "usually, students from these institutions are good professionals". You don't see things like "institution X has a nice pedagogical process, I checked their documents...", managers don't care about this, they care about the overall quality of the outputs of these institutions.

It is curious too see the same manager that behaves like this as a "costumer", saying that it is important to fill in dozens of forms showing "how good is their development process...". Funny, isn't it? As a costumer the main thing is the final product, but as a manager it is the process(?). Someone would promptly answer: it's obvious, as a manager you are responsible for the process. Yes, that's right, and we must take care of our management process as a precious thing.

However, what I am discussing here is a question of focus. I am not saying that correctly managing the development process isn't important, in fact, it is very important. I am saying that you should map every process activity into the client's value chain. The client is the important part, and executing tasks that don't aggregate value to the final product is simply waste. Doesn't matter if you "process framework" says they are necessary, it is waste. Sorry for that. Quality must me measured on top of the product, process quality is an indirect measure and don't supply any real proof of product quality. Yes, leadtime and cost are important process metrics but they don't say much about the quality of the product. And unnecessary tasks raise leadtime and costs without aggregating value to the client.

Here at NSI, we know some people who work for certified companies, and they frequently report informally some horrible real-world practices at these companies. The fact that your company got certified only prove that you could fill-in a lot of forms, and while the auditors where in place you managed to show that everything was really perfect. Sorry again, but the only thing that really matters for your customers is the delivery of good quality products, on-time and on-cost! Your certification today can reduce the chances of occurring a disaster tomorrow, but it doesn't give any guarantee to you, your company, and your clients. 

Only a process that focuses on keeping product quality - and not showing how good it(self) is - can give you more security that a disaster won't occur. At this point I introduce Parnassianism. According to Wikipedia, "Gerard Manley Hopkins used the term Parnassian to describe competent but uninspired poetry, where a talented poet is merely operating on auto-pilot." When I was in high school I learned that it was "the art for the art" and "poetry for poets (not for the public)."

Certification sometimes becomes Project Management Parnassianism.

You can say that auto-pilot is good, meaning that your process is industrialized, standardized and works smoothly. Ok. Now read the first part: uninspired. Do you really think that in times of highly dynamic Economy and Technology you won't need to be creative to attain competitive advantage? And do you really believe that processes based on taking notes, scheduling and running long meetings, and reporting to various hierarchical levels will allow you to be creative? Processes where workers cannot create, test, and put in production new stuff quickly are really the answer for market shifts? Moreover, what about activities for process control that increase costs and leadtimes, without improving the product?

The truth is that Certification Industry became "an end in itself", you do things because someone said to do that. And the time it takes from a good new idea to appear and to reach the certification documentation is too long to help innovation, someone will have the same idea and will apply it before you do - while you wait for the "new, official, certified documentation."

The best way of detecting that you are not innovating in management anymore is when someone ask you "why you do that?" and you answer "because our certified process says so." Or, even worse, "how this activity aggregates value to the final product?", and you answer "Oh, yeah, not directly, but this is necessary for us too <do something internally>."

That's Parnassianism in Project Management: doesn't matter the people's (that read your poetry) needs or wishes, what matters is to follow standardized metrics.

Beautiful, isn't it?

But who is going to buy pretty formated but non-inspiring poetry?

Sunday, September 12, 2010

Why Agile for EIS? - Part III

There is an "ultimate" argument for Agile for EIS: need for speed.

Although this is not mainstream yet, more and more EIS are becoming in reality systems of systems. People talk about Enterprise Mashups and Dynamic Process Integration. During a dinner in Confenis 2010 some of us were discussing about the huge amount of disparate data sources and business processes that must be integrated to achieve better enterprise management, better decisions, and better understanding of customer expectations - in other words, to attain and keep competitive advantage. And new sources appear almost every day.

Although  there is a lot of work on automating and reducing the effort for the integration of business processes and data sources, I personally believe that programmers will still have a key role in all this. And for us to give quick response to these new integration needs ("which appear almost every day"), we cannot count on heavy and slow project management processes. Moreover, we cannot count on filling forms to guarantee software quality, simply because forms are not software, and there is no guarantee that the form is telling the truth.

Enter Agile...

Ps.: there is one Agile technique that I am not convinced yet of its applicability in EIS: Open Scope Contracts. I believe that is too hard for most law departments, in special in big companies and government, to accept this. In fact my team faced this problem when we tried to use OSC in an small ERP project for a foreign government some time ago.

Mapping Agile Methods to ERP

SlideShare paper presentation at Confenis 2009 by the blogger and colleagues. This paper briefly presents some ideas that are discussed in this blog.

Saturday, September 11, 2010

Effort Metrics Sophisma (in Portuguese)

A SlideShare presentation on why we at NSI believe that traditional effort metrics simply don't work. In fact, why trying to plan months ahead in software development doesn't work. In Portuguese, in the next days I will start translating it.

Why Agile for EIS? - Part II

In the previous post my arguments were a bit intimate - yes, I was called "too nerd"/"too academic"/"too non-real-world" when started using object oriented programming, then free software, then open source ERP, and now Agile methods in EIS.

Now I am going to use more technical arguments (or questions to make you think) in favor of Agile Methods:
1) What guarantees that your code is working well and according to client's requirements when your Quality techniques are based on filling a lot of forms? Filling a form makes the code correct?

2) Do you really need to keep all your models always in synch with code, even if nobody is using these models? Have you ever heard of Reverse Engineering?

3) Do you know that putting a Quality Team to test software after it is "finished" is called "Quality Control"? And do you know that people in (tangible goods) industry know for decades that this is not the better way of assuring quality, because you are going to waste time and money correcting things that should have been corrected steps ago (and you have already consumed resources without aggregating value to the product) ?

4) Do you believe that the better way of checking if a requirement is "finished" (and thus can be delivered to the customer) is by hand, reading a document? Have you ever heard of automatizing acceptance tests?

5) Do you think that the better way of keeping track of changes in you code base is by filling forms? What guarantees that the person that checked the change worked properly?

6) Do you really believe that a lot of internal documents, used only by the development team (to check things that should have been checked automatically), will aggregate value to the user?

Finally, here goes some challenges (related to the process, not the product, like the previous ones):
a) Show me where PMBoK says that you must fill all those documents of each knowledge area to have a successful project.

b) Tell me how can you use metrics such as Use Case Points in an (really) Iterative & Incremental project (remember that real I&I cycles only details Use Cases when needed, so how are you going to count the number of steps for every use case and then get a "precise" effort estimation?).

Why Agile for EIS? - Part I

Because of the "E" in EIS, these systems are usually associated to big enterprises. And for many people, a big enterprise means bureaucracy and lots of documents for doing everything. This is curious since Japaneses have been running big and efficient enterprises for the last 60 years (ok, Toyota is now going through quality problems but this cannot bury decades of success).

The question is in fact cultural: like Free/Open Source Software a decade ago, Agile Methods is still considered "nerd stuff" for many people from the EIS area. On the other hand, Model Driven Development (MDD) + project-management-methods-with-a-lot-of-bureaucracy appear to match Enterprise Engineering philosophy of modeling and documenting Enterprise aspects under different views and abstraction levels.

Even more curious is the fact that ERP implementation statistics show a lot of disasters, cost & time overruns and alike, but people still think that they need a lot of documents... Ok, someone can show various examples of successful bureaucratic ERP projects, but if you check their costs, the truth will appear.

Another human-related problem is that Agile put the focus - and consequently the decision power - on the producer (the programmer), not in the "modeler". And many people, although graduated in Computer Science, never liked programming... "I am an analyst, I am not a programmer." This is also funny, I remember, during my trainee days in Datamec (1993), when Essential Analysis and Waterfall life-cycles was the default way of developing, of a manager saying that "programmers and analysts are dead, long live designers... the software  development professional of the future is the one that will be able of modeling and programming, improving software quality and reducing lead-times and costs." He was a visionary, object oriented programming was still (another) "nerd stuff" and the Agile manifesto came some eight years latter.

As you can see, object orientation was "nerd stuff", free software was "nerd stuff", and now, Agile is "nerd stuff"...
So, are you going to wait a decade to accept Agile Methods in EIS Development???

Pyramid: BDD in Python

NSI started developing in March 2009 a set of tools for using BDD in Python language. They are briefly described in the technical report A tool stack for implementing Behaviour-Driven Development in Python Language, which needs to be improved and updated, however, it gives a summary of this work. This project started as an desire of using BDD in a real-world Python application: the Federal EPCT Network Digital Library (BD-EPCT) project. By the time we started, with should-dsl, Pyramid was the only set of integrated tools for BDD in Python.

Thursday, September 9, 2010

Abstraction and Test By Example in the EIS Case

Another point to consider in BDD for EIS is the question of Abstraction during testing. The same way we expect to describe requirements through BPM, we probably would like to write tests not using examples, but parameters instead. I mean, instead of declaring things like

 Given that a client buys a jumper
           And I have an inventory of three jumpers
 When the client returns a jumper
 Then I will have four jumpers in inventory

I would have something like

 Given that a client buys a given product
 When the client returns this product
 Then I will add this returned product to my inventory

Since tests must deal with examples, parameters must be marked for parsing and when we run tests we must supply these parameters.  Many automated tests tools already do that.

I think parameterizing is a must in the case of EIS development and customization. Think in terms of an ERP and its different adopters, probably they will demand different test sets during customization or even system evaluation. Actually, during deployment, you are going to import a lot of legacy data, thus it is interesting to use this data for testing.
See? Parameterization is a must for Agile EIS testing.

Ubiquitous Language or a Composition of Languages?

According to Wikipedia, "BDD focuses on obtaining a clear understanding of desired software behavior through discussion with stakeholders. It extends TDD by writing test cases in a natural language that non-programmers can read." Also, Dan North says that "BDD provides an Ubiquitous Language for Analysis" and then presents the Given-When-Then (GWT) triad.
Thinking on these definitions, I proposed the use of Business Process Modeling (BPM) artifacts for describing requirements in BDD. Things like Petri Nets and Statechart or Activity Diagrams are a "natural language" for many users, in special for those that interact with EIS.
I understand that GWT is an UL for System Requirements Elicitation. In fact,  I  think that ULs should behave as a composition of languages, something like (some language that can describe system requirements) + (the user's business language).
For EIS, I envision something like some BP notation with a textual language that is simultaneously understandable by users, parseable, and that we can write code that is the closest to the user's way of writing about their business, using expressions such as "Vectorial Calculus is_a_pre-requisite_of Linear Algebra" (Yes, that's what people call DSL - Domain Specific Language).
Ideally this textual language should go from the requirements to the code, letting the program's structures be described through the BPM notation.
I wrote all this text to say only this:
-UMLers would say "Activity Diagrams + OCL" is the answer, and I say, yes, that's an option, but OCL is not the right answer, it isn't natural!
So, we need to go further...

Monday, September 6, 2010

BPDD | BLDD| CLDD | SLDD, Dynamic Business Process Composition, The Future of ERP, REST, and Here We go!

The economic dynamism has been pushing a lot enterprise systems development. Some business processes must be altered and/or re-composed frequently to meet changing business conditions. Dynamic Business Process (or Service) Composition represents a set of techniques that try to provide the basis for achieving this dynamism.
Also, some say the future of ERP is to become process integration frameworks. Instead of integrated modules, ERPs would provide the means for quickly integrating disparate business processes living in different systems - in fact, this is an old Supply Chain Management need, far away from be quickly (and cheaply) fulfilled in real world applications.

Many proposals are based on the use of Intelligent Agents, Ontologies and other automated decision making supportive methods. Agents and ontologies can be very, very useful for filtrating, among hundreds or thousands of options, the few dozen that may interest to a given organization. Ontologies are very, very useful for helping finding the points of integration for these process. However, I still think that in the end, it will be a human-decision-maker and a human-programmer who will choose the best combination of processes and implement their integration respectively.

Experience says that even if the leading-edge organizations can reach the Nirvana of totally automated dynamic composition of business process, we are going to find most of smaller organizations striving to do this "manually".

Therefore, I think there is still room (and a lot of it) for developing supportive techniques for "semi-automated" Business Process Composition (BPC). And I believe our proposal of having a Business Process extension of BDD (we are deciding which name to use - BPDD, BLDD, CLDD, or SLDD - see my previous post) associated to tools like Restfulie may offer a powerful method to implement executable documentation for business applications that demand integration of disparate applications and their business processes.

I can envision a business analyst building a graphical representation of the macro-process, using a shared development language - understandable by all stakeholders (see my previous post),  and, after implemented the automated tests, checking the live application running as he/she follows (clicks) on the different paths of a workflow representation. As the macro-process changes, the tool will visually identify where and what to change accordingly so that the system becomes business compliant again. That would be a way of building a "next-generation ERP" or "next-generation Enterprise Information Systems" through a combination of visually manipulating BP representations and Test Driven Development (TDD).

Well, that was one more teaser to the matters I want to discuss here, I will go back to it latter.

Business Process Driven Development - BPDD

At NSI we are working on adding Business Process (BP) representations to Behavior Driven Development (BDD). I mean, substituting the "classic" Given-When-Then (GWT) by some BP graphical representation, such as Petri Nets or UML Statechart Diagrams.

The core idea, already working as a proof of concept using  Woped, is to allow a final user to run a business process in a step by step way and then see the living system pop up as he/she clicks on the activity elements, such as edges in a Petri Net. In that way we provide executable documentation using BP graphical representations, delivering BDD to the realm of Enterprise Information Systems.

Initially, we thought of calling this extension to BDD "Business Process Driven Development" (BPDD), however, we realized that a lot of related but different methods use this name with many different meanings. Then I thought of other names, and leave to my readers the choice of the best one:


a)Common Language Driven Development (CLDD)
The use of common language for users and developers (yes, another name for UL...)
b)Business Language Driven Development (BLDD)
The use of the user's business language as a common language.
c)Shared Language Driven Development (SLDD)
The use of a shared language, resulting of an agreement between users and developers.


(b) is interesting because keep our focus on business systems. However, I think we can use the concept of UL for scientific systems, for instance. So "business" may not map well for these systems.
(c) is interesting because it shows that users and developers agreed to use a shared language.
Yes, I know that UL's definition may cover all that, but I don't want to call ULDD!

We prepared two technical reports and posted them at Arxiv.org, please check them and help us to find a nice name to this BDD extension:
a) Filling the Gap between Business Process Modeling and Behavior Driven Development
b) Mapping Business Process Modeling constructs to Behavior Driven Development Ubiquitous Language

PS: in fact I believe that the Ubiquitous Language (UL) is not a single language, but a composition of languages. For instance, we can mix mathematical formulas and plain text into a process representation so that this representation is the closest possible to the "real thing". In other words, since there are languages such as should-dsl, we could also have things like math-dsl, or logistics-dsl, used externally and internally.
I will go back to this later on...