Ideas on Enterprise Information Systems Development

This blog is devoted to ideas on Enterprise Information Systems (EIS) development. It focuses on Lean Thinking, Agile Methods, and Free/Open Source Software, as means of improving EIS development and evolution, under a more practical than academical view. You may find here a lot of "thinking aloud" material, sometimes without scientific treatment... don't worry, this is a blog!
Every post is marked with at least one of Product or Process labels, meaning that they are related to execution techniques (programming and testing) or management techniques (planning and monitoring), respectively.

Sunday, December 5, 2010

Assorted Thoughts on Agilism X Traditionalism - Part IV

Some days ago I bumped again into an "old" internal discussion by email on the Agile 2009's talk "Lets stop calling it 'agile' ", which summary I copy below:
"
Agile development has grown a lot since its rebeleous 2001 start. In fact, it has grown to be the mainstream way of developing software. The time has come to drop the word "agile." Agile development is just modern practices in software development. There is no need to explicitly mark practices as Agile. There is no need anymore for opposing camps. Keeping the word Agile and things like "the Agile conference" is holding the development of modern SW development practices back. This session will be in debate form to discuss the above mentioned motion.
"
Some can find this radical. When I first read it one year ago, I also thought the same. However, reading it now I can clear understand the proposal, and it made me remember of my way through software processes.

During my undergraduation course I had contact with Yourdon's Structured Analysis as well as Object Oriented Analysis & Design, by that time (1990-1994), the first was mainstream and the second a new proposal which needed to be improved. However, the second one attracted me more, given that I wanted to use Object Oriented concepts the most, since my first contacts with C++ in 1991 I was already convinced that OO was a better way of programming.

First Experiences
Outside the classroom, my first professional contact with "analysis" occurred in 1993, when I took part of a Management Information System (MIS) development in Clipper and Structured Analysis and Design. On those days of waterfall cycles, someone did the analysis and design and I implemented the system. As expected, after deploying it, users asked for changes almost every week. Firstly I  implemented them and once a week updated the modeling artifacts, for maintenance purposes . Yes, first I changed the programs, because my users were eager to get the software running with the changes, and them, when had time, updated the models. Looking back I see how natural is the idea of using models only to understand the problem, and updating them when needed.

In early 1994 I inverted positions, acting as the analyst for another MIS, using Structured Analysis and a classical waterfall cycle. The implementation was successful until at least its deployment, later in the year. After that, I lost contact with it.

In the second semester of 1994 I entered a bigger Enterprise Information System project, with circa 15 developers and two consulting companies working on it. It was very uncommon environment, using IDEF artifacts for modeling, C++ for programming, and Sybase. Transformation from functional models to C++ methods were done during interesting CRC Cards sessions, and class diagrams were derived from Entity-Relationship Diagrams. Object-relational mapping was done according a set of very basic rules, created internally by the team. The only problem we had was the fact that some process models were prepared almost one year before we started to touch them. However, we had a very good business analyst (he was an Industrial Engineer) who re-checked all business process with the clients before they enter the programming phase. Therefore I can say that the process was interesting, however, the "waterfall way" of doing things created a lot of rework.

Before I forget, effort estimating in all cases was done by "expert opinion",  people didn't believe in Cocomo's use of constants and in Function Points counting - which needs a lot of things already defined to have a chance of being "accurate".  I could only remember my Software Engineering classes and started to see them as something closer to Alice in Wonderland than to an Engineering discipline.

Enters Object Modeling Technique (OMT)
However, it was only in 1995 that I had the first chance of using professionally an OO method in the whole process, specifically, I used the OMT method. OMT was interesting because it used a mixture of class and state diagrams with Data Flow Diagrams (DFD's) to describe its Functional Model. OMT was a product of its time: OO languages were becoming mainstream, but Structured Analysis was still the mainstream process paradigm.

Our Environment in 1995
In this year I moved back to my hometown, aiming to take part of the creation of the first big and public university in the region. We were a two men team, in partial time, working at the Math Laboratory - from where the Computer Science Laboratory would emerge years after. The mission was to develop a library management system to be used by the three campus' libraries. The university was new and many, many information systems were necessary. Library Management Systems where really expensive by that time, thus the university's president asked us to create the system.


Our Process in 1995
We decided to use a mixture of Prototyping Life Cycle with OMT's artifacts to build the system, since we had very few knowledge in the problem area (libraries). For more complex operations, we would use CRC cards to define how classes would interact. We decided to use Delphi 1.0, which allowed us to create nice GUI as well as use Pascal, instead of Visual Basic (which I hated). Moreover, my colleague was developing an object-relational mapping library in Pascal for his Master's Thesis, which gave us a cleaner and powerful way of doing Object-Relational Mapping (ORM). We resisted the temptation to use Delphi's components for direct access to the database for the sake of really using object orientation for coding business logic.


Firstly we drafted a list of high level requirements - separated in modules, and then we asked the librarian to prioritize them. Then we started to follow a process that we built on top of our experiences in previous systems:
1) Get the next priority.
2) Detail its requirements.
3) Create a prototypical GUI and a draft database to reflect the requirements.
4) Re-check requirements during GUI presentation to the user.
5) Model stuff in OMT.
6) When appropriate, use CRC cards to define classes' responsibilities and collaborators.
7) Write code ignoring the database*.
8) Write ORM and database code.
9) Present the implementation to the user.
10) If the user was happy with the results:
a) we deployed the new module and went back to 1)
else
b) we looped back to 5) if modeling was needed, 6) if changes in classes were likely to occur, even without new modeling, or 7) if only changes in algorithms were needed.


* We used an ancestor technique of what is called today "test doubles". In our case, collections of objects with typical values were created "by hand", so that we could test the algorithms. When we put the algorithms to work properly, we then implemented the persistence code, and finally we took way these "fake objects". Sometimes I developed algorithms and my colleague put things to work with the database, since he was the ORM library developer. We didn't want to waste time with persistence problems while we were trying to solve users' problems. Persistence was a pure technical question, not a main development concern.

Effort estimation was done before each of these "proto"-iterations, by experience, because I kept the same opinion regarding effort metrics. The project was really successful, we both left the university in the end of 1996, however it was used for years, until the university bought a new system, with external support from a development company.

I used this process more or less unchanged until 2000, when I had access to my first UML Case Tool, and started reading about the Unified Process and Eriksson-Penker's business modeling extensions for UML. I knew UML since 1997, however, by that time, and until 1999, I had only one project, which used OMT already, and I had no reasons to change from it to UML. I used UP variations until 2006/2007, sometimes without the expected output. And all times not fully implemented, since I thought many things were too bureaucratic to be used.


The main thing I learned from these days was that you must mix techniques and even create new ones to make the development process work for your environment, and not the opposite. Hardly you will find a process that fits exactly to your environment, team, culture, and problem. I've seen a lot of people eager to follow some guru or certification, and when things go wrong they blame themselves for not being able to do exactly "what must be done." They should blame themselves for not being able to adapt the process to their environment.


Another thing I can clear identify now is that some recent Agile techniques are modern versions of old programming tricks, those kind of hints shared during the coffee break, and that never appeared in the "software engineering" books.


Going back to the "drop the word agile"
I think my intimate history in dealing with development process is the same of a lot of others developers. In early 1990's, there were a lot of different Object Oriented notations and processes, all seeking to solve the same problems, in slightly different ways. Hybrid methods such as OMT (and its DFDs) were told to be the ideal solution for a world in transition. And for me they seemed to be.

Then came UML and UP, both bringing some standardization to the software development world. Finally, Model Driven Engineering (MDE) and Model Driven Development (MDD) could realize their full potential. And they did, given that I think their potential has finished. At least, the potential to be innovative in research terms, and to answer to constant changes in practice terms.

I agree with the phrase "Agile development is just modern practices in software development." Yes, one should try to list a single innovation that came from the MDD world in the last few years. Every "new" MDD technique that I saw recently, either was repetition, either was "extreme cases", hardly useful.

Many people still resist to Test Driven Development (TDD), for instance, because they can't understand how tests can be used for design. There is a common misunderstanding that design is related only to blueprints, thus, design = modeling in software development. What about Stress Analysis, mostly based on Math and Physics, instead of drawings... so design is not only about blueprints, and, actually, if you can rely on Math to do the design, it is more likely that your design will work. As I said before, coding is closer to Math than modeling.

For a developer to understand how TDD can drive design, he/she must practice it. You can draft a model to understand the problem in the beginning, however, given that you are going to interact with tests - which are code - and the code that answer to them, you are going to evolve the design as the need for testing evolves. An tests evolve as the implementation of requirements evolves. The problem is that many decision makers are those that "don't program anymore", so will "never" be convinced, unless some programmer show to them how things work.

There is also an economic and cultural reason for people not recognizing agilism as the "modern" software engineering: agile techniques mostly were - and still are - nurtured by individuals and communities of developers, in opposition to big consultancy companies. Because of this, Agile techniques will take longer to enter the EIS world. People will repeat to the exhaustion "Why big companies like A or B don't use a lot of Agile if it is so good?", and will answer to themselves "because it doesn't scale!". Isn't because you don't need expensive tools and related consultancy to adopt them? Isn't because of culture? Isn't because a lot of people who stopped programming after becoming project managers are afraid of seeing part of their daily activities labeled "waste", thus endangering their jobs?

I witnessed similar reactions in the early 1990's regarding Object Orientation, and if the "legacy guys" were right, I would be saying stuff like "Structured Analysis rules!".

So, drop the word Agile!

9 comments:

  1. I liked the discussion, however there is one point that I must disagree: The use of Agile term for software development is concurrent with the Agile Industry paradigm, which came first and has built the broader environment where the evolution occured first.
    As developers, we may not participate or even track the events happening in industry.
    Agile Industry became in the 90's an important paradigm, promoving changes in processes, operations and way of thinking.

    ReplyDelete
  2. The intention is to state that modern software engineering = agile. For instance, when we say "modeling" we automatically think of Object Oriented Modeling, but we don't have to say all that. Anyway, I understand your point.

    ReplyDelete
  3. I would say that dropping the word "Agile" wouldn't be the smartest thing to do.

    There are plenty of reasons why we still need to have a stake in the ground as to what Agile is and what it isn't. There's a reason the Agile Manifesto is called AGILE Manifesto.

    A starting point. We shouldn't forget what began the movement to better iterative development.

    ReplyDelete
  4. Also, not quite sure where you got:

    "There is also an economic and cultural reason for people not recognizing agilism as the "modern" software engineering: agile techniques mostly were - and still are - nurtured by individuals and communities of developers, in opposition to big consultancy companies."

    Big consulting companies are embracing Agile as a consulting product more and more these days.

    ReplyDelete
  5. Hi, firstly, nice to hear from you!
    I agree that big consulting companies are embracing Agile, however, in the EIS arena, many still live in the MDD world.
    The "agreement" around the need of BP modeling in the EIS arena has pushed people to MDD, something like "if I start by modeling I have to finish modeling."

    ReplyDelete
  6. Regarding the "drop the world agile", I think the Agile Manifesto was a manifesto 9 years ago. Now it consolidated into modern SE practices.

    ReplyDelete
  7. Good points here. Thanks for the follow up!

    ReplyDelete
  8. Boa a discussão e boa a sua história passando pela a evolucão dos processos de desenvolvimento de software. Infelizmente só vi agora! Acho que o termo "desenvolvimento ágil de software" as vezes espanta as pessoas. Na verdade esse representa um modelo enxuto de produção. Seja qual for o processo, creio que ele deve ser flexível e proporcionar vantagens a produção e não simplesmente seguir um formalismo.

    ReplyDelete
  9. I suggest you read the whole series, plus the ones entitled On Certifications and Tribal Culture.

    ReplyDelete