Adaptable Designs for Agile Software Evolution

Posted on May 5, 2013 by

bird beaks“It is not the strongest of the species that survives, nor the most intelligent that survives. It is the one that is the most adaptable to change.” – Charles Darwin

Big Design Up Front (BDUF) is considered a very bad practice in Agile software development. BDUF is problematic because it assumes frozen requirements. The Agile approach is that software must be developed iteratively since the requirements will change and the system will evolve. So we need some Design Up Front, but this must be Adaptable design that supports change: ADUF.

Adaptable Design Up Front

We should never work on the design of a specific system. Instead, we should generalize, designing for a family of systems. Applying object-oriented concepts, our design should represent a class of systems, and each version during the system evolution should be an instance of this class. Thus each new version may have modifications and extensions when compared to previous versions, but if it still fits the same generic design, then it is still an instance of the same class.

Adaptable Software Design: A generic software design for a family of systems which does not need to be changed to accommodate new requirements.

The challenge of creating an adaptable design is making it change-resilient. Adaptable designs should evolve and accommodate new requirements through specialization. When designing for a family of systems, there are two main such approaches: The framework approach and the platform approach.

Frameworks

Framework: “A software framework is an abstraction in which software providing generic functionality can be selectively changed by additional user-written code, thus providing application-specific software.”

The framework defines the system structure: The main system components and the relationships among them. Each one of these framework components should have an abstract interface, and several possible implementations for these interfaces. If the design is sound, the components implementation may evolve over time, but the relationships among them should be preserved.

Each component in a framework should be designed to be highly cohesive, and the framework itself should be designed so that components are weakly coupled. If the component interfaces are indeed abstract and if the coupling among these components is weak, each type of component in the framework can be seen as a kind of plug-in.

Plug-in: “A plug-in is a software component that adds a specific feature to an existing software application. When an application supports plug-ins, it enables customization.”

If we are able to define such framework with diverse plug-ins, the system evolution occurs through the implementation of new versions of components. Thus, instead of having a new version of the entire system, we may have new versions for each one of the plug-ins. Building the system means simply combining these plug-in versions in an appropriate configuration.

Platforms

Platform: “Technology that enables the creation of products and processes that support present or future development.”

A platform should support the basic services which are required by the system to provide its functionality. When these basic services can be well-defined and implemented independently of the system which will use them, we have reached an ideal environment for adaptability:

  • The platform may evolve independently of the systems that are built on top of it. If the interfaces are kept intact and if the behavior is preserved, the platform developers are free to improve its efficiency and other non-functional attributes.
  • Several systems may be built on top of the same platform, using its services to provide diverse functionalities. These systems may each evolve independently of the others, and independently of the evolution of the platform itself.

Building platforms is a very effective approach for experimentation. If an essential part of the system behavior can be captured as basic services being provided by the platform, then each version of the system may be seen as an experiment and the cost of failure is low. The resources invested in the development of the platform are not lost in the case one of the system versions is not deployed.

Adaptable Software Design in Practice

Question: How do you identify the components in your framework or the services in your platform?

Answer: We want to think generally about a family of systems, and avoid defining requirements for a specific system. To identify the main entities that should compose any system in this family, we should apply domain modeling.

Domain Model: “A domain model is a conceptual model of all the topics related to a specific problem. It describes the various entities, their attributes, roles, and relationships, plus the constraints that govern the problem domain.”

Question: How do you decouple the components in your framework?

Answer: To reduce coupling, one should look for appropriate Design Patterns. Most of these patterns are related to avoiding the direct links between concrete classes, through the introduction of interfaces and abstract classes.

For example, when instantiating an object, one should avoid creating an instance of some specific concrete class. This can be done using the Factory pattern.

Summary

Software systems must be able to evolve over time. This evolution should be planned and supported trough Adaptable Software Designs. There are two main approaches for creating such designs: Frameworks and Platforms. Both require the identification of domain entities and the usage of abstract interfaces to minimize coupling between components.

What do you think? What is your personal experience with software evolution, changing requirements and adaptable designs? Please share your thoughts in the comments below.

Posted in Agile, Design Patterns, Software Architecture, Software Evolution | Tagged , , , | 17 Comments

Outliers? The Myth of the 10,000 Hours Rule

Posted on April 4, 2013 by

neutralFollowing the “Outliers” book by Malcom Gladwell, some people are convinced that if they “just” invest 10,000 hours in something they will become really good at it. I think that this is not always the case. I don’t believe any person is able to invest 10,000 hours in anything he/she decides. I believe that challenging activities require special aptitude. And I think there are situations in which someone may spend lots of time stuck in some place, doing the same mediocre job for years. Like driving in neutral while pressing down the gas…

Question:  Can you just decide to invest 10,000 hours in anything you feel attracted to?

I believe you can only invest a lot of time in some activity if you really enjoy doing it. And I also believe that you can really enjoy an activity only if you got some talent in performing it. In other words, if you don’t have the innate ability to do some activity you will not be able to enjoy it, and thus even if you try you will simply quit much earlier than you expect.

You certainly know people who have tried all kinds of activities and then decided to quit after a relatively short period of time. And you probably also have your own personal experience of trying to do something and then deciding to quit. That may be trying to learn a foreign language, trying to play some musical instrument, trying to exercise more frequently or even trying to write your own blog.

If you had these experiences, you can ask yourself why you decided to quit. Was learning a foreign language more difficult than you expected? Was playing a musical instrument less fun than you thought? Did you enjoy exercising more frequently? Did you feel satisfaction from your blog writing?  If you decided to quit, this probably means that for these particular activities you did not have enough talent to enjoy and persevere.

One case in point is that of College drop-outs. It’s clear that if a person chose some profession is because he/she felt attracted to it, and was ready to invest a lot of time learning it, believing that with the right amount of dedication and effort success was guaranteed. However, statistics show that 33% of the students drop-out of College. Of course some of these students may have personal reasons to drop-out, such as health problems, but for most of them the reality was different from their expectations, and they simply decided to quit.

Question: May you invest 10,000 hours and still see no improvement?

I believe people can get stuck in some position, for example performing the same job for years without any personal progress or improvement in their competencies. Unfortunately, this probably happens more frequently than it should, since after a certain age people may have very limited opportunities to try something new. So they simple continue doing the same mediocre job they did in the past, because this is the only thing they know.

This phenomenon is known as the “Peter Principle”:

“In a hierarchy, members are promoted so long as they work competently. Eventually they are promoted to a position at which they are no longer competent (their ‘level of incompetence’), and there they remain, being unable to earn further promotions.”

The Peter Principle has two corollaries:

“In time, every post tends to be occupied by an employee who is incompetent to carry out its duties.”

“Work is accomplished by those employees who have not yet reached their level of incompetence.”

These people who have reached their levels of incompetence are probably not enjoying their jobs anymore. But they will continue working because they are being paid and they may not have any other alternative. So in this case they may actually spend 10,000 hours doing something, not enjoying it and not improving.

In Summary:

  • Aptitude is required: To invest 10,000 hours doing some activity you must enjoy it, and to enjoy it you need talent.
  • No guarantee of success: If people reach their incompetence level they may spend 10,000 hours doing the same mediocre job, not enjoying it and not improving.

People like to believe that if they work hard enough they can become very good at anything they want. I think that people should first understand what are the few things they are already good at, and then work hard to become even better.

What do you think? Do you agree or do you have different experiences?

Posted in Efficacy | Tagged | 7 Comments

A Social Watching Experience for Yahoo! Connected TV

Posted on March 18, 2013 by

Connected TV is one of Yahoo!’s most innovative products. The basic idea is that your TV set should be as smart as other devices such as your smartphone or your tablet. In other words, the Connected TV is always connected to the Internet and is able to run its own Web-enabled applications. And, of course, the Connected TV also has a store from which you can download these applications.

CTV boxingOne of the most interesting technologies in Yahoo!’s Connected TV is called Broadcast Interactivity (BI). BI allows the automatic identification of the show that is being watched in the TV set. This real-time knowledge about the show being watched creates a context for the enrichment of the user experience. For example, it is possible to supply additional content that complements the TV show, or targeted advertisements that take in consideration the show being watched.

We at Yahoo! Labs Haifa decided to develop an application to investigate the possibilities of BI combined with social networks. Our idea was to help the user to share his TV watching experience, through the automatic location of related content that could be posted in social networks. For example, if the user is watching a movie, he may want to post on his Facebook wall a YouTube link to the trailer of this movie. In a different scenario, the user may want to check the Facebook page for this movie and read what other users have been saying about it.

This application was called the Social Companion, and it was developed in one semester by two bright Technion students, Eitam Amit and Roy Velich, under my supervision together with my colleague Amit Kagian. I must say that Eitam and Roy did an outstanding work, learning in a short time how to develop applications for Connected TV, combining content from multiple sources and proposing several additional features to our initial concept. You can check their project site for more information, and see below a screen shot of the Social Companion application.

CTV social companionWe hope that this project will motivate other young programmers to develop applications for Connected TV, and that in the future it will become a platform as popular as the other smart devices. If you want to learn more about Connected TV or about the Social Companion application, please use the comments below.

Posted in Social Networks, Web Development, Yahoo! | Tagged , , | Leave a comment

On Developer Wisdom and Software Quality Attributes

Posted on February 7, 2013 by

EcclesiastesWhat is wisdom? In order to answer this question, we will look into some ancient Jewish texts.

From the Talmud (Tamid 32A), compiled 1500 years ago:

“Who is wise? He who discerns what is about to come to pass.”

In other words, the wise man is the one who sees the consequences of his actions.

From Ecclesiastes (14:1), composed 2300 years ago:

“The wise man, his eyes are in his head; but the fool walks in darkness.”

Rashi, the medieval commentator (France, 1000 years ago), interprets his eyes are in his head to mean: “In beginning of the matter he foresees the outcome.”

From Pirkei Avot (2:10), written 1800 years ago:

“[Rabbi Yochanan] said to them: Go and see which is the best trait for a person to acquire… Said Rabbi Shimon: To see what is born [out of one’s actions]…

He said to them: Go and see which is the worst trait, the one that a person should most distance himself from…. Said Rabbi Shimon: To borrow and not to repay…”

In this case, a person who accumulates a debt which he will not be able to pay is the ultimate example of not being wise enough to foresee the future implications of your actions.

Wisdom and Decision Making

From the sources above, we conclude that the wise person is able to foresee the future consequences of his actions. Of course this ability is useful only if it is applied to the decision making process.

If at the present moment you have several possible courses of action and if you are able to predict the implications of each alternative, then you can choose the option that will produce the more favorable outcome.

chessThe decision making process is very similar to a game of chess. It is a well-known fact that chess masters have the ability to foresee several moves in advance. This vision allows them to build a successful game strategy. In the case a chess master must react to a surprising move from his adversary, he is able to rapidly consider several alternatives, analyzing their likely outcomes, and choosing the best one.

Some artificial intelligence algorithms are also based on this notion of exploring the future consequences of present actions. For example, the Minimax algorithm builds a game tree in which each node represents a move. The algorithm tries to maximize its chances of winning the game, while at the same time it minimizes the chances of its adversary to win the game. This algorithm has been used to implement chess-playing programs, employing heuristics to evaluate non-final game states.

Wisdom in Software Development

But how is this ability to foresee the consequences of our decisions related to software development? From Wikipedia:

“Effective software design requires considering issues that may not become visible until later in the implementation.”

We could also ask: What characterizes high-quality software? Is this the ability to satisfy customer needs? Is quality the correct implementation of the requirements?

The answer is that correctness is just one of several software quality attributes. What about other attributes such as maintainability, extensibility and reusability? All of them have in common the fact that they address future needs:

Maintainability: In the future, the system will need to be maintained, for example to accommodate changes in requirements or to improve its performance. We would like this maintenance to be as easy as possible.

Extensibility: In the future, the system will need to be extended, for example to add new features or to support new functions. We would like these extensions to require minimum effort.

Reusability: In the future, we would like to reuse some modules of our system, for example to create a new product or support a new customer. We would like this reuse to be as simple as possible.

In other words, designing a system to be maintainable, extensible and reusable requires from software developers the capacity to plan for future needs.

But what happens if a team implements a system without taking in consideration these future needs? The answer is that this team is accumulating Technical Debt.

Technical Debt: The consequence of implementing poor design decisions in order to save development time. Like a debt is a value you know you will have to pay, technical debt is code that you know you will have to refactor.

As we have learned above from Pirkei Avot, a person who accumulates a debt which he is not able to pay cannot be considered wise. The same can be said about software developers who accumulate technical debt.

Conclusion

As we have learned from the ancient Jewish sources, wisdom means knowing the future consequences of our actions. Accordingly, developer wisdom means understanding the future implications of our design decisions. A programmer who is only concerned with the correct implementation of the current requirements cannot be considered wise.

What about you? Are you a wise software developer?

Posted in Agile, Efficacy, Software Evolution | Tagged , , | 8 Comments

My first computer, 30 years ago…

Posted on January 27, 2013 by

CP 500So we have reached the year 2013, and this reminds me of my first computer, which I got in 1983, or 30 years ago. It was a Brazilian CP-500, compatible with the original TRS-80 model III. It had 64K of RAM, two drives for 5¼ floppy disks and a monochromatic monitor. Each floppy disk could store 180K on each side, and this was enough for saving several programs.

I was 13 when my father bought me this computer. Of course, it was a gift for my Bar-Mitzvah, and I was one of the first boys in my class to have a computer at home. I was very excited, but did not have any software to run on it. So I did what I could: I read the two manuals that came with the computer, and tried all the commands.

The first manual was for the operating system, the TRS-DOS. I learned everything about manipulating files on the disk. This was useful, but not very fun.

The second manual was for the BASIC programming language. I immediately started writing my own tiny programs, copying the examples from the book and making small modifications. This was much more fun, but still very limited.

SinclairThings got more interesting when one of my friends gave me a book with printed listings of source code for games in BASIC. I tried to copy them, typing line after line, but got many errors. I soon understood that these games were written for the Sinclair computer, which was not compatible with mine.

Then I decided to fix these games so they would run on my computer. I remember spending many hours trying the different instructions, especially the ones related to the graphic display, which was completely different in the two computers.

In the end I was able to convert all games and had them running properly on my machine. It was a double win: Now I had games to play with, and now I had learned to program in BASIC. And this was the very start of my career as a software developer.

I was very effective in my early years as a teenage programmer. I was extremely curious and looking for challenges. My first computer was a great source of new things to learn and problems to solve, and I had lots of free time.

Today I’m still curious and looking for challenges, but now of course I do not have the free time I had as a teenager. Thus I’m very happy to be still learning new things and solving interesting problems at work. Currently, Recommender Systems for millions of users instead of small games in BASIC.

I feel lucky for having this passion for programming, which is already 30 years old. What can be better than your hobby becoming your profession? Is there anything nicer than being paid to do what you love?

Posted in Programming | Tagged | 4 Comments

IASA IL meeting with Prof. Rick Kazman

Posted on December 7, 2012 by

KazmanThe International Association of Software Architects (IASA) in Israel organized a special event with the participation of Prof. Rick Kazman, who talked about ”The Metropolis Model for Software Development”.

Dr. Rick Kazman is a Professor at the University of Hawaii and a Visiting Scientist at the Software Engineering Institute. His primary research interests are software architecture, design and analysis tools, software visualization, and software engineering economics. He is the author of over 100 papers, and co-author of several books, including “Software Architecture in Practice“, and “Evaluating Software Architectures: Methods and Case Studies“. Kazman was one of the creators of the SAAM (Software Architecture Analysis Method) and the ATAM (Architecture Tradeoff Analysis Method). Dr. Kazman received B.A. and M.Math degrees from the University of Waterloo, a M.A. from York University, and a Ph.D. from Carnegie Mellon University.

The Metropolis Model for Software Development

Abstract

We are in the midst of a radical transformation in how we create our information environment.  This change—the rise of large-scale cooperative efforts, peer production of information—is at the heart of the open-source movement but open source is only one example of how society is restructuring around new models of production and consumption.  This change is affecting  not only our core software platforms, but every domain of information and cultural production.  The networked information environment has dramatically transformed the marketplace, creating new modes and opportunities for how we make and exchange information.  “Crowdsourcing” is now used for creation in the arts, in basic research, and in retail business.  These changes have been society-transforming.  So how can we prepare for, analyze, and manage projects in a crowdsourcing world?  Existing software development models are of little help here.  These older models all contain a “closed world” assumption: projects have dedicated finite resources, management can “manage” these resources, requirements can be known, software is developed, tested, and released in planned increments.   However, these assumptions break in a crowdsourced world.  In this talk we will present principles on which a new system development model must be based.  We call these principles the Metropolis Model.

These are the original slides of Prof. Kazman’s presentation:

Here is the video of the entire talk and the Q&A session:

This event was hosted by LivePerson at Raanana.

The audience was composed of experienced software professionals, who clearly identified with the subject and asked many questions.

To participate in our future meetings, please join the IASA IL group on LinkedIn.

Feel free to share your comments below. Thanks!

Posted in IASA Israel, Software Architecture | Tagged , | Leave a comment

On Information Hiding and Encapsulation

Posted on November 25, 2012 by

This month I participated in IBM Haifa’s Programming Languages and Software Engineering (PLSE) Seminar. There I had the opportunity to have lunch with David Parnas, one of the world pioneers in the field of Software Engineering. Parnas is the father of Information Hiding, a term he coined and which became popular through his seminal paper “On the Criteria to Be Used in Decomposing Systems into Modules“, published in 1972.

My personal conversation with Parnas at lunch was mostly about places to visit in Israel, but during the seminar it was great to hear him talking about the challenges of modern software development. He expressed his view that software developers must have a solid education on the principles of software engineering, and focus less on particular technologies which will soon become obsolete. This is my motivation to write this post about Information Hiding and Encapsulation.

In the conclusion of his above-mentioned paper, Parnas writes: “… it is almost always incorrect to begin the decomposition of a system into modules on the basis of a flowchart. We propose instead that one begins with a list of difficult design decisions or design decisions which are likely to change. Each module is then designed to hide such a decision from the others. Since, in most cases, design decisions transcend time of execution, modules will not correspond to steps in the processing.”

This idea that modules should not “correspond to steps in the processing” can be considered one of the principles in object-oriented design. In OOD, each class has some responsibility and different classes collaborate with each other, as proposed by the class-responsibility-collaboration (CRC) approach. The methods provided by a class may be used at different times, on diverse parts of a system. Thus it is clear today that in OOD we do not decompose the system according to the steps in the processing, but in 1972 this was a revolutionary idea.

In another section of his paper, Parnas describes the Information Hiding decomposition criterion: “Every module … is characterized by its knowledge of a design decision which it hides from all others. Its interface or definition was chosen to reveal as little as possible about its inner workings.”

The ideas expressed in these two lines are the essence of what today is called encapsulation. We know that in OOP each class should have a clear and well-defined interface that hides its implementation details. Modern object-oriented languages provide constructs which allow the programmer to define the visibility of members of a class, such as public or private.

When a programmer uses encapsulation to hide the “inner workings” of a class, he is assuring that this class will be:

  • Easier to maintain, because changes will not be propagated to other parts of the system.
  • Easier to extend, by adding new functionality while preserving the original interface.
  • Easier to reuse, since this class will not depend on the details of the rest of the system.

These benefits of encapsulation are well-understood by programmers today. It is natural to hide the details of a class when using object-oriented programming languages. But in the past people did not have appropriate abstractions to implement it.

Because Information Hiding is a principle, it was true back in 1972, it continues to be true today and it will still be true in the future. It could be applied in technologies that are now obsolete, it is an essential part of modern object-oriented technologies, and it certainly will continue to play an important role in new technologies in the future.

Software Developers must learn about the principles of Software Engineering. This is a principle. Specific technologies are good for the implementation details.

Posted in OOD, OOP, Software Architecture | Tagged , , | 4 Comments