Antifragility and Component-Based Software Development

AntifragileIn his book “Antifragile: Things That Gain From Disorder”, Nassim Taleb introduces the concept of Antifragility, which is the opposite of Fragility. Antifragile things are able to benefit from volatility. In a previous post, I explained how in the field of software development the main source of volatility are the changes in requirements, and how it is possible to make software systems more robust through the appropriate usage of abstractions.

However, robustness is not the same as Antifragility. A robust system may survive the impact of changes, but our goal is to develop systems that actually benefit and improve as a consequence of these changes. In this article we will analyze a strategy that may be adopted to develop such Antifragile systems.

It is common knowledge that systems should be divided into components. The reaction to a change in the environment should only affect a few components, and not the entire system. Thus, component-based systems are more robust than monolithic systems.

However, to be Antifragile, a system must be able to benefit from changes in the environment. This Antifragility may be achieved when several systems share the same components. Thus, when a specific component is improved, all systems using this component can benefit from this improvement.

One concrete example is AA batteries. During many years AA batteries were the standard for millions of different devices. Then, energy-consuming devices drove the development of rechargeable AA batteries. Now all devices can benefit from this improvement, and in this sense they are Antifragile.

Component-Based Software Development

Since the first years of software development it was understood that systems should be divided into components, and thus software engineers tried to identify the right principles to perform this division, which was once called modularity. Metrics such as coupling and cohesion and principles such as information hiding were defined to guide the decomposition of large software systems into modules.

Over the decades the advances in programming languages and software engineering practices were followed by better ways to perform this decomposition. Object-oriented programming languages allowed software developers to create classes of objects and organize them into hierarchies. Then, larger distributed systems were built based on Object-Request Broker (ORB) platforms. More recently Service Oriented Architectures (SOA) became popular and now the current trend is Microservices.

In all the examples above, the evolution was based on the same original principles: Focusing on low coupling, high cohesion and defining clear and abstract interfaces. All of them contributed to make software systems more robust. It is easier to change the implementation of a class than to modify a data structure and its associated procedures. Services can be deployed independently of each other, and without disrupting the rest of the system.

The next step, which is to develop Antifragile systems, going beyond robustness, will happen when many different systems share the same components. Then, if a particular component is improved to satisfy the requirements of a specific application, all the systems may enjoy the improved component at no additional cost. This is true Antifragility.

Consider for example a system that is based on a SOA architecture. At any point in time it is possible to deploy an enhanced version of one of the services without affecting the other ones, and thus the system is very robust. But if there are several systems based on shared services, each time one of these services is improved all the systems will be able to immediately benefit from the improvement. Thus while each system is robust, the collection of systems is Antifragile, because they benefit from the same changes at no added cost.

Another modern application of this approach is the idea of a Software Product Line (SPL), in which a series of software products are based on the same components and differ only on their configuration. In the case of SPLs there may be several coexisting versions for each component. Each time a new component version is created, all the products using previous versions may benefit through simple configuration updates.

In summary

A simple strategy for Antifragility is: Build component-based systems with shared components. When one of the shared components is improved all the systems will be able to benefit from this improvement at no additional cost.

What do you think? Have you experienced the benefits of building several systems on top of the same components? Please share your experience with us in the comments below.

Posted in Antifragility, Software Architecture, Software Evolution, Software Reuse | Tagged , , , | 16 Comments

Agile Practices and Social Nudges in the Workplace

Nudge-coverIn their best-selling book “Nudge: Improving Decisions About Health, Wealth, and Happiness“, Richard Thaler and Cass Sunstein propose the adoption of interventions to “attempt to move people in directions that will make their lives better.” A nudge “alters people’s behavior in a predictable way without forbidding any options or significantly changing their economic incentives.”

One classic example of a nudge is the “fly in the urinal”. Many public bathrooms face the problem of men’s lack of attention. To solve this problem, they place a sticker with the picture of a fly inside the urinal, and the result is that men aim at the fly as a target. Trials showed that this simple idea reduced spillage by 80%!

A special kind of nudge is the “Social Nudge”, which results from the interaction with other people. According to Thaler and Sunstein:

“Social influences come in two basic categories. The first involves information. If many people do something or think something, their actions and their thoughts convey information about what might be best for you to do or think. The second involves peer pressure. If you care about what other people think about you, then you might go along with the crowd to avoid their wrath or curry their favor.”

I believe that the success of Agile practices may be partially attributed to the Social Nudges created by changes in the team dynamics and by the richness of interactions among team members. Let’s consider the two social influences categories as discussed above:

Exchange of Information

Agile practices increase the amount of interactions among team members and also the depth of information exchanged among them. For example:

  • Daily stand-up meetings allow all team members to know what their peers are doing and to become aware of their particular contribution to the project.
  • Retrospective meetings allow the team members to think about ways to improve their own processes based on past experience.
  • Code reviews allow team members to learn from each other and develop improved coding standards and shared patterns.

In this environment rich of information exchange opportunities, the natural tendency for the team members is to adopt best practices and increase both their productivity and their satisfaction.

Peer Pressure

Agile practices increase the visibility of the work being performed by each team member, and thus increase also the peer pressure. For example:

  • Burn-down charts display clearly if a team member is making progress as originally estimated or if he is late in his work.
  • Velocity charts show the productivity of the team and will be negatively impacted if there is a team member who is working slower than others.
  • In the daily stand-up meetings each team member must say what he is going to do today, which serves as a public commitment to perform his task.

In this environment in which there is so much visibility of the progress and the commitments of each team member, a person certainly will not feel comfortable if he is under-performing in comparison to other members of the team.

In summary

Agile practices promote more opportunities for the exchange of information and for peer pressure, and thus create social nudges that result in improved performance and satisfaction.

What do you think? Did you experience these social nudges? Please share your opinion in the comments below.

Posted in Agile, Efficacy, Psychology of Programming | Tagged , , | 2 Comments

IASA Israel Meeting – Dror Helper on TDD as an Approach for Software Design

The International Association of Software Architects (IASA) in Israel organized a special event with the participation of Dror Helper, who talked about the “TDD as an Approach for Software Design”.

Title: Designing with Tests

Abstract: Unit tests are great, they help a developer gain control over his code and catch bugs and regression issues. Test Driven Development is a methodology that uses unit tests but it’s not about writing unit tests – in fact the tests are only a design tool. TDD is a methodology that solves problems in an iterative way, it’s about emergent design that creates a maintainable solution. In this session I’ll talk about common mistakes and misconceptions, how to benefit from TDD and show how to design your code using unit tests.

Bio: Dror Helper is a senior consultant at CodeValue specializing in engineering practices. He has been writing software professionally for more than a decade, during which he has worked for industry giants such as Intel and SAP as well as small start-up companies. Dror has designed and developed software in various fields including video streaming, eCommerce, performance optimization and unit testing tools. His first encounter with agile happened a few years ago while working for Typemock Ltd, since then he has been evangelizing agile wherever he goes — at his work, speaking at conferences and today as a consultant. In his blog (http://blog.drorhelper.com) Dror writes about programming languages, software development tools, unit testing and anything else he finds interesting.

These are the original slides of Dror’s presentation:

Here is the video of the talk (in Hebrew):

This meeting was organized with the cooperation of ILTAM and sponsored by SAP Labs Israel. Special thanks to Dima Khalatov who helped with the organization.

To participate in our future meetings, please join the IASA IL group on LinkedIn.

Feel free to share your comments below. Thanks!

Posted in IASA Israel, Software Architecture, TDD | Tagged , , | 1 Comment

Contextual Recommendations in Multi-User Devices

yahoo_labsMy last project at Yahoo! Labs was focused on Recommender Systems for TV shows. In the presentation below, Ronny Lempel (who was my manager at this project) discusses the challenges of producing personalized recommendations in multi-user devices. This is joint work with my colleagues Raz Nissim, Michal Aharon, Eshcar Hillel and Amit Kagian.

Abstract: Recommendation technology is often applied on experiences consumed through a personal device such as a smartphone or a laptop, or through personal accounts such as one’s social network account. However, in other cases, recommendation technology is applied in settings where multiple users share the application or device. Examples include game consoles or high end smart TVs in household livingrooms, family accounts in VOD subscription services, and shared desktops in homes. These multi-user cases represent a challenge to recommender systems, as recommendations exposed to one user may actually be more suitable for another user.

This talk tackles the shared device recommendation problem by applying context to implicitly disambiguate the user (or users) that are being recommended to. Specifically, we address the household smart-TV situation and introduce the WatchItNext problem, which — given a device — taps the currently watched show as well as the time of day as context for recommending what to watch next. Implicitly, the context serves to disambiguate the current viewers of the device and enables the algorithm to recommend significantly more relevant watching options than those output by state-of-the-art non-contextual recommenders. Our experiments, which processed 4-months long viewing histories of over 350,000 devices, validate the importance and effectiveness of contextual recommendation in shared device settings.

The presentation slides are available for download here.

Feel free to share your comments below. Thanks!

Posted in Data Mining, Recommender Systems, Research, Yahoo! | Tagged , , , | 1 Comment

IASA Israel Meeting – Lior Israel on TDD as an Approach for Software Design

The International Association of Software Architects (IASA) in Israel organized a special event with the participation of Lior Israel, who talked about the “TDD as an Approach for Software Design”.

Title: Providing Business Value Through TDD

Abstract: Our goal is to provide to our customers business value. In my daily work as an architect I show how to provide business value with quality in short feedback time. One of the best ways to provide business value is TDD, and this means adopting TDD for all the game player’s not only for developers! The presentation will be based on code examples. Outline:
– TDD overview as way of thinking – I am going to demo the simple truth: TDD is NOT unit testing.
– Let’s start coding code Kata – The best way to understand it is through a real example.
– Stop! Are we going to meet the DoD? We are going to see that TDD lead us to verify that we understand the business value not at the end of development but at the beginning.
– Architecture & TDD – demonstrate that TDD puts us on the right side of the sun (Architecture). We only need to do it as TDD way.

Bio: Lior Israel is an experienced software and hardware developer, designer and System Architect. Lior started his studies of Electronics in the early 1990’s and has a degree in Computer Sciences. He has a practical formal experience of over 19 years with software and hardware development and design. Currently working as a Software Architect on large scale IT applications with cutting edge technologies. Lior has a blog in which he writes about software development: http://blogs.microsoft.co.il/lior_israel/

These are the original slides of Lior’s presentation:

Here is the video of the talk (in Hebrew):

This meeting was organized with the cooperation of ILTAM and sponsored by SAP Labs Israel. Special thanks to Dima Khalatov who helped with the organization.

To participate in our future meetings, please join the IASA IL group on LinkedIn.

Feel free to share your comments below. Thanks!

Posted in IASA Israel, Software Architecture, TDD | Tagged , , | Leave a comment

IASA Israel Meeting – Hayim Makabee on the Role of the Software Architect

The International Association of Software Architects (IASA) in Israel organized a special event with the participation of Hayim Makabee, who talked about the “Role of the Software Architect”.

Title: The Role of the Software Architect

Abstract: In this talk Hayim will present the practical aspects of the role of the Software Architect, including:

  • The four areas of expertise: Design, Domain, Technology and Methodology.
  • The cooperation with stakeholders: Developers, Team Leaders, Project Managers, QA and Technical Writers.

Understanding the expected areas of expertise is essential for the architect to develop his/her professional skills.
Understanding how to cooperate with the diverse stakeholders is essential to improve the architect’s impact and effectiveness.

Bio: Hayim Makabee was born in Rio de Janeiro. He immigrated to Israel in 1992 and completed his M.Sc. studies on Computer Sciences at the Technion. Since then he worked for several hi-tech companies, including also some start-ups. Currently he is a Research Engineer at Yahoo! Labs Haifa. He is also a co-founder of the International Association of Software Architects (IASA) in Israel. Hayim is the author of a book about Object-Oriented Programming and has published papers in the fields of Software Engineering, Distributed Systems and Genetic Algorithms. Hayim’s current research interests are Data Mining and Recommender Systems.

These are the original slides of Hayim’s presentation:

Here is the video of the talk (in Hebrew):

This meeting was organized with the cooperation of ILTAM.

To participate in our future meetings, please join the IASA IL group on LinkedIn.

Feel free to share your comments below. Thanks!

Posted in IASA Israel, Software Architecture | Tagged , | 1 Comment

Reducing Technical Debt

logo_white_towerLast week I traveled to Thessaloniki to participate in the Conference on Advanced Information Systems Engineering (CAISE’14). I presented a paper at the Workshop on Cognitive Aspects of Information Systems Engineering (COGNISE). This is joint work with Yulia Shmerlin and Prof. Doron Kliger from the University of Haifa.

Title: Reducing Technical Debt: Using Persuasive Technology for Encouraging Software Developers to Document Code

Abstract: Technical debt is a metaphor for the gap between the current state of a software system and its hypothesized ‘ideal’ state. One of the significant and under-investigated elements of technical debt is documentation debt, which may occur when code is created without supporting internal documentation, such as code comments. Studies have shown that outdated or lacking documentation is a considerable contributor to increased costs of software systems maintenance. The importance of comments is often overlooked by software developers, resulting in a notably slower growth rate of comments compared to the growth rate of code in software projects. This research aims to explore and better understand developers’ reluctance to document code, and accordingly to propose efficient ways of using persuasive technology to encourage programmers to document their code. The results may assist software practitioners and project managers to control and reduce documentation debt.

You can freely download the paper.

Below are the slides of my presentation at the workshop:

Posted in Psychology of Programming, Research, Technical Debt | Tagged , , | 11 Comments

Antifragile Software Design: Abstraction and the Barbell Strategy

AntifragileIn his book “Antifragile: Things That Gain From Disorder”, Nassim Taleb introduces the concept of Antifragility, which is the opposite of Fragility. The main question is how things react to volatility (such as randomness, errors, uncertainty, stressors and time). According to him, there is a Triad – Antifragility, Robustness, and Fragility:

  • Fragile things are exposed to volatility, meaning that volatility is prejudicial to them.
  • Robust things are immune to volatility, meaning that volatility does not affect them.
  • Antifragile things enjoy volatility, meaning that volatility is beneficial to them.

In Taleb’s words:

“Antifragility is beyond resilience or robustness. The resilient resists shocks and stays the same; the antifragile gets better.”

In order to reduce fragility, Taleb proposes the adoption of a Barbell Strategy:

“Barbell Strategy: A dual strategy, a combination of two extremes, one safe and one speculative, deemed more robust than a ‘monomodal’ strategy.”

In the case of software systems, volatility appears in the form of changes over time. These changes are unavoidable, as expressed in one of the Lehman laws of software evolution:

“Continuing Change — A software system must be continually adapted or it becomes progressively less satisfactory.”

There are two main types of changes that are required in a software system over time:

  • Functional changes: These are changes required to implement new requirements, or to implement modifications in the requirements. These changes have an impact on the system’s behavior and functionality.
  • Non-functional changes: These changes are required to improve the quality of the design. They are normally the result of Refactoring and focus on the reduction of Technical Debt. These changes should not affect the system’s behavior or functionality.

Now that we have identified the sources of volatility (changes) the question is: What is the software equivalent of a Barbell Strategy?

The answer is: Abstraction. Software systems should have a structure and organization based on different levels of abstraction. In this case the duality of the Barbell Strategy is expressed as the separation between high-level abstract elements and concrete implementation details.

  • Abstract elements: Are robust, and should not be easily affected by changes.
  • Concrete implementation details: Are fragile, directly affected by changes.

In other words, software systems should be built in such a way that the volatility (changes) does not affect its structure and organization, preserving the main, high-level abstractions and requiring modifications only on the low-level, concrete implementation details.

Below are several examples of this duality between abstractions and concrete details.

Information Hiding and Encapsulation

Information Hiding was defined by David Parnas as a criterion to decompose a system into modules. According to him: “Every module … is characterized by its knowledge of a design decision which it hides from all others. Its interface or definition was chosen to reveal as little as possible about its inner workings.”

The mechanism of Encapsulation in Object Oriented Programming (OOP) is a direct application of this principle. When applying encapsulation, the interface of a class is separated from its implementation, and thus the implementation may be changed without affecting the clients of the class. In other words, thanks to encapsulation, the clients of a class become less fragile to changes in its implementation details.

Open-Closed Principle (OCP)

The Open-Closed Principle (OCP) was defined by Bertrand Meyer as: “Software entities (classes, modules, functions, etc.) should be open for extension, but closed for modification.” In practice, the implementation of the OCP requires the definition of an abstraction that allows a module to be open to extensions, so that each possible extension is a specialization of this abstraction.

strategyFor example, in the Strategy design pattern, the Strategy itself should be defined as an abstract interface that may be implemented by several possible concrete subclasses. Each specific strategy is fragile and may need to be replaced, but the object using the strategy is not fragile, thanks to abstraction from implementation details.

Test-Driven Development (TDD)

In the case of Test-Driven Development (TDD) the fundamental idea is to write the tests before writing the code. This requires the definition of abstract interfaces for the classes being tested, with a well-defined behavior. Since TDD forces developers to thinks in terms of interfaces and behaviors before writing the code, it actually focuses on the early separation between abstractions and concrete implementation details. This allows the code to be easily refactored in the future, because the unit tests would catch any mistake resulting from changes in the implementation. In other words, the unit tests are less fragile to volatility (changes) because they are based on abstractions.

Inheritance Hierarchies and Frameworks

In OOP, an inheritance hierarchy organizes classes according to a generalization-specialization rationale: the classes near the root of the hierarchy should be more abstract than the classes on the leaves of the inheritance tree. Accordingly, the abstract classes should be less fragile than the concrete classes, meaning that they should be less subject to impact caused by volatility, such as changes in the requirements or changes in the implementation details.

In general, Frameworks use inheritance hierarchies to provide reusable code that is not application-specific. Frameworks define abstractions and the relationships among them. It is the responsibility of the software developer to extend the framework, deriving concrete subclasses, and writing the implementation details to satisfy the requirements of his specific application.  These implementation details are obviously fragile, but the framework is not fragile, and may be reused with no changes on many applications.

Conclusions

Software systems are subject to volatility in the form of changing requirements. Concrete implementation details are fragile and directly affected by these changes. In order to reduce the fragility of the system as a whole, it is important to adopt a Barbell Strategy and define these concrete details as the specialization of higher-level abstractions. Proper abstractions should be robust, surviving the impact of changes. Thus, while the details will change over time, the system structure and organization will be preserved because it is based on abstractions.

Please notice that the examples discussed in this post (encapsulation, OCP, TDD, frameworks) mostly increase the robustness of software systems. Thus, in the original Triad we moved away from Fragility towards Robustness, but we still cannot claim we reached true Antifragility. This would require a system that benefits from volatility, or, in our case, that benefits from changes. This will be the subject of a future post.

Posted in Antifragility, OOD, OOP, Software Architecture, TDD | Tagged , , , , | 29 Comments

IASA Israel meeting – Atzmon Hen-Tov on the Adaptive Object Model

The International Association of Software Architects (IASA) in Israel organized a special event with the participation of Atzmon Hen-Tov, who talked about the “Adaptive Object Model”.

Title: Adaptive-Object-Model – Empower your users to evolve the system

Abstract: In this talk Atzmon Hen-Tov will introduce the Adaptive-Object-Model (AOM) architecture style.

Architectures that can dynamically adapt to changing requirement are sometimes called reflective or meta-architectures. We call a particular kind of reflective architecture an Adaptive Object-Model (AOM). An Adaptive Object-Model is a system that represents classes, attributes, relationships, and behavior as metadata.

Users change the metadata (object model) to reflect changes to the domain model. AOM stores its Object-Model in XML files or in a database and interprets it on the fly. Consequently, the object model is adaptive; when the descriptive information for the object model is changed, the system immediately reflects those changes.

“If something is going to vary in a predictable way, store the description of the variation in a database so that it is easy to change” — Ralph Johnson

Bio: Atzmon Hen-Tov is the VP R&D at Pontis, where he leads the development of the company’s Marketing Platform, iCLM™ on top of the ModelTalk AOM engine. Atzmon has led the development of the ModelTalk AOM Engine since its inception and is co-founder of the Ink AOM open-source framework. Atzmon is interested in Model driven software development and other methods for software development industrialization. Contact him at atzmon@ieee.org.

These are the original slides of Atzmon’s presentation:

Here is the video of the entire talk (in Hebrew):

And here is an animated version of the slides at Prezi.

This meeting was organized with the cooperation of ILTAM and sponsored by LivePerson. Special thanks to Dima Khalatov who helped with the organization.

To participate in our future meetings, please join the IASA IL group on LinkedIn.

Feel free to share your comments below. Thanks!

Posted in Adaptable Design, IASA Israel, Software Architecture | Tagged , , | Leave a comment

Lean Software Development: Before and After the Last Responsible Moment

The Last Responsible Moment (LRM) is a lean development principle defined as:

“A strategy of not making a premature decision but instead delaying commitment and keeping important and irreversible decisions open until the cost of not making a decision becomes greater than the cost of making a decision.”

Let tbe the specific point in time for a concrete design decision Di. The LRM principle tells us that tshould be deferred as much as possible. One of the reasons is to avoid making decisions based on assumptions, so that we should decide when we have optimal information about the requirements and there is a reduced risk that our decisions will need to be changed.

LRM1During the incremental software development process we will implement a set of concrete design decisions D1 … Dn, in which D1 … Di-1 are the decisions implemented before Di, and Di+1 … Dare the decisions made after Di. This order of decisions is not arbitrary and will follow some design rationale.

LRM2But of course these design decisions are not independent of each other. Thus, among the decisions D1 … Di-1 there will be some of them that depend on Di. For example, if each design decision Dj is encapsulated in a class Cj, we must know at least the interfaces of the other classes being used by Cj.

Before the Last Responsible Moment

This means that for each design decision Di there must be a point in time, before ti, in which we define an abstraction for Di. This abstraction allows the implementation of all design decisions that depend on Di but must be made before it. If Di is represented as a class, the abstraction would be its interface, and it should be defined at a time tAi < ti.

LRM3This definition of abstractions before implementations is exactly the result of Test-Driven Development (TDD). When we write unit tests for a class we are defining its interface, both at the syntactic and the semantic levels. By the time all unit tests are written for class Ci we have reached tAi, but we are still before ti, the last responsible moment in which we will decide how to implement Ci.

After the Last Responsible Moment

No matter how long we were able to defer a concrete design decision Di, there will probably be a moment in which it will need to be changed. The last responsible moment is the first time we provide an implementation for an abstraction, but certainly this does not mean that there is any guarantee that this implementation will never be modified.

LRM4For every design decision Di there may be a time tCi > ti in which its implementation is changed. We do not want this change to affect the modules that depend on Di, and this can only be done by preserving the interface. Therefore, the importance of abstractions is twofold: they allow concrete design decisions to be deferred until the LRM, and they also allow these design decisions to be changed after the LRM.

Conclusions

In order to adopt the Last Responsible Moment principle it is essential to focus on the definition of abstractions representing the diverse design decisions. These abstractions serve as the placeholders that will allow the concrete implementation details to be deferred as much as possible. Moreover, good abstractions are essential to support maintainability and allow these design decisions to be easily changed.

Posted in Lean Development, TDD | Tagged , | 3 Comments