Introducing Significant Architectural Change within the Agile Iterative Development Process

Abstract

As an architect working within an iterative agile environment, it’s important to understand how significant architectural decisions need to be made as early in the development process as possible to mitigate the high cost of change of making these decisions too late in the game. In iterative development, it’s important to realize the distinction between requirements that require significant consideration to the architecture and those that are insignificant. This article contrasts the difference between significant and insignificant requirements and demonstrates the best approach to implementing both types of requirements. In agile environments, it’s important to realize that significant architecture decisions will sometimes need to be determined iteratively and late in the development cycle. Guidance is provided on determining how to move forward by considering many factors including ROI, risk, cost of change, scope, alternative options, regression, testing bottlenecks, release dates, and more. Further guidance is provided on how the architect can collaboratively move forward with an approach to implementation while ensuring team vision and architectural alignment with the business requirement.


Introducing Significant Architectural Change within the Agile Iterative Development Process

As agile methodologies such as scrum and XP focus on iterative development (that is, development completed within short iterations of days or weeks), it’s important to distinguish the requirements that are significant to your architecture within the iterative development process.  Iteratively contributing to your software architecture is very important to maintaining your architecture, ensuring the right balance between architecture and over architecture, and ensuring that the architecture is aligned with the business objectives and ROI throughout the iterative development process

Many agile teams are not making a conscious effort to ensure significant decisions relating to software architecture are being accounted for iteratively throughout the software development process.  Sometimes it’s because there is a rush to complete features as required within the iteration and little thought is given to significant architectural changes, or sometimes it is lack of experience or lack of a team vision.  There is usually a lack of understanding how the important guiding principles of the architecture need to be continually established and how they shape the finished product and align with the business objectives in order to see a return on investment. This is where the architect plays a huge role within the iterative development process.

The lack of attentiveness to significant architecture decisions whether at the beginning or mid-way through a release cycle can cause significant long term costs and delay product shipping significantly. Many teams are finding out too late in the game that guiding principles of their architecture are not in place and strategies to get to that point still need to be put in place at the expense of time, technical debt, and being late to ship.  When requirements from the product team involve core architecture changes or re-engineering, changes are sometimes done so without recognizing the need to strategize and ensure that guiding principles of your core-architecture are in place to ensure on-going business alignment and minimal technical debt cost.

Within the iterative development process, it is important that the agile development teams (including the architect) learn to recognize when new requirements are significant and when they are not.  Deciding which requirements are significant and will be carried on as guiding principles of your architecture can be worked out collaboratively with the developers and architects during iteration planning or during a collaborative architectural review session.  This will help ensure that development is not started without consideration to the architecture of these new significant requirements, and that there is time to get team buy-in, ensure business alignment, and create a shared vision of the new guiding principles of your architecture.

In addition, to help prevent surprises during iteration planning, the architect can be involved and work with the product team when preparing the user story backlog to help identify stories that could have a significant impact on the architecture prior to iteration planning.  Steps can be put in place to help the product team understand the impact and to assist the product team in understanding what the ROI should be in contrast to the cost to implement major architectural changes.

So, what distinguishes significant architectural requirements with insignificant ones?

Separate which requirements have architectural significance with ones that do not where significant is distinguished by the alignment of business objectives and a high cost to change, and insignificant is more closely aligned with changing functional requirements within the iterative development process.

An insignificant architectural requirement will be significant as it retains to the functional requirements’, but not significant in terms of the core architecture.  To further contrast the difference between which decisions to consider significant and which to consider insignificant take a look at the following table.

Significant Insignificant
A high cost to change later if we get it wrong We write code to satisfy functional requirements and can easily refactor as needed
Functionality is highly aligned to key business drivers such as modular components paid for by our customers and customer platform requirements A new functional requirement that can be improved or duplicated later by means of refactoring if necessary.
The impact is core to the application and introducing this functionality too late in the game will have a very high refactoring and technical debt cost Impact is localized and can be refactored easily in the future if necessary.
Decisions that affect the direction of development, including platform, database selection, technology selection, development tools, etc. Development decisions as to which design patterns to use, when to refactor existing components and decouple them for re-use, how to introduce new behavior and functionality, etc.
Some of the ‘ilities’ (incl: Scalability, maintainability/testability and portability). Some of the ‘ilities’, such as usability depending on the specific case can be better mapped to functional requirements.  Some functional requirements may require more usability engineering than others.

It is best to handle the significant decisions as early as possible in the development process.  As contrasted in the table below, you can see how the iterative approach lends itself well to requirements that have an insignificant impact on the architecture.  You can also see how significant architectural requirements really form guiding principles of your architecture and getting it right early on lessons the impact of change on the product.

Insignificant Decisions How to Approach
Examples:

  • New functional requirements (ex: to allow users   to export a report to PDF. There is talk in the future about allowing export to Excel as well, but it is currently not in scope.)
  • Modifications and additions to existing   functionality and business logic

 

Use an agile iterative approach to development.  This is a functional requirement, with a low cost to change and a low cost to refactoring.  Write the component to only handle its specific case and don’t plan your code too much for what you think the future requirements might be.  If the time comes to add or improve functionality, then we refactor the original code to expose a common interface, use repeatable patterns, etc, In true agile form, this will prevent over-architecture if future advances within the functional requirements are never realized, and there is minimal cost to refactor if  necessary.
Significant Decisions How to Approach
Examples:

  • Our customers are using both Oracle and SQL Server
  • Performance and scalability
  • Security Considerations
  • Core application features that have a profound   impact on the rest of the system (example, a shared undo/redo system across all existing components) 
These decisions need to be made early as possible and an architectural approach has to be put in place to satisfy the business and software requirements’.  These decisions are usually significant as there is a huge cost to change (refactoring and technical debt) and potential revenue loss if not put in place correctly, or needs to be refactored later.  These decisions are core to the key business requirements and will have a huge cost to change if refactoring is required later.

Agile Isn’t Optimized For Significant Architectural Change

It’s unfair to assume that the agile development process is built to excel at the introduction of significant functionality and architecture changes without a large cost.  This is why some requirements are significant and need to be made as early on in the development cycle as possible while ensuring there is alignment with the business objectives.

As great as it would be to map out every possible significant requirement as early on as possible, there are sometimes surprises.  This is agile development after all.  We need to understand that along with changing functional requirements, significant business changes part way through the development process can occur and could still have a significant impact on our core architecture and guiding principles, so we need to mitigate and strategize how to move forward.

Certainly, it’s possible to introduce significant core-architecture changes by means of refactoring, or scrapping old code and writing new functionality – that’s the agile approach to changing functional requirements and how agile can help prevent over-architecture and ensure we are only developing what’s needed.  The problem is that it doesn’t work well for the significant decisions of your architecture, and when we do refactor, the cost can be so high that it will exponentially increase your time to market, cause revenue or customer loss, and potential disruption to your business.  In these cases, the architect, along with the product and development team need to create a plan to get to where they need to be.

Mitigating the Cost of Significant Change

There is always a cost to introducing core architectural functionality too late in the game.  The higher the cost of the change and the higher the risk of impact, the more thought and consideration needs to be put into the points below.  Here are some points that need to be considered.

  • The refactoring cost will be high.  Is there an alternative way we can introduce this functionality in a way that will have a minimal impact now without affecting the integrity of the system later?
  • This change is significant and will require a huge burden by the development team to get it right. Will we have a significant ROI to justify the huge cost to change?  For example, is Oracle support really necessary after developing only for Sql Server for the first 6 months or is it just a wish from the product team?  Do we really have customers that will only buy our product if it’s Oracle, and what are the sales projections for these customers?  Is there a way we can convince these customers to buy a Sql Server version of the product?  The architect needs to work with the product and business teams to determine next steps.
  • How will this affect regression testing?  Are we creating a burden for the testing team that will require a massive regression testing initiative that will push back our ship date even further?  Is it worth it?
  • How close are we to release?  Do we have time to do this and make our release ship date?
  • What is the impact of delaying our product release because of this change?
  • Is it critical for this release or can it wait until a later release?
  • Can we compromise on functionality?  Can we introduce some functionality now to satisfy some of the core requirements and put a plan in place to slowly introduce refactoring and change to have a minimal impact up front, but still allow us to meet our goal in the future?
  • What is the minimal amount of work and refactoring we need to do?
  • What is the risk of regression bugs implementing these major changes late in the game?  Do we have capacity in our testing team to regression test while also testing new functionality?
  • Are we introducing unnecessary complexity?  Can we make this simple?

All individuals involved in the software need to be aware of the impact and high cost that significant late to game changes will have on the system, development and testing teams, ship dates, complexity, refactoring, and technical debt that could be introduced.  There are strategies that can be used, and the points above are a great start in determining how to strategize the implementation of significant architecture changes.  One of the roles of the architect is to help facilitate and create the architecture and guiding principles of the system and ensure its long term consistency.  As the system grows larger and more development is complete, introducing significant architecture changes becomes more complex.  The architect needs to work with all facets of the business (developers, qa, product team, sales, business teams, business analysts, executives, etc) to help ensure business alignment and a solid ROI of significant architecture decisions.

Moving Forward

Once a significant decision is made that will form part of your architectures significant guiding principles, the architect needs to understand the scope of work, determine what will be included and what won’t, collaboratively create a plan on how we will get there, and understand how the changes will fit within the iterative development cycle moving forward.  The architect needs to ensure that the product and development teams share the vision, understand the reasons we are introducing the significant change, and understand the work that will be required to get there.  If your team is not already actively pairing, it may be a good time to introduce it or alternatively introduce peer reviews or other mechanisms to help ensure consistent quality when refactoring existing code to support significant architectural changes.

Depending on the level of complexity, the testing team may need to adjust their testing process to ensure adequate regression testing takes place that tests new and existing requirements that are affected by the significant architecture change.  For example, if we make a significant change to support both Oracle and Sql Server we need to ensure existing functionality that was only tested for Sql Server support is now re-tested in both Oracle and Sql Server environments.  The architect or developers can work with the testing team for a short time to help determine the degree of testing and which pieces of functionality specifically need to be focused on and tested to ensure the QA/testing teams are correctly focusing their efforts.

Summary

It’s important to distinguish significant architecture decisions from requirements that are insignificant as they relate to the core architecture of your system.  When introducing significant architecture changes iteratively within an agile environment, it’s important to understand the impact and complexity that significant changes have when introduced late in the game.   It’s important to understand the business impact of the changes and it’s important that the architect works with the rest of the organization to determine business alignment, risk, and ROI of these changes while understanding the cost of change before moving forward with a plan to introduce the significant architectural changes.


Dan Douglas is a professional independent Software Consultant and an experienced and proven subject matter expert, decision maker, and leader in the area of Software Architecture. His professional experience represents over 12 years of architecting and developing highly successful large scale solutions.  Dan has been the architect lead on over 15 development projects.

Advertisements

About dandouglas
Dan Douglas is based in Toronto, Ontario, Canada and is a professional independent Software Consultant and an experienced and proven subject matter expert, decision maker, and leader in the area of Software Development and Architecture. His professional experience represents over 15 years of architecting and developing highly successful large scale solutions. Dan also believes that properly empowering teams with trust and responsibility yields the greatest results. | For inquiries about Dan's software consulting practice, please see the contact page.  Dan has also built up a network of highly successful and professional Software Developers and Architects who are highly respected and can be called upon and used in conjunction with his own consulting practice to handle the largest of consulting projects.

One Response to Introducing Significant Architectural Change within the Agile Iterative Development Process

  1. Pingback: Agile Software Architectural Governance in the Real World | Dan Douglas - Articles on Software

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s