Your Team’s Ongoing Vision Will Determine the Long Term Consistency and Alignment of Your Software Architecture

Abstract

The ongoing consistency of your software architecture depends on its alignment with your business objectives, how suitable the technology choices are, and that your team has bought into and is following the architectural vision. This article demonstrates key factors in determining how suitable your software architecture is in being able to sustain a collaborative long term vision and growth of your software along with the difficulties of trying to ensuring business value without a shared vision or collaborative team input. Real world examples from the field are used to demonstrate successful scenarios, the thought behind them, how buy-in was attained, and how they complemented the QA testing strategy of the business objectives. An approach is given on how to recover failing projects where there was no consistent strategy and to turn chaos into a coherent strategy that is aligned with the business objectives.


Your Team’s Ongoing Vision Will Determine the Long Term Consistency and Alignment of Your Software Architecture

I’ve written a recent article that focused on ensuring that your architecture was aligned with your business objectives.  See  Aligning Your Software Architecture With Your Key Business Objectives and Why Your Business Needs It.  This article goes a step further and demonstrates how team buy-in and a shared team vision contributes to the consistency of the architecture and business value throughout the development cycle.

One of the problems that the architecture of your software should address is consistency. Consistency with technology, design patterns, approaches, layers, frameworks, etc.  As software projects evolve, it’s important to ensure that the architecture, especially as aligned with your business objectives, remains consistent.  Functional requirements can certainly change along the way, but the key is to do enough architecture work, and ensure consistency, for those big design decisions that need to be made as early in the game as possible.

Ensuring consistency in this capacity isn’t about guidelines for best practices such as re-use of existing components and developing components that are decoupled from one another.  These best practices should be part of most (all) development projects.  Ensuring consistency of the architecture is about ensuring that the guiding principles of your core-architecture have buy-in, are clear, are being followed, and are driving the long term success of your software in relation to your critical business objectives.

Ensuring consistency can require a certain level of control in some scenarios, however more often than not, very little controls are required when you have team buy-in, and a shared vision from the beginning as to how the business value is being provided.  This is especially true once you have a team that is consistently delivering business value with the software.  Enforcing too strict controls  on teams can be demoralizing for most, and I’m very opposed to forcing development teams to do things a certain way or dictating how work will be done.  I’m all for ensuring consistency and a good architecture across the application, but this can be done without forcing, controlling, and dictating how it will be done.

To help ensure consistency and buy-in across the board, it is important to consider the following when coming up with your architecture:

  1. How crucial are the recommendations at hand to the business?
  2. Will we see a business benefit by evaluating and following guidelines set around our evaluation?
  3. What is the long term detrimental impact to the software of not doing this or doing it too late in the game?
  4. Will following these recommendations eliminate refactoring costs and technical debt later on?

These considerations will help ensure buy-in and ease collaborative agreements with the team as the team will have a better understanding and vision as to the importance of the architecture to the business.  Having a consistent vision of the business value helps ensure consistency with the architecture moving forward.

Be sure that team members who want to participate can help collaborate on the architecture or guidelines.  This provides ownership by the team members which helps drive and ensure continued consistency.  I’ve said many times that architects should not dictate requirements, rather they should create recommendations facilitated by understanding the software, technology, business, customers, etc.  These recommendations should involve collaboration and review with the other team members before final architecture decisions are agreed upon and finalized.

Let’s look at some real life examples from the field

Example 1) The sales team had trouble in the past selling to some large customers who used primarily Oracle database servers.  The architect discussed the scenario with the business leaders who made up a business case for supporting both Oracle and SqlServer.  Collaboratively with the development team, the architect determined that the code base of the entire application could remain the same, but the data layer could be swapped out to support different platforms.  The business case helped ensure buy-in of an ORM mapping tool to be used as a data layer that supported both platforms.  The team collaborated and evaluated different options finally selecting a tool that met all of the business requirements for performance, scalability, and multiple platforms.  In fact, the tool selected would work easily with SqlServer and Oracle platforms with little overhead.  It was clear to the entire team how important the ongoing use of this ORM framework throughout development would be to the business and that deviating from this could cause considerable damage to the product and business model.  The team was completely onboard with the vision.  In addition, this drove the QA testing team to put controls in their testing processes to ensure compliance with this business requirement.  The QA testing team made sure, as part of their process, to test the software on both SqlServer and Oracle platforms and multiple platform testing environments were created.

Example 2) Another scenario allowed customers to pay for specific application modules , but not others.  It was evident that the architecture needed to adhere to this business model that consistency of the architecture throughout the development process would ensure ongoing compliance with the business objectives.  Collaborative team buy-in of the business benefit of this along with the dependency injection and inversion of control framework ensured that the components being built were modular were able to be swapped in and out of the application easily.   This would also drive QA testing initiatives that would ensure test plans accounted for and tested this modularity as it is a core part of the business model.  From a development standpoint, the team came to a shared vision as to the business reason components are built using this approach.  Teams understand that by not doing this or by deviating from this approach that they are creating a refactoring and technical debt cost to fix this later on.  Of course, teams are free to improve upon this when new functional requirements are added within the iterative development process.

In these two examples, it was clear to see that team buy in and a shared vision were established because the architectural approaches were well thought out, evaluated, aligned to the business requirements, and had a huge cost to change if we got it wrong.  Importantly, the entire team had an opportunity to be involved collaboratively in coming up with the architecture which further strengthened their commitment as they implicitly took ownership among themselves in coming up with the architecture.

It is much more difficult to ensure consistency across development teams without a shared vision about the value being delivered.  For example, dictating certain design patterns to be used over other patterns is a subjective decision that will likely fail buy-in as the business value isn’t clear.  A forced buy-in approach which will likely fail and will likely lead to team demoralization.

Guidance, recommended patterns, approaches, and coding standards can all be put in place.  In reality, it means very little unless we are leading by example and have shown through practice, team buy-in, and business value why we are using said approaches and what the business advantage is.  Instead of working on aligning architecture with business requirements, I’ve seen teams spend weeks coming up with coding practices (how to declare variables, which variable naming pattern to use, etc) for new projects.  The problem is that most people don’t read or care to look at the documents created from these team sessions, and ensuring compliance for compliance-sake can be difficult and is a waste of time.  Even if there is a little bit of value in ensuring how code is written and that variable naming is consistent across the board, I don’t believe documents standardizing the approach provide the value or incentives to do this.

Sometimes, to rescue a failing project, you may need to assert more control and constraints in order to get to a point where the software is coherent and beginning to meet the business objectives.  This is a state that we are trying to avoid by doing our up-front architecture work and ensuring consistency with a shared vision.  However, as consultants, sometimes we are brought into the project too late in the game.  If this happens, trying to fix the solution may require short term measures and controls, but don’t lose sight of the fact that the real value is in ensuring consistent business value through architecture and a shared vision.  There may be a lot of refactoring that needs to be done, but the teams still need to share a vision as to the business value of what is trying to be accomplished.  My experience is that dictating control will only work in emergency scenarios in the short term just to get to a stabilization point, but for the long term, the team needs to work with a shared vision and understanding of the business value to make consistent progress.

Getting the team’s buy-in and creating a shared vision may be a bit challenging and may take longer in failing projects where the vision wasn’t there from the beginning, but it’s the best bet for the long term success of your software and for the consistency and sake of your business objectives.  Once the team has a shared vision and is consistently contributing to the business value, less controls will be required.  Your software will be continuously aligned with your business objectives as both the development and testing teams work together to ensure that your software is adhering to your critical business objectives.

Dan Douglas is a professional independent Software Consultant and an experienced and proven subject matter expert, decision maker, and leader in the area of Software Architecture. His professional experience represents over 12 years of architecting and developing highly successful large scale solutions.  Dan has been the architect lead on over 15 development projects.

Advertisements

Aligning Your Software Architecture With Your Key Business Objectives and Why Your Business Needs It

Abstract

Software development projects need to be aligned with your key business objectives.  The key business objectives that have the biggest cost to change need to be baked into your core architecture as early in the development process as possible.  Key business objectives relating to what the customer is paying for, such as, modularity of components (ex: paying for some components, but not others), performance and scalability, and data accessibility, are a few of a plethora of possible key business objectives that need to be baked into the core-architecture.  Failure to do so can double or triple your development costs, leading to months of refactoring and potential customer and revenue loss.  In agile environments, this article places importance on ensuring that significant core-architecture decisions are not made too late in the game as the myth of “no upfront architecture in agile” is debunked with real world examples where user stories, while aligned with functional requirements, were not aligned with the key business objectives nor the core-architecture.


Aligning Your Software Architecture With Your Key Business Objectives and Why Your Business Needs It

Software projects need an architecture – a core architecture, but defining that architecture and ensuring it meets the business and technology objectives is a bit more thought out than just doing “architecture”.  This article will focus on specific areas in creating a core-architecture as related to key business objectives.

Listing some design patterns and layers for your new system and presenting to the team might have some technical merit, but it isn’t enough if you want to ensure a well thought out and successful software system that meets your business objectives. Architecture isn’t just design patterns, layers, and code design. In fact that’s a very small part of it, if it even qualifies at all. Architecture is more about making very significant decisions that will help ensure the alignment of your completed software with your business and technology objectives.

Creating the right architecture requires business and domain knowledge, product and customer knowledge, research, communication, technological evaluations, technical agility, and expert experience in software development.  The decisions made here will shape the final software solution, so your core-architecture really represents these decisions that have been made during this architecture creation phase.

Now, separate this from day to day software development.  Teams make “architectural-like” decisions all the time when they determine how to implement specific functionality.  Ideas will get tossed around about how many layers of abstraction will be implemented, which design patterns to use, and so forth.  Although some would categorize this solely as design, in a general sense this is still architecture, but maybe not your core-architecture. You could certainly say that not all of these design decisions will have a significant impact on the software.

Grady Booch states that “All architecture is design, but not all design is architecture”.  He also states that “Architecture represents the significant design decisions that shape a system, where significant is measured by cost of change”

Your core-architecture needs to represent those significant decisions and most importantly of all, they need to be aligned with the business requirements.

Some examples of these significant decisions that become part of your core architecture include technology selection (development languages, frameworks, server platforms, etc), application deployment considerations, and technology considerations for key business objectives.

Some examples of architectural decisions to support technology considerations for key business objectives are as follows:

  • The system must work on both Oracle and SQL Server databases. -> An ORM tool is selected that will allow the application to easily swap DBMS vendors.  The ORM tool that has been selected has been evaluated against other tools and also meets the performance requirements. Using this tool will also lead to faster data layer development time over the alternative tools.
  • Customers can use and pay for a variety of modules that need to be plugged in at run time.  The modules need to work together to share information, but also operate independently if related modules are not available. -> Dependency Injection and Inversion of Control frameworks are evaluated against writing an internal DI/IoC framework, and an approach is selected to control how and when the components get loaded and used.  The components will adhere to a specific interface to ensure compatibility and modularity within the system.
  • The system has to be fast – a single implementation must handle a minimum of 100 customers and 5,000 simultaneous users with no performance degradation.  This performance must be maintained with over 100,000,000 database records in the core database tables, and will be measured by… etc… -> Performance considerations relating to how data is retrieved and stored, caching, scalability, and data load operations are reviewed.  Frameworks and architecture decisions related to this are reviewed and selected to ensure that performance considerations are baked into the core architecture. Coding standards and design patterns are put in place to ensure the UI is always responsive and UI data is loaded asynchronously to not freeze the user experience for any amount of time.
  • Customers are paying to be able to access their data in a standard way using 3rd party tools, and want to write automated scripts to retrieve and access this data. -> Business layer will be accessible to and have a corresponding REST API where all customer data will be accessible through this secure REST API.
  • We have licensed 3rd party vendors that pay a yearly fee to write and sell reports to our customers.  We need an open reporting tool with easy access to report data. -> A technology evaluation is completed and multiple technology vendors are evaluated.  The choice is made to use SQL Server Reporting Services to allow other vendors to easily create and sell reports to our customers. Reporting will not be done on the live database to minimize performance impact and mitigate the risk of rogue reports making their way into the reporting module.  A separate star-schema analytics server will be deployed that contains aggregated customer data that is suitable for very fast customer reporting with no impact to the live production system.

As part of this process you may need to include specific details about how this will be implemented, measured, how the testing team will test the performance, the specific technology in question and how it will be used, etc.  Your project will also need a shared vision of this architecture as the success of the project will depend on ensuring this architecture is maintained throughout the development process.

It should be clear to see that the core-architecture will represent these decisions that will have a big impact on the completed software.  Trying to change or implement these core-architecture decisions mid-way through the development process can have vigorous consequences and there will be a refactoring cost that could require months of development time. The worst case scenario is that it could be deemed to be virtually impossible without a major re-write if you have moved too far in another direction.

Could you imagine discovering close to a customer release that your software that was supposed to scale, doesn’t scale?  Or that specific functionality doesn’t work when more than a few users are accessing it at the same time?  It happens often when the software team was not mature enough to ensure that there was a core-architecture aligned with the key business objectives and ensure that the developed software code was aligned with the core-architecture.  These things must be accounted for and mitigated against by your core-architecture and the concepts of the architecture need to be baked into all development decisions made by the team throughout the development process.

If the problems mentioned above happen on your development project it is likely that your core-architecture was incorrect or never established, never followed, or effectively established too late in the development process. These problems alone could easily double or triple your total development costs, cause considerable delay to product shipping, and cost your organization customer and profit losses.

“What about Agile?  We don’t do up front architecture in Agile, we do it as needed during our iterations and we follow the last responsible moment design principle in doing so.”

I’m a proponent of agile methodologies, and I’ve seen the great benefits that effective agile teams can have on a development project, but one thing I’ve seen over and over again in many agile environments is the assumption that less (sometimes zero) time should be spent on the architecture up front. The principle itself is sound as removing most up-front design for functional requirements to be developed as slices of functionality throughout the iterations provides a benefit of not over complicating and over architecting code. However, the assumption that we throw away all up-front design is incorrect. Even on agile projects there are significant decisions relating to your core-architecture and key business requirements that need to be made before development begins and other significant decisions that need to be made as early in the development process as possible. These decisions and related architecture need to be reviewed, updated, and maintained throughout the iterative development process.

Agility is fantastic when dealing with functional requirements and the need to respond quickly to changing functional requirements. However, this agility needs to be kept separate from the up-front core-architectural decisions that need to be made that are aligned with the key business objectives and that will help ensure your software products success and conformance to these key business objectives.

Even in agile environments where YAGNI (You aren’t gonna need it) and “build now, refactor later” are the trends, refactoring code too late in the development process in order meet significant design decisions that align with the business objectives is going to cost you. As mentioned earlier, refactoring costs of months and months of development time is the norm for organizations that didn’t account for core-architecture decisions in their development – especially in agile environments where there is a misconception that these architecture decisions were supposed to be made as late in the game as possible.

In many agile implementations, these core-architecture decisions are made too late as they don’t typically relate to a single user story and they end up having a huge technical debt cost because there is a huge cost to change and cost to refactor existing code once the architecture decisions get made. Days, weeks, or months, can be lost to refactoring.

Another problem on some agile teams is what I call “the race to the finish”. Development teams race to satisfy the requirements of the user stories as quickly as possible to ensure they complete the stories within their iteration and to keep their velocity up (average user story points completed per sprint). And although the functional requirements of the user stories are solid and intact, thought isn’t always given to core-architecture decisions such as modularity, scalability, performance, etc, even though the core-architecture is aligned in parallel with the business objectives. Especially, if the core-architecture hasn’t been or has only loosely been defined, you can expect even less consideration, as the focus turns to completing the work as described instead of ensuring the development is in alignment with the core-architecture. To mitigate this in true agile form, agile teams and the product owners need to ensure that business objectives relating to the core architecture are part of the acceptance criteria for the user stories, and that they are thoroughly tested before the testing team can give the “Ok” on the completed user stories.

Depending on the agile team, how experienced they are, and how senior they are, the focus on how and when design decisions are made can also vary, and not every decision needs to be made during the inception of the project. Typically the biggest decisions with the biggest cost to change should be made as early in the software development process as possible. I would make sure 100% that you don’t lose sight of the “architecture” and the significant design decisions of that architecture and how these decisions are aligning with the key business requirements.

In summary, part of a successful architecture for a successful software product will require significant design decisions to be made up front.  The bigger the cost of change for the design, the sooner the decision needs to be made and implemented within the solution.  Not establishing an architecture can lead to months of wasted development time refactoring code that wasn’t originally aligned with key business objectives such as modularity, performance, and scalability objectives.  In agile environments, it’s especially important to ensure that thought is given to up-front architecture to mitigate the cost of refactoring in the future.  Establishing and maintaining a core-architecture which is tied in parallel with the key business objectives will go a long way to ensuring a successful product rollout.

Dan Douglas is a professional independent Software Consultant and an experienced and proven subject matter expert, decision maker, and leader in the area of Software Architecture. His professional experience represents over 12 years of architecting and developing highly successful large scale solutions.  Dan has been the architect lead on over 15 development projects.

Perhaps you want to learn more about how to establish the architecture, how to present and collaborate with your development team to finalize the architecture and create a shared vision, how to ensure consistency across the architecture, how to bake in non-business requirements such as logging, cross-cutting, and other technical concerns into the architecture.

Or maybe your architecture is established, and your team has done a damn fine job of ensuring the architecture is solid and that it will meet the key business and technology objectives.  How do you maintain this? How do you cope with architecture change when it is warranted or when the business changes?

Stay tuned, as I am writing a series of articles on these topics which will be available in the future.

Technical Debt In Software – The Fine Line Between No Architecture and Over Architecture

As I am writing this, I am sitting on my condo’s terrace in downtown Toronto, baking in the sun while I take in the noise of the city and cars and people down below at the street level.  There is a helicopter that seems to be circling the downtown core for quite a while now.  It’s freaking hot today and I could probably use some water.  Be right back.

At the time of writing...

At the time of writing...

(1 minute later…)

Ok – water has been acquired….. I also grabbed a Kilkenny and poured it into my Kilkenny glass (Kilkenny requires the right glass to be enjoyed properly)

It’s been a while since I’ve posted, and career wise a lot of big and exciting changes have been made.  I made the move to independent consulting, and I am enjoying it big time!  Now my efforts are shifting from helping one organization develop bleeding edge scalable systems to helping many organizations with development, architecture, minimizing technical debt, and team building.

Along the lines of the type of work I’m focusing on, I want to write a bit about bad architecture, over architecture, and technical debt.

Ok, so I’m going to talk about the benefits of a “value added approach” to software.  I’ve seen a lot of systems in my day – ranging from poorly architected apps that deliver high amounts of business value to over architected systems that, instead of delivering their expected business value, became a total utter failure for a multitude of reasons… and everything between.

To the business, a successful system is typically one that delivers on it’s promises on value to the business.  The negative impact of technical debt is not always seen by the business teams and is sometimes seen, albeit indirectly, as “necessary”.  So, in these scenarios – why is it necessary?  is it job security for the dev team? Lack of training?  Lack of standards?  There could be a plethora of reasons, but in the end the technical debt introduced by these systems is high and could cost the organization millions of dollars.

High business value systems and mission critical systems can suffer from an endless amount of technical debt due to lack of design standards and architecture.  This technical debt is not always apparent to the senior business teams. The business loves the system however, but they sure don’t understand why it takes such a long time to add new features or track down bugs.  The business leaders at the top see the system as great too, but the fact that it’s overly fragile, requires daily maintenance and an overly large team just to support it seems somewhat necessary – plus it’s “just the way it is”, right?

The real deal here is that technical debt, and overly large dev support teams are just not necessary at all.  The right people, training, technical skills, system architecture, and business leaders have the potential to create the right systems and find that fine line between a proper development architecture and business value.

Yet, there is another extreme…..Over architecting and big ego’s….

People have ego’s… Fact of life .. When architecting a system, I’ve seen many software architects with an ego.  Their system is great, using all the right design patterns.. Look at how cool my undo system is with the command pattern implementation I came up with.  Watch how I can add one entry to the configuration file and all the sudden the entire behavior and business logic of the app can be changed… Our customers can now create their own custom modules using my handy dependency injection techniques and augment the app with their own fancy things they want to do…..  Pretty sweet eh?

I agree, ok it’s pretty sweet (and fun to work on) and I’ve seen and created my fair share of ‘coolness’ in my systems.  There are always business cases for these types of things and having the right technical team and abilities to implement them properly is key.  The problem is over architecting when they aren’t needed or for the dreaded reason ‘just in case in the future’.  The fact is, this can sometimes delay shipping the product and complicate the development.  There is typically a big disconnect between the development team and the business value in these scenarios.  You end up having a technical team who is more focused on architecting than on providing business value.  Focus on what’s needed, and if it is, and there is a case for having it… Build away and have fun!

Both of these scenarios above can lead to long term technical debt.  The trick in software is building a team who can leave their ego’s out of it, share knowledge of the system and the business value proposition, and focus on what’s important for the long term success of the project while minimizing technical debt.  Doing this has the potential to create phenomenal systems that are well architected, bugs become easy to track down, the system can grow organically the way it needs to, the size of the development team can remain minimal, and the business is happy knowing that maintenance costs are reduced and new features can be added to the system quickly.  There is no more fragility – the system becomes clear and small changes are less likely to have unexpected bad consequences.

There is a fine line between both of these scenarios and it requires practice and discipline to build the right team who can achieve and continually create well architected solutions that maximize business value and eliminate technical debt.  It can be done and can be done very successfully.  To truly master this as a software/business team, it needs to be instilled as part of the culture of the team.  The right leaders and people are very important in truly creating world class software solutions.