Midwest IT Survival

The challenge to integrate Architecture and Agile

Recently on Simon Brown’s blog “Coding the Architecture”, Simon posed are interesting challenge of which this article provides my response. I enjoy Simon’s blog for he presents very well thought out articles on the challenging overlap between Agile software development and the value of architectural principles in providing software solutions. As those that frequent my blog have seen, I’ve written about the challenges of integrating pure Agile development methodologies into non-Agile organizations such as legal services, manufacturing as well as financial services. I’ve approached the topic of work estimation here which is a frequent challenge in non-agile organizations that make project and resource prioritization decisions based loosely on the estimated work effort. I’ve tried to successfully blend Agile story and blacklog project management with the preponderance for waterfall-esk program management here. Based on the comments to that article I didn’t succeed in my goal. I’ve even broached the Agile versus architecture question before here but the focus was on the Agile reducing over architect-ed solutions rather than taking Agile versus architecture head on. I’ve even discussed budget challenges before.

Simon’s challenge: Given a four month window to deliver a new software project in an Agile shop, how would one go about doing so given some interesting constraints? The below graphic is from his blog post that captures the unique constraints for this challenge:

Based on my multiple years in the financial services industry working for more than one bank, I feel the need to outline some immediate assumptions that pop into my mind that are tainted by my direct banking experience:

1. Design Firms – Simon’s challenge involves the bank undergoing a new branding effort. In my experience, a bank’s internal marketing department partners heavily with an outside marketing firm for larger banding initiatives. Additionally, new branding efforts usually involve getting that new brand message out through a variety of channels.
2. Middle-ware/SOA capability – Given the overall growing maturity in the service oriented architecture space and the relative IT budgets of most financial institutions, I am going to assume there is some middle-ware, messaging and/or enterprise service bus-like capability to leverage.
3. Content Management System/CMS – Given the explanation of the current state online banking system, I am going to assume the bank does not currently have a mature content management platform which can be extended to provide online banking functionality.
4. Backoffice – Also, I am assuming that the bank currently has a backoffice transaction system that currently performs the bank’s core bank product transaction needs. For the sake of this exercise, a “legacy” mainframe set of applications are in place that run the core bank transactional services that has some capability to expose those transactions to the middle-ware/SOA platform.
5. Build versus Buy – A classic challenge for banks has been the decision to put the focus on building in-house capabilities versus buying mature vendor solutions and integrating those into the technology portfolio. Depending on the financial health of the bank itself as well as the predilection of the CIO, the build versus buy pendulum can swing to either extreme. Add the frequent turn over of senior IT leadership and this can be a real challenge for enterprise architectural patterns. For the sake of this exercise, the assumption is that the bank is willing to invest in the middle-ware transactional capabilities in-house as well as the supporting online banking application user experience but prefers to purchase a new content management solution to host the UI.
6. Big Bang versus Functional Sequence based rollout – I am also assuming product management is risk adverse and open to sequencing in the new functionality rather than demanding a big bang approach. More on the sequencing later in this post.

Given the architecturally significant changes requested in 4 months, specifically the read-only to real-time transactions and the complete hosting re-platform need, I would convince management that it would be infeasible to deliver everything in 4 months. Yes, the project management side of my brain says don’t over commit given the technical complexity involved and the linkage to a significant customer impacting change.

Thus convinced, I would then break the effort into multiple parallel technical efforts:

Re-brand current site = I would spin up an Agile team including the online banking product management/manager to focus on re-branding the current site between now and the 4 month end date. The team’s priority would be:

1. Integrate the new branding into the current site working heavily with the outside design firm
2. Make changes, assuming no conflict with #1 above, to improve W3C compliance

Create re-useable transaction services = I would spin up another Agile team that has the mission of exposing the backoffice transactional services as “enterprise” consumable web services through the middle-ware platform. The stories would mix both the technical needs such as “Develop a way for the middle-ware components to authenticate to the mainframe components” as well as business capabilities “have the ability to transfer unlimited funds, subject to current available account balances, from one customer owned account to another account owned by the same customer”. The prioritization of the stories would be as follows:

1. Architecturally needed stories such as component to component authentication, high availability, etc. Basically, everything needed to support #2 below.
2. Transactions needed in the new online banking product based on the roll-out schedule.

One of the challenges for this team is the likelihood that the mainframe platform participants are not familiar with Agile practices. Thus, extra coaching is probably a must for this team.

Purchase Content Management Product = I would task the architecture group to work with procurement to select a vendor’s content management product that will become the framework for the new transactional customer user experience. The classic steps would be undertaken: RFI/RFP, product evaluation, proof of concept, vendor pricing negotiations and ultimately product selection. This needs to be completed in enough time for the next team to be able to deliver some functionality in the new CMS product.

New UI Development = Finally, I would spin up another Agile team to map out and build the new transactional UI. Their first challenge would be to reverse engineer the customer authentication system to determine how the system authenticates users and then links them to their accounts and products with the bank. They would need to work closely with the re-branding Agile team to determine the most seamless way to add navigation to the current site to link over to the new site for the real-time transactional capabilities. Additionally, this team would also have to work with the middle-ware team in order to understand the re-usable services being created and how to consume them. Lastly, this team would also have to keep tabs on the content management team to follow the procurement process and start to understand the framework capabilities of the new CMS product and how to link those to the navigational needs and transactional needs of the final online banking product.

This last team would have a heavy architectural staffing up front and transition to a more development heavy staffing towards the end for delivery. In a perfect world, the project management resources from this team would be the most senior in the organization and consistently work with this team as the skill-set transition occurs.

The below Gantt chart reflects the multiple parallel efforts and when they ultimately need to come together to deliver the final product.

The functional roll-outs would be linked to the customer marketing campaign in order to provide sequential customer communication on new product features as well as permit the new technology “kinks” to be addressed. Also, this allows call centers to ramp up temporary staff to address the increased call volume due to the changes impacting customers familiar with the prior system.

The actual sequence would need to be agreed upon by IT, product and call center management, but potentially the following sequence might fly:

1. Launch the old site with the new branding.
2. After a brief window of time to let the call center call volume return to normal levels, replace the link to current “account balance” information in the read-only system to the same “page” in the new system that requests that same balance through the new middle-ware services. The customer experience would be marginally impacted in that instead of seeing yesterday’s balance details they would be seeing their current account balances. Lastly, address any negative performance or functional kinks.
3. Add account transfer links to the old site that take users to the new site to perform the transactions [first real new customer functionality]
4. Same call center pause.
5. Repeat the sequence for additional transactions such as external account-to-account, bill payment, bill presentment, etc.
6. When all transactions are being handled both UI and middle-ware by the new site and the old site is essentially a set of static pages with links, cut the login process over to the new new site and disable the old site when web logs show no stray bookmarked pages are being accessed. Porting over the authentication system was part of the deliverables of the third Agile team.

Sure, this sounds like a full program of interdependent project activities, but from my experience, banks want to provide highly available and reliable services to their online customers which tend to be their highest revenue generating customers as well. Thus, all of the incremental functional roll-outs which come with the additional cost of testing and program management overhead deliver on the risk adverse, reliable and measured change.

I look forward to all the alternative approaches to this challenges Simon has presented. How far am I from the response norm? Any fundamental flaws?

Organization Silos Impeding your Enterprise Architecture?

There are countless sources extolling the benefits of a strong enterprise architecture strategy. The experts all agree on an effective enterprise architecture and even more so the larger the organization’s consumption of IT services. But recently, I’ve been reminded that the IT organizational structure, especially the structure aligned to providing new projects and new technical solutions has a dramatic impact on the ability for an organization to realize the benefits of established enterprise architecture.

In short, the more the IT new project and solution delivery organization is directly aligned with the individual business unit or group it supports, the more likely a spaghetti architecture will be the result. To help outline this point, below is a quick graphic of a sample organization chart that shows the two extremes:

Extreme A = Business group/unit functionally aligned:

Extreme B = Common function/service aligned:

The “extreme A” example is functioning with each vertical group acting as silo. Each silo is held accountable for delivering solutions to their business unit. In turn, each business unit will drive their IT silo to meet their needs exclusively. There is no inherent need to collaborate with their peers supporting other business units even if there are “enterprise architecture goals”. In my observation, collaboration might actually be viewed as a distraction to getting work done. There is essentially no organizational driver to force standard solutions and re-use of technology assets.

Sure, an architect in one group might have a strong personal relationship with an architect in another group, but unless they are trying to share ideas that happen to be at relatively the same level of “maturity” for each silo’s needs, they will unlikely be able to produce a common, re-useable technology used across both silos. Architect A attempts to work out a common solution with Architect B but suddenly Architect B’s project gets accelerated. Suddenly Architect B has to quick assemble a slimmed down solution that can’t be dependent on Architect A’s requirements or time-line. And sure, everyone might meet and agree to “retrofit” Architect B’s project with a more standardized solution with Architect A’s project in a later phase/iteration, but it would take an amazing level of cross-silo organizational governance to make sure that happens. If that standardization would in any way delay Architect A’s silo from delivering from the business, the standardization most likely gets pushed far and far out until no one remembers or even exists that remembers why the architecture alignment was needed in the first place.

Other potentially non-technology specific negative byproducts can evolve from this structure:

• Strong versus weak silos

One silo maybe more effective at deploying more current technology by the very nature of the business unit’s needs. As an example, consider a customer product or engineer unit compared to say, finance or one that uses in-house developed technology versus another with outsourced/SaaS technology. This creates a problem of talented architects and developers/others actively seek out roles within the perceived “strong silo” creating an even stronger silo compared to the other silos.

• Overall increased IT cost of ownership

As each silo is standing up technology to meet the business unit’s needs, they are all solving basically a significant number of the same problems with different solutions. Those solutions come with reoccurring maintenance, product end of life and all the traditional support and vendor management overheard. (I’ve written extensively about vendor management in past articles.) Then, to drive up costs even further, as some business need to see a “single view of the customer” or some other cross silo business challenge pops up, the cost to map all the data across the disparate systems is exceedingly high. In addition, not only does the mapping need to occur, but the technical (and probably some business) workflow needs to be altered to continue to keep all the data in sync. Someone suggests we need “Master Data Management” and now you have the cost and complexity of implementing a system to manage the data across all your systems.

The “extreme B” example brings the solution needs from each separate business unit/group into a common functionally aligned project delivery discipline. The goal is to leverage best practices/success stories from working with one business unit/group across to the other business units/groups via the common management structure by discipline. More specifically, the goals and objectives of each IT service discipline or function can be aligned to efficiencies and re-use within the specific discipline. Project Managers ensure they have common mechanisms to track and report on the project process. Business Analysts make sure they have a common framework for capturing requirements. Architects, develop unique frameworks for common requirements across all projects. Developers follow a consistent coding standard and reuse objects for data access, error reporting, and instrumentation, etc. The same goes for the other disciplines. Each discipline can focus on efficiencies aligned within their area of responsibility. And since each discipline has a somewhat singular work input and output for all business units, there is complete enterprise visibility to what is needed per each discipline. Thus, ultimately, the architecture team is looking across the entire enterprise at all the requirements and multi-year plans and can be charged with common frameworks for essential IT needs:

• Authentication
• Authorization
• Entitlement Management
• Business Rules/Workflow Management
• Business Event Management
• Reporting
• Data Structure, Storage, Management, Retention, Archiving
• Auditing and Compliance
• Capacity Planning
• Disaster Recovery

… and probably a bunch more that don’t immediately come to mind!

In conclusion, a strong enterprise architecture strategy gets a significant boost from a discipline aligned organizational structure rather than a business unit/group aligned structure.

Anyone have any thoughts to support and/or contradict my thoughts here?