Midwest IT Survival

Over Architected?

First off, I am by no means an “Agile” (nor even “agile” with a small “a”) software development expert.  There are many more individuals out there that are way more experienced and in a much better position to speak authoritatively about the Agile methodology and its associated principles on driving efficient and effective software development.  A few blogs where I consistently find great Agile perspectives are included at the end of this article.  But as I’ve participated in Agile and agile-like development efforts over the years, I’m finding an interesting pattern.  Agile and agile-like approaches have a positive by product of reducing the occurrence of over architect-ed software solutions. Over architect-ed solutions put stress on the delivery of a software application project as well as drive up the cost of software development and maintenance, in general, disproportionately to the business value produced.

As an example, a sample development effort starts out with:

Product: “We need a super widget in the product by next release, can we have that?”

Project: “We are going to need detailed requirements for the super widget in order to start developing it.”

Product: “Oh, it needs to be able to interface with the dry cleaners to know when it is time to pick up the laundry as well as make coffee for the customer before they get out of bed in the morning.  Basically the same features as our competitor’s recent release, but with these additional benefits. <Or some other description that is actionable at a high level, but lacks the detailed requirements needed to feed a development team with actionable development tasks>”

Development: “Ok, we’ll get started since we don’t have much time before the code freeze for the next release.”

… Time goes by …

A project status meeting occurs sometime in the future.

Product: “So, where are we with that super widget?”

Development: “We have the basic framework setup but it isn’t going to have all the features working by the next release.”

Product: “But I thought …”

Development: “But you said …”

Project: “What happened?  How come we are X days from the code freeze and we don’t have a viable solution?  I thought …”

Now sure, this isn’t the most perfect example of capturing how the requirements drift between stakeholders leads into an over architect-ed solution, but hopefully you get the scenario.  Or possibly another example would be when a solution is developed and released into the production environment.  Extending the example above, weeks later, when enhancements, tweaks or feature extensions are requested, a tense conversation occurs around:

Product: “But I thought the super widget would do X?  How come I am hearing it will take 30 hours of development to get X?  The testing cycle is already elongated due to the complexity of the super widget thus I thought it included X?”

Development: “But we said that the framework would support X, but we never said X would actually function without more development!  We developed the super widget to do W, Y and Z but only stubbed out X.”

Project: “But according to the requirements, it says X should be …”

And yes, an argument can be made that if:

• more effective requirements gathering occurred
• more effective project management captures more depth of what would be developed and available when
• more effective product management defines a more exhaustive product feature road map that more clearly outlines what would be available and not available when, feature-wise

… These problems wouldn’t have occurred.  But the nature of an agile-like approach puts a tighter focus on all the stakeholders:

Product: They can share the “overall vision” of what they ultimately desire the product to do, but they are forced to consider what they really need within the shorter duration of the agile-like release schedule.  Thus, product walks away with a clearer picture of what they are really getting in the next release.

Development: They get the benefit of the product’s “overall vision”, yet, they get to quickly dive into the critical features and start the dialog of how long different feature components will take to develop.  Thus, development knows exactly what they need to do now for the next release, yet they benefit from knowing where this product feature is going in the future.

Project: As long as they keep the product and development stakeholders talking about granularly defining what needs to really be built by when, the project management function has much greater clarity into what is going on and what details to track.

From what I am observing, all of the above create a stakeholder forum of information sharing that reduces the likelihood that an over architect-ed application will get developed. Most importantly, instead of leaving the feature set open and vague enough to allow a creative and motivated development team to start building and building and building only to re-surface with a highly complex solution to a loosely defined problem or need, it brings more cohesion between what is really needed first.  Once the “first” has been built, the “seconds” and “thirds” get built inline with the product roadmap.

In researching this theory, I wasn’t able to find any articles that linked agile-like development efforts to a direct reduction in software over architect-ing.  This article entitled “Agile Architecture: Strategies for Scaling Agile Development” had some interesting content on baking architecture into an agile-like effort.

Anyone else have any direct experience in agile-like compared to waterfall-like development efforts yielding less application over architecture?  Can anyone share any links to good web articles on this topic?

Agile related blogs I follow:

David’s Software Development Survival Guide

http://softwaresurvival.blogspot.com/

NOOP.NL

http://www.noop.nl/

Software Project Management

http://blog.brodzinski.com/

fragile

http://fragile.org.uk/

Regular Geek

http://regulargeek.com/

Critical Results

http://criticalresults.com/

Know when to call in some help during an outage

Anyone that works in the Information Technology field knows that production technology systems, from time to time, will have problems. From a functional defect that has everyone scratching their heads as to how it wasn’t discovered by seemingly endless rounds of QA to full blown hardware failures that take down entire suites of applications, no matter how much is invested in “highly available” and “redundant” technologies, failures are bound to occur. For IT Managers and IT Engineers, how one handles these failures from inception through service restoration and finally root cause analysis is critical. Sure, the priority is to restore full service availability as soon as possible. But, if you neglect some key technical support quality attributes in the process, which I’ll highlight in this series of articles, you may find you both succeeded and failed in restoring service at the same time. Succeeded and failed at the same time you wonder? Please read on and I will attempt to shed some light on this success with failure construct and considerations on how to avoid the failure “pitfalls”.

Pitfall = Challenges in an Extended Outage

So, you’ve bought into the need to be response based on a previous article touting the benefits to you (being viewed as a leader and raise and bonus positives) and your organization (calmly restore production IT services to normal working order). You’ve communicated in a personal style with incremental positive facts and indicated at what timing points you will be updating the stakeholders on your progress as indicated in the previous article. If the problem can be easily identified and corrected quickly including a rather direct way to explain why it happened, pat yourself on the back for a job well done. Now get ready for the after math of re-explaining what happened a hand full of times over and possibly participate in some post issue shoring up of the technology (see root cause analysis considerations posted here previously). But what happens when the status reporting is going on longer and longer and you can tell that the natives are getting restless as they are starting to grow concerned at the length of the outage and at the lack of a clear “it will be fixed in 5 minutes” status report? When an outage becomes an extended outage, time to ratchet up the communication plan and bring in some help.

Problem isn’t Obviously Fixable in Short Order? Get Help

Most likely, as time is going by, more people are aware of the outage and thus the list of stakeholders is growing larger. Also, the likelihood those stakeholders are senior technical people offering to give you a hand is slim and none … and slim left town as the saying goes. I would venture to say that the stakeholders are a growing list of non-technical people that are impacted in some way by the production situation continuing to be a problem. More and more managers on the operations and product side of the service are getting engaged as possible customer complaints are mounting or call center call volumes are reaching levels of concern. There maybe more people engaged to discuss what to do if the outage continues and an alternative, possibly more manual means is needed to meet customer SLAs. By the way, manual usually means more work done by people, hence more people getting engaged to see if they have to bring in even more people to ensure the alternative service delivery option has the right, skilled and trained staff. Company marketing resources could be engaged to offer advice on how best to let customers know the service is having a greater than normal duration outage and what the company plans to do to service their needs. I am not trying to paint a picture of doom and gloom for the primarily technical audience for this article. I know the technical mind wants to have all the people just stop talking so the real work of fixing the technology can take place. But on the business side of the technology in trouble, there are company stakeholders and customers of some form or another that are materially impacted in some way by having the usually highly reliable technology fail to function correctly.

Thus, as time goes by, your incrementally positive but not “it’s fixed” communications aren’t enough to appease the masses. You are either going to have to spend more and more time explaining to new people joining the situation what happened when, what has been ruled out, what is next to investigate, etc. or risk becoming non-communicative in order get some focused time to fix things, thus putting all your hard work at risk as outlined in this previous article. It is time to ask for some help.

Hopefully you have already engaged your management to keep them apprised of the situation as suggested in this previous series of articles. Thus, you may already be getting asked if you need help because you have informed your management and thus they are starting to ask the “hey, you are doing a good job, but can we help?” type of questions.

Ask for and accept help

I can’t stress it enough: avoid the notion that the fix is “just around the corner and if I only spend 10 more minutes researching …”. Ask for and accept help. To start, get someone engaged to be the status communicator so you have less distractions and more time to dig into the problem. The status communicator needs to have level of competence in the following skill areas:

1. Enough of a technical background to take technical status bits from you and quickly understand what you are saying without a 5 hour white-board deep dive session.

2. Ability to communicate in “business speak” not “techno-speak”.

3. Enough understanding of the players involved organizational chart-wise to know how and when to communicate with stakeholders and when to recognize the VP of Product is looking for status and it is time to get your VP peer manager involved.

Your manager is in the best position to act in this capacity if they aren’t already doing so. As managers, you stand to lose huge management credibility and leadership points of you just sit on the sidelines and hope the problem goes away or you are somehow hoping for plausible deny-ability to relieve you of your responsibility in this situation. Roll up your sleeves and get engaged. Start sharing what is going on in a polite but authoritative tone to build confidence and most importantly, buy more time for your engineers to dig in and figure out what is going wrong and fix it.  This previous series of articles offers additional tips.

In summary, as the outage is dragging on, be mindful that not everyone involved has the priority of discovering the coveted technical root cause. For engineers, as an extended outage is building, don’t keep trying to take on the rolls of technical investigator and communications expert. Get help. Managers, get involved and start shielding your engineers from the constant barrage of status requests and allow them more focused attention on digging in and finding out what is really going on and get it fixed.

We’ve extended the need for responsiveness to reports of production support problems to include an initial take on the art of creating an effective status communication approach as well as when to admit your need help and get your manager and/or team lead involved directly. Look for additional articles to identify more technical support pitfalls and steps to take to avoid them.

Respond and forget, right?

Anyone that works in the Information Technology field knows that production technology systems, from time to time, will have problems.  From a functional defect that has everyone scratching their heads as to how it wasn’t discovered by seemingly endless rounds of QA to full blown hardware failures that take down entire suites of applications, no matter how much is invested in “highly available” and “redundant” technologies, failures are bound to occur.  For IT Managers and IT Engineers, how one handles these failures from inception through service restoration and finally root cause analysis is critical.  Sure, the priority is to restore full service availability as soon as possible.  But, if you neglect some key technical support quality attributes in the process, which I’ll highlight in this series of articles, you may find you both succeeded and failed in restoring service at the same time.  Succeeded and failed at the same time you wonder?  Please read on and I will attempt to shed some light on this success with failure construct and considerations on how to avoid the failure “pitfalls”.

Pitfall = Providing Status

So, you’ve bought into the need to be responsive based on the previous article touting the benefits to you (being viewed as a leader and raise and bonus positives which are always good) and to your organization (calmly restoring production IT services to normal working order).  So, all you have to do is “respond” by sending an email right away, jumping on a conference line quickly or changing a status in a production trouble ticket tracking system promptly and you are done, right?  You can now disappear into the depths of your logs files and your performance counters and your packet traces only to resurface when you have found the real cause of the problem, right?  Never under estimate the extent to which people, lacking timely information people, will panic.

To help illustrate, we can extend the example from the responsiveness article of needing a plumber to call you back quickly to address the hot water heater that is pouring water all over your basement floor and not delivering any hot water to any faucet in your house.  Consider that a plumber does call you back promptly to indicate they are able to start looking into your leaking hot water tank right away.  But after that responsive call back, time keeps ticking by without any indication if your tank can be fixed or needs to be replaced or is about to explode and flood your basement in the process.

Note: Yes, you can walk down into the basement to physically see the plumber’s progress or lack there of, but pretend you can’t easily do that to allow this extended plumbing example to help frame the context for this article.  Let’s say you left your home for work right after you confirmed the plumber was engaged to fix your problem.

So, without any further status from the plumber besides his or her initial: “Yes, I look into your hot water tank problem right away”, how do you know what is going on?  The plumber could be minutes away from turning off the water main to stop the river forming in your basement followed quickly by unloading a delivery truck approaching your house with a brand new hot water heater or sitting down on the couch to catch a baseball game on TV completely ignoring your water dilemma.  Thus, how do you know what is going on?  You don’t, unless you are physically watching the plumber’s every move or the plumber is providing frequent status as to what is going on with your hot water crisis.

Frequently provide status

So, how does one keep the panic to a minimum once initially responding to the production issue?  Reduce panic by frequently communicating status of what is going on in the troubleshooting process.  This sounds simple enough, just keep everyone informed:

• “I just VPNed into the network”
• “I am pulling up a terminal session with the server now.”
• “I am typing my user name.”
• “I am typing my password.”
• “Ooops, wrong password, trying again.”
• “I am now at a command shell …”

Obviously, that is going too far into the over communication side of the status equation.  What you are trying to find is the artful balance in the level of detail and frequency to share status.  As in all things technological, there is no silver bullet, no industry established check list and no “do this and it will work for every situation written on a stone tablet somewhere to implement with guaranteed success.  One has to put some energy into looking for clues as to what is going to work best in the given situation and then constantly monitor the results of the your communication approach to tweak as necessary.

But this sure seems like a lot of work that doesn’t get directly at fixing the true technical problem?  Correct.  As I mentioned previously, you can dedicate all your efforts to fixing the problem as quickly as possible, but be prepared for the consequences of various negative backlashes surrounding non-technical and peer management’s frustration of being left in the dark for who knows how long starting from problem occurrence and ending at problem resolution.  Plus, you can safely anticipate the root cause analysis aftermath being painful and extended due to this lack of communication frustration you have helped create.  Thus, I am arguing the time invested up front in an effective communication approach will pay large dividends in avoiding post service restoration negativity and an elongated investment in root cause analysis malaise.

Art of an Effective Status Communication Approach

So how does one determine a successful status communication approach?  First, suspend your technical or engineering brain that puts speedy problem resolution as the highest priority in any production outage situation.  Recall that once you put aside the technology, people are involved in the production outage.  Harkin back to the plumbing crisis example above, if you are at work wondering how much your water bill is going to be as your basement floods, what would be your reaction to getting call or a note from your plumbing saying:

“Hey, this is Bob the plumber, just wanted to let you know I stopped the geyser erupting in your basement.  A replacement water tank is on a delivery truck and should be arriving at your house within the hour.  I’ll let you know when it gets here and what the next steps are in about an hour or so.”

Imagine the feeling of relief at getting such an update at work.  Now, carry those feelings of relief over to the other people involved in the production outage situation.  They are fretting over lost revenues or having to explain to their management what happened, why and what is going to be put in place so it never happens again with absolutely no clue at this moment on answers to any of those questions at the moment.

Can you make everyone relax and go about their day with a smile with a few simple sentences on what is going on?  Not a chance, but you can help keep the people involved more calm and less likely to break out in irrationality by providing indications of where you are in the troubleshooting process.

Consider this revision to the step by step over communication example from above:

“Everyone, this is Bob from systems support.  I was able to get online and successfully access the production server that is hosting the application that is involved in the production outage.  This is a good sign in that we able to start debugging immediately without any infrastructure barriers at this point.  I will now start investigating the error logs that should give some further technical direction on what is going on.  I will let everyone know what I discover in 15 minutes from now.”

Similar to the status update from your plumber, there are key elements in this status message that address the human side of the outage:

1. Saying your name

Saying your name seems over simplistic, but giving your name instead of hiding behind the anonymity of an artificial company group such as “systems support” makes a small but important personal connection to all of the people involved that possess likelihood to panic at a moments notice.  This is similar logic as to why people prefer talking to a human rather than interacting with an automated “push or say 1 and then entering your 45 digit account number” system when calling to resolve an incorrect cell phone, gas or electric bill.

1. Providing legitimate positive news, even if it is somewhat insignificant to correcting the real problem

Again, seems simplistic, but by indicating you were able to get online and get into some level of technology to begin troubleshooting, it helps to give additional confidence to the non-technical individuals participating in the outage that some potential barriers to real problem resolution have been crossed.  Look for opportunities to share facts that narrow the problem down, even if they only narrow the problem down ever so slightly.  The increased feeling of progress that the elements of narrowing down the problem create help to continue to enforce feelings of increasing control over a seemingly out of control situation to the non-technical people involved.  Again, you are looking for balance.  “I successfully typed my password” does no invoke that much confidence.  Thus look for real progress facts that can be shared that focus on narrowing the problem scope rather than just facts for the sake of facts.  Lastly, I chose the word “facts” specifically.  Make sure you communicate facts and not speculation at this early problem engagement level.  I’ll cover some suggestions on how to share speculation in another article.

1. Indicate when the next status communication will occur

Giving people an indication of when they can anticipate an update on what is going on or what you are doing provides two significant benefits.  The first is it allows everyone participating in the outage who is not directly involved in restoring service the ability to relax just a bit and prepare for the when they need to be engaged next.  They know there is nothing tactically they can really do to solve the immediate problem.  They know they are effectively 100% dependant on technical resources to do the real work of finding the problem and fixing it.  They desperately want to hear: “the problem is X and I’ve fixed it.”  But since you nor anyone else is at that point in the troubleshooting process, a time in the not too distant future where such a phrase might be uttered is the next best thing.

The second is it gives you much needed breathing room.  Instead of hearing “Is it fixed yet? How about now?  Now?  Maybe now?” every couple of minutes, you’ve clearly set the expectation that you need some uninterrupted time to do some digging in order to provide anything valuable as far as investigative analysis.  Thus, you now have some time to completely disengage from the noise associated with the problem and roll up your sleeves and immerse yourself in performance and log data to try and figure out what is going wrong with the technology.

Communicating Status – Approach in Summary

1. Use your name and thus communicate in a more personal tone to increase confidence in non-technical participants … avoiding the opposite completely impersonal tone of “tech resource number 12”
2. Provide positive news to further increase confidence and reduce the panic building in others with facts (not opinions), even if those facts are small troubleshooting milestones and not grandiose “ah ha!” findings.  Make sure to balance the too small “I pressed enter and …” type facts.
3. Indicate you need time to dig deeper and set the timing expectations of when others can await the next element of status from you to buy uninterrupted investigation time and allow others to put off panicking for a period of time.

We’ve extended the need for responsiveness to reports of production support problems to include an initial take on the art of creating an effective status communication approach.  Look for additional articles to identify more technical support pitfalls and steps to take to avoid them.

Respond ASAP!

Anyone that works in the Information Technology field knows that production technology systems, from time to time, will have problems.  From a functional defect that has everyone scratching their heads as to how it wasn’t discovered by seemingly endless rounds of QA* to full blown hardware failures that take down entire suites of applications, no matter how much is invested in “highly available” and “redundant” technologies, failures are bound to occur.  For IT Managers and IT Engineers, how one handles these failures from inception through service restoration and finally root cause analysis is critical.  Sure, the priority is to restore full service availability as soon as possible.  But, if you neglect some key technical support quality attributes in the process, which I’ll highlight in this series of articles, you may find you both succeeded and failed in restoring service at the same time.  Succeeded and failed at the same time you wonder?  Please read on and I will attempt to shed some light on this success with failure construct and considerations on how to avoid the failure “pitfalls”.

The First Pitfall = Responsiveness

The very first pitfall in providing any level of technical support to a production system is to fail to be responsive.  Imagine for a minute that you are a home owner and you don’t know the first thing about plumbing (and maybe you don’t know much about plumbing).  You turn on the facet expecting to feel some soothing hot water spray over your hands and yet, the water remains freezing cold.  You wait the typical number of seconds by when you would expect to at least sense the water turning from freezing cold to mildly tolerable; yet it isn’t happening.  You turn off the facet in disgust and trudge down to the basement to confront the source of pleasant, warm water: the hot water heater.  To your surprise, the hot water heater is where you expected it to be physically, but what you didn’t expect to discover is a steady stream of water pouring from underneath the unit and running a short length to your basement floor drain.  Since you don’t know the first thing about plumbing, your immediate and only thought is to call a plumber to come over as soon as possible.

Assume you have contact information for three plumbers in the area that you either have had satisfactory work completed prior or had reliable information from friends and neighbors that indicated they were prompt and reliable.  You hastily dial each plumber only to be greeted by pleasant yet unhelpful answering machines:

“… your call is very important to us.  Please leave a message and someone will be with you shortly <beep>”

You leave a hasty yet detailed message about the unnatural spring that has sprung forth from your hot water heater in the basement.  You provide a home phone, cell phone and your spouse/roommate/significant other’s contact information to ensure there are multiple ways your urgently needed plumber can contact you.

Minutes go by.

Hours go by.

Hours turn into days …

Okay, you get the picture.  What you and your lack of hot water disaster need is some initial response from a plumber.  Without any response of any kind, one can only plunge deeper and deeper into panic that you will forever by taking cold showers and personally draining the local fresh water lake via your leaking hot water heater.

I believe this analogy translates well to the notion of being responsive to production support issues.  Replace the panicked home owner with the leaking hot water heater and no response from any plumbers with an IT manager that is responsible for a technology service that is aware the technology is broke, but has no clue if/when it will be attended too by an engineer and you can image the panic the IT manager is feeling.

Thus, to not respond to a production support call or page promptly when the technology you are responsible for could be broken is liken to a previously reliable plumber never even returning your call about your broken hot water heater.

To Avoid the First Pitfall = Responsiveness = Respond in a Timely Fashion

Yes, this may seem ever so simple, but by responding to a call or page related to a production support issue in the expected manner will go an exceedingly long way to avoid the first pitfall and put the IT manager’s mind at ease.  Respond in the “expected manner”?  Yes, if the expectation is that you verbally answer a call or dial into a conference line or bridge to announce your availability or log into an automated support management system and perform some simple acknowledgement that you are aware your services are needed, and then do just that.  Nothing will be gained by sending an email to your manager when the clear expectation is you join a conference bridge line to be informed of the situation.  It may seem painful, irritating, not worth your energy, etc.  Allowing your immediate distaste for the production support situation that you are about to be drug into block you from “doing the right thing” well create the perception that you are not reliable and thus not a leader.  You don’t want those negative perceptions to be linked to your professional image, especially around raise and bonus time.  Plus, once you have been linked to those perceptions, it is going to take above and beyond effort for some time to reverse them.

Know the response expectations

As mentioned above, make sure you know what the response expectations are.  Make sure you have a clear understanding what the SLAs** are for the services for which you will be contacted.  If you have 30 minutes to respond, then make sure you make every effort to respond within that 30 minutes; sooner the better.  Are you supposed to respond via phone, email or join a conference line?  Make sure you are clear on what you are supposed to do.  Are you supposed to login to a problem management system and update the status on a support ticket?  Make sure you have confirmed you can do this remotely as well as in the office (assuming you are providing off hours support).  Know the customer SLAs for the service you are supporting.  If the service is available to customers 24/7 but the real customer service agreement is from 7am to 7pm, know that so if a call comes in before 7pm, be of the mindset that the system needs urgent attention until someone of authority indicates the problem can be handled the next day, not the other way around.  Are there priority levels assigned to the problems that get communicated out?  If so, make sure you know so that you are confident you can ignore a priority twelve problem till the next day, and so on.

Additionally, even though there are SLAs with different response times by priority, make every effort to understand what really constitutes a priority for the service rather than just arbitrary numbers.  Is there a particular “high value” customer or customer group that requires high touch service?  Is there a particular business function that is mission critical or if not completed successfully in a timely fashion, will create a rippling effect of additional problems within the support organization?  Develop a firm grasp of these unique support situations.  Even through they technically might not match the “priority 1 – entire system is down” criteria, they still are viewed by senior stakeholders as important to the business.  Hence, treating them as such will go along way to create the perception you care about your role, the company and have leadership potential.  Alternatively, think positive perception for raise and bonus time can’t be a bad thing.

Lastly, consider that response is just that: response.  Compare these two examples of responsiveness to the same problem:

Example A

Bob on his cell phone calling into the production situation conference line: “Hey, this Bob from FlimFlam support, I just got a text that there is a problem and to join this line.  How can I help?”

Voice on conference line: <Briefly explains that the production system is throwing error codes left and right and the system is essentially unusable>

Bob: “Hmm, I don’t know what could be causing that situation off the top of my head.  I am in the car and about 30 minutes away from being online to begin troubleshooting.  Is that problem?”

Example B

Bob, checking his cell phone for a text message to join a production situation conference line, thinks to himself: “I bet FlimFlam is throwing errors again.  I’ll get home, get online, see what might be going on and then join the conference line.”

In both examples, the time to resolve the production situation is probably the same.  I would actually argue that the time to restore service is probably quicker in option B than in A due to less communication and interaction time compared to hands on technical troubleshooting time.  But if it takes 60 minutes to restore service in example A compared to only 45 minutes in example B, the perception of the quality of technical support provided in example A is much higher than example B due to the higher level of communication and responsiveness involved in example A.  Back up to the leaking hot water heater example from the beginning of this article, that 30+ minute driving commute from receiving the text message to join the call to get engaged is similar to leaving a message for a plumber and not hearing back.  The perceived lack of responsiveness will work against any heroic technical feats of system restoration because those that don’t fully appreciate that you pulled off a systems miracle behind the scenes are only aware of the stress you caused them by not responding promptly to their communication needs first, technical second.

Look for additional articles to identify more technical support pitfalls and steps to take to avoid them.

* QA, Quality Assurance, the process or set of processes used to measure and assure the quality of a product.

*SLA, Service Level Agreement is a part of a service contract where the level of service is formally defined. In practice, the term SLA is sometimes used to refer to the contracted delivery time (of the service) or performance.

Whether you are working in a complete custom software development shop with little vendor interaction or a technology integration shop with vendor solutions integrated with other vendor solutions on top of yet other vendor solutions, you will have to manage vendor relationships to some degree as an IT manager in a MidWestern company.  This series looks at the complex arena of IT vendor management and offers some tips to make the arduous process a bit less arduous and possibly discover some additional benefits along the way.

Vendor Management Categories

• Role of the Sales Rep

Leverage your Vendor Sales Cheese to avoid the Architecture Astronauts!

In the previous article [], we explored how this role can benefit the IT engineer by cutting through the bologna that is the normal route to getting technical support from a vendor and putting one in direct contact with a peer senior technical resource that can solve tough problems quickly.  But what about the benefits to an IT manager?

IT Manager Dividends

Scenario 1 – You Don’t Have Bob the Engineer from the Previous Example

Well, let’s start with that same hypothetical problem scenario in the previous post [] with a production problem being reported in the service you and your team are responsible for, in this case the FlimFlam software.  You don’t have a Bob that has introduced himself to the Vendor Sales Cheese and established himself as the guy, that when he reports a problem with FlimFlam, there is indeed a problem with FlimFlam and Bob needs senior tech support ASAP.  It not, the Vendor Sales Cheese is going to spend his or her value time not selling but rather in fire suppression mode.

Once you notice your team is stuck in the troubleshooting process with the FlimFlam software and the trouble ticket that is open with the vendor is going no where, have your Vendor Sales Cheese contact info handy:

Sally the Manager: “Hey, Vendor Sales Cheese, it is Sally at ABC Company.  Well, no, everything isn’t quite fine.  We’ve got a problem.  FlimFlam is causing customer pain and is throwing an error 57 that no one, not even the tech folks behind your support web site are able to figure out.  You know I don’t waste your time with the trivial stuff, thus this isn’t trivial.  I am about to get on a conference call to explain to my peer management that we have our top engineers working on the problem but I don’t have a good answer when they ask what the vendor of FlimFlam is doing to aide us.  Yes, you can help, that is why I called.  Can you get a senior tech person to look at support ticket <blah> and then have that senior tech person call Joe on my team at <blah> to start resolving the issue?  I have your commitment someone is going to call Joe, right?  Good … now I have a much better story to tell everyone on the conference call.  I’ll be in touch.”

Similar to the previous example, you can leverage your relationship with the Vendor Sales Cheese to get priority service.  From a management perspective, you have just increased the technical capacity of your team without incurring any additional cost by leveraging the notion that the Vendor Sales Cheese would rather have a happy customer for which they can manage this problem more proactively since they were engaged early in the problem resolution process.  Any experienced Vendor Sales Cheese that been set on proverbial fire coming into a customer hot zone with threats of having the vendor and all provided products and services throw out because high level management had to get involved with a problem that, to them and rest of the organization, should never have occurred in the first place.  High level management sees the $$$ from their budget going to these monthly maintenance fees to their vendors as insurance that they will never have to directly deal with a problem caused by the vendor.  The Vendor Sales Cheese that can get engaged in a hot problem early, bring the right level of technical support and relationship support at the highest level and make the problem go away quick has actually earned positive face time with higher levels in the customer organization.  Their bet is the higher levels in the customer organization will see the support value in addition to product feature set from the vendor and look for that vendor to provide future solutions.

Thus, if the issue is heating up in your organization and you don’t have a Bob that has dragged the vendor into the troubleshooting mix, get on the phone to the Vendor Sales Cheese and give the Vendor Sales Cheese the opportunity to augment your team’s technical capacity as well as the opportunity to manage the customer relationship early rather then when asbestos underwear will be required later.

IT Manager Dividends

Scenario 2 – You Have Bob the Engineer

In this scenario, Bob on your team has already brought the vendor into the troubleshooting effort.  So, all you have to do is put your feet up on your desk and watch the magic happen as Bob and the senior vendor resource solve the issue before the day is over.  Well, you could, but what if it is 6PM and Bob and the vendor are stuck after hours of troubleshooting?  Once Bob has reached out to the Vendor Sales Cheese, wait a brief amount of time and then make a follow-up call yourself.  Let the Vendor Sales Cheese know that of all the problems you wrestle with every day, this one has your highest priority attention.  This further emphasizes to the Vendor Sales Cheese that this problem could become the problem that has the Vendor Sales Cheese sending $300+ an hour vendor solution architects to your company for free to appease the post resolution craziness that will surely develop if this problem isn’t given top priority and resolved in short order.

Sure, the Vendor Sales Cheese will quickly become high maintenance because they have had to work with customers in postmortems where a bunch of company architecture astronauts are swooping in after the system has been stabilized to philosophize on how a completely different solution that doesn’t even involve the vendor’s technology would be the best approach for senior management to implement to avoid this type of problem in the future.  The experienced Vendor Sales Cheese knows that once they are up against the customer’s architecture astronauts, the customer’s senior management has lost faith the vendor can save them from getting involved in distracting support issues.  Plus, senior management gets to throw out a vendor that failed and proceed to be courted by new vendors salivating at getting the opportunity to make the sale and have the “we replaced vendor X at company Y” story to enhance their sales pitches of the future.

Again, be prepared for the Vendor Sales Cheese to be high maintenance, but take comfort in the constant polling for status can easily be a positive when you need the vendor to jump higher and you already have the Vendor Sales Cheese with full issue context ready to say “how high?”

This concludes the perspectives on the role of the sale representative short of their role in the sales cycle and pricing.  I’ll dive into this aspect in the next article with more MidWestern IT perspectives on the topic of the “Sales Cycle and Pricing” in the spectrum of vendor management.

Whether you are working in a complete custom software development shop with little vendor interaction or a technology integration shop with vendor solutions integrated with other vendor solutions on top of yet other vendor solutions, you will have to manage vendor relationships to some degree as an IT manager in a MidWestern company.  This series looks at the complex arena of IT vendor management and offers some tips to make the arduous process a bit less arduous and possibly discover some additional benefits along the way.

Vendor Management Categories

• Role of the Sales Rep

The Vendor Sales Cheese can keep you off the phone and on your keyboard

In the previous article [], we explored this role more deeply and how, as an IT manager or IT engineer in a MidWestern company, you need to partner with this role to be successful in delivering you and your team’s services to the company.  We established the notion that as either an IT manager or IT engineer, instead of bolting for the nearest keyboard, there is benefit to spending five minutes introducing yourself to the Vendor Sales Cheese and giving him or her a clear understanding of your role within the company and how you are linked to the product or service the Vendor Sales Cheese is representing.  We left off suggesting that this brief exchange will pay off in tactical dividends.  So, enough with the preview, what are these so called dividends?

IT Engineer Dividends

In short, someone to suck into your troubleshooting effort to get you the technical expertise you need without having to sit on the phone on endless hold finding it yourself.  Someone you can say: “Hey, I did everything I was supposed to do to get help and I am stuck.  Get someone who can speak at my level that knows your product and can help me get this working ASAP.”  Take this typical scenario:

A technical problem makes its way through your business call center through the IT technical support helpdesk to your inbox.  Based on the brief explanation of the problem and the levels of “reboot your PC and try again” and “close your browser, clear your cache and try again” that have been tried with no success, this hypothetical situation suggests you are going to have to roll up your sleeves and figure this out because likely, no one else can in the company.  So, after much wailing and grinding of teeth, you are able to reproduce the problem in a test environment and have ruled out everything except the FlimFlam product.  From everything you can tell, you can now get the problem to occur at will, but all the FlimFlam system does is throw an “Error Code 57: Process died unexpectedly”.  The almighty Google is no help with error code 57.  The vendor’s tech support web site or “knowledge base” (which you have now dismissed as an oxymoron) completely mocks you with no reference whatsoever to error code 57.  So, no instant problem resolution gratification today, you have to log in to the vendor’s support web site with your company’s magic trouble ticket account to open a support issue.  You have been down this road plenty of times before, so you succinctly enter the exact end user steps to reproduce the problem and generate an error code 57.  You cut and paste in a copy of the system log that says “yep, I’m definitely throwing an error code 57 … and unexpectedly as well!”  You provide your platform and vendor product version, revision, and patch level down and dump the configuration out to the final detail.  You get back a trouble ticket number which you write down in the false hope your next email from the vendor’s support site will have the magic cure.

Off to lunch at the default route … err, food court at the mall

When you get back to your desk, you see an email from the vendor’s support site indicating your ticket has been closed.  You log back into the vendor’s site to see the last ticket status entry read:

“Upgrade to the latest version by applying patch 59837”

You switch over to the download tab and punch in “patch 59837”.  You quickly skim the release notes only to find absolutely no reference to error code 57 or anything that even resembles the problem you reported.  You’ve played this game many times before.  But you know, you have to play it or you get stuck at this level in the game, forever unable to advance.  So, you download patch 59837.  You install it in the test environment.  No install errors.  You re-test the system and low and behold, on the first attempt, you generate error code 57 with the same “unexpected-ness”.  So, you go back to the vendor support site.  You re-open your ticket; indicate you did exactly as told with no success.  Again, you’ve been down this road, so you re-state the platform and product versions showing the new patch applied.  You re-attach the log files and a dump of your configuration settings.  You re-attach the steps to create the error.  You raise the ticket to highest priority level you can.  You submit it back to the vendor.

Time goes by.

People in the company, including your boss, start asking: “Hey, when is that problem going to get fixed, people are complaining.” or “customers are getting irritated” or “we are starting to experience high call center call volume related to this problem.” Or whatever constitutes the inter-company fervor building to where you will soon be joining conference calls to explain what is going on and where things are at … rather than being allowed to actually fix the problem.

In anticipation of that first “please join the problem resolution conference line” alter, you re-check your ticket status online and see:

“Ticket Status = Pending”

… and nothing else.

Your world is about to get even more unpleasant as you see you’re frustrated and exhausted boss heading to your cube.

Wouldn’t it be great to have some human at the vendor to reach out to who is motivated to keep your company happily paying the monthly maintenance fees to help cut through the bureaucracy and get your technical peer at the vendor working on a fix for this problem?  Someone who can find that singular vendor engineer, that upon seeing your configuration can immediately go:

“Geez, they are running on OS 34 in 61-bit mode?  They need to add the ‘no cache during day light savings time=yes’ setting to their config file or else they will throw error 57 every time someone presses the ‘Q’ then ‘k’ keys.”

Here is where having the contact info for the Vendor Sales Cheese handy and having had that five minute conversation not too long ago with the Vendor Sales Cheese pays big tactical dividends.

Bob the Engineer: “Hey Vendor Sales Cheese, it is Bob at ABC Company.  Hey, I am getting the run around on support ticket <blah>.  I did everyone as instructed but we are still getting errors from FlimFlam.  A whole bunch of managers are going to get together and start talking about this problem with FlimFlam which means they are probably going to call you at some point if this problem doesn’t get resolved.  What do you need from me to get a senior tech guy to call me ASAP to avoid this pending mess?”  (Note the clever use of language to make your problem the Vendor Sales Cheese’s problem as well.)

The Vendor Sales Cheese doesn’t want to spend time putting out a fire at a customer’s site due to his product or service.  He or she wants to out selling their product or service to a new customer.  Plus, the Vendor Sales Cheese knows you from that fire minute conversation at which they “Check!” linked you to the top tech guy who knows his stuff and only needs help when something is going horribly, horribly wrong at ABC Compnay.  The Vendor Sales Cheese starts lighting fires within his company for get their FlimFlam tech expert on your phone ASAP.

That five minute awkward conversation with the Vendor Sales Cheese pays off big time when that FlimFlam senior tech guy or gal calls you with the magic config file setting that makes the whole problem go away.  And equally important, stops you from having to join the company trouble call and spend countless hours trying to explain to a conference call full of managers.

IT Manager Dividends

Ok, I see the value to the IT engineer, but what about the IT manager?   I’ll dive into this value proposition in the next article with more MidWestern IT perspectives on the topic of the “Role of the Sales Rep” in the spectrum of vendor management.

Whether you are working in a complete custom software development shop with little vendor interaction or a technology integration shop with vendor solutions integrated with other vendor solutions on top of yet other vendor solutions, you will have to manage vendor relationships to some degree as an IT manager in a MidWestern company.  This series looks at the complex arena of IT vendor management and offers some tips to make the arduous process a bit less arduous and possibly discover some additional benefits along the way.

Vendor Management Categories

• Role of the Sales Rep

You want to get to know the shinny suit

In the previous article, I concluded thoughts on vendor service integration challenges.  I made a cavalier reference to the “Vendor Sales Cheese” role.  This article will explore this role more deeply and how, as an IT manager in a MidWestern company, you need to partner with this role to be successful in delivering you and your team’s services to the company.

I don’t think there are more diametrically opposed roles in business than the IT engineer and the IT vendor sales representative.  One of the best descriptions of the complex persona that is the IT engineer is The Nerd Handbook.    IT engineers look at the world as an ever unfolding flow chart of logical constructs built on top of more logical constructs.  They are constantly learning and building.  They prioritize human interactions based on a peer level of technical appreciation and comprehension.  If someone isn’t at their level of knowledge on the subject at hand, the value of the exchange diminishes rapidly in their mind.  On the other hand, the Vendor Sales Representative or as I’ve affectionately relabeled as “Vendor Sales Cheese”, as viewed through the IT engineer lens, couldn’t be worth even a nod in the conversation spectrum.  If you align yourself more with the IT engineering mindset, I bet you are getting ready to HTTP 302 yourself off this article and on to something more technical.  I beg you to continue reading in the hopes I can influence you to consider a logical argument for the value of the Vendor Sales Cheese in your technical and/or management function.

So, as a typical IT engineer or engineering manager, your initial interactions with the Vendor Sales Cheese have you thinking: “This person is way too positive and friendly.  That sure is a slick and way too shinny suit.  I need to get outta this conversation and back to my keyboard ASAP.”  Yes, the Vendor Sales Cheese meets new people every fifteen minutes of every day.  Those people could be the tier one HelpDesk technician or the president of the company.  Hence, they error on the side of potentially meeting the president and bust out the shinny suit.  In meeting people, they need to quickly determine your role in the customer vendor relationship ASAP since there is going to be someone new to meet in another fifteen minutes.  Thoughts going through the Vendor Sales Cheese’s mind:

• How do you align in the organization against the product or service the Vendor Sales Cheese represents?
• Are you an end user that is going to be a source of complaints?
• Are you a decision influencer that won’t make the final purchase decision but could influence the decision maker and possibly tank the deal?
• Or are you’re the golden role, the decision maker that is the person between the Vendor Sales Cheese and closing the deal to get the big compensation bonus?

The Vendor Sales Cheese is trying to determine this as quickly as possible in the limited interaction time they are given.

Sure, you can return to the safety of your keyboard and the logical and controlled in order to avoid the seemingly unpleasant and awkward conversation.  But is another five minutes of conversation really going to kill you?  My recommendation is make this five minutes tactically productive by immediately describing your role within the organization and how it aligns with the product or service the Vendor Sales Cheese represents:

IT Engineer: “Hi, I’m Bob and I am the lead engineer in ABC Company that has the job of taking your FlimFlam software product and cramming it into our enterprise IT environment.  I can’t sign a PO and can’t buy anything.  But, when there is a tough problem with the FlimFlam software here, I get the call.  I’ve been working with it for X years.  So when I need tech support, I don’t need the 1-800 number level 1 tech.  I need access to the guy who, like me, knows how FlimFlam works inside and out.  How do I get that tech access so I don’t waste your company’s time?”  (Note the clever use of language to suck the “vsc” into the need to solve a problem for his company and help a customer at the same time.  How can the “vsc” resist this?)

IT Manager: “Hi, I’m Sally and I am the manager over the team that integrates your FlimFlam software with the rest of the technology here at ABC Company.  Let me start with the fact that I am not the guy that signs the PO, but I have the Director’s ear does.  We’ve had great success with FlimFlam but we know there is plenty of competition in this product space.  My biggest challenge with your company is X.  What is the best way to improve the X situation?”  (Again, sucking the “vsc” in by creating a scenario he/she can’t possibly walk away from since they are ever so relationship positive focused)

For the IT engineer, you have given the Vendor Sales Cheese exactly what they need to know:

• Bob = tech guy at ABC Company that can sing the praises of FlimFlam or make a lot of noise when we drop the ball failing to supporting his priority support needs.  Check.

For the IT manager, the Vendor Sales Cheese knows:

• Sally = manager at ABC Company that shouldn’t get the loge tickets to Sunday’s game at the stadium, but he needs some above average attention because she could tank the current/next deal by pitching to the VP/Director/Other that another company with a competing product could be integrated quicker/faster/cheaper is giving her more respect and support.  Check.

Ok, you are scratching your head … “ok, I see how the stage has been set for some tactical value from this Vendor Sales Cheese exchange, but what does this really do for me?  Doesn’t this just lead to more annoying conversation?”  I’ll dive into this value proposition in the next article with more MidWestern IT perspectives on the topic of the “Role of the Sales Rep” in the spectrum of vendor management.

Anyone that has had to participate in a meeting to determine why some IT system went down is echoing a collective groan as they read this title.  For both IT managers and engineers alike, it is the least desired activity following a system failure of any kind.  Business and/or product owners outside of IT are waiting, after the dust settles and the system is restored to working condition, to have primarily two questions answered

1. Why did the system go down in the first place?
2. What is IT going to do to make sure this doesn’t happen again?

In the first article, I outlined the business context of the root cause analysis exercise in general and the complexities in clearly and logically arriving at a true root cause for a system outage due to the interconnected players involved.  In the previous article, I outline a particular IT engineering resource approach entitled “Openly Be the Hero” to participating in the root cause analysis process.  This article introduces “Play it Safe”.

IT Engineering Participatory Approach C = Play it Safe

Play it Safe!

Having seen the potential pros and cons of approaches A and B, I assume you are wondering is there any way to play the root cause situation safely?  There is, but you are going to have to put your engineering brain and ego on hold a bit.

Step 1 = Resist the urge to be either “surprised and confused” or the Hero.

At the onset, avoid meetings, emails, hallway conversations and basically any situation that might put you in a position to start down the road of approach A or B.  Reply with vague “I’m not sure.  I think we are still looking into that.  I am waiting on <whatever>, let me get back to you” type answers.

Step 2 = Get with your management ASAP and give them a full run down of what is going on, the situation and the players involved.

As succinctly as humanly possible, state the problem “we may very well be part of the root cause for this outage”.  That should get management’s attention very quickly.  Follow-up with “here is what I know, stop me if I am going to fast or you already know all this already” and then quickly and briefly step through the problem clearly indicating where you believe/feel/think each “fact” ranks in authority.  In other words, don’t claim something is a fact unless you hold a log file printout in your hand that date and time stamps what you are saying.  “The commonly held view is the temporary storage volume filled up before anyone could purge files as the system needs, etc., etc., etc.”  “I am 50% confident based on this log entry that disk space was an issue.”  Be prepared to be stopped and asked all kinds of questions pertaining to how you know this, from whom, who else knows, etc.  What is happening is management is starting to build the story of what is taking place factually, the black and white versus gray-ness of those facts and how all the players are positioned to take the blame.

Step 3 = If management doesn’t define your role and thematically what to say and not to say, suggest your role and seek confirmation

Equally as important as step 2, confirming how management wants you to proceed is critical.  If you complete step 2 but then go off and “Being the Hero”, you will be susceptible to all of the cons associated with being the hero. Rather, if you are going forward and executing your role under the clear direction of management, as long as you indeed execute and seek clarity when a unclear situation presents itself, it will be exceedingly difficult to fall victim to the cons associated with “Being the Hero”.

Step 4 = Execute your role and keep management informed of major milestones

Go forth and help the post outage root cause investigation effort always being mindful of your role as indicated by your management.  As you are made aware of “major” milestones, make sure you go back and update management as soon as possible.  The timely updates directly assist in reshaping the story and may be accompanied by some tweak in direction to your role.  “Major” represents any event or new information that changes the shape of events.  “Bob in infrastructure just shared that the daily disk utilization report was indeed showing a reduction in free space for the last two weeks” = share ASAP.  “Bob just shared he forgot his lunch at home” = ignore.  Yes, these are rather obvious examples of what to share and not to share, but the goal here is to develop your own system for listening strategically to all the information that is being shared in order to parse out the noise and direct significant facts back to management.

In summary, the approaches and recommendations here may seem a bit extreme to many.  If you are lucky enough to belong to an organization that is culturally rational and fact based, you may not be forced into these scenarios.  Yet, all it takes is one situation to get out of control and the scenarios above become reality.  The rational, fact based logical analysis of an outage is replaced by the panicking, irrationality and emotion of engineers and managers faced with the notion of job loss due to failure to prevent an outage disaster that had major reputational and/or financial impacts.

Anyone that has had to participate in a meeting to determine why some IT system went down is echoing a collective groan as they read this title.  For both IT managers and engineers alike, it is the least desired activity following a system failure of any kind.  Business and/or product owners outside of IT are waiting, after the dust settles and the system is restored to working condition, to have primarily two questions answered:

1. Why did the system go down in the first place?
2. What is IT going to do to make sure this doesn’t happen again?

In the first article, I outlined the business context of the root cause analysis exercise in general and the complexities in clearly and logically arriving at a true root cause for a system outage due to the interconnected players involved.  In the previous article, I outline a particular IT engineering resource approach entitled “Surprised and Confused” to participating in the root cause analysis process.  This article introduces “Openly Be the Hero”:

IT Engineering Participatory Approach B = Openly Be the Hero

“I know what happened, the temporary storage volume …..”

I will save the day with facts no one can refute!

This approach, which is diametrically opposed to the surprised and confused approach, comes with some different risks.  By standing up and sharing every technical fact you can get your hands on to point out what really is going on can back fire in exactly the opposite way as the surprised and confused option.  People will tend to latch on to the one spouting off all the undeniable facts and suddenly the masses will associate the one with all of the answers as the one being in a position to have avoided the problem all together.  As far as your management goes, if they aren’t on board, you’ve placed them in a difficult spot to be supportive if the tide turns towards the root cause being the hero’s perceived lack of involvement.  Your peers, fearing their job might be in some jeopardy, will most likely slink down in their chairs to remain quiet and allow you to stand tall to take the proverbial daggers of blame.

Now if you are one that has put in the extra energy to understand how the system or systems were constructed, the “why” behind the seemingly architecturally backwards ways certain business processes are completed you may struggle with avoiding the hero trap.  You may be thinking: “The facts that I possess clearly indicate without compromise that what I known to be the root cause is the root cause.  Why can’t everyone just go with the facts and be done with it?”  Not everyone is comfortable accepting the facts even if they are the facts.  What if the facts suggest a particular individual or group of individuals have been linked to the last five system outages?  Maybe these five outages are legit and the individual or group is trying desperately to improve their system management activities.  The last thing they need is another problem piled on top of their previous problems to further put pressure on management to take some action.  In an effort to save their jobs and buy more time to get out from underneath their pile of problems they can redirect the masses to focus on the hero’s involvement and thus take the heat off themselves.

“Let me understand, the Hero knew that this problem was going to happen but didn’t do anything to stop it?  Why is the Hero hiding knowledge that would help the company?  This is yet another example of the Hero not sharing and not partnering.  How can the Hero just sit idly by and allow this to happen.  Something needs to be done about the Hero …”

And this “something that needs to be done” … and get ready, this is going to make any logical thinking IT engineer’s head spinning … could be as severe as disciplinary action cast upon the Hero.  Why is such an illogical outcome such as the individual that amassed such valuable knowledge to be able to assemble together all the puzzle pieces of the problem become the victim of some disciplinary action?  The answer falls more on the organizational hierarchy than on conventional logic.  If the individual that is uttering those statements about the Hero is significantly high on the organizational chart, then the layers below, who have been focusing on all sorts of other fires, are caught without a good story as to why this situation occurred and why the Hero is not the root of all evil.  Not being armed with a story that shields the Hero, the management layers in between are somewhat constrained and thus the blame lands on the Hero.  For more of the management side of the Hero’s plight, see the articles that cover this in the management section.

Sure, your peers might find you after meetings and give you kudos for standing up for the facts, but is being technically “right” worth the cost of being put through this ancillary pain?

The next article introduces the hybrid approach which I’ve entitled “Play it Safe”

Anyone that has had to participate in a meeting to determine why some IT system went down is echoing a collective groan as they read this title.  For both IT managers and engineers alike, it is the least desired activity following a system failure of any kind.  Business and/or product owners outside of IT are waiting, after the dust settles and the system is restored to working condition, to have primarily two questions answered:

1. Why did the system go down in the first place?
2. What is IT going to do to make sure this doesn’t happen again?

In the previous article, I outlined the business context of the root cause analysis exercise in general and the complexities in clearly and logically arriving at a true root cause for a system outage due to the interconnected players involved.  In this article, I outline a particular IT engineering resource approach to participating in the root cause analysis process.

IT Engineering Participatory Approach A = Surprised and confused

“There is a temporary storage volume?  Really?  What is it used for?  You mean it can fill up?  Wow … I’m surprised!  If someone had told me about the temporary storage volume I would have obviously blah, blah, blah”

What technology? I'm supposed to know about this "technology"?

This approach is synonymous with playing dumb about the whole situation as well as the technology itself that was aligned with the outage.  If you haven’t experienced someone using this technique, you might be thinking to yourself: oh come on, can this really work, doesn’t this just make one come across as inept?  The answer is: absolutely.  So why does this approach based in ineptitude actually work.  I will put forth two arguments based on human thought processes and the pressures of time.

I’ve witnessed time and time again where the immediate reaction to the surprised and confused IT engineer is to get caught up in following that line of reasoning.  People do not by and large enjoy being surprised at work with questions about their performance or work quality or responsiveness, etc.  I am not a sociologist, but in my experience, when experiencing someone else suggesting they were surprised and caught off guard by this work event, they immediately identify with the notion of dreading the feelings associated with being surprised themselves and implicitly support this individual’s claim of surprise and confusion.  Rarely have I witnessed the alternative reaction supersede the previous with: “wait a minute … isn’t it your job to be responsible for health of this IT service and thus why are you surprised at how the service you are responsible for functions?”  If someone indeed starts to go down this logic road, the confused aspect starts to take precedence of the surprise with “well, I obviously would have known and fixed the temporary storage problem if someone had told me about it … but I’m confused, who should have notified me ….”  The perception of the problem again, has been shifted from the individual off to some more nebulous area that suggests there was nothing the individual could have done because of this nebulous third party’s role as a barrier for the individual to do their job.  The confusion can keep growing: “wasn’t there a project that was supposed to fix this alerting problem?  Wasn’t <insert random but related engineering group name here> working on this?  Isn’t there a ticket open with the vendor on this issue?  Wasn’t <insert random IT resource name here> working on a fix for this?”  The larger the organization, the more effort it will take to follow up on each accusation in order to find validity.  And that validity gets more and more difficult as the word spreads that the proverbial witch hunt has begun:

<insert random IT resource name here>: “Working on a fix for that?  Yah, but that got assigned to what’s-his-name in the quality team.  You might want to follow up with whose-his-face in testing services because I think they now are responsible for the quality team.”

So this option sure sounds great since it appears the perception is always some external force that can’t be controlled is restricting one from doing their job.  With the converse being if the restriction didn’t exist, I would have done my job and there wouldn’t have been a problem in the first place.

Finally, for this option to be successful, you have to stay completely away from the problem resolution itself as much as possible.  If you are visibly involved in finding and fixing the problem or if you are poking around systems related to the problem and thus are appearing in audit and history logs, you can lose plausible deniability.  Someone could surface at the least opportune time and reveal your involvement in the resolution process, hence reducing your legitimate claim to surprise and confusion.

My suggestion, don’t pursue this option unless explicitly directed by management with some explanation of what role you are playing in the greater root cause exercise.  Why since it seems such an easy out?  In one word: reputation.  You will quickly be branded inept by your peers and management will see you as weak in the sense you are junior, not worthy of being trusted with an important assignment and finally, not promotion material; the latter being the most difficult to rescind if you have your heart on a different job within the organization.  Ok, so you might be thinking, right now, just staying where I am is just fine.  That might be true right now … but what if some new project comes along or the company purchases some cool new technology that you might want to participate in the near future?  The likelihood that you will get such an opportunity given your propensity to be surprised and confused about your job assignments is exceedingly low.  I will finally venture to say that when the company is facing difficult financial stress and the option of work force reduction seems eminent, guess who falls into the X percent category of people the organization can survive without: those that are perceived as surprised and confused by work.

In the next article I’ll outline the pros and cons associated with “Openly Be the Hero”.