Midwest IT Survival » priority

Instant IT Gratification

jfbauer — Thu, 27 Jan 2011 05:18:48 +0000

"I just signed the sales contract yesterday. This needs to be up and running by the end of this week!"

In talking to others recently, we’ve discovered an unfortunate pattern across multiple companies where the partnership between the business units and IT isn’t particularly strong. When I say “partnership” I mean the level of collaboration on business activities that involve IT in some manner. Prior to contracts being signed, has IT been directly involved in the new business opportunity? Does IT have a voice in the major initiative delivery date setting process? Is IT’s resource capacity considered at the same time the business’s resource capacity is being evaluated in moving forward with a major effort? Does IT have access to the business’s multi-year plan? Are IT initiatives peppered throughout that multi-year plan?

There are a seemingly endless list of articles available on the Internet that talk of the need for a strong partnership between IT and the business units IT supports thus I won’t extol those benefits here. Rather, what can an IT manager do in a company that is taking strides towards but is still working on forming a strong business and IT partnership? Or …

What to do when the Business wants “Instant IT Gratification”?

By “Instant IT Gratification” I mean the corporate culture where IT is expected to delivery what the business is asking for, how they are asking for it and when they want it done all without collaborative dialog. Contracts are signed without IT. Due dates are determined in advance and IT is expected to meet these deadlines without being consulted. The results of such interactions between IT and business requesters varies, but the thematic results are the most frustrating:

Sub-optimal technology implemented: Instead of matching the need with the optimal technology, the quickest/fastest must be chosen to hit the date.
Many post-deployment follow-up activities needed: Instead of having a healthy collaborative discussion about all the “requirements”, only what the business shares is implemented requiring many “oops, we forgot we also need X”
Inability to upgrade/enhance existing technical capabilities: always focusing on the urgent needs of the business without allocating a percentage of time to invest in more current technology assets.
Work never done: Constantly jumping to the next hot request leaving only phase one of Y implemented from the previous hot, now cooling request.

What can an IT manager do in this kind of culture?

Anyone who skimmed the above bullets and identified with even one point knows these are cultural problems that a single IT manager won’t be able to solve over night. So how does one not pull their proverbial hair out of their head trying to get anything proactive accomplished in such a re-active business model? Consider some of the options below; just know that none is a silver bullet of success.

Leverage the formal charge back model.

If your organization has a formal IT work charge back model, become intimately familiar with how the process formally and informally works. Employ the charge back model in every possible situation to ensure there is fully accountability and record-ability for all business requested IT work.

No formal charge back model, then use time as the “IT currency”.

If no formal charge back model exists to leverage, then the only real charge back model or IT “currency” is time. Make sure you have full data based reporting on all the work your team is performing. When a new “hot” request comes in, update your single view of the work and present the new “hot” request alongside all the previous “hot” requests and ask the business to prioritize.

Like it or not, get some formal work estimation model in place

If all the business requests “just get handled” then the business has no governor for making requests. Without a charge back model to where the business has to make the conscious decision to spend on one thing compared to something else, there is no barrier for the business to make every conceivable request to IT. One approach is to “just handle it” and try to service every request as quickly as possible so to not have to engage in any prioritization or work break down discussions. But as I have written prior, having some formal work estimation process allows for an intelligent, data and date based discussion with the business. When presented with duration and possible delivery dates, business users can more easily take low priority work off the request list. Low priority work can easily be removed once it is known by all what is consuming time and delaying the delivery of what the business truly needs in a hurry.

Don’t blindly start working on requests, ask probing questions

From a previous article on my learning in attending a formal Agile training class, asking why multiple times (per the instructor, ask “why?” five times to get to the bottom of the need for a request) in order to dig into the real reason for making a request for work to IT. You will be amazed how many times the request hasn’t been fully vetted by the business. Asking why to get to the real business case to support a request will help to reveal what is truly a priority that IT is in the best position to deliver on compared to a frivolous request that if not executed, actually reduces technical debt without harm to the requester. Invest 40 IT hours in order to avoid a situation that happens maybe once a year and consumes 4 hours of the business’s time? Assuming all costs being equal between the business and IT (usually IT costs more per hour) , a ten year return on investment? It would seem those 40 IT hours should be invested elsewhere.

As I said, these clearly aren’t silver bullets that eliminate the challenges of a less than strong partnership between IT and the business units. Yet, by consistently improving ones capability to work with business requesters through careful implementation of the bullet points above, one can establish a level of credibility for your team’s services back to the business to better align the limited IT resources with the highest priority work for the company as a whole.

The Dreaded Root Cause Meeting for the Manager, Part 6

jfbauer — Thu, 12 Nov 2009 05:46:49 +0000

For both IT managers and engineers alike, it is the least desired activity following a system failure of some kind, coming up with the root cause. Business and/or product owners outside of IT are waiting, after the dust settles and the system is restored to working condition, to have primarily two questions answered:

Why how did the system go down in the first place?
What is IT going to do to make sure this doesn’t happen again?

The urgency can return without warning!

The need for answers to these two seemingly straight forward questions generates an urgency that has all the IT stakeholders rallying together in camps. This series of articles look at this challenging exercise from an engineering management perspective with the first article introducing the “80% accurate technique” and previous article focusing on avoiding the spinning wheel of blame. This article considers how to approach the inevitable post outage “ruggedization” efforts.

So, you have survived with minor bumps and bruises from a service outage. The service is now restored. But does everyone just go about their regular work and forget this grueling event? Nope … here come the “ruggedization” efforts. I’ve covered one angle to the project involvement perspective in a previous post. In summary from that post to set the tone for this extension: “ruggedization” projects tend to have strong support immediately following the outage but as time marches on and new problems and priorities pop up, the “ruggedization” effort loses momentum. Strong resources move on to the problems of the moment and the challenges of the future leaving weaker resources behind to struggle to move forward on the “ruggedization” effort of the past. What ultimately puts a nail in the coffin of the “ruggedization” effort is when real capital dollars need approval in order to buy new equipment and/or additional software licenses when many have forgotten the event ever occurred.

Thus you, as a manager, are faced with the strong potential for your team’s resources and your energy to get pulled into this likely to eventually stall out effort. Walking away from these “ruggedization” efforts initially will brand you and your team as ones that don’t partner with the rest of the organization. Assigning your top engineers and keeping tabs on all the throws of the ensuring project process could put you at even greater risk of not paying enough attention to new and in flight projects. Thus you need a strategy to maintain a partnering perception while not losing your strategic focus.

Approach? In a word: balance

Balance in the sense that you need to balance you and your team’s involvement in the effort with the priority the rest of the organization is applying to the effort. In the beginning, everyone will be running around with a sense of urgency about the effort and you need to be applying an equal amount of urgency. Everyone external needs to have a similar sense that the urgency they feel is matched by the urgency you and your team export. But as you get a sense that the organization is beginning to lose the momentum and participants are dropping off to focus on more urgent matters, begin to echo that same level of decreasing involvement. Depending on your risk tolerance level, you can immediately start pulling back at the sign the others are doing the same. I prefer a slightly more risk adverse approach. I have found you get an even more concrete sense of the dissipating urgency as you directly interact with people that were demanding licensing costs, hardware estimates and testing schedules yesterday, but when you follow up with them with more questions today, they noticeably baulk at returning your calls, emails and IMs. Plus, with this approach, if something goes wrong and everyone is brought back up to the level of urgency (example: system shows signs of potential doom and gloom again), you have a steady flow of “pre-tasks” (introduced in this article) you can reference that has “you” waiting on “them”. These “pre-tasks” help both with your interactions with external parties and your management. As they rush to get back into the hyper –urgent state and begin to thrash you and your team with requests, you have immediate responses that redirect them back to their world allowing you to more calming ramp backup. The same applies to your management as they hear things are picking back up, they want/need a sense that you and your team are on top of the situation. Nothing conveys that message as when, as you are fighting the current fires and this old fire flares back up, you have this at the ready:

“The customer service quality team is asking where the “ruggedization” project is at? Well, we are waiting on a quote for the two different server config options the platform team recommended to add capacity from IT procurement. We have questions out to the enterprise support team for them to confirm what data they are looking to pull from the system logs that they said don’t have the info they need. And finally, we are waiting on Testing Services to provide a performance testing window to test if the vendor recommended performance tuning settings will have any effect. So, we are ready to re-engage, but right now, we are waiting on these items in order to proceed.”

Want to be even more proactive? Then email each of the contacts in the above example and check in on how they are progressing with your request as soon as you get wind the fire has re-ignited. Then you can add the following to you response to your management:

“In addition, I’ve ping-ed each of those groups to see if they need anything from us at this point.”

This further solidifies you are on top of the situation when you can respond with this vote of confidence.

Thus, in summary, by keeping a pulse on the level of involvement and urgency of external stakeholders and metering you and your team’s involvement to a similar level you can maintain a sense of partnership with the rest of the organization. In addition, if you are positioned with “pre-tasks” and an at-the-ready response to your management when the “ruggedization” effort goes from cool to cold to instantly hot again, you will respond to their desire to have confidence and trust that you are on top of the situation. Maintaining both achieves the required sense of balance to maintain the appropriate level of involvement in the “ruggedization” effort along with not neglecting the new and emerging request for attention.

The Dreaded Root Cause Meeting for the Manager, Part 5

jfbauer — Mon, 09 Nov 2009 05:18:27 +0000

Why how did the system go down in the first place?
What is IT going to do to make sure this doesn’t happen again?

Can you avoid the spinning wheel of blame?

The need for answers to these two seemingly straight forward questions generates an urgency that has all the IT stakeholders rallying together in camps. This series of articles look at this challenging exercise from an engineering management perspective with the first article introducing the “80% accurate technique” and previous article focusing on communication strategies with your management when the priority of the organization is restore service at all costs, but don’t neglect data critical to finding the root cause. This article considers an even more challenging element … avoiding the spinning wheel of blame.

What is the priority of the organization when it comes to systems outages?
Does someone/group need to be blamed for the outage?

Is the priority to restore services as fast a humanly possible, but with this ever present fear of the inevitable “spinning wheel of blame” along the way? If so, then you have your work cut out for you. Hopefully this article provides some helpful tips for this most unpleasant IT cultural scenario.

Those working or having worked in an IT culture that embraces what I call the spinning wheel of blame immediately know to what I am referring. It is that sense that as the duration of the outage increases, proportionally, the need to cast blame on a particular entity for the cause of the outage also increases. This proportional increase results in significant downward pressure on everyone involved not to be remotely close to the impending blame assignment. In the opposite culture, though an organization does not enjoy a systems outage, they take a more healthy approach liken to a previous article: restore service quickly, but learn why the outage occurred in the first place so rational steps and associated investments can be made in order to reduce the likelihood of future outages. Again, in this counter case, the priority shifts to restoring service as quickly as possible but at the same time, building a case to point the finger of blame as far from one’s team and one’s department as possible. This is where throwing the technology vendor under the proverbial bus comes in very handy. Look for more on this challenging dynamic for the engineering team dependent on a vendor in a future article.

So, you have made it this far and you are either still groaning at the thought of your most recent experience avoiding the wheel of blame in your organization or curious how such an unhealthy culture can actually manifest itself in IT which is known for embracing constant change and the bumps and bruises along the way. Below is a modification to the two pronged approach I mentioned previously:

Prong One – Keep Your Team Focused

Similar to the approach in this article, identify team member competencies on juggling the multiple priorities involved in restoring service and gathering data and manage accordingly. But considering the wheel of blame element, coach the senior members to keep you abreast of the current buzz on who the wheel is pointing at before and after each major milestone in the restoration effort. Instruct them to give you a heads up as soon as their confidence nears 80% your team’s service is likely the root cause candidate: preferably before external parties catch on to this likelihood. For junior members without leadership provided by a senior member, though it may come across as a little bit of micro managing, step in frequently to get a pulse on their discoveries and remind them to inform you of incriminating facts prior to sharing with a larger audience. You’ll need to absorb as much of the pulse and data of what is going on as to predict where the spinning wheel is pointing at any given moment and if it is potentially going to point at you as the next log entry is revealed.

Lastly, and very important, make sure you instill trust in your team that you have their back. If they know the wheel exists and they get a sense you will throw them under the bus at any opportunity, they will quickly adopt non-supporting and counter productive behaviors making your job significantly harder. Be prepared to go to bat for them when others might like to take the easy route and blame one of your team members in the blame assignment phase.

Prong Two – Keep Your Management Team Informed

While you cringe at the energy you have to expel to keep up with all the activities in flight plus the spinning wheel, your management is crossing their fingers that the wheel will land on someone else with equal fervor. In addition to providing the information in the thematic format I proposed in this previous article, with each communication, consider including a “likelihood it is us at XX%” indicator, preferably at the top of each communication. Strive to not have XX go from 5% to 95% in between a 5 minute communication string. It is best to start with some assumed outage responsibility since your team is being called into the restoration and root cause effort for a reason. If data even smells like you might have some culpability, start showing it in the XX% indicator right away. Nothing will grab attention like an XX% going from 50% to 60% to 70% as this is a clear indication the wheel of blame is definitely spinning in your department’s direction. This gives your management the opportunity to get involved if only to be prepared to erect the blame shields. Another positive to having your management get involved as the percentage increase is that they can give direction if they see fit. You have most probably been heads down, focusing on the tactical. Your management has had the opportunity to be looking more broadly at the situation and can provide some valuable feedback from this more external perspective.

In extreme cases, not keeping your management informed opens the door for the wheel of blame to land on you directly from them. If you haven’t brought your management in early, then when something goes wrong procedurally or otherwise, you are going to have to retroactively explain. Unless you have a great story, and more than likely you don’t, you set yourself up for enabling your management to have little choice but to leave you out in the proverbial cold. Where as, if you have been in regular communication with them and they are interacting in some manner, then they are implicitly part of the situation, not abstracted from it. Taking a more harsh angle: you have removed their plausible deniability and significantly reduced the “surprise and confusion” opportunity as their out.

In the next article in this series … now that service is restored and a brief sense of calm has returned, how to approach to spirited post disaster “ruggedization” efforts.

Anyone have an example of a do or a don’t when it comes to how you handle these situations? Anything you did that was helpful or hurtful during these events you can share?

The Dreaded Root Cause Meeting for the Manager, Part 4

jfbauer — Wed, 04 Nov 2009 05:38:11 +0000

Why how did the system go down in the first place?
What is IT going to do to make sure this doesn’t happen again?

Fix ASAP, but don't miss the how and why?

The need for answers to these two seemingly straight forward questions generates an urgency that has all the IT stakeholders rallying together in camps. This series of articles look at this challenging exercise from an engineering management perspective with the first article introducing the “80% accurate technique” and previous article focusing on communication strategies with your management when the priority of the organization is restore service at all costs, then try to find the root cause. This article considers a less “at all costs” culture.”

What is the priority of the organization when it comes to systems outages?

Is the priority to restore services as fast a humanly possible, but with attention paid to what changes are being made, when and what are we learning about the system along the way? If so, you need a two pronged approach:

Prong One – Keep Your Team Focused

Taking a quick assessment of your team members, you can probably quickly determine who can successfully balance competing priorities and who is overwhelmed when multiple goals are up in the air at the same time. For those that have proven the ability to successfully balance these competing priorities, minor reminding to be cognizant of the need to balance the urgency to get things fixed against the need to capture the high level steps taken both towards ultimate success as well as towards knowledge that ultimately leads to success. Overly reminding these individuals of these goals will be perceived as micro managing. Thus, politely remind them, then proceed to monitor without nagging. For those that have proven to struggle with the “troubleshooting 101” concepts of value derived from both fixing the problem quickly and gathering knowledge during the fixing process, you will need to get more involved. One approach is to link these individuals to more senior team members that can take the lead and leverage these less skilled resources as a personal support arm to their resolution efforts. If you are unable to link these individuals to ones that do possess these skills, you will need to provide further instruction. Here, what skilled team members would view as micromanaging is what these team members would view as helpful, clear and focused direction. Consider a quick electronic template for individuals to use with columns such as:

Date/Time, Activity Performed, Knowledge/Result of Activity, By Your or Other?

Example entries:

10/01/2009 8:00am, Joined troubleshooting conference call, nothing yet,

10/01/2009 8:10am, Restarted BLAH service, no change for the system still crashed under load,

10/01/2009 8:15am, Increased Available Threads in Thread Pool and Restarted BLAH service, no change for the system still crashed under load but the system supported 10k more users than before this change,

Also, consider pre-populating the template with other relevant data such as a drop down list of business units impacted or applications/services involved or other support groups involved. Include as many pre-populate-able attributes as needed to assist you in strategizing your communications.

Prong Two – Keep Your Management Team Informed

If you feel some what tactically helpless, image your management’s level of helplessness in these situations. If you are new to your management role, this is a great opportunity to demonstrate your leadership capabilities and build confidence and trust in you from your management. Structuring a communication frequency that provides timely, but not thrashing, updates of major milestones in the troubleshooting and root cause effort will go along way to build that confidence and trust. A theme sequence to consider is:

Reported problem, your team’s initial engagement (don’t forget to mention the urgency of your team’s involvement), other teams engaged, more details forthcoming
Quickly following, initial assessment of the systems and end users impacted, what they are experiencing, what the initial take is on what the culprit is, temperature check of the players involved, more details forthcoming
Major milestones of knowledge discovery or change in the reported problem from the last report, confidence assessment of next steps equaling resolution and root cause
- Consider attaching your most recent template as an appendix/supporting material
- Final resolution, root cause with confidence assessment, degree of involvement in the cause of the problem, next steps now that service is restored
  - Consider attaching your most recent template as an appendix/supporting material

With a two pronged approach of balancing the directional needs of your team to juggle competing priorities factoring in their individual skill sets plus an organized thematic approach to communicating to your management, you add considerable value in the root cause analysis process even though your hands are not directly solving the technical issues.

In the next article in this series … what if the priority to restore services is as fast as humanly possible but under the overwhelming fear that the spinning wheel of blame has to land on someone for this disastrous event?

Anyone have an example of a do or a don’t when it comes to how you handle these situations? Anything you did that was helpful or hurtful during these events you can share?

The Dreaded Root Cause Meeting for the Manager, Part 3

jfbauer — Fri, 30 Oct 2009 07:33:04 +0000

Why how did the system go down in the first place?
What is IT going to do to make sure this doesn’t happen again?

Get the system restored at all costs?

The need for answers to these two seemingly straight forward questions generates an urgency that has all the IT stakeholders rallying together in camps. This series of articles look at this challenging exercise from an engineering management perspective with the first article introducing the “80% accurate technique” and the second focusing on interacting with your team. In this article I’ll cover considerations on how to interact with your management during the outage and crucial fact gathering post outage activities.

Considering you have had a hands-on engineering role in the past but have now transitioned fully into management, you probably remember the first major systems outage you participated in as a manager. Now if you were managing a system that you had very recent hands on experience working on, you probably felt more comfortable digging into error logs and debugging lines of code than communicating to outside stakeholders. One the most important stakeholders is your management structure. If you have drifted from being the hands on guy who knows all about the system to the manager, you probably have come to grips with not being able to immediately diagnose every problem and thus have to put trust in your team members (as mentioned in the previous article). And most challenging, if you find yourself managing a service which you did not have a technical hand in designing and building, you are completely unable to rely solely on your brain power to dig into the problem and fix it without serious technical help. Yet, in all the above situations, your role as manager requires having a solid understanding of what the problem is at any given moment, what impact the problem is causing and what steps are planned to make life grand again.

Keeping your management informed of what is going on in a manner which gives them the timely information they need to act at their level is curtail. Keeping them in the dark about what you and your team are doing to work the problem and restore services by feverishly fixing things does not bode well for you being seen as a leader. Also, you may need some assistance from your management chain when other groups are being impacted by your service outage and increasingly higher levels of their management start asking tough questions. On the other side, sending your management details of new found cryptic error log data every two minutes is going to have a similar perception result … your distinct lack of leadership.

I wish I could produce a single check list of activities that would work for every organization, every culture and every one of your managers. Rather, as I look back at past companies, managers and their associated styles and cultures, there is no one size fits all. Thus, instead of a check list, I thought the best method would be to look at organizational attributes via a series of questions.

What is the priority of the organization when it comes to systems outages?

Is the priority to restore services as fast a humanly possible, regardless of the steps taken? If so, then the information flow up the chain would be catering towards creating confidence in your team’s focus on the urgency of getting things working. At the same time, coach your team to look for every option to get it running and figure out the why later.

Experienced engineers know the best time to capture useful data is when the system is hemorrhaging error information during the failure. In high volume systems, restarting processes or rebooting systems clears a good portion of this invaluable real-time problem data from the crash. This begs the obvious contradiction: if you are rushing to get things running at all costs, isn’t one of those costs the loss of critical data that might point squarely at the root cause? The answer is “Yes”. Thus, in your communications upwards, strategically force into the communication stream the notion that as the team is rushing, the ability to slow down and interpret data for root cause is being sacrificed. That way, once everyone temporarily relaxes when the system is restored, then switches to why did it crash in the first place, you have a proverbial leg to stand on when there is a lack of critical data to support a real root cause determination. Sure, the “I told you so” conversation is never pleasant. What is worse is the “why didn’t you tell me” conversation. Choosing between the lesser of two evils, I would rather quietly and politely refer to mentioning the cost of rapid restore versus methodical data gathering first, and then restore, rather than “Oh, um, yah, I forgot to mention that when we rebooted the box, we lost all the error logs in memory thus we have no clue why the service was taking up all the CPU.”

In addition, don’t neglect keeping your upward communication stream of urgent service restore in sync with your download stream. Know your team members’ approaches involved in the system restore and ultimately the root cause exercise. You may need to help them refocus themselves on the priority of system restore at all costs. Engineers tend to want to figure out the “why” which could eat up precious time against the goal of service restoration. Plus, they know what follows the restore, thus they naturally want to continue to be viewed as a knowledge expert. They want to get their hands on as much data to process as possible to maintain that image.

In the next article, I’ll build on this theme of organization and culturally aligned approaches to management communication.

Anyone have an example of a do or a don’t when it comes to how you handle these situations? Anything you did that was helpful or hurtful during these events you can share?