Midwest IT Survival

Anyone that has had to participate in a meeting to determine why some IT system went down is echoing a collective groan as they read this title.  For both IT managers and engineers alike, it is the least desired activity following a system failure of any kind.  Business and/or product owners outside of IT are waiting, after the dust settles and the system is restored to working condition, to have primarily two questions answered:

1. Why did the system go down in the first place?
2. What is IT going to do to make sure this doesn’t happen again?

In the previous article, I outlined the business context of the root cause analysis exercise in general and the complexities in clearly and logically arriving at a true root cause for a system outage due to the interconnected players involved.  In this article, I outline a particular IT engineering resource approach to participating in the root cause analysis process.

IT Engineering Participatory Approach A = Surprised and confused

“There is a temporary storage volume?  Really?  What is it used for?  You mean it can fill up?  Wow … I’m surprised!  If someone had told me about the temporary storage volume I would have obviously blah, blah, blah”

What technology? I'm supposed to know about this "technology"?

This approach is synonymous with playing dumb about the whole situation as well as the technology itself that was aligned with the outage.  If you haven’t experienced someone using this technique, you might be thinking to yourself: oh come on, can this really work, doesn’t this just make one come across as inept?  The answer is: absolutely.  So why does this approach based in ineptitude actually work.  I will put forth two arguments based on human thought processes and the pressures of time.

I’ve witnessed time and time again where the immediate reaction to the surprised and confused IT engineer is to get caught up in following that line of reasoning.  People do not by and large enjoy being surprised at work with questions about their performance or work quality or responsiveness, etc.  I am not a sociologist, but in my experience, when experiencing someone else suggesting they were surprised and caught off guard by this work event, they immediately identify with the notion of dreading the feelings associated with being surprised themselves and implicitly support this individual’s claim of surprise and confusion.  Rarely have I witnessed the alternative reaction supersede the previous with: “wait a minute … isn’t it your job to be responsible for health of this IT service and thus why are you surprised at how the service you are responsible for functions?”  If someone indeed starts to go down this logic road, the confused aspect starts to take precedence of the surprise with “well, I obviously would have known and fixed the temporary storage problem if someone had told me about it … but I’m confused, who should have notified me ….”  The perception of the problem again, has been shifted from the individual off to some more nebulous area that suggests there was nothing the individual could have done because of this nebulous third party’s role as a barrier for the individual to do their job.  The confusion can keep growing: “wasn’t there a project that was supposed to fix this alerting problem?  Wasn’t <insert random but related engineering group name here> working on this?  Isn’t there a ticket open with the vendor on this issue?  Wasn’t <insert random IT resource name here> working on a fix for this?”  The larger the organization, the more effort it will take to follow up on each accusation in order to find validity.  And that validity gets more and more difficult as the word spreads that the proverbial witch hunt has begun:

<insert random IT resource name here>: “Working on a fix for that?  Yah, but that got assigned to what’s-his-name in the quality team.  You might want to follow up with whose-his-face in testing services because I think they now are responsible for the quality team.”

So this option sure sounds great since it appears the perception is always some external force that can’t be controlled is restricting one from doing their job.  With the converse being if the restriction didn’t exist, I would have done my job and there wouldn’t have been a problem in the first place.

Finally, for this option to be successful, you have to stay completely away from the problem resolution itself as much as possible.  If you are visibly involved in finding and fixing the problem or if you are poking around systems related to the problem and thus are appearing in audit and history logs, you can lose plausible deniability.  Someone could surface at the least opportune time and reveal your involvement in the resolution process, hence reducing your legitimate claim to surprise and confusion.

My suggestion, don’t pursue this option unless explicitly directed by management with some explanation of what role you are playing in the greater root cause exercise.  Why since it seems such an easy out?  In one word: reputation.  You will quickly be branded inept by your peers and management will see you as weak in the sense you are junior, not worthy of being trusted with an important assignment and finally, not promotion material; the latter being the most difficult to rescind if you have your heart on a different job within the organization.  Ok, so you might be thinking, right now, just staying where I am is just fine.  That might be true right now … but what if some new project comes along or the company purchases some cool new technology that you might want to participate in the near future?  The likelihood that you will get such an opportunity given your propensity to be surprised and confused about your job assignments is exceedingly low.  I will finally venture to say that when the company is facing difficult financial stress and the option of work force reduction seems eminent, guess who falls into the X percent category of people the organization can survive without: those that are perceived as surprised and confused by work.

In the next article I’ll outline the pros and cons associated with “Openly Be the Hero”.

Anyone that has had to participate in a meeting to determine why some IT system went down is echoing a collective groan as they read this title.  For both IT managers and engineers alike, it is the least desired activity following a system failure of any kind.  Business and/or product owners outside of IT are waiting, after the dust settles and the system is restored to working condition, to have primarily two questions answered:

1. Why did the system go down in the first place?
2. What is IT going to do to make sure this doesn’t happen again?

Everything is broken!

Since the business folks have had their otherwise perfectly aligned piles of work to do that doesn’t involve IT completely interrupted by IT, they start the root cause analysis process by contacting that person in IT that represents their “relationship” with IT.  That IT person is usually high enough in the management hierarchy that their need for answers to these two seemingly straight forward questions generates an urgency that has all the IT stakeholders rallying together in camps.  As each camp is forming, one underlying theme prevails: no one wants to be the engineer that broke the system and no manager wants to be responsible for that engineer and thus the breakage itself.

Engineering Perspective:

The last thing an IT engineer wants to hear from his boss is that after being up all night fixing the problem, that he or she has to come into the office to participate in a root cause or root cause analysis meeting to talk about what happened.

Bob the Engineer: “But we all know what really happened … Storage Support was supposed to monitoring the temp storage volume and when it starts to fill up, get someone on the DBA team to start dumping transaction history data … but since Storage Support let the volume fill up, of course the DB transactions are going to halt, which backs up everything in the system and then it crashes!”

Sure, that could very well be an accurate root cause to the problem.  Take a deeper dive into the group suggested to be dropping the proverbial ball on this one: Storage Support could be waiting on Storage Engineering to build/provide/implement a monitoring and alerting solution due to some past recognition that there doesn’t exist a reliable way to alert appropriate folks when a storage volume reaches a certain threshold.  And yet, Storage Engineering is actually actively working on a solution as part of a formal IT project already in flight based on some past disaster that kicked of said “ruggedization” project.  Thus, once “projectified”, the ownership of the delivery of an alerting and monitoring solution for the temporary storage volume is arguably not Storage Support, nor Storage Engineering but rather the more nebulous ruggedization project itself.  Thus, if the ruggedization project has been providing frequent and accurate updates to project sponsors and stakeholders as to the status and ultimate delivery dates of the project itself, one could argue that the project sponsors failed to assign appropriate priority to the ruggedization effort itself.  And since the project sponsors are most likely IT managers the situation becomes increasingly complex in that the logic (or illogic if you may) proposes that IT management, the same people that very well could be getting the phone calls from the business people, are actually the root cause of this hypothetical outage.

Whew … now if you are still reading and haven’t passed out yet, congratulations!  As an engineer caught in the middle of this interconnected web of essentially competing priorities of limited resources, you have a couple different ways to participate.  Each participatory approach has its positives and negatives, thus choose the approach you are most comfortable with to achieve the approaches associated outcome.

The next article will outline some participatory approaches and their associated pros and cons.

I think it is safe to say that anytime any company wants to get something new accomplished they “kick off” this thing called a “project”.  Now if you have spent any time in IT you have probably had a role on one of these so called projects.  You know when you are on a good project:

• Everyone knows what the end goal is.
• Everyone knows each others roles.
• Everyone has a sense that all participants are accountable for their tasks.
• Things are getting done.
• Everyone feels a sense of progress.
• Everyone feels a sense of team.

You also know when you are on a bad project:

• Everyone is seemingly bumping into each other.
• No one can really articulate the goal of the project
• No one knows what the end state is supposed to look like.
• Time is going by with meetings and minutes and status reports but no real tangible work is getting done.

This series of articles will cover different aspects on how to survive as an engineer that has a role to play on an IT project.

In the previous article, the concept of a “pre-task” was introducing a gap in the project plan and introducing gaps into the plan causes panic.  Thus, how can you redirect this panic to work for you?  That is were the ability to introduce a gap or a “pre-task” from the example in the previous article that redirects the panic on to another individual or entity while at the same time, implying the gap should have been already accounted for because everyone knows it is a required tasks works.  Breaking down the previous example:

How does a pre-task help me?

“…I can’t finish until the Architect signs off on my design document” introduces a gap in the plan that doesn’t involve or imply more work on your part.  Rather, it brings in a task that is a barrier for you to complete your task.  Thus, and here is the selfish benefit, this technique allows you to work on your task without someone hovering asking every five minutes … “is it done yet?  How about now, is it done yet?” as project managers tend to do.

“All designs must have Architect sign-off” is the follow-up sentence implying that it is the project manager’s job to know this standard operating procedure or process.  Thus, it isn’t your job and thus you can take no blame for not having this step in the plan from the beginning.

In summary, whether you find yourself with too many chiefs or too many indians, using the tools above should help you succeed in your role on a project.  Now if you have read this far and are thinking to yourself, “geez, this sure sounds like a lot of work above and beyond doing real engineering work.” my response is yes, it is.  And yes, none of it is seemingly appealing to the engineer that just wants to do engineering work.  But, in my experience, I have seen, time and time again, the good natured engineer get himself, his boss and his team in hot water because all the chiefs were pointing the finger of blame squarely on the good natured engineer that became the almighty reason for the project delay.  The finger is pointed regardless of all the prior missteps that get conveniently forgotten in these scenarios or have already been covered up.  To better position yourself not to get caught up in these nontechnical exercises with chiefs, it pays dividends to invest some time understanding what the formal and informal processes are in relation to your project role and get into the practice of always having a pre-task in your back pocket to use when chiefs are starting to dig into the project tasks.

Key points from this series of articles:

• Know your role on the project and ensure your boss is in agreement.
• Always have your boss in 110% support of your performing tasks/functions outside of your traditional role
• Have a working knowledge of the official project and engineer processes leveraged by your company
• Always have a pre-task ready to articulate when ever possible to create that external dependency on any tasks you are performing

I think it is safe to say that anytime any company wants to get something new accomplished they “kick off” this thing called a “project”.  Now if you have spent any time in IT you have probably had a role on one of these so called projects.  You know when you are on a good project:

• Everyone knows what the end goal is.
• Everyone knows each others roles.
• Everyone has a sense that all participants are accountable for their tasks.
• Things are getting done.
• Everyone feels a sense of progress.
• Everyone feels a sense of team.

You also know when you are on a bad project:

• Everyone is seemingly bumping into each other.
• No one can really articulate the goal of the project
• No one knows what the end state is supposed to look like.
• Time is going by with meetings and minutes and status reports but no real tangible work is getting done.

This series of articles will cover different aspects on how to survive as an engineer that has a role to play on an IT project.

In the previous article two concepts were introduced to help you survive.  Now it is time for the most powerful concept, the “pre-task”.  As much as possible at all times, create the perception that you are waiting on something from someone else in the organization.  Make sure in every situation possible that the project manager or development lead or whoever is managing the checklist of project tasks believes someone else needs to complete their task before you can complete your task.

Not having a “pre-task”:

Sally the Project Manager: “So Bob, according to my plan, you should be working on that widget … are you done yet?”

Bob: “Um, yah, I’m working on it but I told everyone it was going to take me five days and this is day two.”

Sally the Project Manager: “Not according to my plan … it says you build the widget and then we start testing.  You are holding up testing.  I’m going to have to escalate!”  And rushes out of your cube before you can formulate a response.

Having a “pre-task” at the ready:

Sally the Project Manager: “So Bob, according to my plan, you should be working on that widget … are you done yet?”

Bob: “Um, yah, I’m working on it but I can’t finish until the Architect signs off on my design document.  All designs must have Architect sign-off.”

Sally the Project Manager: “Um …” <has no clue what that means but now knows there is a delay building> “I better go find our Architect on this project and get him or her involved”.  And again, rushes out of your cube.

Now, in this example, there are multiple issues generating confusion.  One is task duration.  The likelihood that someone actually recorded your need for five days to actually do work is very low.  I am continually amazed that the less technical a person is the greater the likelihood they will assume a technical task takes an amazingly short amount of time.  Another is the propensity for project managers or people whose primary job it is to track other peoples work are quick to react less than rationally when their plan gets messed up.  They have their plan and they remain rational when, as time goes by, the plan remains unchanged and tasks are sequentially marked complete.  But when the plan itself appears to have a gap, panic and irrationality can ensue.

Ok, why does someone whose sole job is to track a checklist loses their marbles when there is a missing tasks or four additional tasks need to be inserted somewhere?  A quick glimpse into the project manager’s world reveals their greatest value to the organization is in their ability to predict the future with their plan.  Based on the future date their plan indicates, all kinds of business-ish things could be set to kick off.  Thus, something as simple as a plan that says when a system upgrade will be deployed could also be the starting point for a marketing team to formally kick off a set of media ads, plus a customer training/education packet could be already in the mail with dates for when each customer will be forced to use the new upgrade system, plus an accounting group could be poised to show a drop in license costs (i.e., save big money) due to the upgrade, plus, some executive is counting on the license money savings making his departments balance sheet come just under a magical finance number that triggers a quarterly bonus.  Hence, when the upgrade implementation date starts slipping from a failure of the future predictor (aka. project manager) being accurate, all kinds of heat from all kinds of folks downstream from any of the technical work associated with the project can come out of the woodwork and come down hard on the project manager for giving bad data.  Hence, fearing such heat, project managers react in a variety of ways and some of which are not particularly 100% rational.

In the next article, I’ll outline how to use this panic to work for you.

I think it is safe to say that anytime any company wants to get something new accomplished they “kick off” this thing called a “project”.  Now if you have spent any time in IT you have probably had a role on one of these so called projects.  You know when you are on a good project:

• Everyone knows what the end goal is.
• Everyone knows each others roles.
• Everyone has a sense that all participants are accountable for their tasks.
• Things are getting done.
• Everyone feels a sense of progress.
• Everyone feels a sense of team.

You also know when you are on a bad project:

• Everyone is seemingly bumping into each other.
• No one can really articulate the goal of the project
• No one knows what the end state is supposed to look like.
• Time is going by with meetings and minutes and status reports but no real tangible work is getting done.

This series of articles will cover different aspects on how to survive as an engineer that has a role to play on an IT project.  The previous article can be found here.

How to survive with too many chiefs or too many indians

First and foremost, make sure your role is clear on the project.  Make sure at all times you are executing the functions associated with your role.  If you are unsure if your role requires you to perform a certain function, ask around to confirm and if you aren’t getting concrete yes or no answers, confirm with your boss.  It might seem like you are annoying or bothering your boss but believe me, your boss would rather answer a 30 second role clarification question than sit in hours of meetings dancing around why someone on his team did or didn’t do a particular function on a project.

Second, if you have opportunity or are asked to perform a function outside the bounds of your role, consider all the angles before just up and completing the function.  It might seem cool to build the widget that connects the two systems together to allow the transactions to flow, but if that is not your explicit role, you might be putting your boss and your team in jeopardy.  How so?  Well, if you team is a support rather than a development group in the organization, you have just forced your boss to have more ownership in the system than he has been charged to have.  Your boss will be in political hot water when you are pulled back into the project make changes to the widget while other systems your boss is responsible for need coverage and you can’t be in both places at once.  The last thing you want to be doing is creating a headache for your boss when your boss has the most direct influence on your job duties and compensation.

Not all functions outside of your role are filled with danger.  Some are filled with opportunity for praise for helping the project move forward.  In the example above, there may not be a role that is supposed to build this widget.  Maybe the project is plugging all the technology together in order to test the final solution and someone didn’t realize these two systems needed a widget.  You could be the engineer that saves the proverbial day.  When everyone is patting each others backs when the project is successfully implemented and everyone is overly positive, someone could exclaim: “Wow, glad Bob pulled the widget out of his ear at the last moment, we thought we had a big delay on our hands!”  Yes, these moments are few and far between, but they do exist, they are fleeting, so enjoy them for the brief moment they do exist.  The key to pulling this off successfully, starting even before building a working widget, is to make 110% sure your boss is completely onboard with what you are proposing to do outside your role and the risks associated.  Be prepared for your boss to agree that you are completely capable of building the miracle widget, but because of company politics that if you tried to put them together in your mind would make you pass out, he asks you not to build the widget.  Don’t get discouraged if this happens.  There will be many projects and many opportunities to stretch your role and position yourself for exceeding expectations.  Learn from these rejections on how to better sell your boss on your ideas for the future.

In the next part of the series I’ll continue cover how to survive with too many chiefs or too many indians with one of the most powerful tools the “pre-task”.

I think it is safe to say that anytime any company wants to get something new accomplished they “kick off” this thing called a “project”.  Now if you have spent any time in IT you have probably had a role on one of these so called projects.  You know when you are on a good project:

• Everyone knows what the end goal is.
• Everyone knows each others roles.
• Everyone has a sense that all participants are accountable for their tasks.
• Things are getting done.
• Everyone feels a sense of progress.
• Everyone feels a sense of team.

You also know when you are on a bad project:

• Everyone is seemingly bumping into each other.
• No one can really articulate the goal of the project
• No one knows what the end state is supposed to look like.
• Time is going by with meetings and minutes and status reports but no real tangible work is getting done.

This series of articles will cover different aspects on how to survive as an engineer that has a role to play on an IT project.  The first article in the series can be found here.

Projects where everyone knows their role on and the goals of the project are rare.  More than likely, the project you are about to join is top heavy in one of two ways.  The first is too many chiefs and not enough indians.  Like the proverbial saying implies, plenty of people taking about the work, defining the work, breaking the work into chunks or phases or milestones or whatever is great, but if there is no one around to actually do the work, you as an engineer or as a do-er is likely to get dumped on.  The second is too many indians and not enough chiefs.  Equally dangerous because the lure of a bunch of engineers doing cool engineering stuff without chiefs provide direction, priority and governance, time and costs are ripe of getting out of control.  The next section will cover each in more detail with tips on how to avoid the prospective pitfalls.

Too Many Chiefs

A project full of managers, project managers, business analysts, requirements documenters, technical writers, testers, QA folks, support people, program office people, release coordinators, business liaisons, time keepers, relationship managers … well, you get the picture: people with functions to perform on the project but no one to actually do the engineering work itself.  As an engineer getting assigned to the project, you run the risk of getting all the “real work” dumped on you.  If you can handle all the work, great, if you can’t, there will be fifteen people with charts and graphs and emails and ten other ways to make it obvious that you dropped the ball.  In subsequent sections, I’ll over tips on how to navigate in this type of project.

Too many Indians

A project full of engineers, developers, architects and coders sounds like utopia with a whole bunch of people all doing actual work.  A bunch of people that think it is great when some new, hot off the Internet coding technique can be used to make some function perform some task at wire speed with four lines of code.  All those people cost money and consume time. People, managers, which are responsible for tracking and managing money and time don’t have an infinite amount.  At some point in time, someone, somewhere in the company is going to ask the dreaded question: “Hey, it has been four months, how close are we to implementing that new FlimFlam product?”  This is the holy crap moment for the managers that are responsible for the resources on the project.  If someone is asking that question, it means the project is in a heap of trouble.  Since that someone doesn’t already know how far along the project is, how much it is costing, how on track the project is to cost as much as originally estimated and exactly when it is going to finish, the project has failed and nothing has even been released yet.  Managers will be swooping in to ask the dreaded questions:

“What have we built?”

“How long have we been building?”

“When are we going to be done building?”

“When can this be implemented?”

If you have a bunch of engineers sitting around trying to answer those questions, the answers are not going to be what managers are expecting to hear.  More than likely, all kinds of cool stuff has been half built with no plan to pull it all together and meet the original goal of the project.  Managers are thus put in the tight spot and will react with decisions on what to do next which you are definitely not going to like.

In the next part of the series I’ll cover how to survive with too many chiefs or too many indians.

I think it is safe to say that anytime any company wants to get something new accomplished they “kick off” this thing called a “project”.  Now if you have spent any time in IT you have probably had a role on one of these so called projects.  You know when you are on a good project:

• Everyone knows what the end goal is.
• Everyone knows each others roles.
• Everyone has a sense that all participants are accountable for their tasks.
• Things are getting done.
• Everyone feels a sense of progress.
• Everyone feels a sense of team.

You also know when you are on a bad project:

• Everyone is seemingly bumping into each other.
• No one can really articulate the goal of the project
• No one knows what the end state is supposed to look like.
• Time is going by with meetings and minutes and status reports but no real tangible work is getting done.

This series of articles will cover different aspects on how to survive as an engineer that has a role to play on an IT project.

Boss: “Bob, looks like the FlimFlam Upgrade project needs a resource from our team and based on our team’s workload, I am going to give them your name.”

Bob:  <forcing a look of enthusiam> “Thanks boss, I’ll get right on it.”

If you already have some knowledge of this project from watercooler discussions or some other means and you see this as a positive assignment, you should take this opportunity to offer some sincere appreciation for the assignment.

Bob: “Boss, I apprecaite the opportunity to work on this project”

Something brief, to the point and low on the proverbial sap works just fine.  Now, if you view this as a negative assignment, I would still strongly encourage you to still offer sincere appreciation for you can’t be working on the cool projects all the time.  Making the best of the assignment will put you in a positive light in your bosses mind and can only increase your chances of being assigned something more postive next time.

If you are 100% clear on your role on this project, you are all set.  If you are unclear, don’t let your boss leave without getting that clarity:

Bob: “Boss, I just want to confirm, I am only fulfilling the project implementation into the test environment role on this project, correct?”

Believe me, your boss probably has ten things on his or her mind at the moment and probably doesn’t realize that you can’t read his or her mind as to what being a resource on the FlimFlam Upgrade project means.  You will cause more problems for your boss and then yourself if you go off and assume you are the test environment implementation person when in fact, you are only supposed to review the design of the system before it is moved into then testing phase.

Making sure you know exactly what role you fill on this particular project is critical.  Everyone on the project will most likely either assume your role is X or want your role to be Y and if you job aboard without knowing exactly what you boss expects, you will most likely fail to meet his or her expectations.

Projects where everyone knows their role and goals of the project are rare.  More than likely, the project you are about to join is top heavy in one of two ways.  The first is too many chiefs and not enough indians.  Like the proverbial saying implies, plenty of people taking about the work, defining the work, breaking the work into chunks or phases or milestones or whatever is great, but if there is no one around to actually do the work, you as an engineer or as a do-er is likely to get dumped on.  The second is too many indians and not enough chiefs.  Equally dangerous because the lure of a bunch of engineers doing cool engineering stuff without chiefs provide direction, priority and governance, time and costs are ripe of getting out of control.  The next part of the series will cover each in more detail with tips on how to avoid the pitfalls.

I bet everyone in IT has had this scenario happen to them at least once if not multiple times at any give company:

Bob the Engineer: <thinking> “This is a no brainer change.  Works fine for me on my PC.  I’ll just copy that new configuration over to the production servers and close out this work item.”

Change is made … some hours go by … then all heck breaks loose …

Frustrated User #3: “I can’t get that document out to the customer because I am getting this ‘Null Exception’ error message on my screen!”

Screaming User #5: “Why is it that every time I try and print I get this ‘Null Exception’ error?”

Irate User #2: “This is ridiculous!  Every time I try and do this my PC crashes.  Someone needs to fix this once and for all!”

Cooling off or Ready to Explode?

Thus that simple change … that no brainer change turns out to have some unforeseen downstream disaster.   To make matters worse, there is always that person that pops their head into the situation and utters that massively annoying phrase:

“Hey, did you test those changes before you implemented them?”

Of course Captain Obvious, these changes should have been tested before they were released in the wild in hindsight.  Plus, now everything that is broke in the near future will be blamed on this change no matter how technically unrelated:

“Hey, my mouse doesn’t work … is it because of that untested configuration change made the other day?”

Every user problem that is called into the IT helpdesk that can be remotely linked to this problem is assigned to you:

Jim the Helpdesk Tech: “Sure Mr. Smith, we will have someone look into this for you right away.” <I don’t want to investigate this … this kind of sounds like the config change from the other day … yah … I’ll just assign this issue to Bob and let him sort it out.>

Your hope here is that a new problem of some magnitude occurs and over shadows your problem so the helpdesk stops assigning problems to you and starts assigning them to the individual or group linked to the new problem.

While this situation is somewhat frustrating to work through to a successful conclusion at your level, it causes an even bigger problem for your boss.  As fate would have it, someone at your boss’s boss’s level was probably impacted.  Instead of calling the helpdesk like everyone else, people at those upper layers of the organization feel they are important enough to skip all the usual IT process and just call some senior IT manager, their peer, to report the problem.  Now your boss’s boss doesn’t particularly want to figure this out and thus picks up the phone and calls your boss to ask why he or she is getting calls about this problem.  Now, your boss needs a story.  If you have read some of my earlier articles, you know that the minute a problem like this pops up, you need to make a bee line to your boss and get him informed of the situation.  Assuming you did this, you still have left your boss in a proverbial pickle.

Good story for your boss to have (which he or she doesn’t in this scenario):

Boss: “Yes big boss, we developed the changes, we had them tested and the testers signed off they were ready to go, thus we are trying to figure out why they still broke in production with full testing signoff”  <the start of confusion and possibly shared blame>

The bad story you have for your boss:

Boss: “Um, yah, big boss, we developed the changes and believed them to be straight forward enough that we didn’t formally engage the testing team and just deployed them in production.” <stop and listen to the obvious “why didn’t you test” feedback> “Yes, I’ll work with the team on making sure all changes are tested going forward.”

What is the morale of the story at this point?  Always, always, always ensure you have followed the testing process of your organization no matter how confident you are in the changes you are making and your assumption that there will be no impact.  Why put yourself in this situation (unless specifically directed by your boss)?  If you don’t get some testing to back you up, you open yourself up for three negative consequences:

1. Extra work and hassel dealing with fixing what you broke and redirecting all the technically unrealted but still assigned to you Helpdesk problems.
2. Causing problems for your boss means your boss has to take some heat because of your actions [never good].  Your boss has a good memory for these events and who caused them.
3. Being labeled as a “cowboy” which is a negative perception amoungst your peers [annoying to have to deal with] and could very well lead to you being passed over by the your boss for new, cool, high profile assignments because you boss can’t risk someone going all “cowboy” on this critical new work [very difficult to get back into your boss’s good graces].

Anyone care to differ?