Midwest IT Survival

I bet everyone in IT has had this scenario happen to them at least once if not multiple times at any give company:

Bob the Engineer: <thinking> “This is a no brainer change.  Works fine for me on my PC.  I’ll just copy that new configuration over to the production servers and close out this work item.”

Change is made … some hours go by … then all heck breaks loose …

Frustrated User #3: “I can’t get that document out to the customer because I am getting this ‘Null Exception’ error message on my screen!”

Screaming User #5: “Why is it that every time I try and print I get this ‘Null Exception’ error?”

Irate User #2: “This is ridiculous!  Every time I try and do this my PC crashes.  Someone needs to fix this once and for all!”

Cooling off or Ready to Explode?

Thus that simple change … that no brainer change turns out to have some unforeseen downstream disaster.   To make matters worse, there is always that person that pops their head into the situation and utters that massively annoying phrase:

“Hey, did you test those changes before you implemented them?”

Of course Captain Obvious, these changes should have been tested before they were released in the wild in hindsight.  Plus, now everything that is broke in the near future will be blamed on this change no matter how technically unrelated:

“Hey, my mouse doesn’t work … is it because of that untested configuration change made the other day?”

Every user problem that is called into the IT helpdesk that can be remotely linked to this problem is assigned to you:

Jim the Helpdesk Tech: “Sure Mr. Smith, we will have someone look into this for you right away.” <I don’t want to investigate this … this kind of sounds like the config change from the other day … yah … I’ll just assign this issue to Bob and let him sort it out.>

Your hope here is that a new problem of some magnitude occurs and over shadows your problem so the helpdesk stops assigning problems to you and starts assigning them to the individual or group linked to the new problem.

While this situation is somewhat frustrating to work through to a successful conclusion at your level, it causes an even bigger problem for your boss.  As fate would have it, someone at your boss’s boss’s level was probably impacted.  Instead of calling the helpdesk like everyone else, people at those upper layers of the organization feel they are important enough to skip all the usual IT process and just call some senior IT manager, their peer, to report the problem.  Now your boss’s boss doesn’t particularly want to figure this out and thus picks up the phone and calls your boss to ask why he or she is getting calls about this problem.  Now, your boss needs a story.  If you have read some of my earlier articles, you know that the minute a problem like this pops up, you need to make a bee line to your boss and get him informed of the situation.  Assuming you did this, you still have left your boss in a proverbial pickle.

Good story for your boss to have (which he or she doesn’t in this scenario):

Boss: “Yes big boss, we developed the changes, we had them tested and the testers signed off they were ready to go, thus we are trying to figure out why they still broke in production with full testing signoff”  <the start of confusion and possibly shared blame>

The bad story you have for your boss:

Boss: “Um, yah, big boss, we developed the changes and believed them to be straight forward enough that we didn’t formally engage the testing team and just deployed them in production.” <stop and listen to the obvious “why didn’t you test” feedback> “Yes, I’ll work with the team on making sure all changes are tested going forward.”

What is the morale of the story at this point?  Always, always, always ensure you have followed the testing process of your organization no matter how confident you are in the changes you are making and your assumption that there will be no impact.  Why put yourself in this situation (unless specifically directed by your boss)?  If you don’t get some testing to back you up, you open yourself up for three negative consequences:

1. Extra work and hassel dealing with fixing what you broke and redirecting all the technically unrealted but still assigned to you Helpdesk problems.
2. Causing problems for your boss means your boss has to take some heat because of your actions [never good].  Your boss has a good memory for these events and who caused them.
3. Being labeled as a “cowboy” which is a negative perception amoungst your peers [annoying to have to deal with] and could very well lead to you being passed over by the your boss for new, cool, high profile assignments because you boss can’t risk someone going all “cowboy” on this critical new work [very difficult to get back into your boss’s good graces].

Anyone care to differ?

No related posts.