Midwest IT Survival

Integration going South? Time to call the Vendor Sales Cheese!

Whether you are working in a complete custom software development shop with little vendor interaction or a technology integration shop with vendor solutions integrated with other vendor solutions on top of yet other vendor solutions, you will have to manage vendor relationships to some degree as an IT manager in a MidWestern company.  This series looks at the complex arena of IT vendor management and offers some tips to make the arduous process a bit less arduous and possibly discover some additional benefits along the way.

Vendor Management Categories

1. Vendor Service Integration Challenges

In the previous article, I suggested some approaches to consider as the project is underway to integrate the vendor’s service with you and your team’s technology service.  This article will explore considerations once you are in the throws of integration.

So, you have your prototyping working to reduce the unknowns associated with how to get the two disparate technologies functionally working together.  You also have your “spaghetti infrastructure” in place to reduce the risk of negative impacts to your service’s ability to continue to perform for existing needs as well as supporting the new vendor integration.  Finally, you have your “vendor simulator” setup to conduct various permutations of performance testing scenarios between you and your team’s service and the vendor’s service.  So what else do you need?

In parallel to all of the above rather technical risk mitigation techniques, you need an additional management tool to address adverse scenarios that will inevitably pop up randomly throughout the integration project.  The tool I am referring to is a thorough understanding of the vendor’s organizational chart.  Specifically, you are looking to identify the following roles within the vendor’s organization:

• Relationship Manager assigned to your company

This is the individual whose job it is to ensure that your company is happy at all times with the service the vendor is providing to your company.  This is an individual who you can leverage any time you believe additional support from the vendor would allow for accomplishing things more efficiently and effectively.  If things are trending poorly and you have exhausted the contacts and procedures to get support from the vendor, don’t hesitate to contact this individual.  They should react immediately to your needs, especially if you can summarize your thorough attempt to use the normal channels and your coming up proverbially empty.  On word of caution, make sure you exhaust the established parameters for vendor resource and assistance engagement.  If you call on this person at every bump in the road, you may get high touch service immediately, but you will quickly be labeled as a “hyper escalator” and your high priority requests will be interpreted as low priority.  This is indeed a balance, for if you don’t leverage this individual and things come to a head down the road, the vendor has an easy out for providing poor service:

Vendor Sales Cheese <role defined later in this article>: “I understand you are frustrated.  You do know you can contact Bob for all your customer service needs.  He is your Relationship Manager dedicated to you and from what I understand; no one reached out to Bob.  If you do contact Bob, he will make sure you are getting the needed service and support from our organization …”

Regardless of your level of credible frustration, you didn’t take advantage of the escalatory options at your disposal from the vendor:

• Senior Technical Person

This is the role you need to find quickly.  The vendor will most likely try to shield this person.  Why shield?  Well, if every customer knew how to get a hold of Senior Technical Person, they would for their every need because they get the real, direct and reliable technical answer quickly.  But like every technical organization, these roles are few and far between and worth protecting so only the most critical problems get into their hands.  This being said, you need to quickly identify this person and establish a back channel communications mechanism between your senior tech person and the vendor senior tech person.  Similar to the Relationship Manager role, this is a “don’t use unless you absolutely have too” communications route.  Make sure you coach your senior tech resource to play along with only contacting their peer at the vendor when the regular channels breakdown.  On the flip side, the vendor senior technical resource will appreciate being able to speak with a technical peer within the customer’s organization.  Just like your senior tech resource, the vendor’s equivalent will be frustrated with the problems that are beneath them to solve crossing their desk.  They will find it refreshing to speak with someone who shares their priority attributed to technical excellence and will provide top notch support when engaged as such.  This back channel communication mechanism can be exceedingly valuable when milestone dates are looming and confounding technical problems block forward progress.

• Vendor Project Manager

Assuming the integration effort is of a scale that involves more than a few humans completing a few tasks to integrate technologies, there will most likely be a project manager within your organization (see other series of articles on interacting within your project management framework for IT Engineers and IT Managers ).  On the vendor side, they will most likely have a Vendor Project Manager role. This role keeps the vendor resource moving forward within the vendor’s organization in parallel with your organization’s project manager.  Don’t be quick to conclude that your company’s project manager will work effectively with their peer within the vendor’s organization.  It is best to assume your project manager will fail to grasp the nuances of the technical tasks and dependencies and thus fail to get critical information passed back and forth between the two organizations successfully.  I am not suggesting you need to absorb the project management function within yourself or your team, rather, develop a communication approach that takes this miscommunication potential into account.  For example, consider adding the Vendor Project Manager as a carbon copy to all emails that make even the slightest reference to questions, answers, tasks or dependencies that involve the vendor.  For example, pass on mentioning that Bob on your team needs to take the internal account access form to the access granters for Bob to get access to that new server he doesn’t have access to yet.  This doesn’t involve the vendor in the slightest.  But, if the vendor is expecting a copy of the contents of a configuration file on that new server, then by all means add the Vendor Project Manager to the cc line of the email to your company’s project manager.  Why this seemingly superfluous communication of ancillary internal access granting minutia?  The perception becomes more clear that the vendor is waiting on the configuration file snapshot, which your team needs to provide, yet clearly can’t until access is granted to Bob.  Without this documented clarity, the Vendor Project Manager can send a heated email to your project manager claiming your company is holding up the project due to not providing the configuration file.  Your project manager, under duress, may be quick to claim you and your team are the hold up within your company.  Without this example email, you have to get on the defensive, especially to the classic “surprised and confused” response from the under duress project manager:

Company Project Manager: “What?  You needed server access in order to send over the configuration file that has been holding everyone up?  I’m surprised that you needed this access and I’m confused that you didn’t tell me about needing it.  For had I known you needed this access, I would have worked to get you that access ASAP.  Since I didn’t know about it, I couldn’t have known that this was a barrier to progress …”

The email including letting the Vendor Project Manager know all of the dependencies in order to provide the deliverable makes it obvious the communication breakdown is outside you and your team so you survive unscathed and don’t have to invest energy in forming a tactical defense or erect proverbial blame shields.

• Vendor Sales Cheese

Similar to the Relationship Manager role, you need to identify the Vendor Sales Cheese or salesperson that stands to loose big in the compensation category if the customer gets frustrated and doesn’t close the deal for him or her to score their bonus.  Depending on the vendor, the Vendor Sales Cheese or the Relationship Manager maybe the same person.  But in the event they are different people, make sure you have a clear understanding of their roles and follow the below sequence for every vendor request, only proceeding to the next step if the prior step failed to generate expected results:

1. Normal communication route (support ticket, email to vendor support mailbox, etc.) keeping the Vendor Project Manager copied on communications
2. Vendor Project Manager directly
3. Vendor Relationship Manager
4. Vendor Sales Cheese

Lastly, knowing who within your company owns the relationship with this vendor, as described in previous articles,  is also important so when you get a sense the communications are getting more heated as you are moving from the normal communication route, past the Vendor Project Manager over to the Vendor Relationship Manager and Vendor Sales Cheese, you can bring the vendor relationship owner into the mix when their vendor is at risk of causing the integration to go off track or miss a major project milestone.  If the integration project is trending in this direction, you will want to have all of the communications documented and reflecting a logical progression through the escalation path confirming to all that you and your team took every opportunity to engage the correct resources to not be the cause of the pending failure.

Knowing the contact information for all the vendor players identified in the roles above will allow you and your team to effectively manage the communications and escalatory options between your company and the vendor’s organization.  Relying on others puts you and your team at risk for being perceived as the service provider that is holding up the integration project and diverts focus from getting the technical integration completed.  The next article will dive deeper into more MidWestern IT perspectives on the topic of the “Role of the Sales Rep” in the spectrum of vendor management.

Whether you are working in a complete custom software development shop with little vendor interaction or a technology integration shop with vendor solutions integrated with other vendor solutions on top of yet other vendor solutions, you will have to manage vendor relationships to some degree as an IT manager in a MidWestern company.  This series looks at the complex arena of IT vendor management and offers some tips to make the arduous process a bit less arduous and possibly discover some additional benefits along the way.

Vendor Management Categories

• Vendor Service Integration Challenges

In the previous article, I described aspects the vendor due diligence process and mitigating technology integration risks with a vendor after a contract is signed with the vendor and  you and your team didn’t get full participation in the vendor selection process.   In this article, I suggest some approaches to consider as the project is underway to integrate the vendor’s service with you and your team’s technology service.

In the previous article, I started to hint at ways to accommodate the usually, less than optimal way one becomes part of a vendor integration effort with my “spaghetti infrastructure” example.  In short, “spaghetti infrastructure” refers to looking to add sub par infrastructure extensions in order to accommodate the variable production volume/usage patterns of this new vendor integrated service to your existing technology platform or service for which you and your team are responsible.  I also mentioned the concept of using a technology prototype to reduce the risk of the technical variables related to integrating two separately owned and designed technology frameworks.  Below is an additional vendor integration challenge and a way to attack this challenge head on:

• Need to confirm the integrated solution will perform to production expectations

“Spaghetti infrastructure” and prototypes are tools to reduce the unknowns associated with integrating you and your team’s technology service with a vendor’s technology service.  But how does one address the question: “Will this integrated solution perform to the level needed to meet production usage SLAs and response times?”  Consider the development of a “vendor simulator”.

What is a “vendor simulator” do you say?  A “vendor simulator” represents all of the integration touch points between you and your team’s technology service and the vendor’s.  Expecting the vendor to have a dedicated test environment that you can use at a moments notice that is pre-populated with synthetic transactions that reflect the sample data you have within you and your team’s test environment including predictive data that runs the gamut of possible query responses is rather unrealistic.  Thus, if you invest some time and energy in building an application or service that can mimic the inputs and corresponding outputs of the vendor’s service, then you have a tool to test your service prior to engaging the vendor’s testing resources.  Below is a simple, high level graphical example of simplistic company/vendor integration architecture:

Note: The above architecture is exceedingly simple.  In real world architectures, there could be many components and technologies between you and your team’s service and the vendor’s technology.

With the above simplistic architecture in mind, below is an example use case scenario to setup for our “vendor simulator”:

Use Case Example #1

1. Create request transaction to pass to the vendor that includes:
   1. Authentication/Authorization information to establish security context between you and your team’s service and the vendor’s service
   2. A “customer identifier” that exists within the vendor’s service
   3. A “Get the last ten transactions” request
2. Request, via the vendor gateway, the above transaction be processed by the vendor
3. Receive a synchronous response from the vendor, for the provided customer identifier, for the ten strings of text that represent this customer’s most recent ten transactions recorded in the vendor’s service
4. Close the connection with the vendor’s service

The above use case could represent a potential real world scenario of a company web site that allows their customers to view all transactions conducted against all services the company offers the customer.  In this case, a particular company product has been “outsourced” to a vendor, but the company desires to have the company web site show the outsourced product transactions within the list of all transactions the customer has performed using all their company enrolled products.

If there were only 1,000 customers involved, the production volumes would be easy to incorporate within a systems design.  But what if there were 100,000?  Or 1,000,000?  Or even 10,000,000?  In the later cases, the production volumes are indeed significant and require a properly designed, optimized and tuned solution.  So where should the “vendor simulator” architecturally exist?  See the below graphic:

Ideally, your “vendor simulator” should exist within your technology framework as close to the end point where your network “links” or “touches” the vendor’s network.  Sure, someone might suggest the “vendor simulator” should hang off the edge firewall in the graphic, but the latency that firewall would add to the simulated request round trips is negligible in the grand scheme of our simulation need.

Great, we have found the ideal location to insert our “vendor simulator”, now what should the “vendor simulator” actually do?

Using our Example Use Case #1 above, the “vendor simulator” would be built to do the following:

1. Accept the first transaction that includes authentication and authorization data and the customer identifier and the request for the 10 transactions
   1. Note: if you would find value from implementing some validation scheme the vendor has indicated they would perform on transactions, feel free to replicate that validation here.  You might find in implementing their validation, your construction of the message payload is faulty and thus you can correct before interacting with the vendor
2. Allow for a random yet within a minimum/maximum range delay latency for the processing of the request within the vendor’s technology and the communications between your company and the vendor
3. Provide 10 artificial transactions to return to you and your team’s service that consist of test data that represents a generous sampling of the plausible data the vendor’s service would return.
4. Close out the request

Note on step 2:  The goal of this step is to provide an ability to simulate the latency or slowness of the network to and from your company and the vendor as well as the time it takes the vendor to receive, validate, process, format and compile the results and return them to your company.  By making those latency elements configurable, specifically identifying a minimum and maximum range and then randomly picking within the ranges per test transaction, the “vendor simulator” can get close to reflecting real world production interactions.

With the “vendor simulator” configured as above, you can run any number of system tests that include interacting with the vendor yet there is no need to engage the vendor for each and every test.

In addition, you can change one variable at a time within the “vendor simulator” to determine how you and your team’s service will perform and/or react to the change in behavior of the “vendor simulator”.  Try having the “vendor simulator” perform close to wire speed.  You will be able to see you your system responds to transactions being requested and completed extremely quickly.  Next, try having the “vendor simulator” perform extremely slowly by greatly extending the minimum simulator response times.  Does your system start backing up transactions as the “vendor simulator” fails to process their requests promptly?  Additionally, create a huge time delta between the minimum and maximum “vendor simulator” response times.  This simulates the vendor having some systems issues where they can process transactions, but their processing is erratic as if they are not having a systems outage, rather, capacity issues that cause requests to be processed with great variety in the response times.  Can you service, under load, successfully “smooth” the variability created by the vendor or does your system begin to back up transactions, then flush them out periodically in parallel to the “vendor simulator”?

All of the above testing data will allow you to tweak and tune you and your team’s service’s operating parameters to achieve the most efficient systems configuration to handle the widest variety of vendor performance scenarios without having to have the vendor initially involved.

Next steps?

When you are confident your system is optimally tuned to handle the chaotic responses from your “vendor simulator”, proceed to engage your vendor in formal integration testing and you will have the ability to relate the combined company and vendor testing results with your “vendor simulator” experience.  Based on the results, you should be able to see problems the vendor is experiencing based on the simulated problems your “vendor simulator” generated for you and your team’s service.  See a performance pattern you didn’t see before?  Go back to the “vendor simulator” and try additional configurations that achieve the performance pattern you experienced with the vendor and proceed to tune and tweak you and your team’s service to handle this scenario in a more efficient manner.

Whew, now you are in production and things seem to be working a-ok, throw the “vendor simulator” away, right?

Absolutely not!  Now you have the great opportunity to observe the production usage patterns and configure the “vendor simulator” to replicate those patterns in your test environment.  Once you have replicated the production patterns in test, you can now perform system maintenance or upgrades and leverage the “vendor simulator” again to predict how your system changes will impact the production vendor integration with a higher degree of confidence.

One caution, the “vendor simulator” isn’t a replacement for testing with your vendor partner.  Rather, it is an efficiency tool that can help rule out problems prior to spending the time and energy (and money depending on your vendor contract arrangement for testing) spinning up all the resources within your company and the vendor’s company to conduct full systems integration testing.

Although a pre-signed contract and delayed vendor integration project involvement can create some challenges, consider some of these sub optimal but success trending options.  You may want to create “vendor simulator” regardless of you ability, up front, to build into the project schedule, a comfortable testing exercise.  The next article will dive into more MidWestern IT perspectives on the topic of “Vendor Service Integration Challenges” in addition to this article.

"Spaghetti Infrastructure" can save the day!

Whether you are working in a complete custom software development shop with little vendor interaction or a technology integration shop with vendor solutions integrated with other vendor solutions on top of yet other vendor solutions, you will have to manage vendor relationships to some degree as an IT manager in a MidWestern company. This series looks at the complex arena of IT vendor management and offers some tips to make the arduous process a bit less arduous and possibly discover some additional benefits along the way.

Vendor Management Categories

• Who Owns the Relationship

In the previous article, I described aspects the vendor due diligence process and mitigating technology integration risks with a vendor prior to signing a contract with the vendor. In this article, I suggest some approaches to consider when the partnership between the business and IT is weak and a contract for services has been signed with the vendor without full IT due diligence participation prior.

Has the Contract been Signed? Yes!

Let us be honest, in a perfect world, all critical stakeholders would be engaged at the onset of the vendor selection process. A disciplined procurement process would unfold measuring and weighting all the pros and cons of each solution. Internally transparent decision matrices would ultimately reflect a cross-functional consensus based vendor selection decision. MidWest IT is definitely not a perfect world. Many factors, most not based on logic or metrics, rather emotion, drive the vendor selection process.

If the contract has already been signed, the rules of engagement are becoming established and the boundaries by which you get to interact with the vendor are becoming even more defined. Unfortunately, defined in the sense that integration expectations have already been set, but the specific details are unknown. If the contract has been signed and work has been underway for some time but you have been recently engaged, you are not going to be very popular when you pop up with:

You: “Oh, we have to integrate our technology with Vendor X to provide that? We are going to have to buy all new servers and upgrade that old version of the FlimFlam platform to 9.0! and … and …”

Budgets have been set, integration costs have been projected and expected revenues already sold to the organization. Although not popular, you have to be somewhat transparent in how the vendor integration will put stress on you and your team’s technical services and how to mitigate that stress. Yes, in my experience in shared IT services, this is a rather frequent occurrence. An effort is underway without involving all the stakeholders at the onset. As you are identified as a stakeholder down stream in the project process, you didn’t get to participate early when including your service impacts into the mix would have resulted in a more rational inclusion of your service extension needs within the effort’s requirements framework. Not to fear, there are some options to allow you to mitigate the risks to you and your team’s service.

Keys to improved success in this hypothetical example would be to smooth the impact in proportion to the usage ramp up. Is the ramp up slow enough to allow use of your service in its current configuration but plan for the next budget cycle to incorporate the purchase of additional capacity to support this service growth? The spreading of the impact to your service over budget cycles allows the initial implementation to continue as planned and essentially pushes the “problem” of the need for additional capacity to a corporate exercise that expects to digest investments in capacity to support continued growth.

Another example of improving the chances of a successful outcome without appearing to need significant unplanned investment on meeting the need is to look to leverage the ever declining cost of infrastructure. Especially with today’s service oriented architecture flexibility, is it possible to add additional infrastructure capacity on the cheap? Is there virtual capacity that you can take advantage of now and then plan to fold your virtual architecture extension back into your overall architecture direction? Are there older, decommissioned servers that, not the latest and greatest, could absorb the initial vendor integration work and usage projections and again, allow for folding back into your overall architectural direction in the future?

Yes, the cost of “spaghetti infrastructure” needs to be weighed against the benefits of architecturally separating the design, development, testing and performance tuning from your existing service. Plus, experience suggests that during the integration and testing phase before deployment, one gets a projection of the production usage pattern but rarely is that projection spot on. More than likely, there are big swings in production usage from those estimated from the integration project. Anywhere from way fewer users performing transactions to a huge spike in volume based on pent up user demand. Your “spaghetti infrastructure” investment can pay off with real production workload pattern data allowing you to strategize on a more strategic and holistic architecture that can support the current workload and this new workload. Don’t be too quick to dismiss this option as a step back on your architectural roadmap. Rather, consider the ability to gather production data as a way to strengthen your architectural roadmap.

Although a pre-signed contract and delayed vendor integration project involvement can create some challenges, consider some of these sub optimal but success trending options. The next article will dive into the MidWestern IT perspective on the topic of “Vendor Service Integration Challenges” touched on at the end of this article.