Whether you are working in a complete custom software development shop with little vendor interaction or a technology integration shop with vendor solutions integrated with other vendor solutions on top of yet other vendor solutions, you will have to manage vendor relationships to some degree as an IT manager in a MidWestern company. This series looks at the complex arena of IT vendor management and offers some tips to make the arduous process a bit less arduous and possibly discover some additional benefits along the way.
Vendor Management Categories
- Vendor Service Integration Challenges
In the previous article, I described aspects the vendor due diligence process and mitigating technology integration risks with a vendor after a contract is signed with the vendor and you and your team didn’t get full participation in the vendor selection process. In this article, I suggest some approaches to consider as the project is underway to integrate the vendor’s service with you and your team’s technology service.
In the previous article, I started to hint at ways to accommodate the usually, less than optimal way one becomes part of a vendor integration effort with my “spaghetti infrastructure” example. In short, “spaghetti infrastructure” refers to looking to add sub par infrastructure extensions in order to accommodate the variable production volume/usage patterns of this new vendor integrated service to your existing technology platform or service for which you and your team are responsible. I also mentioned the concept of using a technology prototype to reduce the risk of the technical variables related to integrating two separately owned and designed technology frameworks. Below is an additional vendor integration challenge and a way to attack this challenge head on:
- Need to confirm the integrated solution will perform to production expectations
“Spaghetti infrastructure” and prototypes are tools to reduce the unknowns associated with integrating you and your team’s technology service with a vendor’s technology service. But how does one address the question: “Will this integrated solution perform to the level needed to meet production usage SLAs and response times?” Consider the development of a “vendor simulator”.
What is a “vendor simulator” do you say? A “vendor simulator” represents all of the integration touch points between you and your team’s technology service and the vendor’s. Expecting the vendor to have a dedicated test environment that you can use at a moments notice that is pre-populated with synthetic transactions that reflect the sample data you have within you and your team’s test environment including predictive data that runs the gamut of possible query responses is rather unrealistic. Thus, if you invest some time and energy in building an application or service that can mimic the inputs and corresponding outputs of the vendor’s service, then you have a tool to test your service prior to engaging the vendor’s testing resources. Below is a simple, high level graphical example of simplistic company/vendor integration architecture:
Note: The above architecture is exceedingly simple. In real world architectures, there could be many components and technologies between you and your team’s service and the vendor’s technology.
With the above simplistic architecture in mind, below is an example use case scenario to setup for our “vendor simulator”:
Use Case Example #1
- Create request transaction to pass to the vendor that includes:
- Authentication/Authorization information to establish security context between you and your team’s service and the vendor’s service
- A “customer identifier” that exists within the vendor’s service
- A “Get the last ten transactions” request
- Request, via the vendor gateway, the above transaction be processed by the vendor
- Receive a synchronous response from the vendor, for the provided customer identifier, for the ten strings of text that represent this customer’s most recent ten transactions recorded in the vendor’s service
- Close the connection with the vendor’s service
The above use case could represent a potential real world scenario of a company web site that allows their customers to view all transactions conducted against all services the company offers the customer. In this case, a particular company product has been “outsourced” to a vendor, but the company desires to have the company web site show the outsourced product transactions within the list of all transactions the customer has performed using all their company enrolled products.
If there were only 1,000 customers involved, the production volumes would be easy to incorporate within a systems design. But what if there were 100,000? Or 1,000,000? Or even 10,000,000? In the later cases, the production volumes are indeed significant and require a properly designed, optimized and tuned solution. So where should the “vendor simulator” architecturally exist? See the below graphic:
Ideally, your “vendor simulator” should exist within your technology framework as close to the end point where your network “links” or “touches” the vendor’s network. Sure, someone might suggest the “vendor simulator” should hang off the edge firewall in the graphic, but the latency that firewall would add to the simulated request round trips is negligible in the grand scheme of our simulation need.
Great, we have found the ideal location to insert our “vendor simulator”, now what should the “vendor simulator” actually do?
Using our Example Use Case #1 above, the “vendor simulator” would be built to do the following:
- Accept the first transaction that includes authentication and authorization data and the customer identifier and the request for the 10 transactions
- Note: if you would find value from implementing some validation scheme the vendor has indicated they would perform on transactions, feel free to replicate that validation here. You might find in implementing their validation, your construction of the message payload is faulty and thus you can correct before interacting with the vendor
- Allow for a random yet within a minimum/maximum range delay latency for the processing of the request within the vendor’s technology and the communications between your company and the vendor
- Provide 10 artificial transactions to return to you and your team’s service that consist of test data that represents a generous sampling of the plausible data the vendor’s service would return.
- Close out the request
Note on step 2: The goal of this step is to provide an ability to simulate the latency or slowness of the network to and from your company and the vendor as well as the time it takes the vendor to receive, validate, process, format and compile the results and return them to your company. By making those latency elements configurable, specifically identifying a minimum and maximum range and then randomly picking within the ranges per test transaction, the “vendor simulator” can get close to reflecting real world production interactions.
With the “vendor simulator” configured as above, you can run any number of system tests that include interacting with the vendor yet there is no need to engage the vendor for each and every test.
In addition, you can change one variable at a time within the “vendor simulator” to determine how you and your team’s service will perform and/or react to the change in behavior of the “vendor simulator”. Try having the “vendor simulator” perform close to wire speed. You will be able to see you your system responds to transactions being requested and completed extremely quickly. Next, try having the “vendor simulator” perform extremely slowly by greatly extending the minimum simulator response times. Does your system start backing up transactions as the “vendor simulator” fails to process their requests promptly? Additionally, create a huge time delta between the minimum and maximum “vendor simulator” response times. This simulates the vendor having some systems issues where they can process transactions, but their processing is erratic as if they are not having a systems outage, rather, capacity issues that cause requests to be processed with great variety in the response times. Can you service, under load, successfully “smooth” the variability created by the vendor or does your system begin to back up transactions, then flush them out periodically in parallel to the “vendor simulator”?
All of the above testing data will allow you to tweak and tune you and your team’s service’s operating parameters to achieve the most efficient systems configuration to handle the widest variety of vendor performance scenarios without having to have the vendor initially involved.
When you are confident your system is optimally tuned to handle the chaotic responses from your “vendor simulator”, proceed to engage your vendor in formal integration testing and you will have the ability to relate the combined company and vendor testing results with your “vendor simulator” experience. Based on the results, you should be able to see problems the vendor is experiencing based on the simulated problems your “vendor simulator” generated for you and your team’s service. See a performance pattern you didn’t see before? Go back to the “vendor simulator” and try additional configurations that achieve the performance pattern you experienced with the vendor and proceed to tune and tweak you and your team’s service to handle this scenario in a more efficient manner.
Whew, now you are in production and things seem to be working a-ok, throw the “vendor simulator” away, right?
Absolutely not! Now you have the great opportunity to observe the production usage patterns and configure the “vendor simulator” to replicate those patterns in your test environment. Once you have replicated the production patterns in test, you can now perform system maintenance or upgrades and leverage the “vendor simulator” again to predict how your system changes will impact the production vendor integration with a higher degree of confidence.
One caution, the “vendor simulator” isn’t a replacement for testing with your vendor partner. Rather, it is an efficiency tool that can help rule out problems prior to spending the time and energy (and money depending on your vendor contract arrangement for testing) spinning up all the resources within your company and the vendor’s company to conduct full systems integration testing.
Although a pre-signed contract and delayed vendor integration project involvement can create some challenges, consider some of these sub optimal but success trending options. You may want to create “vendor simulator” regardless of you ability, up front, to build into the project schedule, a comfortable testing exercise. The next article will dive into more MidWestern IT perspectives on the topic of “Vendor Service Integration Challenges” in addition to this article.