Application Management : Automated Business/IT Service Management

HP Software Universe - day 1

Michael_Procopio — Tue, 16 Jun 2009 02:11:00 GMT

by Michael Procopio

Today was the first day of Software Universe. I had customer meetings all day today. Here are some interesting items from my conversations.

Most said budgets were down in 2009 and will be flat to down in 2010. But a few who were related to government stimulus said theirs will be up.
Co-sourcing and outsourcing continue as ways to reduce costs
A few were focusing on asset management with the express purpose of getting rid of things in the environment they don’t need anymore. They know they are out there but they need to find them first.
Most customers I spoke to said they keep aggregated performance data for 2 years the range was 18 months to 5 years.
There was an interesting discussion about the definition of a business service versus an IT service. The point being made was a business service by definition involves more than IT. While I agree this is a good point, I think the IT industry has focused on business service as a way to say - “I’m thinking about this IT service in the context the business thinks about it not just from my own IT based perspective”
A number of customers have or are about to implement NNMi. If this is something you are interested in check out the NNMi Portal
Many customers are moving to virtualized environment highest percentage I heard was 70%. Another customer forces all internal developers to deliver software as a virtual image.
Another topic was how to monitor out tasked items. For example, some part of what you offer is delivered by a third party - how do you make sure they are living up to your standards. Two methods I heard were 1/ use HP Business Process Monitor 2/ get the 3rd party to send you alerts from their monitoring system.
On the question does your manager of managers send back data to sync the original tools 1 did, 1 didn’t. For the one who did it was part of a closed loop process.
- Monitor tool finds problem send alert to MOM (Manager of managers).
- MOM send event ID to monitoring tool
- Subject matter expert uses monitoring tools to diagnose problem
- Once diagnosed updates monitoring tool which updates MOM

A very productive day for me. I hope some of this is useful information to you.

For additional coverage my blogger buddy Pete Spielvogel is also here and beat me to the first post. You can read his posts at the ITOps Blog.

There are a variety of Twitter accounts you can follow as well as the hashtag #HPSU09

HPITOps – Covers BSM, Operations and Network Management

HPSU09 – show logistics and other information

HPSoftwareCTO

informationCTO

HPSoftware

BTOCMO – HP BTO Chief Marketing Officer

For HP BSM, Michael Procopio

Advanced analytics reduces downtime costs - detection

Michael_Procopio — Wed, 15 Apr 2009 18:01:00 GMT

by Michael Procopio

In the world of advanced analytics, two areas that are of interest to the IT management world are: detection of a problem and isolation of a problem. In this post I'll cover detection.

Problem detection, typically called anomaly detection, in the analytics circles started in a very basic way. Take a metric, say CPU utilization, set a threshold for it and anytime the threshold is crossed, we have an anomaly.

This, of course, has many problems:

· How do I know where to set a threshold

· The right level may be different at different times

· If there is a one sample spike above the threshold is that really an anomaly I care about (for some it is but not for most in my experience)

The next step in setting thresholds was using standard deviation (STD). I will create a sleeve of upper and lower bounds that cover a major percentage of the situations (+/- 1 STD covers 68.2%) I have measured and use that. This has some of the same problems as above. However, let’s focus on the time period problem.

The next step is to set thresholds by time of day. With this added capability I can set a reasonable threshold for the typical 10am and 2pm peak traffic periods separately and alerts still come if there is unusual behavior at 8am. This quickly leads to “my Mondays are busier than most of my other days”. To avoid false alerts, this leads us to time of day and day of week where we keep the standard deviation for each of the 168 hours of the week.

The next problem, the end of quarter booking and shipping madness or it is black Friday (largest shopping day of the year) and we realize we need to add in a seasonally adjusted set of thresholds as well. And something that seasonality can't take into account are macro events such as a weak economy affecting purchasing.

Of course, none of these will take into account the spikes mentioned above. How to solve all these problems -- hmmm.

There is a completely different approach, for which HP Labs has a patent filing, which uses more sophisticated machine learning. Like the other approaches, it breaks time up into segments, which, in the paper Achieving Scalable Automated Diagnosis of Distributed Systems Performance Problems, are called epochs. Unlike the other approaches, it does not simply compare now to a predetermined set of threshold levels.

This method compares ‘now’ to recent epochs as well as previous learning and makes a determination of what behavior is good and bad. While a spike is bad, if the epoch is behaving well overall it is considered to have good behavior. There is a lot of math behind this but I find looking a picture much more obvious.

Notice the gray outline the looks like a city skyline. This is what the algorithm has determine is ‘normal’ or good behavior. The hatch lines on the right show where it found an anomaly.

These advanced algorithms are implemented in HP Problem Isolation. In the next post, I’ll discuss how analytics are used to find the source of the problem.

tweet this!

BSM Evolution Paths: Financial Services Example

adsey007 — Wed, 18 Mar 2009 11:16:00 GMT

When two Fortune 500 companies merge the IT convergence can feel like two high speed trains on parallel tracks speeding toward a single-track tunnel. Not only is IT tasked with maintaining or increasing quality of service, but the CEO’s are quite impatient to quickly rationalize the IT operating expense equation of “1+1=1.25”. Maybe 1.50 if you have an extremely benevolent Board of Directors.

Unlike the Automotive Industry example posted earlier (BSM Evolution Paths: Auto Industry Sample), this Financial Services example has much less tops-down roadmap direction, and much more independent parallel paths. Let’s take a look at three of the key personas and evolutions within these parallel paths.

Data Center Operations Manager; Infrastructure Operations path:

The new Data Center Operations Manager (DCOM; reporting to VP of IT Ops) commissioned a tools architecture analysis. They inventoried their management tools and counted over 80 major “platforms” in the fault, performance and availability category alone!

The DCOM empowered a Global Software Management Architect to drive a “limited vendor” strategy to simplify and standardize the tool environment. Although there were many individual domain experts bent out of shape, this standardized environment limited the vendor touches, enabled renegotiated license/support contracts, concentrated tool expertise and resulted in improved quality of service.

The fault, performance and availability architecture was boiled down to three major vendors covering three broad categories (plus device specific element plug-ins):

System Infrastructure (Server, OS, Database, storage middleware, LAN feeds, etc.)
Network Services (WAN, LAN, advanced protocols, route analytics, etc.)
Enterprise Event (consolidated event console, correlation, filtering, root cause)

The DCOM could have pushed harder for a single vendor covering all three categories, but it was a matter of time-to-deploy pragmatism. A vendor could only be selected as category solution if the product was successfully deployed previously, and internal deployment expertise existed to lead the global implementation. This “survival of the fittest” approach did not necessarily drive the most elegant architecture, but it did speed deployment and limit risk.

Independent roadmaps and key integration capabilities were developed for each category to meet 6, 12, 18 and 24 month milestones.

CTO; Business Service Oversight path

Early on in the merger process, there was a power struggle to own the business service visibility and accountability solution. The VP of IT Operations wanted the tools, process and organizational power, but the Lines of Business insisted on a more independent group that would sit between IT Operations and the business-aligned Application Owners.

The Online Banking Group from one of the pre-merger divisions had successfully implemented a business service dashboard and Service Level Agreement reporting solution (based primarily on end-user experience monitoring). Using an “adopt and go” strategy, the CIO empowered the CTO to develop an end-to-end group and expand the solution to all six major business units.

This business unit expansion rolled out over 12-18 months and was successful, but limited to monitoring and reporting. Over the next 12 months, Application Owners, Line of Business CIO’s and VP of IT Operations all wanted to extend the business service monitoring to:

Problem isolation, application diagnostics, and incident resolution
In-depth transaction management of composite applications

Director Service Management; Enterprise CMDB path

The Director of Service Management, reporting to VP IT Ops, drove two major initiatives over the first 12 months of the merger.

Consolidate to a single, global, follow-the-sun service desk
Rationalize and standardize the request and incident management process

I could easily spend an entire blog post discussing the IT process convergence and standardization, but I refuse! Instead, I’ll focus on what happened in the 12 months following the service desk consolidation.

The Director of Service Management launched a CMDB RFP which was originally grounded in incident, problem and configuration management. The RFP touched off an enterprise-wide nerve, not to mention a flurry of vendor responses. The project quickly expanded, and changed focus to the “hotter” driver of change and (release) risk management, and how to drive all IT process from an enterprise service model.

Once the application owners got involved (from a change/release control perspective), and the infrastructure operations got involved (from a change and performance/availability perspective), and the CTO got involved (from a business service reporting and accountability rperspective) all of a sudden incident management took a back seat in the decision process.

In the end, a service discovery, dependency mapping and change/release management solution was selected that was a different vendor all together from the incumbent service desk solution.

An interesting journey… so far

The three paths described above are clearly a small subset of the overall work done for this corporate merger, but hopefully gives a glimpse into the BSM evolution dynamics. By all accounts, this company has been successful in their journey; you may be interested to know that this financial services company is not participating in the government bail-out program.

The lack of a tops-down “enterprise IT transformation” roadmap did not hinder their progress… in fact some will argue it enable their progress! You can observe, however, that at the end of each path there is a drive towards further integration and cross-IT dependence. It will be interesting to watch this company, and see how their approach evolves as they continue down the intersecting evolution paths.

Bryan Dean, BSM Research

Prediction BSM Evolution

adsey007 — Tue, 10 Mar 2009 12:32:00 GMT

I usually do not like making public predictions because I hate being wrong. But I was discussing my last blog post (Business Service Visibility & Accountability: Where is it Homed?) with a colleague and he pressed me for my prediction of where I thought this function will eventually live in the organization. Maybe more of you have the same question, so in this post I will lay out what I believe is the compelling evidence… and I might even make a prediction.

Let’s take a quick look some of the key evidence or clues:

CIO role continues to shift

This has been researched to death, but is still true. CIO’s are spending more time on business innovation and less time on production IT operations. The range of issues that CIO’s drive and influence is staggering. Does this mean they don’t care about production operations and business service accountability? No, they care greatly; it is just that most CIO’s have learned that having a top-notch, empowered VP of IT Operations is the only way to be a proactive CIO.

Application owners want to focus on development

My previous post looked historically at application owners and line of business CIO’s buying their own business service visibility and accountability tools because the pressure they felt from the business. This did happened and continues to happen in many organizations, but research shows that after a couple years of owning, architecting and maintaining these tools the application owners realize that production management tools takes valuable time away from their primary goals.

They are primarily goaled to get new functionality out the door that meets business requirements for function, quality, performance and security. It is still vitally important to the Application Owners to maintain visibility and accountability once their applications are in production. They will continue to be a catalyst in purchasing performance tools, and providing the intellectual property for rules, thresholds and reporting metrics. But ownership of the tools, configuration, vendor management and ongoing maintenance of the tools is clearly shifting to the production operations teams.

Line of Business CIO’s don’t own enough

Line of business CIO’s love to have business visibility and accountability tools in their hot hands, but they also recognize the issues of owning tools without owning the IT infrastructure. Security access and rights is a constant issue for them. Management process and tool architecture is also becoming a more standardized, centralized function that the line of business IT participates strongly in, but really is not in a position to own.

Successful customers adding problem resolution

Something I have observed in customers, who have successfully implemented a business service visibility and accountability solution, is that the next step in their evolution is to tie issue visibility to issue diagnostics and resolution. They find it is wonderful that they now have a business relevant way to measure IT service performance, but their constituents quickly move to, “Ok, now fix it when it breaks”.

Nobody in IT will be shocked to hear business takes a, “what have you done for me lately”, stance. So, the tool owners now find themselves sorting through how to integrate into the established event, incident and problem management processes. Depending on where they sit organizationally, this can be a painful yet necessary adjustment when trying to improve efficiency and time to diagnose/repair.

VP of IT Operations taking on end-to-end responsibilities

Seven years ago we conducted an extensive ITSM customer research project. At that time, there were a large number of CIO’s and industry pundits who had taken on the mantra, “Run IT like a business”. IT consolidation, adoption of ITIL process standards, organization alignment and tool deployment were all solid benefits from this era (and continue as we speak), but did not solve the issue of managing end-to-end business services.

At the time, too many IT Operations managers became “infrastructure service providers”, and when polled did not feel responsibility for the application, the end user experience, or the final business service. The CIO, and many of the application owners ended up shouldering the business service responsibility. Today, this is radically changing and ITIL V3 clearly reflects this evolution.

Application development teams continue to be organizationally aligned to the line of business more often than not, but research is showing a dramatic shift in mindset as to whom is responsible for the end-to-end application performance. The majority of IT Operations organizations today own Level 1 application monitoring and often own level 2 application support. level 3 application support typically remains aligned with the development teams, but there is no doubt that the VP of IT operations is taking on end-to-end responsibility.

Tools vendors are getting their act together

Alas, I must at least touch on technology… but only briefly! The major tools vendors have done a commendable job putting together portfolios of solutions that span the BSM/ITSM lifecycle. Plenty of improvement can still be made on integration, interoperability and ease of use; but I think it is fair to say IT finally has access to a management technology architecture that can be leveraged and multi-purposed to serve a wide range of persona needs and management disciplines.

The Prediction

You have probably guessed my prediction by now based on my biased presentation of the six clues above. I believe the ownership of the business service visibility and accountability solution will be homed under the VP of IT Operations, and purpose-specific instances will be customized for the application owners, business relationship managers, line of business CIO’s and executive IT management.

The VP of IT Operations – empowered by the CIO - will continue to drive compliance to a single, standardized IT process and software management architecture (not “single vendor”, but “limited vendors”). This will irritate many ‘best-of-breed’ fans, but in the end it will pay off.

The business service visibility and accountability function will be a module of a more comprehensive fault, performance and availability solution set that effectively ties together discovery, visibility, accountability, issue detection, isolation, diagnosis, business impact analysis and direct connection to the enterprise service model.

Implementing this cross-IT management solution will not be easy. Organizationally, the VP of IT Operations will have to empower an independent executive-level manager to drive, similar to an ERP Application owner. This will fail if owned by an “ivory tower” type, but must be practically driven by trading off the lobbying of existing IT domain and function specialists with the need to consolidate, standardize and implement a modular, multi-purposed solution.

Ok, maybe I got a little carried away at the end there, but one cannot ignore the demonstrable evidence of business, organization and persona driver dynamics. I would be surprised if we do not see a pragmatic, yet steady evolution toward this ultimate model.

BSM Evolution Paths: Auto Industry Sample

adsey007 — Tue, 24 Feb 2009 10:24:00 GMT

In the last post, Bryan Dean, our research expert in the BSM team, outlined the different ways in which customer evolve towards Business Service Management. In the next few posts, Bryan will give an example of each of the different types of evolution. Over to you Bryan ....
_______About three years ago, the business division managers of a multinational automobile manufacturing company planned a bold transformation of their distribution network to leapfrog the competition. They enthusiastically laid out a roadmap for business process innovation and aggressive customer/dealer satisfaction initiatives.

Only one real problem; the CIO knew that building, rolling-out, and operating the underlying IT for this future business vision exceeded their current capabilities. The CIO eventually had to raise the red flag and explain to the executive committee why IT was the bottleneck. Ouch, not a good day.

In the previous post BSM Evolution Paths: Samples and Observations, we talked about five common evolution paths, the organizational and persona dynamics of an Automated BSM/ITSM journey. In this post we will overview a specific example.

To be fair, the CIO spent years driving significant investment in process, tools and the organization. Let’s look at a subset of key personas and BSM/ITSM foundation:

Director of Infrastructure (reporting to the VP Global IT Ops):

Enterprise-class central event/performance platform and console
WAN/LAN network management platform
Basic, component level performance and availability reporting
Dozens of vendor-specific configuration and admin tools

Director of Service Management (reporting to the VP Global IT Ops):

Global, consolidated helpdesk/service desk
Well defined and automated incident process; basic level problem, configuration, and a manual change process

Director of Applications (development, test & level 3 support. Reports to business divisions):

Suite of pre-production stress-test quality and performance tools
End-user and application performance/diagnostic tools (test environment)

The Key Evolution Steps

Step 1: CIO empowers and holds the VP of Global IT Operations (VPITops) accountable for end-to-end business service responsibility. Imagine the panic on his face! VPITops launches key lieutenants on quick gap analysis.

Step 2: The VPITops needed a quick win. He believed that visually demonstrating and reporting performance and availability from a business service perspective -versus an infrastructure perspective- would be a catalyst for driving “aligned” IT behavior. The current network and infrastructure products didn’t have this capability, so VPITops leveraged the tools already proven by the application test and level 3 support team.

VPITops established a new team within Operations (parallel to infrastructure event management) to own and run the end-to-end business service visibility/accountability solution. Integration was established between the two teams and tools.

Step 3a: VPITops took his new business service visibility/accountability tool (in dashboard/report form) to key business division managers, and established a business relationship management function. This converted the conversation from anecdotal complaints, to measurable service levels. The CIO had tangible proof of progress.

Step 3b: While engineering step 2, the Tools and Process Architect realized they needed a better means of discovering and maintaining the IT/Business service models. Their infrastructure environment was shared, complex and dynamic enough that static service models were not effective, so they brought in an application dependency mapping technology. This success spawned a serendipitous benefit to another team in step 4a.

Step 4a: The application quality/test and release team realized the service model could be utilized in the service transition process. They previously had several very painful episodes of moving complex applications from test into production. With an accurate, up to date service model of the production environment they could better identify dependency issues before roll-out. Speed and accuracy... Happy CIO.

Step 4b: The Director of Service Management and the architect evaluated how to federate the data between the application dependency mapping service model and the CI configuration data in the helpdesk. The software vendor provided a federation / reconciliation adaptor, so the helpdesk was able to leverage the CI relationships and operate off a “single version of the truth” (sounds eerily like an ITIL V3 CMS!).

Near Term Roadmap

· Automate change/configuration workflow and provisioning

· Upgrade/replace enterprise event and performance console to leverage service model for root cause analysis and business impact assessment

· Apply business service relationship management to additional business divisions

· End-to-end visibility of composite MQ application business transactions

The Verdict of the Journey so far

The CIO still has a job, and has a funded roadmap. One might ask why they didn’t start with step 4b, and establish the CMDB and service model first? Well, the CIO was on the hot seat, and they were concerned about getting bogged down in an enterprise-wide CMDB architecture project.

This exemplifies the unpredictable and unique nature of evolution paths. More can be said about the delicate balance between tops-down guidance, and fostering organic innovation from within the ranks of IT. In future posts, I will discuss and analyze this further, as well as introduce other examples.

BSM customer evolution paths: Samples and observations

Michael_Procopio — Thu, 05 Feb 2009 01:41:00 GMT

When developing and marketing products, we often have questions which can only be answered by going out there and seeing what people are doing. We have a guy on the BSM team who does this for us. His name is Bryan Dean. I've worked with Bryan for many years and I've always been impressed by his objectivity and the insight he brings to his analysis (i.e. he doesn't just present a set of figures - he gets behind the figures).

At the end of last year, we asked Bryan to analyze the top 20-odd BSM deals of 2008. He formed a number of conclusions from this research. One set of conclusions concerned how people "get to BSM" - how they evolve towards an integrated BSM solution. I asked Bryan to help me with a series of posts to share what he learnt about evolutions towards BSM because I think that knowing what our other BSM customers are doing may help you.

________

Mike: Bryan, can you give a summary of what you learnt?

Bryan: There is no one evolution path. It's fascinating to me that a hundred different IT organizations can have virtually the same high-level goals, fundamentally agree on the key factors for success, and yet end up with a hundred unique execution paths.

Before I answer your question, can I create a definition? The term "BSM" is very poorly defined within the IT industry - different vendors have different versions, and so do the industry analysts (in fact, some other research I did last year concluded that very few people had a clear idea of what BSM means). So, I'd like to introduce the term "Automated Business/IT Service Management" or AB/ITSM.

Back to your question, I think I can group all the different evolution paths into five key types:

ITSM incident, problem change & configuration: this evolution is driven out of the need for process-driven IT service management with the service desk as a key component
Consolidated infrastructure event, performance and availability: this is driven by a recognition that having a whole ton of event management and performance monitoring systems is not an efficient way to run IT, and so there is a drive to consolidate them into one console.
Business service visibility & accountability: this is more of a top-down approach - start with monitoring the customer's quality of experience and then figure out what needs to happen underneath. This is popular in industries where the "web customer experience" is everything - if it's not good, you lose your business
Service discovery & model: this is where evolution towards integration is driven from the need for a central model (the CMDB). Often, the main driver for such a central model is the need to control change
Business transaction management: today, this is the rarest starting point. It's driven by a need to monitor and diagnose complex composite transactions. We see this need most strongly in the financial services sector

Mike: How about the politics of such AB/ITSM projects? (I don't see the AB/ITSM term taking hold, by the way :-) )

Bryan: Politics (or, most specifically, the motivational side) is important. I think many heavy thinkers in our industry have the mistaken assumption that that there is a single evolution path, controlled from the top on down by the CIO following a master plan. Trying to manage such a serialized, mega project is a huge challenge and too slow, not to mention that 99% of CIO’s are not in the habit of forcing tactical execution edicts on their lieutenants (I know I’ll get some argument on that one :-) ).

What I see from my research is that the most successful IT organizations are those who have figured out how to balance between discrete doable projects, and an overall AB/ITSM end-goal context and roadmap. Typically, the CIO lays down a high-level vision that ties to specific business results, and then allows key lieutenants to assess and drive a prioritized set of federated, manageable projects that independently drive incremental ROI. Some IT organizations may have a well-defined integrated roadmap, but the majority of IT run federated projects in a fairly disjointed fashion.

These parallel paths are owned by many independent personas within IT, each trying to solve the specific set of issues at hand. For them, being bogged down in how their federated project aligns and integrates with all the other AB/ITSM projects is daunting… if not fatal.

And on reflection this makes sense to me - the human side of things plays a large role in such endeavors.

Mike: What do you mean?

Bryan: IT organizations of all shapes and sizes have goals to reduce costs, increase efficiency, improve business/IT service quality, and mitigate risk all while applying technology in an agile way to boost business performance. What I find interesting is how specific, funded initiatives are created by specific personas to achieve the goals.

In future posts, I will share some specific examples of how customers evolved through these paths, the key driver personas, the core motivations and how these paths come together.