Application Management : application performance management

RUM the Real User Experience Manager

Michael_Procopio — Wed, 07 Oct 2009 02:20:00 GMT

by Michael Procopio

RUM or Real User Monitor is a tool to monitor actual user traffic running over your network.

Its part of our EUM or end user management suite. In the area of EUM there are two primary ways to monitor 1/ synthetic, which is covered by BPM or Business Process Monitor and 2/ real user monitoring.

Each has its place in a monitoring strategy. BPM is good for making sure things are up 24x7, even when no users are using your applications. Real user monitoring can give you information down to the specific user.

When I first moved over to BAC group and heard about RUM, I was impressed. One of the things it can do is replicate a users web session click by click. This allows someone troubleshooting a problem to see exactly what happened and what the error message was the user saw – no guessing. (sensitive data like passwords and credit cards are filtered out in memory before writing to a disk). Further, if you do find a problem it can turn the session into a script that can be passed to the QA team to replicate the problem, if they are using LoadRunner.

How does it work? It starts by capturing packets as they go over the network. This is done by a RUM Probe, which is software that runs on a dedicated piece of x86 hardware (typically). The Probe passes the relevant data to the RUM engine.

The RUM engine stores the data and key performance metrics are aggregated before being sent up to BAC for reporting and alerting. For example, an alert might be round trip time for the Savings Deposit transaction is taking too long. Here are some of the reports RUM provides:

Global Statistics
Page Summary
Transaction Summary
End User Summary
End User Over Time
Server Over Time
Session Analyzer
TCP Application Summary
TCP Application Over Time
Event Summary
Business Process Distribution

Figure: Example RUM deployment configuration

Originally RUM strictly focused on HTTP/S traffic. But a while back support was expanded to due general tracking of TCP traffic, both streaming and non-streaming. In more recent releases additional upper level protocol analysis has been added. Beyond HTTP/S current support includes:

XML/SOAP
Siebel
WebSphere
MPLS

Related Items:

RUM webinar
HP Real User Monitor product page

For Business Availability Center, Michael Procopio

Get the latest updates on our Twitter feed @HPITOps http://twitter.com/HPITOps

Join the HP Software group on LinkedIn and/or the Business Availability Center group on LinkedIn.

Fighting or friendly, Problem Isolation and OMi

Michael_Procopio — Tue, 22 Sep 2009 20:27:00 GMT

by Michael Procopio

In the post Event Correlation OMi TBEC and Problem Isolation What's the Difference, my fellow blogger, Jon Haworth, discussed the differences between TBEC and Problem Isolation. To be consistent, I'll use the acronyms PI for Problem Isolation and TBEC to refer to OMi (Operations Manager i series) Topology Based Event Correlation.

Briefly, he mentioned that TBEC works “bottom up”, that is starting from the infrastructure, with events. PI works “top down”, that is, starting from an end user experience problem, primarily with metric (time series) data.

Jon did an excellent job describing TBEC; I’ll do my best on PI because like Jon I have a conscience to settle.

Problem Isolation is a tool to:

1. automate the steps a troubleshooter would go through

2. run additional tests that might uncover the problem

3. look at all metric/performance data from the end user experience monitoring and all the infrastructure it depends

4. find the infrastructure metric the most closely matches the end user problem using behavior learning and regression analysis techniques (developed by HP Labs)

5. bring additional data such as events, help/service desk tickets and changes to the troubleshooter

6. allow the troubleshooter to execute Run books to potentially solve the problem

Potentially the biggest difference in the underlying technology is that Problem Isolation does not require any correlation rules or thresholds to be set for it to do the regression analysis to point to the problem. Like TBEC, it does require that an application be modeled in a CMDB.

An example: Presume a situation with a typical composite application - web server, application server and database. No infrastructure thresholds were violated; therefore, there are no infrastructure alerts. Again, as mentioned in the previous post, end user monitoring (EUM) is the back stop. EUM alerts on slow end user performance, now what?

Here is what Problem Isolation does:

1. determines which infrastructure elements (ITIL configurations items or CIs) support the transaction

2. reruns the test(s) that caused the alert – this validates it is not transient problem

3. runs any additional tests defined for the CIs

4. collects Service Level Agreement information

5. collects all available infrastructure performance metrics (web server, application server, database server and operating systems for each) and compares them to the EUM data using behavior and regression analysis

Problem Isolation screen show performance correlation between end user response and SQL Server database locks

-------------------------------------------------------------------------------------------

6. determines and displays the most probable suspect CI and alternates

7. displays run books available for all infrastructure CIs for the PI user to run directly from the tool

8. allows the PI user to attach all the information to a service ticket, either existing or create a new one

Another key differentiator of OMi/TBEC and PI is the target user. There is such a wide variance in how organizations work that it is hard to name the role but let me do a brief description and I think will be able to determine the title in your organization.

There are some folks in the organization whose job is to take a quick look (typically < 10 minutes, in one organization I interviewed < 1 minute) at a situation and determine if they have explicit instructions on what to do via scripts or run books. When they have no instructions for a situation they pass it on to someone who has a bit more experience and does some free form triage.

This person might be able to fix the problem or may have to pass it on to a subject matter expert, for example if they believe it is an MS Exchange problem to an Exchange admin. It is this second person that Problem Isolation is targeted at. This is helping automate her job, reducing what might take tens of minutes to hours and performing it in seconds. If it ends up she can’t solve the problem it automatically provides full documentation of all information collected. That alone might take someone five minutes to write-up.

OMi’s target is the operations bridge console user. Ops Bridge operators tend to be lower skilled and face hundreds if not thousands of events per hour. Jon described how OMi helps them work smarter.

TBEC and Problem Isolation both work to find the root cause of an incident but in different ways. Much like a doctor might use an MRI or CAT scan to diagnose a patient based on what the situation is, TBEC and Problem Isolation are complementary tools each with unique capabilities.

Problem Isolation will not find problems in redundant infrastructure that OMi will. Conversely, OMi can’t help with EUM problems when no events are triggered, where Problem Isolation will.

We know this can be a confusing area. We welcome your questions to help us do a better job of describing the difference. But these two are definitely friendly.

For Business Availability Center, Michael Procopio

Get the latest updates on our Twitter feed @HPITOps http://twitter.com/HPITOps

Join the HP Software group on LinkedIn and/or the Business Availability Center group on LinkedIn.

Related Items

Business and IT closer but still not on the same page

Michael_Procopio — Mon, 20 Jul 2009 20:51:00 GMT

by Michael Procopio

Network World editor Denise Dubie assessed a report by the analyst firm Aberdeen Group in her recent article Application performance management: Keeping an eye on the end-user prize. Her comments mirrored many of those I have spoken with recently who have said their top priorities this are year are revenue and removing any distractions from making revenue.

What was pleasant to see was data showing the end user performance is still important. More frustrating to see was that business and IT managers still differ in their priority of what needs to be measured, according to the article. As you might guess, business folks are more concerned with business processes while IT folks are more concerned with the infrastructure. One-step down from that the article covers the specifics of how each group prioritized how to do application performance monitoring.

Both end user and infrastructure monitoring are critical. Recently, at HP Software Universe in a talk I gave with a customer he showed this picture.

This does a good job of making the point that end user monitoring is critical. Since there are multiple pieces to the IT part of the puzzle, cumulative effect is important.

This customer said when he took his job there was a Severity 1 (highest priority) problem meeting everyday with typically greater than 10 "Sev 1" items. Today he has no meetings and approximately one Sev 1 per week.

How did he get here, monitoring the infrastructure. He said when they tracked down the source of the problem they put in a new infrastructure monitor for the item that failed, so he got early warning. But even in his current state he commented that end user monitoring is important because things change, whether it is a new version of the application or a change in the infrastructure that create a situation where something can go wrong that isn't currently being monitored.

For the Business Availability Center, Michael Procopio

How long between the problem and the first phone call?

adsey007 — Fri, 12 Jun 2009 13:01:00 GMT

By Mike Shaw, BSM Product Marketing

Last year, we did a series of in depth interviews with customers (28 of them, actually). As part of these interviews, we asked if people did proactive user experience monitoring - either using synthetic scripting technology to pretend to be a user, or using a probe on the network to look at the data going to the users' screens and monitoring the response time.

About half the respondents said they did. This ties in with a recent Aberdeen study that found 57% of companies didn't do user experience monitoring.

So, we asked on IT manager who didn't do user experience monitoring, why had he not invested in this technology. "Because we respond very quickly when the first customer rings in", was his response.

And since that day, I've been on a quest to get a magic number. How many minutes, on average,elapses between a business service giving a poor user experience and the first customer calling in. I have only three data points, but no definitive study. The average seems to be about 30 to 45 minutes.

To get another angle, whenever I present to a friendly audience, I'll ask them how often they have called a company when they have had problems with a user interface (e.g. on an ordering web site). Of the 160 people I've asked, just two had actually picked up the phone, and in both situations, it's been something critical like sorting out a mobile/cell phone bill.

I have another data point. A study by Corporate Executive Board in 2004 found that the average cost to a company of down-time is $1.3m per hour. That's $27,000 per minute.

So, we have 30 to 45 minutes (very rough estimate). We have $27,000 per minute. Being conservative (and very rough), we have a cost of 30 minutes X $27,000 or $81,000 per poor user experience problem.

Do you have any data on the average time between a poor user experience situation starting and the first customer calling in? If you do, could you please post a comment with the data - it amazes me that such data is not readily available "out there".

Advanced analytics reduces downtime costs – isolation

Michael_Procopio — Fri, 08 May 2009 15:40:00 GMT

by Michael Procopio, Product Manager, BAC

In the world of advanced analytics, two areas that are of interest to the IT management world are: detection of a problem and isolation of a problem. Previously I wrote Advanced analytics reduces downtime costs – detection; in this post I’ll cover isolation.

In the previous post, I covered how advanced analytics finds an anomaly, potentially before a threshold is crossed.

Problem Isolation is the process of determining which component in the infrastructure is causing the problem* or incident* that we found. We will presume we are monitoring the service that is having the issue.

If one had no management tools (amazingly I have spoken to customers in this situation) the method of trying to find a problem is to login to each system, router, switch and potentially application (ex: Oracle) look at the items with whatever tools are available (ex: Windows Perfmon)and hopefully you find it. If you are interested in advanced analytics, this is probably not your situation.

The more typical case is you have multiple management tools, network, system, virtualization, database and perhaps others. So if you know the domain the problem exists in you have a good place to start. I’ve listened to podcasts / read reports which bring up few problems with this: (if you know of any good IT podcasts please send them along)

~80% of problems are sent to the network team with only ~20% being network issues
~60% of problems take >10 experts to resolve
~80% of the time to restore service is spent isolating the problem

Here is an analogy I use with my non IT friends on why this area is needed. You are monitoring the speed of a car going across the country (pick your favorite country). You are separately monitoring the infrastructure, all:

roads
bridges
ferries to take cars across the water

What you don’t know is where the car is (old car, no GPS). You are getting many alerts from the roads, bridges and ferries. Which one is affecting the car? Since you don’t know what road the car is on you don’t know if any given alert is the one affecting your car.

This is where the CDMB comes into the isolation process. The CMDB has the route the car is taking or, in our case, the items in the IT infrastructure that make up the service that has the problem.

Part one of the isolation process is to restrict what we are looking at to the relevant IT items. This greatly reduces the computational power required. For example, one customer I recently visited told me he has 2000+ servers. If we can reduce that to a few app servers and a few database servers (isn’t SOA wonderful for we operations types) that is a factor of ~200 reduction.

Part two of the isolation is the heavy math from HP Labs, with more patent filings. It is a form of regression analysis, where application or end user response time monitoring is the dependent variable and all the infrastructure metrics are independent variables. In plain terms, if end user response gets worse find the infrastructure metrics that get worse. When end user response gets better find the metrics that get better. The more closely an infrastructure metric tracks the end user response the more likely it is to be the cause.

Again, while the math is interesting, pictures work better for me.

The thick grey line is the end user response, the red-purple line is the most closely correlated metric -- in this case a database metric. Just so you don’t have to strain your eyes we provide a table like this (from a different problem) showing the weighted correlations score.

Isolation part 3 is to include non-time series data. In the screen capture below you see planned changes and incident details (think alerts) on the timeline. Unplanned changes can also be displayed. Changes are pulled from the CMDB and incidents can come from any management system that can send alerts. And since we know that most problems occur from changes that is an important component. Finally tickets from the helpdesk are included on the timeline, for the case where users are doing the monitoring.

All together this automates a number of things the operations teams already do and some math help isolating problems.

*Incident and problem are ITIL terms. There may be many incidents that are symptoms of an underlying problem.

tweet this!

BSM Evolution: Small Enterprise Example

Michael_Procopio — Fri, 10 Apr 2009 04:31:00 GMT

My previous BSM evolution postings focused on mega-corporations and large IT organizations with a myriad of personas. In this post, I will contrast the experience of a relatively small IT shop of roughly 30 full time IT operations personnel.

Back when the economy was cooking along, an up and coming commercial construction company grew right out of their business model. Historically, they utilized a decentralized model, setting up and staffing a stand-alone onsite operation for each new project. This model was excellent at delivering customized project support, but lacked scalability and leverage; with remote site spin-up slow and error prone.

From an IT perspective, the CIO realized they needed to, in his words, "Consolidate and professionalize the IT operations", with the following goals:

1. Improve quality of service and experience for worksite users & applications
2. Contain IT costs and efficiently scale current IT personnel to meet growth
3. Improve speed, accuracy, and agility of spinning up new project worksites

Key Personas:

CIO

Many years of commercial construction experience
Personally drove IT consolidation / professionalization strategy and roadmap
Directly engaged in evaluating and selecting the solution vendor/consultant

VP of IT

"Co-pilot" for CIO on strategy, drove project deployment and vendor engagement

Subject Matter Experts (SME)

One for performance and availability tools / architecture
One for service management process workflow and automation (helpdesk)

Two Key Parallel Evolution Paths:

Path A: Performance, availability, and quality of experience monitoring

Step 1: -Deployed synthetic end-user / application monitors, agentless remote site infrastructure monitoring, and general WAN/LAN management

-Basic service experience reporting, and per-site performance dashboards

Step 2: -Enterprise infrastructure fault/performance (agent based system, OS, DB)

-Central "IT Command Center" event console with trouble ticket integration

Step 3: -In-depth application management modules (exchange, SAP)

-Advanced network services (route analytics, performance)

Path B: Service management process workflow and automation

Step 1: -Single call/request center organization established

-Incident management (utilized pre-packed ITIL module)

Step 2: -Knowledge management process, analytics and automation modules

Step 3: -Configuration and change management process/automation

-Service Level Management definition and basic reporting

An Uncommon Sequence of Evolution Steps

Notice the interesting order of the steps. The CIO dictated that the performance monitoring path start with remote site end-user / application experience monitoring. The original roadmap proposed by the system integrator recommended starting with basic data center tools, advancing through central event console, then application and database management, and finally end user experience. This is a traditional evolution path, but the CIO was adamant that, "what happens at the remote work-sites IS the business". So, he wanted an immediate awareness of remote site experience to drive the design of every step in the roadmap.

There was a similar "cultural" direction from the CIO on the service management workflow path. Again, the CIO insisted that Knowledge management be moved up in the evolution before configuration, change, and service level management. Typically, significant knowledge management execution is viewed as "icing on the cake" by most organizations, and only implemented after all the other core ITIL processes.

This CIO believed that analyzing and formalizing knowledge learned from successes and failures of spinning-up remote sites and dealing with issues was the best early investment. This approach immediately became part of the standard IT culture, and played a significant role in guiding change and configuration management process definition.

The CIO's Project-Based perspective

This CIO is indeed very ITIL savvy, but I think living and breathing the commercial construction business had a significant impact on his choice of system integrators. During the bidding process for the ITSM/BSM contract, it came down to three competitors in a direct "shoot-out". System integrator number one and two brought product and ITIL experts to the shoot-out, concentrated very heavily on features and functions, and gave a fixed-price bid of 200 deployment days.

System integrator number three brought a project manager to the shoot-out, and changed 75% of the discussion to, "here is how we will navigate the project and be successful". Can you guess who won? It shouldn't be news to anyone that a CIO's background alters the decision criteria, or the roadmap vision.... But it is always interesting to observe it in action. Maybe I will write a post about that someday

Conclusion

This IT organization is relatively small, so the decision making process and personas are greatly simplified compared with the large corporations previously analyzed. Despite the CIO's unique influence on approach and deployment sequence, in the end, the same fundamental truths of BSM/ITM evolution apply.... Just on a different scale, agility and timeframe.

Bryan Dean - BSM Research

Tweet this!

BSM Evolution: The CIO/Ops Perception Gap

adsey007 — Fri, 03 Apr 2009 10:31:00 GMT

There are many potential culprits for why IT organizations struggle to make substantive progress in evolving their ITSM/BSM effectiveness. A customer research project we did a few years ago offered an interesting insight into one particular issue that I rarely see the industry address. The research showed that most CIO’s simply had a different perception – when compared to their IT operations managers- of their IT organization’s fundamental service delivery maturity and capability. This seemingly benign situation often proved to be a powerful success inhibitor.

The Gap:

A substantial sample size of international, Global 2000 enterprise IT executives participated in the study. When asked to prioritize investment priorities on a broad range of IT capabilities, we saw a definite gap. IT Operations managers consistently ranked, “Investing to improve general IT service support and production IT operations” in their top 1 or 2 priorities, where CIO’s ranked this same capability much lower as a priority 6 or 7.

The Perception:

When pressed further, CIO’s believed that the IT service management basics of process and technology were already successfully completed, and the CIO’s had mentally moved on to other priorities such as rolling out new applications, IT financial management, or project and portfolio management.

Most of the CIO’s in the study could clearly recall spending thousands of dollars sending IT personnel to ITIL education, and thousands more purchasing helpdesk, network, and system management software. Apparently, these CIO’s thought of their investment in service operations as a onetime project, rather than an ongoing journey that requires multiple years of investment, evolution, reevaluation, and continuous improvement.

IT operations managers –on the other hand- clearly had a different view of the world. They were generally pleased with the initial progress from the service operations investments, but realized they were far from the desired end state. The Ops managers could plainly see the need to get proactive, to execute advanced IT processes and more sophisticated management tools, but could not drain the proverbial swamp while fighting off the alligators.

The Trap:

We probed deeper in the research, diligently questioning the IT operations managers on why they didn’t dispel the CIO’s inaccurate perception. In order to secure the substantial budget, these Ops managers had fallen into the trap of over-promising the initial service management project’s end-state, ROI and time to value. (I wouldn’t be surprised if they had been helped along by the process consultants and software management vendors!)

These Ops managers saw it as “a personal failure” to re-approach the CIO and ask for additional budget to continue improving the IT fundamentals. Worse yet, they had to continually reinforce the benefits from the original investment so the CIO didn’t think they had wasted the money. So, the IT operations staff enjoyed the result of reactively working nights and weekends to meet business’ expectations, and make sure everyone kept their jobs. Meanwhile, the CIO’s slept well at night thinking, “Hey, we are doing a pretty darn good job”, but faced the next day asking, “Why are my people burnt out?” A vicious cycle.

Recommendation through Observation:

I’m not wild about making recommendations since I merely research this stuff… not actually perform hands-on implementation. Instead, I will offer some observations of best practices from companies who appear to be breaking through on BSM, lowering costs, raising efficiency and improving IT quality of service.

Focus on Fundamentals: It is boring and basic, but absolutely critical to continually look for ways to improve the foundational service management elements of event, incident, problem, change, and configuration management. Successful IT organizations naturally assume that if they implemented these core processes more than 3 years ago, they likely need to update both technology and process. If FIFA World Cup Football clubs and Major League Baseball teams revisit their fundamental skills each and every year, why wouldn’t IT?

Assume a Journey: IT leaders who develop a step-wise, modular path of realistic projects that deliver a defined ROI at each step have the best track record of securing ongoing funding from the business. The danger here is defining modular steps that are so disconnected and silo’d, that IT never progresses toward an integrated BSM/ITSM process and technology architecture. This balance continues to be one of the most difficult to manage.

Empowered VP of IT Operations: The advantages of a CIO empowering a VP of IT operations and holding them accountable for end-to-end business service has been discussed in previous posts. The practice of having a strong VP of operations who has executive focus on service operations and continual service improvement, while having end-to-end service performance responsibility does appear to be a growing trend and success factor.

Focus on the Applications: In the same research study that showed the perception gap on, “Investing to improve general IT service support and production IT operations”, there was consistent agreement on, “Investing to improve business critical application performance and availability”. The CIO’s, Ops Managers and Business Relationship managers all ranked this capability as a top 1 or 2 priority.

Successful BSM implementations focus on the fundamentals of process and infrastructure management, but do so from a business service, or an application perspective. This approach not only enables an advantageous budget discussion with the business, but it also hones the scope and execution of projects.

It is difficult to assess the relative impact of this CIO/IT Ops perception gap, considering the wide variety of challenges that IT faces. But hopefully, this post gives you something to consider when assessing your own IT organization’s situation and evolution.

Let us know where your organization fits – please take our two question survey (two demographics questions also). We’ll publish the results on the blog.

Describe the perception of your IT's fundamental service delivery process
How often does your IT organization significantly evaluate and invest to update your fundamental IT process

Click Here to take survey

Bryan Dean – BSM Research

Monitoring your cloud computing as easy as calling an airport shuttle

Michael_Procopio — Tue, 31 Mar 2009 05:15:00 GMT

HP made an announcement about new cloud computing management capabilities today: HP Unveils "Cloud Assure" to Drive Business Adoption.

HP currently offers Software-as-a-Service (SaaS) for individual management applications such as HP Business Availability Center (BAC) and HP Service Manager primarily for intranet and extranet applications.

HP Cloud Assure helps customers validate:

Security – by scanning networks, operating systems, middleware layers and web applications. It also performs automated penetration testing to identify potential vulnerabilities. This provides customers with an accurate security-risk picture of cloud services to ensure that provider and consumer data are safe from unauthorized access.
Performance – by making sure cloud services meet end-user bandwidth and connectivity requirements and provide insight into end-user experiences. This helps validate that service-level agreements are being met and can improve service quality, end-user satisfaction and loyalty with the cloud service.
Availability – by monitoring cloud-based applications to isolate potential problems and identify root causes with end-user environments and business processes and to analyze performance issues. This allows for increased visibility, service uptime and performance.

HP Cloud Assure provides control over the three types of cloud service environments:

For Infrastructure as a Service, it helps ensure sufficient bandwidth ability and validates appropriate levels of network, operating system and middleware security to prevent intrusion and denial-of-service attacks.
For Platform as a Service, it helps ensure customers who build applications using a cloud platform are able to test and verify that they have securely and effectively built applications that can scale and meet the business needs.
For Software as a Service, it monitors end-user service levels on the cloud applications, loads tests from a business process perspective and tests for security penetration.

A diagram showing the differences in the services is at Cloud Computing Basics.

In the end it doesn't matter where the service is; you need to be sure it is available and performing to expectations. Cloud Assure provides the capability in a way that is very agile. You say "I need this service monitored" and it is monitored. Its just like calling for an airport shuttle -- you call, they show up.

For Business Availability Center Michael Procopio, product manager HP Problem Isolation.

OpEx versus CapEx

adsey007 — Mon, 30 Mar 2009 08:06:00 GMT

Forrester just posted on how the recession is hitting capital budgets (CapEx) and that you should consider using operating expenses (OpEx) to purchase software (http://www.idc.com/getdoc.jsp?containerId=lcUS21765009).

About 18 months ago, we introduced one year term licenses on the Business Availability Center (application and business transaction management) software so that it is more likely to fit within OpEx budgets.

Mike Shaw.

BSM Evolution Paths: Financial Services Example

adsey007 — Wed, 18 Mar 2009 11:16:00 GMT

When two Fortune 500 companies merge the IT convergence can feel like two high speed trains on parallel tracks speeding toward a single-track tunnel. Not only is IT tasked with maintaining or increasing quality of service, but the CEO’s are quite impatient to quickly rationalize the IT operating expense equation of “1+1=1.25”. Maybe 1.50 if you have an extremely benevolent Board of Directors.

Unlike the Automotive Industry example posted earlier (BSM Evolution Paths: Auto Industry Sample), this Financial Services example has much less tops-down roadmap direction, and much more independent parallel paths. Let’s take a look at three of the key personas and evolutions within these parallel paths.

Data Center Operations Manager; Infrastructure Operations path:

The new Data Center Operations Manager (DCOM; reporting to VP of IT Ops) commissioned a tools architecture analysis. They inventoried their management tools and counted over 80 major “platforms” in the fault, performance and availability category alone!

The DCOM empowered a Global Software Management Architect to drive a “limited vendor” strategy to simplify and standardize the tool environment. Although there were many individual domain experts bent out of shape, this standardized environment limited the vendor touches, enabled renegotiated license/support contracts, concentrated tool expertise and resulted in improved quality of service.

The fault, performance and availability architecture was boiled down to three major vendors covering three broad categories (plus device specific element plug-ins):

System Infrastructure (Server, OS, Database, storage middleware, LAN feeds, etc.)
Network Services (WAN, LAN, advanced protocols, route analytics, etc.)
Enterprise Event (consolidated event console, correlation, filtering, root cause)

The DCOM could have pushed harder for a single vendor covering all three categories, but it was a matter of time-to-deploy pragmatism. A vendor could only be selected as category solution if the product was successfully deployed previously, and internal deployment expertise existed to lead the global implementation. This “survival of the fittest” approach did not necessarily drive the most elegant architecture, but it did speed deployment and limit risk.

Independent roadmaps and key integration capabilities were developed for each category to meet 6, 12, 18 and 24 month milestones.

CTO; Business Service Oversight path

Early on in the merger process, there was a power struggle to own the business service visibility and accountability solution. The VP of IT Operations wanted the tools, process and organizational power, but the Lines of Business insisted on a more independent group that would sit between IT Operations and the business-aligned Application Owners.

The Online Banking Group from one of the pre-merger divisions had successfully implemented a business service dashboard and Service Level Agreement reporting solution (based primarily on end-user experience monitoring). Using an “adopt and go” strategy, the CIO empowered the CTO to develop an end-to-end group and expand the solution to all six major business units.

This business unit expansion rolled out over 12-18 months and was successful, but limited to monitoring and reporting. Over the next 12 months, Application Owners, Line of Business CIO’s and VP of IT Operations all wanted to extend the business service monitoring to:

Problem isolation, application diagnostics, and incident resolution
In-depth transaction management of composite applications

Director Service Management; Enterprise CMDB path

The Director of Service Management, reporting to VP IT Ops, drove two major initiatives over the first 12 months of the merger.

Consolidate to a single, global, follow-the-sun service desk
Rationalize and standardize the request and incident management process

I could easily spend an entire blog post discussing the IT process convergence and standardization, but I refuse! Instead, I’ll focus on what happened in the 12 months following the service desk consolidation.

The Director of Service Management launched a CMDB RFP which was originally grounded in incident, problem and configuration management. The RFP touched off an enterprise-wide nerve, not to mention a flurry of vendor responses. The project quickly expanded, and changed focus to the “hotter” driver of change and (release) risk management, and how to drive all IT process from an enterprise service model.

Once the application owners got involved (from a change/release control perspective), and the infrastructure operations got involved (from a change and performance/availability perspective), and the CTO got involved (from a business service reporting and accountability rperspective) all of a sudden incident management took a back seat in the decision process.

In the end, a service discovery, dependency mapping and change/release management solution was selected that was a different vendor all together from the incumbent service desk solution.

An interesting journey… so far

The three paths described above are clearly a small subset of the overall work done for this corporate merger, but hopefully gives a glimpse into the BSM evolution dynamics. By all accounts, this company has been successful in their journey; you may be interested to know that this financial services company is not participating in the government bail-out program.

The lack of a tops-down “enterprise IT transformation” roadmap did not hinder their progress… in fact some will argue it enable their progress! You can observe, however, that at the end of each path there is a drive towards further integration and cross-IT dependence. It will be interesting to watch this company, and see how their approach evolves as they continue down the intersecting evolution paths.

Bryan Dean, BSM Research

Business Service Visibility & Accountability: Where is it Homed?

adsey007 — Wed, 04 Mar 2009 12:25:00 GMT

Virtually every customer that I have studied has a critical moment in their BSM evolution where they realize the need for viewing, measuring and reporting business service performance in a business-relevant way. We could discuss the technical complexities of integrating service model discovery, end-user experience, transaction management, performance and event data to develop this business service view, but in this post I’m going to examine the most common key personas, core motivations, and organizational impact.

In the previous post, BSM Evolution Paths: Auto Industry Sample we saw how the core motivation came tops-down from senior IT management. Let’s compare three different models.

Line of Business / Application Driven

Key Personas: Application owner, Business Relationship Manager, Business Unit CIO

Core Motivations: These personas are typically closest to how business utilizes IT to execute a business process or function. They usually report into the business unit itself, rather than into IT Operations. They have responsibility for the application, but the business perceives them as owning the end-to-end service performance, even though they often have little control of the underlying IT infrastructure and service delivery processes.

At some point, a business critical service melts-down, or endures a never-ending spree of performance degradations where Global IT Operations says, “All the systems and network are green”. This is the point where many business unit managers take matters into their own hands and fund a significant investment in End-to-End business service visibility tools.

Software: Since they do not control the infrastructure, the application owners often look for tools that require minimal agentry and do not require a lot of feeds from the individual domain management tools. They gravitate towards sophisticated end-user experience tools, probes, application diagnostics, and the ability to traverse composite application middleware.

Organization: They use these tools to prove accountability to the business units, but they also use the tools -not always politely- to hold infrastructure operations accountable. The animosity usually wanes, and the separate IT groups work out the process integration… but often not the tool integration. This leaves the end-to-end group outside of IT Operations. We also see this model where the infrastructure operations are outsourced, and the service provider is held accountable to specific Service Level Agreements.

Infrastructure Operations Hero

Key Personas: Infrastructure Operations Manager, Data Center Manager, NOC manager

Core Motivations: These personas traditionally have the responsibility for care and feeding of the vast shared-service IT infrastructure environment. They have likely done a reasonable job of consolidated event management, and domain-level configuration, performance and capacity management. But, they have a vision of elevating IT to demonstrate the value delivered to the business, and proactively solve issues before end users report them. This effort can be either in conjunction or parallel to an ITIL-driven service management initiative.

Software: Often very budget constrained, they don’t always have the funding that the application owners do. They look first toward leveraging investment of their existing tool set, gathering agent-based data from their infrastructure and augmenting with lighter-weight end-user experience tools. Converting this data to business-relevant information is difficult, as they often don’t have the deep business process or application knowledge, but it is much better than the previous IT element statistic data.

Organizational: The Hero Operations manager then faces the daunting task of taking the new service oriented visibility and reporting capability to upper management and business unit managers. Sometimes they yawn. Sometimes the strategy is embraced, and the operations manager is elevated to strategic status. The Operations manager keeps both tools and processes very integrated. New end-to-end skill sets are developed, but usually not new organizational groups.

Tops-down Service Management
Key Personas: CIO, CTO, VP IT Operations

Core Motivations: These personas have the luxury of controlling the organization, budget and overall priority of IT, yet their job is likely on the line. Pressure from the business units, a personal drive to elevate IT to a strategic partner and sometimes fear of being outsourced are the powerful drivers.

Business service visibility and accountability is usually part of a larger, multi-project, multi-step roadmap that includes a hefty process component. Since these initiatives tend to be “horizontal” in nature across all IT, many companies fall into the trap of trying to institute end-to-end business service performance tools too broadly. The successful organizations focus on a discrete business service and satisfy key metrics that are specific to the particular business and application.

Software: These personas tend to focus on service level management, and the ability to demonstrate the value IT is delivering to business. Typically requires a substantial investment in tools that can abstract the business services into something meaningful to business, looks hot to business stakeholders, yet also improves service delivery time to diagnose and repair. This ends up requiring a rationalization of the service discovery model, CMDB, and the enterprise operational tools.

Organization: I’ve seen some CIO’s form executive business relationship management functions, keeping the team independent from both business and IT Operations. Other CIO’s formally extend the VP of IT operations charter to include this new end-to-end function that bridges the infrastructure operations teams and the helpdesk/service desk teams.

Conclusion?
Here’s a news flash…there is a wide variety of organizational models. But there are some definite patterns, and in my next post, I will offer some evidence that the model will be more predictable in the future.Bryan Dean, BSM Research

BSM Evolution Paths: Auto Industry Sample

adsey007 — Tue, 24 Feb 2009 10:24:00 GMT

In the last post, Bryan Dean, our research expert in the BSM team, outlined the different ways in which customer evolve towards Business Service Management. In the next few posts, Bryan will give an example of each of the different types of evolution. Over to you Bryan ....
_______About three years ago, the business division managers of a multinational automobile manufacturing company planned a bold transformation of their distribution network to leapfrog the competition. They enthusiastically laid out a roadmap for business process innovation and aggressive customer/dealer satisfaction initiatives.

Only one real problem; the CIO knew that building, rolling-out, and operating the underlying IT for this future business vision exceeded their current capabilities. The CIO eventually had to raise the red flag and explain to the executive committee why IT was the bottleneck. Ouch, not a good day.

In the previous post BSM Evolution Paths: Samples and Observations, we talked about five common evolution paths, the organizational and persona dynamics of an Automated BSM/ITSM journey. In this post we will overview a specific example.

To be fair, the CIO spent years driving significant investment in process, tools and the organization. Let’s look at a subset of key personas and BSM/ITSM foundation:

Director of Infrastructure (reporting to the VP Global IT Ops):

Enterprise-class central event/performance platform and console
WAN/LAN network management platform
Basic, component level performance and availability reporting
Dozens of vendor-specific configuration and admin tools

Director of Service Management (reporting to the VP Global IT Ops):

Global, consolidated helpdesk/service desk
Well defined and automated incident process; basic level problem, configuration, and a manual change process

Director of Applications (development, test & level 3 support. Reports to business divisions):

Suite of pre-production stress-test quality and performance tools
End-user and application performance/diagnostic tools (test environment)

The Key Evolution Steps

Step 1: CIO empowers and holds the VP of Global IT Operations (VPITops) accountable for end-to-end business service responsibility. Imagine the panic on his face! VPITops launches key lieutenants on quick gap analysis.

Step 2: The VPITops needed a quick win. He believed that visually demonstrating and reporting performance and availability from a business service perspective -versus an infrastructure perspective- would be a catalyst for driving “aligned” IT behavior. The current network and infrastructure products didn’t have this capability, so VPITops leveraged the tools already proven by the application test and level 3 support team.

VPITops established a new team within Operations (parallel to infrastructure event management) to own and run the end-to-end business service visibility/accountability solution. Integration was established between the two teams and tools.

Step 3a: VPITops took his new business service visibility/accountability tool (in dashboard/report form) to key business division managers, and established a business relationship management function. This converted the conversation from anecdotal complaints, to measurable service levels. The CIO had tangible proof of progress.

Step 3b: While engineering step 2, the Tools and Process Architect realized they needed a better means of discovering and maintaining the IT/Business service models. Their infrastructure environment was shared, complex and dynamic enough that static service models were not effective, so they brought in an application dependency mapping technology. This success spawned a serendipitous benefit to another team in step 4a.

Step 4a: The application quality/test and release team realized the service model could be utilized in the service transition process. They previously had several very painful episodes of moving complex applications from test into production. With an accurate, up to date service model of the production environment they could better identify dependency issues before roll-out. Speed and accuracy... Happy CIO.

Step 4b: The Director of Service Management and the architect evaluated how to federate the data between the application dependency mapping service model and the CI configuration data in the helpdesk. The software vendor provided a federation / reconciliation adaptor, so the helpdesk was able to leverage the CI relationships and operate off a “single version of the truth” (sounds eerily like an ITIL V3 CMS!).

Near Term Roadmap

· Automate change/configuration workflow and provisioning

· Upgrade/replace enterprise event and performance console to leverage service model for root cause analysis and business impact assessment

· Apply business service relationship management to additional business divisions

· End-to-end visibility of composite MQ application business transactions

The Verdict of the Journey so far

The CIO still has a job, and has a funded roadmap. One might ask why they didn’t start with step 4b, and establish the CMDB and service model first? Well, the CIO was on the hot seat, and they were concerned about getting bogged down in an enterprise-wide CMDB architecture project.

This exemplifies the unpredictable and unique nature of evolution paths. More can be said about the delicate balance between tops-down guidance, and fostering organic innovation from within the ranks of IT. In future posts, I will discuss and analyze this further, as well as introduce other examples.

BSM customer evolution paths: Samples and observations

Michael_Procopio — Thu, 05 Feb 2009 01:41:00 GMT

When developing and marketing products, we often have questions which can only be answered by going out there and seeing what people are doing. We have a guy on the BSM team who does this for us. His name is Bryan Dean. I've worked with Bryan for many years and I've always been impressed by his objectivity and the insight he brings to his analysis (i.e. he doesn't just present a set of figures - he gets behind the figures).

At the end of last year, we asked Bryan to analyze the top 20-odd BSM deals of 2008. He formed a number of conclusions from this research. One set of conclusions concerned how people "get to BSM" - how they evolve towards an integrated BSM solution. I asked Bryan to help me with a series of posts to share what he learnt about evolutions towards BSM because I think that knowing what our other BSM customers are doing may help you.

________

Mike: Bryan, can you give a summary of what you learnt?

Bryan: There is no one evolution path. It's fascinating to me that a hundred different IT organizations can have virtually the same high-level goals, fundamentally agree on the key factors for success, and yet end up with a hundred unique execution paths.

Before I answer your question, can I create a definition? The term "BSM" is very poorly defined within the IT industry - different vendors have different versions, and so do the industry analysts (in fact, some other research I did last year concluded that very few people had a clear idea of what BSM means). So, I'd like to introduce the term "Automated Business/IT Service Management" or AB/ITSM.

Back to your question, I think I can group all the different evolution paths into five key types:

ITSM incident, problem change & configuration: this evolution is driven out of the need for process-driven IT service management with the service desk as a key component
Consolidated infrastructure event, performance and availability: this is driven by a recognition that having a whole ton of event management and performance monitoring systems is not an efficient way to run IT, and so there is a drive to consolidate them into one console.
Business service visibility & accountability: this is more of a top-down approach - start with monitoring the customer's quality of experience and then figure out what needs to happen underneath. This is popular in industries where the "web customer experience" is everything - if it's not good, you lose your business
Service discovery & model: this is where evolution towards integration is driven from the need for a central model (the CMDB). Often, the main driver for such a central model is the need to control change
Business transaction management: today, this is the rarest starting point. It's driven by a need to monitor and diagnose complex composite transactions. We see this need most strongly in the financial services sector

Mike: How about the politics of such AB/ITSM projects? (I don't see the AB/ITSM term taking hold, by the way :-) )

Bryan: Politics (or, most specifically, the motivational side) is important. I think many heavy thinkers in our industry have the mistaken assumption that that there is a single evolution path, controlled from the top on down by the CIO following a master plan. Trying to manage such a serialized, mega project is a huge challenge and too slow, not to mention that 99% of CIO’s are not in the habit of forcing tactical execution edicts on their lieutenants (I know I’ll get some argument on that one :-) ).

What I see from my research is that the most successful IT organizations are those who have figured out how to balance between discrete doable projects, and an overall AB/ITSM end-goal context and roadmap. Typically, the CIO lays down a high-level vision that ties to specific business results, and then allows key lieutenants to assess and drive a prioritized set of federated, manageable projects that independently drive incremental ROI. Some IT organizations may have a well-defined integrated roadmap, but the majority of IT run federated projects in a fairly disjointed fashion.

These parallel paths are owned by many independent personas within IT, each trying to solve the specific set of issues at hand. For them, being bogged down in how their federated project aligns and integrates with all the other AB/ITSM projects is daunting… if not fatal.

And on reflection this makes sense to me - the human side of things plays a large role in such endeavors.

Mike: What do you mean?

Bryan: IT organizations of all shapes and sizes have goals to reduce costs, increase efficiency, improve business/IT service quality, and mitigate risk all while applying technology in an agile way to boost business performance. What I find interesting is how specific, funded initiatives are created by specific personas to achieve the goals.

In future posts, I will share some specific examples of how customers evolved through these paths, the key driver personas, the core motivations and how these paths come together.

There are a number of ways of populating the service dependency map

adsey007 — Tue, 03 Feb 2009 13:13:00 GMT

In a post two weeks on this blog, I listed all the ways that we use service dependency maps (model-based event correlation, service impact analysis, top-down performance problem isolation, SLAs, etc). What can be used to discover service dependency information?

OperationsCenter Smart Plug-ins (SPIs) now discover to the CMDB

If you're using the agent-based side of OperationsCenter (OpC), then each managed node will have an agent on it. You can put a smart plug-in (SPI) onto that agent. SPIs have specialized knowledge of the domain they are managing. There are many SPIs for all kinds of things from infrastructure up to applications like SAP. Many of the SPIs discover (and continue to discover) the environment they are monitoring. This is agent-based discovery using all the credentials you've already configured into the OpC agent.

The OMi team are working on putting SPI-based discovery information into the HP CMDB (the Universal CMDB or uCMDB).

Agentless monitoring populates the uCMDB

If you have agentless monitoring (HP SiteScope) this will populate the uCMDB too (as of SiteScope version 10).

Whatever SiteScope monitors you have configured will send their configuration information to the uCMDB. So, if you're monitoring a server with a database on it, all the information about the server and its database will be sent to the uCDMB.

Network Node Manager populates the uCMDB

As of the latest version of Network Node Manager (NNMi 8.10), discovered network end-points are also put into the uCMDB. "Network end-points" are anything with a network terminator - network devices, servers, and printers. NNMi provides no service dependency information, but it does provide an inventory of what's out there.

This inventory discovery is useful for rouge device investigation - noticing an unknown device, creating a ticket to the group responsible for that type of device so they can look into it.

Standard Discovery

Our Standard Dependency Discovery Mapping product (DDM-Standard) will discover your hosts for you. This also discovers network artifacts (but, see NNM discovery above - if you have NNMi, this is a more detailed network discovery mechanism).

Advanced Discovery

Advanced Dependency Discovery Mapping will discover storage, mainframes, virtualized environments, LDAP, MS Active Directory, DNS, FTP, MQSeries buses, app servers, databases, Citrix, MS Exchange, SAP, Siebel, and Oracle Financials.

You can also create patterns for top -level business services and DDM-Advanced will discover those too.

Transaction Discovery

Our Business Transaction Management product, TransactionVision, deploys sensors to capture application events (not operational events) from the application and middleware tiers. These sensors feed the events to the TransactionVision Analyzer which automatically correlates these events into an instance of a transaction. TransactionVision also classifies the transactions by type - bond trade, transfer request, etc. Thus, TransactionVision is discovering transactions for you.

TransactionVision puts this transaction information into the CMDB. In other words, the CMDB doesn't just know about "single node" CI types like servers, it also knows about flow CI types - transactions.

Also, if the CMDB notices that the transaction flows over a J2EE application, it links the transaction to information in the CMDB about this J2EE application - the transaction step and the J2EE app are now linked in the model. .

__________

By the way, my colleague Jon Haworth has just posted on the value of discovery in the realm of Operations Management at ITOpsBlog (28th January, "Automated Infrastructure Discovery - Extreme Makeover").

Answers to questions on "what's new in Business Availablity Center 8.0?"

adsey007 — Wed, 28 Jan 2009 07:19:00 GMT

I recently mentioned about a whats new webinar conducted on BAC v 8.0. You can now access this on-demand webinar at

https://h30406.www3.hp.com/campaigns/2008/events/sw-01-20-09/index.php?rtc=3-2CDASIY

Here are some of the questions which came up during the live webinar.

Q: When will 8.0 be available?

A: The 8.0 release will be made available the first week of February

Q: How will the new modeling changes affect my current custom views?

A: There are no more instance views, it’s just views and custom perspectives that provide the content in the view, the upgrade for most customers should be straightforward, unless they have changed the model, created new CI types with custom links or are heavily using pattern views with impact analysis, correlation rules and alerts.

Q: How about integration with HP Operations Manager? Can we leverage our current HPOV infrastructure monitoring capabilities and marry data with BAC application monitoring?

A: Yes with HP problem Isolation we have support of OM (Operations Manager) through event correlation to application problem/ anomaly start time.

Q: Does v 8.0 support oracle 11g platform

A: Yes with v 8.0 is it supported.

Q: I was told that the DDM portion of the new 8.0 can discover WebLogic 10.x iis it true?

A: Yes with v 8.0 is it supported.