Application Management : Michael Procopio

BAC 8.03 release - what's new

Michael_Procopio — Thu, 08 Oct 2009 22:56:00 GMT

by Michael Procopio

Release Highlights

Service Level Management (SLM) enhancements:

SLM now enables customization of the First day of month setting which is used in calculating agreements.

A new business rule, EMS Events Availability, calculates status for the System Availability KPI assigned to an EMS Monitor CI. This enables you to include EMS events as part of the SLM calculation.

Platform enhancements:

Update of Java Runtime Environment running on Business Availability Center servers to JRE 1.5.0_18

Mutual authentication: Users can set up authentication for both the Discovery and Dependency Mapping (DDM) Probe and Business Availability Center running an embedded UCMDB, with certificates. The certificate for each side is sent and authenticated before the connection is established.

Real User Monitor Engine enhancements:

Support for Oracle Forms NCA version 10g

System Availability Management enhancement:

Added the ability to control the location of the ellipsis ("...") used in long labels in System Availability Management Reports. This enables you to move the ellipsis from the end to the middle of the label.

Dashboard enhancement:

Added the ability to drill down from Dashboard to the Event Log report from Windows and UNIX CIs.

TransactionVision enhancement:

Support for LW-SSO 2.2 has been added to the UI/Job Server.

Database Admin privileges are no longer required to set up a database for use with TransactionVision.

For Business Availability Center, Michael Procopio

Get the latest updates on our Twitter feed @HPITOps http://twitter.com/HPITOps

Join the HP Software group on LinkedIn and/or the Business Availability Center group on LinkedIn.

RUM the Real User Experience Manager

Michael_Procopio — Wed, 07 Oct 2009 02:20:00 GMT

by Michael Procopio

RUM or Real User Monitor is a tool to monitor actual user traffic running over your network.

Its part of our EUM or end user management suite. In the area of EUM there are two primary ways to monitor 1/ synthetic, which is covered by BPM or Business Process Monitor and 2/ real user monitoring.

Each has its place in a monitoring strategy. BPM is good for making sure things are up 24x7, even when no users are using your applications. Real user monitoring can give you information down to the specific user.

When I first moved over to BAC group and heard about RUM, I was impressed. One of the things it can do is replicate a users web session click by click. This allows someone troubleshooting a problem to see exactly what happened and what the error message was the user saw – no guessing. (sensitive data like passwords and credit cards are filtered out in memory before writing to a disk). Further, if you do find a problem it can turn the session into a script that can be passed to the QA team to replicate the problem, if they are using LoadRunner.

How does it work? It starts by capturing packets as they go over the network. This is done by a RUM Probe, which is software that runs on a dedicated piece of x86 hardware (typically). The Probe passes the relevant data to the RUM engine.

The RUM engine stores the data and key performance metrics are aggregated before being sent up to BAC for reporting and alerting. For example, an alert might be round trip time for the Savings Deposit transaction is taking too long. Here are some of the reports RUM provides:

Global Statistics
Page Summary
Transaction Summary
End User Summary
End User Over Time
Server Over Time
Session Analyzer
TCP Application Summary
TCP Application Over Time
Event Summary
Business Process Distribution

Figure: Example RUM deployment configuration

Originally RUM strictly focused on HTTP/S traffic. But a while back support was expanded to due general tracking of TCP traffic, both streaming and non-streaming. In more recent releases additional upper level protocol analysis has been added. Beyond HTTP/S current support includes:

XML/SOAP
Siebel
WebSphere
MPLS

Related Items:

RUM webinar
HP Real User Monitor product page

For Business Availability Center, Michael Procopio

Get the latest updates on our Twitter feed @HPITOps http://twitter.com/HPITOps

Join the HP Software group on LinkedIn and/or the Business Availability Center group on LinkedIn.

Fighting or friendly, Problem Isolation and OMi

Michael_Procopio — Tue, 22 Sep 2009 20:27:00 GMT

by Michael Procopio

In the post Event Correlation OMi TBEC and Problem Isolation What's the Difference, my fellow blogger, Jon Haworth, discussed the differences between TBEC and Problem Isolation. To be consistent, I'll use the acronyms PI for Problem Isolation and TBEC to refer to OMi (Operations Manager i series) Topology Based Event Correlation.

Briefly, he mentioned that TBEC works “bottom up”, that is starting from the infrastructure, with events. PI works “top down”, that is, starting from an end user experience problem, primarily with metric (time series) data.

Jon did an excellent job describing TBEC; I’ll do my best on PI because like Jon I have a conscience to settle.

Problem Isolation is a tool to:

1. automate the steps a troubleshooter would go through

2. run additional tests that might uncover the problem

3. look at all metric/performance data from the end user experience monitoring and all the infrastructure it depends

4. find the infrastructure metric the most closely matches the end user problem using behavior learning and regression analysis techniques (developed by HP Labs)

5. bring additional data such as events, help/service desk tickets and changes to the troubleshooter

6. allow the troubleshooter to execute Run books to potentially solve the problem

Potentially the biggest difference in the underlying technology is that Problem Isolation does not require any correlation rules or thresholds to be set for it to do the regression analysis to point to the problem. Like TBEC, it does require that an application be modeled in a CMDB.

An example: Presume a situation with a typical composite application - web server, application server and database. No infrastructure thresholds were violated; therefore, there are no infrastructure alerts. Again, as mentioned in the previous post, end user monitoring (EUM) is the back stop. EUM alerts on slow end user performance, now what?

Here is what Problem Isolation does:

1. determines which infrastructure elements (ITIL configurations items or CIs) support the transaction

2. reruns the test(s) that caused the alert – this validates it is not transient problem

3. runs any additional tests defined for the CIs

4. collects Service Level Agreement information

5. collects all available infrastructure performance metrics (web server, application server, database server and operating systems for each) and compares them to the EUM data using behavior and regression analysis

Problem Isolation screen show performance correlation between end user response and SQL Server database locks

-------------------------------------------------------------------------------------------

6. determines and displays the most probable suspect CI and alternates

7. displays run books available for all infrastructure CIs for the PI user to run directly from the tool

8. allows the PI user to attach all the information to a service ticket, either existing or create a new one

Another key differentiator of OMi/TBEC and PI is the target user. There is such a wide variance in how organizations work that it is hard to name the role but let me do a brief description and I think will be able to determine the title in your organization.

There are some folks in the organization whose job is to take a quick look (typically < 10 minutes, in one organization I interviewed < 1 minute) at a situation and determine if they have explicit instructions on what to do via scripts or run books. When they have no instructions for a situation they pass it on to someone who has a bit more experience and does some free form triage.

This person might be able to fix the problem or may have to pass it on to a subject matter expert, for example if they believe it is an MS Exchange problem to an Exchange admin. It is this second person that Problem Isolation is targeted at. This is helping automate her job, reducing what might take tens of minutes to hours and performing it in seconds. If it ends up she can’t solve the problem it automatically provides full documentation of all information collected. That alone might take someone five minutes to write-up.

OMi’s target is the operations bridge console user. Ops Bridge operators tend to be lower skilled and face hundreds if not thousands of events per hour. Jon described how OMi helps them work smarter.

TBEC and Problem Isolation both work to find the root cause of an incident but in different ways. Much like a doctor might use an MRI or CAT scan to diagnose a patient based on what the situation is, TBEC and Problem Isolation are complementary tools each with unique capabilities.

Problem Isolation will not find problems in redundant infrastructure that OMi will. Conversely, OMi can’t help with EUM problems when no events are triggered, where Problem Isolation will.

We know this can be a confusing area. We welcome your questions to help us do a better job of describing the difference. But these two are definitely friendly.

For Business Availability Center, Michael Procopio

Get the latest updates on our Twitter feed @HPITOps http://twitter.com/HPITOps

Join the HP Software group on LinkedIn and/or the Business Availability Center group on LinkedIn.

Related Items

Freddy Kruger, James Bond and IT Cost Optimization

Michael_Procopio — Thu, 17 Sep 2009 17:33:00 GMT

by Michael Procopio

Top 10 Ten ways to improve IT Cost Optimization. Another fun video from Mark Leake, this time with an appearance from Freddy and James.

Get the latest updates on our Twitter feed @HPITOps http://twitter.com/HPITOps

Join the HP Software LinkedIn Group

For the Business Availability Center, Michael Procopio

Announcing a New BSM Solution Offering for Virtualization

Michael_Procopio — Tue, 01 Sep 2009 18:02:00 GMT

by Michael Procopio

The new offerings include enhancements to HP Server Automation, HP Client Automation, HP Storage Essentials, HP Network Automation, and HP Operations Manager software.

According to a recent report from Gartner⁽¹⁾, "Virtualization’s impact on the overall IT industry has been dramatic, and virtualization will continue to be the leading catalyst for infrastructure and operations software change through 2013. Organizations are looking at ways to cut costs, better utilize assets, and reduce implementation and management time and complexity."

Although virtualization is often adopted to help reduce capital expenditures, it can trigger increased management expenses and lead to more pronounced organizational silos. The new HP offerings bridge all the physical and virtual data center silos through management and automation. This reduces complexity and ultimately management costs.

"HP business service automation software helps us eliminate manual, error-prone tasks by automating server lifecycle management, including provisioning multiple operating systems, software installation, deployment of patches, configuration management and audits," said Ron Cotten, senior manager IT OSS Engineering, Level 3 Communications, a leading international provider of voice, video, and data communications services. "With HP Server Automation, we are able to patch over 1800 servers in 24 hours, which helps us reduce scheduled downtime."

The updated HP business service offerings help you:

· Increase administrator effectiveness with HP Operations Manager for virtualization by monitoring the availability and performance of all virtual and physical assets through a common dashboard.

· Reduce the risk of downtime with HP Network Automation, which for the first time gives the network administrator control of the VMware vSwitch in addition to the physical network environments.

· Provision the right amount of storage to keep applications performing properly without overspending on excess storage with HP Server Automation, the first solution in the industry that gives server administrators this capability.

· Reduce problem resolution times with HP Storage Essentials Performance Edition by quickly identifying, troubleshooting and reporting performance metric related trends in physical and virtual environments.

“To make virtualization cost effective, customers must minimize operating expenses and have seamless management of infrastructure silos,” said Erik Frieberg, vice president of Product Marketing, Software & Solutions, HP. “Our newly enhanced HP business service offerings help customers manage all aspects of the physical and virtual application infrastructure to unlock the true promise of virtualization.”

HP Software Professional Services provides solution consulting services to accelerate the value of business service automation and business service management software investments.

HP Server Automation, HP Client Automation, HP Storage Essentials and HP Network Automation are available now. HP Operations Manager Virtualization Smart Plug-In will be available next month.

HP Business Availability Center 8.02 What's New

Michael_Procopio — Fri, 24 Jul 2009 01:17:00 GMT

by Michael Procopio

BAC 8.02 is now generally available. Here are the highlights of what's new. There are also a number of defect fixes in this release.

Netuitive integration:
Enables integrating Netuitive alarms into Dashboard
System Availability Management (SAM) enhancement:
SAM administration now displays SAM points in use, enabling more effective management of SiteScope license points by increasing visibility into point consumption
Apache Web server upgrade:
Upgrade of Apache Web server to version 2.2.11 keeps Business Availability Center current with the latest industry release and compliant with security requirements
End User Management (EUM) enhancements, including:
Real User Monitor (RUM): Additional OS support for RUM probe (Windows 2003 and RHE5 64 bit); Improved SSL traffic handling for special environments; Seven protocol decoder packs for slow requests: MSSQL, LDAP, MySQL, IMAP, POP3, SMTP, FTP; Improved system health; Keystore management improvements
Business Process Monitor (BPM): Support for VuGen 9.5; BPM support for Windows Vista; New MSI installation replaces InstallShield on Windows platforms; New Solaris packaging (SPARC) installation replaces InstallShield on Solaris platforms
Business Process Insight enhancement:
Implemented a way to link sub-processes to parent processes.
Interactive integration documentation for Business Availability Center-Service Manager/Service Center integration:
New interactive document that enables selecting specific integration parameters and viewing only the documentation relevant for the integration specified

For the Business Availability Center, Michael Procopio

Business and IT closer but still not on the same page

Michael_Procopio — Mon, 20 Jul 2009 20:51:00 GMT

by Michael Procopio

Network World editor Denise Dubie assessed a report by the analyst firm Aberdeen Group in her recent article Application performance management: Keeping an eye on the end-user prize. Her comments mirrored many of those I have spoken with recently who have said their top priorities this are year are revenue and removing any distractions from making revenue.

What was pleasant to see was data showing the end user performance is still important. More frustrating to see was that business and IT managers still differ in their priority of what needs to be measured, according to the article. As you might guess, business folks are more concerned with business processes while IT folks are more concerned with the infrastructure. One-step down from that the article covers the specifics of how each group prioritized how to do application performance monitoring.

Both end user and infrastructure monitoring are critical. Recently, at HP Software Universe in a talk I gave with a customer he showed this picture.

This does a good job of making the point that end user monitoring is critical. Since there are multiple pieces to the IT part of the puzzle, cumulative effect is important.

This customer said when he took his job there was a Severity 1 (highest priority) problem meeting everyday with typically greater than 10 "Sev 1" items. Today he has no meetings and approximately one Sev 1 per week.

How did he get here, monitoring the infrastructure. He said when they tracked down the source of the problem they put in a new infrastructure monitor for the item that failed, so he got early warning. But even in his current state he commented that end user monitoring is important because things change, whether it is a new version of the application or a change in the infrastructure that create a situation where something can go wrong that isn't currently being monitored.

For the Business Availability Center, Michael Procopio

HP Software Universe - Mainstage Robin Purohit

Michael_Procopio — Thu, 18 Jun 2009 01:19:00 GMT

After a break Mainstage split into a Business Intelligence session and a Business Technology Optimization (BTO) session, I went to the later hosted by Robin Purohit, vice president and general manger, Software Products, HP Software & Solutions

Here are some bits I found interesting:

There are 15M virtual servers now and it is expected to double in 2 years.
Three key areas of change IT will see are: virtualization, SaaS and Web 2.0.
Web 2.0 screws up a lot in management, HP is building Web 2.0 compatibility into BTO tools.
Dataprotector, monitoring tools and mapping tools are including or working to include virtualization capability.
Service Manager and Asset Manager are now available as SaaS.
35% of flash apps violate Adobe best practices. HP released a testing tool which you can download for free called SWFScan. The same technology is built into some of HP’s BTO products. Customers can also do a free trial of our scanning SaaS.
One customer cut $5M in costs just in automating incidents with HP Operations Orchestration.
HP Software recently announced a suite of software for operations of Blackberry implementations. See HP and RIM Announce Strategic Alliance to Mobilize Business on BlackBerry for more details.
HP Software is publishing configuration management best practices later this month.
Brian Byun VP Global Alliances from VMware came on stage and announced announced an expanded partnership with HP to jointly develop software to manage VMware hypervisor technology. Denise Dubie at Network World wrote HP, VMware team to manage virtual servers.

There are a variety of Twitter accounts you can follow as well as the hashtag #HPSU09. Search Twitter for #HPSU09.

HPITOps – Covers BSM, ITFM, ITSM, Operations and Network Management

HPSU09 – show logistics and other information

HPSoftwareCTO

informationCTO

HPSoftware

BTOCMO – HP BTO Chief Marketing Officer

For HP BSM, Michael Procopio

HP Software Universe – Mainstage Andy Isherwood

Michael_Procopio — Wed, 17 Jun 2009 03:57:00 GMT

Andy Isherwood VP, Support & Services kicked off Mainstage.

There are four key areas shown in the picture above. HP announced this week its IT Financial Management offering. Andy likened ITFM to an ERP system for IT. Information Management magazine wrote an article on HP ITFM.

HP has had offerings in IT Performance Analytics and IT Resource Optimization for awhile. HP Cloud Assure was announced was announced in May 2009, HP Unveils “Cloud Assure” to Drive Business Adoption of Cloud Services

Some key points from his opening remarks:

Prepare for coming out of the recession when cutting costs and innovating.
Best in class means being good at all four - aligning to the business, taking out costs, increasing efficiency and consolidation.
Jetblue, Altec and T-Mobile were the winners of the HP Software Award of Excellence.
As an example of the quick ROI companies can get, Altec produced 10% application downtime reduction, 20% faster response time, 15% increase in customer satisfaction and a 300% improve application transaction time in 6 months.
Last year we were HP Software, this year HP Software and Solutions. This was the combining HP Software with HP Consulting and Integration. The net result increased our delivery options. In addition to offering software for in-house use, HP now has EDS, SaaS and continues with it Partners
HP SaaS business is seven years old this year and has 650 customers.

You can read other coverage of HP Software Universe in the ITOpsBlog. There are a variety of Twitter accounts you can follow:

HPITOps – Covers BSM, ITFM, ITSM, Operations and Network Management

HPSU09 – show logistics and other information

HPSoftwareCTO

informationCTO

HPSoftware

BTOCMO – HP BTO Chief Marketing Officer

as well as the Twitter hashtag #HPSU09

For HP BSM, Michael Procopio

HP Software Universe - day 1

Michael_Procopio — Tue, 16 Jun 2009 02:11:00 GMT

by Michael Procopio

Today was the first day of Software Universe. I had customer meetings all day today. Here are some interesting items from my conversations.

Most said budgets were down in 2009 and will be flat to down in 2010. But a few who were related to government stimulus said theirs will be up.
Co-sourcing and outsourcing continue as ways to reduce costs
A few were focusing on asset management with the express purpose of getting rid of things in the environment they don’t need anymore. They know they are out there but they need to find them first.
Most customers I spoke to said they keep aggregated performance data for 2 years the range was 18 months to 5 years.
There was an interesting discussion about the definition of a business service versus an IT service. The point being made was a business service by definition involves more than IT. While I agree this is a good point, I think the IT industry has focused on business service as a way to say - “I’m thinking about this IT service in the context the business thinks about it not just from my own IT based perspective”
A number of customers have or are about to implement NNMi. If this is something you are interested in check out the NNMi Portal
Many customers are moving to virtualized environment highest percentage I heard was 70%. Another customer forces all internal developers to deliver software as a virtual image.
Another topic was how to monitor out tasked items. For example, some part of what you offer is delivered by a third party - how do you make sure they are living up to your standards. Two methods I heard were 1/ use HP Business Process Monitor 2/ get the 3rd party to send you alerts from their monitoring system.
On the question does your manager of managers send back data to sync the original tools 1 did, 1 didn’t. For the one who did it was part of a closed loop process.
- Monitor tool finds problem send alert to MOM (Manager of managers).
- MOM send event ID to monitoring tool
- Subject matter expert uses monitoring tools to diagnose problem
- Once diagnosed updates monitoring tool which updates MOM

A very productive day for me. I hope some of this is useful information to you.

For additional coverage my blogger buddy Pete Spielvogel is also here and beat me to the first post. You can read his posts at the ITOps Blog.

There are a variety of Twitter accounts you can follow as well as the hashtag #HPSU09

HPITOps – Covers BSM, Operations and Network Management

HPSU09 – show logistics and other information

HPSoftwareCTO

informationCTO

HPSoftware

BTOCMO – HP BTO Chief Marketing Officer

For HP BSM, Michael Procopio

BSM at HP Software Universe

Michael_Procopio — Thu, 11 Jun 2009 04:44:00 GMT

by Michael Procopio

HP Software Universe is next week, 16-18 June, in Las Vegas. Business Service Management (BSM) will be well represented.

In the Business Transaction Management area there are 13 sessions. Most of them are lead by customers. The sessions are listed below.

In the Network Management track Aruna Ravichandran is speaking in three sessions, you can see information on those at her post HPSoftware Universe/HP Technology Forum (HPTF) - Network Management sessions. The rest of the track is listed in the post Network Management at HP Software Universe.

Amy Feldman, Dennis Corning and Peter Spielvogel the ITOps bloggers has covered a number of the sessions in the Consolidated event and performance management. Here are a list of the posts:

Business Transaction Management Track

Session ID	Title	Presenting company
1114	Confessions of a product manager: get the real scoop on the latest HP Business Availability Center	HP
1165	The MITRE Corporation: higher operational effectiveness at lower cost through automated alert management	MITRE Corporation, AlarmPoint
1233	Key decisions and practical techniques in configuring business transaction management
1236	Real User Management: know how your TCP/IP applications perform for your users	HP
1267	Using HP Business Availability Center to analyze and triage application and infrastructure anomalies and problems	BCBS of Florida

1303	Sodexo: partnering with HP Software-as-a-Service to ensure critical e-business application performance and availability	Sodexo

1342	Wrigley: HP Business Availability Center deployed on Software-as-a-Service yields big improvements in IT monitoring without increasing staff	Wrigley


1360	Lockheed Martin: deploying HP Business Availability Center in a virtual environment and forwarding alerts through an iPhone Twitter-based application	Lockheed Martin


1363	DIRECTV: an HP Business Availability Center and HP operations implementation	DIRECTV
1401	Liberty Life: taking the fast track to implementing HP Business Availability Center and gaining business value in 6 months	Liberty Life
1425	Sentara Healthcare: improving the availability of critical business services and fixing IT problems before they impact customers	Sentara Healthcare
1436	Lockheed Martin: practical advice for configuring and operating HP End User Management solutions	Lockheed Martin
1452	Vale: deploying HP Business Availability Center solutions to monitor applications and systems and to help ensure availability and performance	Vale

You can get the details of all the BSM sessions at the HP Software Universe Track Session Catalog.

I hope to see you there, but if you can’t make it we will be doing follow-up posts. You can also follow on Twitter, the hashtag is #HPSU09. There are already a number of Tweets and the show hasn’t started yet. The Twitter account for the show is HPSU09, if you’d like to follow us. Or visit the HP Software Universe Facebook page.

For the Business Availability Center, Michael Procopio

Advanced analytics reduces downtime costs – isolation

Michael_Procopio — Fri, 08 May 2009 15:40:00 GMT

by Michael Procopio, Product Manager, BAC

In the world of advanced analytics, two areas that are of interest to the IT management world are: detection of a problem and isolation of a problem. Previously I wrote Advanced analytics reduces downtime costs – detection; in this post I’ll cover isolation.

In the previous post, I covered how advanced analytics finds an anomaly, potentially before a threshold is crossed.

Problem Isolation is the process of determining which component in the infrastructure is causing the problem* or incident* that we found. We will presume we are monitoring the service that is having the issue.

If one had no management tools (amazingly I have spoken to customers in this situation) the method of trying to find a problem is to login to each system, router, switch and potentially application (ex: Oracle) look at the items with whatever tools are available (ex: Windows Perfmon)and hopefully you find it. If you are interested in advanced analytics, this is probably not your situation.

The more typical case is you have multiple management tools, network, system, virtualization, database and perhaps others. So if you know the domain the problem exists in you have a good place to start. I’ve listened to podcasts / read reports which bring up few problems with this: (if you know of any good IT podcasts please send them along)

~80% of problems are sent to the network team with only ~20% being network issues
~60% of problems take >10 experts to resolve
~80% of the time to restore service is spent isolating the problem

Here is an analogy I use with my non IT friends on why this area is needed. You are monitoring the speed of a car going across the country (pick your favorite country). You are separately monitoring the infrastructure, all:

roads
bridges
ferries to take cars across the water

What you don’t know is where the car is (old car, no GPS). You are getting many alerts from the roads, bridges and ferries. Which one is affecting the car? Since you don’t know what road the car is on you don’t know if any given alert is the one affecting your car.

This is where the CDMB comes into the isolation process. The CMDB has the route the car is taking or, in our case, the items in the IT infrastructure that make up the service that has the problem.

Part one of the isolation process is to restrict what we are looking at to the relevant IT items. This greatly reduces the computational power required. For example, one customer I recently visited told me he has 2000+ servers. If we can reduce that to a few app servers and a few database servers (isn’t SOA wonderful for we operations types) that is a factor of ~200 reduction.

Part two of the isolation is the heavy math from HP Labs, with more patent filings. It is a form of regression analysis, where application or end user response time monitoring is the dependent variable and all the infrastructure metrics are independent variables. In plain terms, if end user response gets worse find the infrastructure metrics that get worse. When end user response gets better find the metrics that get better. The more closely an infrastructure metric tracks the end user response the more likely it is to be the cause.

Again, while the math is interesting, pictures work better for me.

The thick grey line is the end user response, the red-purple line is the most closely correlated metric -- in this case a database metric. Just so you don’t have to strain your eyes we provide a table like this (from a different problem) showing the weighted correlations score.

Isolation part 3 is to include non-time series data. In the screen capture below you see planned changes and incident details (think alerts) on the timeline. Unplanned changes can also be displayed. Changes are pulled from the CMDB and incidents can come from any management system that can send alerts. And since we know that most problems occur from changes that is an important component. Finally tickets from the helpdesk are included on the timeline, for the case where users are doing the monitoring.

All together this automates a number of things the operations teams already do and some math help isolating problems.

*Incident and problem are ITIL terms. There may be many incidents that are symptoms of an underlying problem.

tweet this!

Advanced analytics reduces downtime costs - detection

Michael_Procopio — Wed, 15 Apr 2009 18:01:00 GMT

by Michael Procopio

In the world of advanced analytics, two areas that are of interest to the IT management world are: detection of a problem and isolation of a problem. In this post I'll cover detection.

Problem detection, typically called anomaly detection, in the analytics circles started in a very basic way. Take a metric, say CPU utilization, set a threshold for it and anytime the threshold is crossed, we have an anomaly.

This, of course, has many problems:

· How do I know where to set a threshold

· The right level may be different at different times

· If there is a one sample spike above the threshold is that really an anomaly I care about (for some it is but not for most in my experience)

The next step in setting thresholds was using standard deviation (STD). I will create a sleeve of upper and lower bounds that cover a major percentage of the situations (+/- 1 STD covers 68.2%) I have measured and use that. This has some of the same problems as above. However, let’s focus on the time period problem.

The next step is to set thresholds by time of day. With this added capability I can set a reasonable threshold for the typical 10am and 2pm peak traffic periods separately and alerts still come if there is unusual behavior at 8am. This quickly leads to “my Mondays are busier than most of my other days”. To avoid false alerts, this leads us to time of day and day of week where we keep the standard deviation for each of the 168 hours of the week.

The next problem, the end of quarter booking and shipping madness or it is black Friday (largest shopping day of the year) and we realize we need to add in a seasonally adjusted set of thresholds as well. And something that seasonality can't take into account are macro events such as a weak economy affecting purchasing.

Of course, none of these will take into account the spikes mentioned above. How to solve all these problems -- hmmm.

There is a completely different approach, for which HP Labs has a patent filing, which uses more sophisticated machine learning. Like the other approaches, it breaks time up into segments, which, in the paper Achieving Scalable Automated Diagnosis of Distributed Systems Performance Problems, are called epochs. Unlike the other approaches, it does not simply compare now to a predetermined set of threshold levels.

This method compares ‘now’ to recent epochs as well as previous learning and makes a determination of what behavior is good and bad. While a spike is bad, if the epoch is behaving well overall it is considered to have good behavior. There is a lot of math behind this but I find looking a picture much more obvious.

Notice the gray outline the looks like a city skyline. This is what the algorithm has determine is ‘normal’ or good behavior. The hatch lines on the right show where it found an anomaly.

These advanced algorithms are implemented in HP Problem Isolation. In the next post, I’ll discuss how analytics are used to find the source of the problem.

tweet this!

Monitoring your cloud computing as easy as calling an airport shuttle

Michael_Procopio — Tue, 31 Mar 2009 05:15:00 GMT

HP made an announcement about new cloud computing management capabilities today: HP Unveils "Cloud Assure" to Drive Business Adoption.

HP currently offers Software-as-a-Service (SaaS) for individual management applications such as HP Business Availability Center (BAC) and HP Service Manager primarily for intranet and extranet applications.

HP Cloud Assure helps customers validate:

Security – by scanning networks, operating systems, middleware layers and web applications. It also performs automated penetration testing to identify potential vulnerabilities. This provides customers with an accurate security-risk picture of cloud services to ensure that provider and consumer data are safe from unauthorized access.
Performance – by making sure cloud services meet end-user bandwidth and connectivity requirements and provide insight into end-user experiences. This helps validate that service-level agreements are being met and can improve service quality, end-user satisfaction and loyalty with the cloud service.
Availability – by monitoring cloud-based applications to isolate potential problems and identify root causes with end-user environments and business processes and to analyze performance issues. This allows for increased visibility, service uptime and performance.

HP Cloud Assure provides control over the three types of cloud service environments:

For Infrastructure as a Service, it helps ensure sufficient bandwidth ability and validates appropriate levels of network, operating system and middleware security to prevent intrusion and denial-of-service attacks.
For Platform as a Service, it helps ensure customers who build applications using a cloud platform are able to test and verify that they have securely and effectively built applications that can scale and meet the business needs.
For Software as a Service, it monitors end-user service levels on the cloud applications, loads tests from a business process perspective and tests for security penetration.

A diagram showing the differences in the services is at Cloud Computing Basics.

In the end it doesn't matter where the service is; you need to be sure it is available and performing to expectations. Cloud Assure provides the capability in a way that is very agile. You say "I need this service monitored" and it is monitored. Its just like calling for an airport shuttle -- you call, they show up.

For Business Availability Center Michael Procopio, product manager HP Problem Isolation.