Tag Archives: Obiee

Visualising OBIEE DMS metrics with Graphite

Assuming you have set up obi-metrics-agent and collectl as described in my previous post, you have a wealth of data at your disposal for graphing and exploring in Graphite, including:

  • OS (CPU, disk, network, memory)
  • OBIEE’s metrics
  • Metrics about DMS itself
  • Carbon (Graphite’s data collector agent) metrics

In this post I’ll show you some of the techniques we can use to put together a simple dashboard.

Building graphs

First off, let’s see how Graphite actually builds graphs. When you select a data series from the Metrics pane it is added to the Graphite composer where you can have multiple metrics. They’re listed in a legend, and if you click on Graph Data you can see the list of them.

Data held in Graphite (or technically, held in whisper) can be manipulated and pre-processed in many ways before Graphite renders it. This can be mathmatical transforms of the data (eg Moving Average), but also how the data and its label is shown. Here I’ll take the example of several of the CPU metrics (via collectl) to see how we can manipulate them.

To start with, I’ve just added idle, wait and user from the cputotals folder, giving me a nice graph thus:

We can do some obvious things like add in a title, from the Graph Options menu

Graphite functions

Looking at the legend there’s a lot of repeated text (the full qualification of the metric name) which makes the graph more cluttered and less easy to read. We can use a Graphite function to fix this. Click on Graph Data, and use ctrl-click to select all three metrics:

Now click on Apply Function -> Set Legend Name By Metric. The aliasByMetric function is wrapped around the metrics, and the legend on the graph now shows just the metric names which is much smarter:

You can read more about Graphite functions here.

Another useful technique is being able to graph out metrics using a wildcard. Consider the ProcessInfo group of metrics that DMS provides about some of the OBIEE processes:

Let’s say we want a graph that shows cpuTime for each of the processes (not all are available). We could add each metric individually:

But that’s time consuming, and assumes there are only two processes. What if DMS gives us data for other processes? Instead we can use a wildcard in place of the process name:

obieesample.DMS.dms_cProcessInfo.ProcessInfo.*.cpuTime

You can do this by selecting a metric and then amending it in the Graph Data view, or from the Graph Data view itself click on Add and use the auto-complete to manually enter it.

But now the legend is pretty unintelligable, and this time using the aliasByMetric function won’t help because the metric name is constant (cpuTime). Instead, use the Set Legend Name By Node function. In this example we want the third node (the name of the process). Combined with a graph title this gives us:

This aliasbyNode method works well for Connection Pool data too. However it can be sensitive to certain characters (including brackets) in the metric name, throwing a IndexError: list index out of range error. The latest version of obi-metrics-agent should workaround this by modifying the metric names before sending them to carbon.

The above graph shows a further opportunity for using Graphite functions. The metric is a cumulative one – amount to CPU time that the process has used, in total. What would be more useful would be if we could show the delta between each occurrence. For this, the derivative function is appropriate:

Sometimes you’ll get graphs with gaps in; maybe the server was busy and the collector couldn’t keep up.

2014-03-28_07-29-47

To “gloss over” these, use the Keep Last Value function:

2014-03-28_07-30-51

Saving graphs

You don’t have to login to Graphite by default, but to save and return to graphs and dashboards between sessions you’ll want to. If you used the obi-metrics-agent installation script then Graphite will have a user oracle with password Password01. Click the Login button in the top right of the Graphite screen and enter the credentials.

Once logged in, you should see a Save icon (for you young kids out there, that’s a 3.5″ floppy disk…).

You can return to saved graphs from the Tree pane on the left:

flot

As well as the standard Graphite graphing described above, you also have the option of using flot, which is available from the link in the top-right options, or the icon on an existing graph:

2014-03-30_21-44-43

Graphlot/Flot is good for things like examining data values at specific times:

2014-03-30_21-47-36

Creating a dashboard

So far we’ve seen individual graphs in isolation, which is fine for ad-hoc experimentation but doesn’t give us an overall view of a system. Click on Dashboard in the top-right of the Graphite page to go to the dashboards area, ignoring the error about the default theme.

You can either build Graphite dashboards from scratch, or you can bring in graphs that you have prepared already in the Graphite Composer and saved.

At the top of the Graphite Dashboard screen is the metrics available to you. Clicking on them drills down the metric tree, as does typing in the box underneath

Selecting a metric adds it in a graph to the dashboard, and selecting a second adds it into a second graph:

You can merge graphs by dragging and dropping one onto the other:

Metrics within a graph can be modified with functions in exactly the same way as in the Graphite Composer discussed above:

To add in a graph that you saved from Graphite Composer, use the Graphs menu

You can resize the graphs shown on the dashboard, again using the Graphs menu:

To save your dashboard, use the Dashboard -> Save option.

Example Graphite dashboards

Here are some examples of obi-metrics-agent/Graphite being used in anger. Click on an image to see the full version.

  • OS stats (via collectl)
    OS stats from collectl
  • Presentation Services sessions, cache and charting
    Presentation Services sessions, cache and charting
  • BI Server (nqserver) Connection and Thread Pools
    BI Server (nqserver) Connection and Thread Pools
  • Response times vs active users (via JMeter)
    Response times vs active users (via JMeter)

The Secret Life of Conditional Formatting in OBIEE

When dealing with conditional formatting the GUI allow us to add as many formats as we want, but every single format has a unique and single condition, not more, not less. In this post I am going to show how it is possible in OBIEE to use multiple conditions to define a special formatting on an analysis column. Sometimes the requirement for conditional formatting is a little more complex, mainly when using pivots, requiring 2 conditions like for example “year = current-year AND country = UK”. In general we try to manage this requirement by a set of conditional formats sorted in a specific order trying to achieve something as close as possible to this, sometimes also setting one format first and then overriding it back in some cells of a pivot to make them looks like the cells without special format. It would be a lot easier to be able to just define a more complex rule for the special format but the GUI doesn’t allow it.

Does it means OBIEE can’t manage complex conditions formatting? No, not at all! Let me introduce you the secret life of conditional formatting…

Before we look at the practical implementation, first let us consider the “history” of conditional formatting, and why we can be optimistic that the method I describe will work.

It must be noted that because it’s not a functionality available through the GUI there is no guarantee it will work correctly or OBIEE will support it in futures releases, but for me it works fine in OBIEE 11.1.1.7 and related patched versions.

Some theory

OBIEE Analyses are stored in an XML format that OBIEE interprets when building and running the report. Using the Advanced tab of an analysis we can examine this XML. The XML can tell you a lot of information and give some hints about functionality the GUI doesn’t allow to perform but based on the XML you can easily guess it will work if coded by hand. Let’s focus on the XML related to conditional formatting as it’s the topic of this post.

Part of the XML standard is an associated XSD file, the XML Schema definition. OBIEE’s XML honours this and we can use this to examine the XML that OBIEE will accept and expect for an analysis. jDeveloper is a good tool to analyze XSD thanks to its built-in “design/source” viewer.
This is the graphical representation of the XSD describing the code behind conditional formatting (click to zoom). If you compare it with the XML you will recognize most of the elements. The important element is into the formatRule block, inside condition: the XSD expect a sawx:expr element there.

cond-format_displayFormat_XSD

Why is it important? Well if you look at a separate part of the XSD, which deals with the filters in an analysis.

cond-format_filter_XSD

Surprise: it’s the same sawx:expr. What does it mean for us? Because the filters accept AND and OR logical operator to join filters together, the XML of the analysis will accept the same syntax in the conditional formatting condition block. There is no guarantee at this point it will work, the XML will be valid but if the code parsing and interpreting the XML doesn’t implement this functionality it will not produce anything (or an error in the worst case). Time to move to the practical part and try it.

Step by step example

I start by creating my pivot where I display Revenues by Country (filtered list to some elements) and Line of Business (LOB).

cond-format_simple_pivot

Now my target is to highlight figures for “Switzerland” in the countries and “Games” in the LOB, so I add a conditional format on the Revenue column and set a red background for Switzerland and yellow background for Games.

cond-format_normal_condition

As expected the result is my pivot with a red row (for Switzerland) and a yellow column (for Games).

cond-format_normal_condition_result

It doesn’t really looks good because the intersection of Switzerland and Games will use the format of the last matched condition, in my case the yellow of Games because it’s the second conditional formatting defined. I would prefer to have this intersection in a nice orange background, but the GUI doesn’t have an option to combine conditions.

The GUI doesn’t do it but it doesn’t mean it’s not possible! Let’s have a look at the XML managing the conditional formatting. First we look an example of the code producing the red background based on the country.

<saw:conditionalDisplayFormat>
  <saw:formatRule>
    <saw:condition>
      <sawx:expr xsi:type="sawx:comparison" op="equal">
        <sawx:expr xsi:type="sawx:columnRefExpr" columnID="c632a4f0428ecfa91"/>
        <sawx:expr xsi:type="xsd:string">Switzerland</sawx:expr>
      </sawx:expr>
    </saw:condition>
    <saw:formatSpec backgroundColor="#FF0000" wrapText="true"/>
  </saw:formatRule>
</saw:conditionalDisplayFormat>

Now, as we saw before, the XSD allow me to use the syntax of filters, including filters containing logical operators (AND, OR) to join conditions together. So if I add the XML with the AND operator I will be able to use the condition country=Switzerland AND LOB=Games to apply a different format.

In a text editor I prepare the new XML for the analysis with an additional <saw:conditionalDisplayFormat> element with my new hand-made condition.

It looks good and I can now try to paste it back in OBIEE and see if the syntax is correct and my XML will be accepted: in the Advanced tab of the analysis I replace the existing XML with my own version of the code.

<saw:conditionalDisplayFormat>
  <saw:formatRule>
    <saw:condition>
      <sawx:expr xsi:type="sawx:logical" op="and">
        <sawx:expr xsi:type="sawx:comparison" op="equal">
          <sawx:expr xsi:type="sawx:columnRefExpr" columnID="c3d00191589cdd4e2"/>
          <sawx:expr xsi:type="xsd:string">Games</sawx:expr>
        </sawx:expr>
        <sawx:expr xsi:type="sawx:comparison" op="equal">
          <sawx:expr xsi:type="sawx:columnRefExpr" columnID="c632a4f0428ecfa91"/>
          <sawx:expr xsi:type="xsd:string">Switzerland</sawx:expr>
        </sawx:expr>
      </sawx:expr>
    </saw:condition>
    <saw:formatSpec backgroundColor="#FF9900" wrapText="true"/>
  </saw:formatRule>
</saw:conditionalDisplayFormat>

When clicking “Apply XML” OBIEE will parse and evaluate the XML. If there is an error a message is displayed and the code is not applied (so as to not break the analysis). If you get an error, double check your XML, make sure every tag you open is closed and read the error message as it says what is the problem.

cond-format_edit_xml

Time to click the Results tab and check the new result of my analysis. YES! There is a nice orange background in the intersection now, proving that even if the GUI can’t do it OBIEE is able to accept complex conditions to define conditional formatting.

cond-format_normal_condition_final_result

What do we see if we check the properties of the column in the GUI? As you can see in the next screenshot we see there is a condition setting an orange background, but the condition itself is just shown as sawx:expr:sawx:logicaland. Where does this text come from? It’s a concatenation of the name of the first XML tag containing the logical operator and the values of the attributes of the tag itself. So if you use a OR instead of AND you will see a condition named sawx:expr:sawx:logicalor.

cond-format_final_condition

Can we edit the condition in the GUI? Not really, the edit window is opened, but the formula will not be recognized and if you set an operator and a value you will lose your hand-made conditional formatting and generate a meaningless piece of XML, so don’t do it for any reason. On the other hand the format can be edited using the GUI with no impact on the rule, so feel free to do it and correct the format if you don’t feel comfortable coding a format by hand in the XML.

cond-format_final_condition_edit

To resume, and some advice

  • It’s possible to have a complex formula as condition in a conditional formatting rule.
  • It’s not supported by the GUI, needs to be done by hand in XML.
  • Do it as one of the last steps when building an analysis because you can’t edit it via the GUI.
  • Be familiar with the produced XML before you play with it.
  • Use the Filters section to create the condition and avoid mistakes, so a copy/paste of the generated rule will help in creating the condition.
  • Document, document, document! As the condition is visible only in the XML document it somewhere, even just in a Static Text view you add to the analysis itself (and don’t display on the screen).
  • Be aware of the maintenance overhead that direct XML manipulation will have – don’t abuse the “cheating”.
  • If you don’t want to hack the XML, an alternative solution can be to use a extra column added to the analysis to evaluate the complex condition and produce a flag used for the conditional formatting. The downside of this is that the condition will be sent to the database as part of the query for evaluation.

Purging corrupted OBIEE web catalog users

Sometimes it can happen that user profiles within a web catalog become corrupted for any number of reasons. In order for these user profiles to be correctly re-initialized, there's more to be done than just drop /users/JohnDoe from the web catalog.

All in all there are three distinct places which need to be cleaned:
  • /users/JohnDoe
  • /system/users/123456
  • /system/acocuntids/987654
This is really important since especially the third place contains the translation between the userid and the effective GUID of the user. I've written a little script which takes the absolute path to the web catalog in question as well as the user to be purged and kills everything that's necessary.

NOTE: This is a quick&dirty solution and I haven't fool-proofed it with any check like "does the folder exist" etc. so use it cautiously and on your own risk.

I may get around to rendering it safer later-on, but since I was asked for it once more just today I thought I'd put it out there.


#!/bin/bash
#
# Purpose: Completely purge a named user from the web catalog with all his content.
#
# Requires absolute path to webcat as param 1 and user name as param 2
#
#
# Author: Christian Berg
# Initial creation: 28/01/2014
# Absolutely no warranty, use at your own risk
# Please include this header in any copy or reuse of the script you make
#
# Current version: 1.0
#
# Change log:
#        CBERG 28/01/2014 Intial creation
#


########################################################################################
# Step 1: Kill the users personal folder plus content with its accompagning .atr       #
########################################################################################

cd $1/root/users

find -type d -name $2 | xargs rm -rf
find -name $2.atr | xargs rm -f

########################################################################################
# Step 2: Kill the users entries in /system/users                                      #
# Two files will be affected "username" and "username.atr"                             #
# Removal happens one-by-one                                                           #
########################################################################################

cd $1/root/system/security/users

find -name $2 | xargs rm -f
find -name $2.atr | xargs rm -f

########################################################################################
# Step 3: Kill the users entries in /system/accountids                                 #
# Two files will be affected. Accountids contains the translation from GUID to         #
# username, so the actual username resides within the files content rather than its    #
# name. Bulk removal.                                                                  #
########################################################################################

cd $1/root/system/security/accountids

grep -r -l $2 . | xargs rm -f

echo User $2 has been purged from the web catalog.

exit 0

Working around opatch – Prerequisite check "CheckSystemSpace" failed.

I recently ran into the situation where the primary mount for a Linux tech account running an OBI install was just way too small to get OBIEE 11.1.1.7.140114 through.
Prerequisite check "CheckSystemSpace" failed.
The details are:
Required amount of space(17499.766MB) is not available.
So with a bit of hacking I got around it by displacing the ./patch_storage directory and forcing opatch to stop doing a file system check (basically no "df -h" )

1.) displace ./patch_storage to a mount with enough space (but keep a backup in place just in case...)
cp -r /FMWH/Oracle_BI1/.patch_storage /data/NASmnt00001/

mv /FMWH/Oracle_BI1/.patch_storage_bkp


2.) create a symbolic link to take the place of ./patch_storage
ln -s /data/NASmnt00001/.patch_storage ./.patch_storage
After this step if you execute opatch normally, it will still fail with "CheckSystemSpace" failed.


3.) Have opatch omit the space check:

opatch napply -silent /data/NASmnt00001/11.1.1.7.140114 -id 16569379,16913445,16997936,17300045,17300417,17922352,17922552,17922577,17922596 OPatch.SKIP_VERIFY_SPACE=true

Done.

(h/t @G_Ceresa for being picky and getting on my nerves with "That's not proper." ... I said it's a hack, mate ;-))

BITeamwork 2.7 is Now Released

BITeamwork 2.7 is Now Released. At the time of this post BITeamwork has seen a solid increase in legitimate download requests for the Oracle BI Tool since last quarter, Q4 of 2013, and we are excited that BI experts in the enterprise are starting to care and believe in Collaborative BI. For us the uptick of […]

The post BITeamwork 2.7 is Now Released appeared first on Art of Business Intelligence Blog.