Tag Archives: 11g
How-to: Hadoop Integration with OBIEE 11g
What is Hadoop?
Hadoop is a framework that enables data to be distributed amongst many servers (nodes), commonly referred to as a 'distributed file system'. The data is not stored in a single database, rather it is spread across multiple clusters.How does Hadoop process data stored in multiple nodes?
Hadoop uses a programming model called 'MapReduce' for parallel processing across multiple nodes. At a high level this is comprised of two steps:- Map step
- The map step takes the data, divides it into smaller sets of data and distributes the result to worker nodes
- Reduce step
- The reduce step collects the data from all of the worker nodes and aggregates it into a single 'output'
What is Hive?
MapReduce functions are generally written in Java and generally require someone with deep knowledge in both Hadoop and MapReduce. The guys over at facebook created a technology called 'Hive' which is a data warehouse infrastructure that sits on top of Hadoop. More simply, Hive does the 'heavy lifting' of creating the MapReduce functions. In order to query a Hadoop distributed file system, instead of having to write MapReduce code, you generate sql-style code in a hive language called 'HQL'Why does this matter in the Oracle Business Intelligence / Analytics space?
The analytics space is experiencing a shift in both technology and function. Traditional BI projects required a 'data warehouse' to store data in a series of star schemas (denormalized models) for quick query generation and data retrieval. The development and support of the data warehouse is achieved through a team of ETL developers whose main focus is to create the mappings that perform the data transformation from the source to the target.
Unless the functional requirements are clearly understood during this phase, value is usually lost in the data transformation and the potential to eliminate relevant data is certainly possible.
Using OBIEE 11g's Hadoop integration via a Hive ODBC, OBIEE can directly query distributed file systems via Hive. What does this mean? The potential now exists to eliminate or reduce the need for ETL as we now have the ability to directly query gigantic file systems.
The saving grace to ETL developers is that a need still exists for someone to create the HQL functions that populate the 'tables' that OBIEE uses. Ultimately, it could be a change in how ETL is developed.
How do you integrate OBIEE 11g with Hadoop?
Step 1: Download the Hive ODBC Drivers from http://support.oracle.com
You can reference Oracle Note 'Using Oracle Hadoop ODBC Driver with BI Administration Tool [ID 1520733.1]'Step 3: Configure Database Connection
Step 4: Create Connection Pool
How-to: Data Visualization with External Javascript Libraries (D3)
Before we get started, you can view all of the D3 visualization methods at their github. In the example below we're going to use airline data to and D3's Calendar View to visualize average flight delays. You will need OBIEE 11.1.1.6.2 or higher (this example uses OBIEE 11.1.1.7.0) and IE 9+.
Step 0: Create an Answers Report
This report should contain a year dimension, a date dimension and an aggregate fact column. In the airline example I've selected 'Date', 'Year' and 'Average Departure Delay'. Take note of the column order as you will have to reference the column number in a narrative.Step 1: Download the D3 Javascript Library from github
user_projectsdomainsbifoundation_domainserversbi_server1tmp_WL_useranalytics_11.1.17dezjlwarresb_mozillacommon
Step 2: Create css file for Calendar Formatting
The Calendar view's javascript code is basically one script, with one function and one css file. These 'chunks of code' are all stored in the index.html using the example located on github, but in order for this view to play nice with OBIEE 11g, we're going to need to dissect components of the code into isolated narratives and css files. The first step is to take the css code:#chart {and save it to its own css file (calendar.css) located at:font: 10px sans-serif;shape-rendering: crispEdges;}.day {fill: #fff;stroke: #ccc;}.month {fill: none;stroke: #000;stroke-width: 2px;}
user_projectsdomainsbifoundation_domainserversbi_server1tmp_WL_useranalytics_11.1.17dezjlwarresb_mozillacommond3examplescalendarcalendar.css (you will need to create the directory as this doesn't exist)
Step 3: Create an Answers Narrative to Execute the Javascript Library
Now that we've laid the groundwork for calling the D3 library, the next step is to integrate the Calendar View code into an Answers narrative.First create the script headers and link type to call the javascript library. This code will be stored in the pre-fix of the narrative:
<script type="text/javascript" src="/analytics/res/b_mozilla/common/d3/d3.js"></script>Next we're going to take the calendar view code and copy the entire code block from the start of the width variable delcaration to the end of the call to the selectAll function. Your code should look similar to:<link type="text/css" rel="stylesheet" href="/analytics/res/b_mozilla/common/d3/lib/colorbrewer/colorbrewer.css"/><link type="text/css" rel="stylesheet" href="/analytics/res/b_mozilla/common/d3/examples/calendar/calendar.css"/>
<script type="text/javascript" src="/analytics/res/b_mozilla/common/d3/d3.js"></script><link type="text/css" rel="stylesheet" href="/analytics/res/b_mozilla/common/d3/lib/colorbrewer/colorbrewer.css"/><link type="text/css" rel="stylesheet" href="/analytics/res/b_mozilla/common/d3/examples/calendar/calendar.css"/><div id="my_chart"></div><script type="text/javascript">var margin = {top: 19, right: 20, bottom: 20, left: 19},width = 720- margin.right - margin.left, // widthheight = 136 - margin.top - margin.bottom, // heightcellSize = 12; // cell sizevar day = d3.time.format("%w"),week = d3.time.format("%U"),percent = d3.format(".1%"),format = d3.time.format("%Y-%m-%d");var color = d3.scale.quantize().domain([5,30]).range(d3.range(9));var svg = d3.select("#my_chart").selectAll("svg").data(d3.range(year_range1, year_range2)).enter().append("svg").attr("width", width + margin.right + margin.left).attr("height", height + margin.top + margin.bottom).attr("class", "RdYlGn").append("g").attr("transform", "translate(" + (margin.left + (width - cellSize * 53) / 2) + "," + (margin.top + (height - cellSize * 7) / 2) + ")");svg.append("text").attr("transform", "translate(-6," + cellSize * 3.5 + ")rotate(-90)").attr("text-anchor", "middle").text(String);var rect = svg.selectAll("rect.day").data(function(d) { return d3.time.days(new Date(d, 0, 1), new Date(d + 1, 0, 1)); }).enter().append("rect").attr("class", "day").attr("width", cellSize).attr("height", cellSize).attr("x", function(d) { return week(d) * cellSize; }).attr("y", function(d) { return day(d) * cellSize; }).datum(format);rect.append("title").text(function(d) { return d; });svg.selectAll("path.month").data(function(d) { return d3.time.months(new Date(d, 0, 1), new Date(d + 1, 0, 1)); }).enter().append("path").attr("class", "month").attr("d", monthPath);var csv =[];
Notes About this Code
Although this code does most of the heavily lifting and can be left unmodified, there are specific lines that can be changed and updated dynamically via the use of presentation variables.Color Thresholds:
The color variable specifies the thresholds for red/yellow/green. In this case I deem the min and max ranges of an airline delay to be between 5 minutes and 30 minutes:var color = d3.scale.quantize().domain([5,30])
Chart Size Adjustment:
By modifying the code for the margin variable:var margin = {top: 19, right: 20, bottom: 20, left: 19},The height/width/cell size can be adjustable by changing the hardcoded values to presentation variables such as:width = 720- margin.right - margin.left, // widthheight = 136 - margin.top - margin.bottom, // heightcellSize = 12; // cell size
- @{Width}
- @{Height}
- @{CellSize}
Date Formatting:
The 'day' variable responsible for date formatting:var day = d3.time.format("%w"),Requires that the format of the date be specified. The Calendar View script by default uses a 'YYYY-MM-DD' format. If your OBIEE data is a MM-YY-DD format or has a timestamp, you will need to modify the column data format to the following:week = d3.time.format("%U"),percent = d3.format(".1%"),format = d3.time.format("%Y-%m-%d");
Modifying the Date Range:
Could be modified to:var svg = d3.select("body").selectAll("svg")
.data(d3.range(1990, 2011))
var svg = d3.select("#my_chart").selectAll("svg")In the upcoming steps I will show how these variables can be called.
.data(d3.range(year_range1, year_range2))
Step 4: Populate the Narrative and Post-Fix
In the narrative you will need to specify the Date and Metric you want to pass to the javascript function using the corresponding column number (see step 0 if you forgot!)The Post-Fix should contain the remainder of the Calendar View code. This can remain unmodified:
var data = d3.nest()Your narrative should be similar to:
.key(function(d) { return d.Date; })
.rollup(function(d) { return d[0].Metric; })
.map(csv);
rect.filter(function(d) { return d in data; })
.attr("class", function(d) { return "day q" + color(data[d]) + "-9"; })
.select("title")
.text(function(d) { return d + ": " + (data[d]); });
function monthPath(t0) {
var t1 = new Date(t0.getFullYear(), t0.getMonth() + 1, 0),
d0 = +day(t0), w0 = +week(t0),
d1 = +day(t1), w1 = +week(t1);
return "M" + (w0 + 1) * cellSize + "," + d0 * cellSize
+ "H" + w0 * cellSize + "V" + 7 * cellSize
+ "H" + w1 * cellSize + "V" + (d1 + 1) * cellSize
+ "H" + (w1 + 1) * cellSize + "V" + 0
+ "H" + (w0 + 1) * cellSize + "Z";
}
</script>
Step 5: Create a Second Narrative for the Date Range
Step 6: View Narratives in Answers
Changing WebLogic Server Deployment Order using MBeans
This post is brought about from some new development on the BITeamwork Collaborative BI solution where one customer had an issue with a delay in the Oracle BI start up supposedly from BITeamwork being deployed as an application with the WebLogic Server default Deployment Order value of 100. This was a valid concern since BITeamwork seeks to call the /analytics JEE application's web services when it starts in order to assess if OBIEE Presentation Services is actually started.
The post Changing WebLogic Server Deployment Order using MBeans appeared first on Art of Business Intelligence.
How-to: Enable Case Insensitive User Name Log-in with OBIEE 11g
In summary, OBIEE 11g authenticates using an OID authenticator via OPSS (Oracle Platform Security Services), followed by a BISQLGroupAuthenticator - both configured in Weblogic Admin Console:
After configuring these authenticators (outlined here and here), you run a test by logging in a user who exists in your LDAP authenticator:
Example 1: Logging in with a valid username, valid password and correct username case sensitivity
- username: cookjr (username is stored as 'cookjr' in external database table)
- member of the following application roles: Authenticated User, BIConsumer, BIAuthor, BIAdministrator
- username: cOoKjR (username is stored as 'cookjr' in external database table)
- member of the following application roles: Authenticated User, BIConsumer, BIAuthor, BIAdministrator
Note how none of the custom application roles were applied to this login attempt even though it's the same user logging in.
WHY?
It is important to remember that the BISQLGroupAuthenticator used for group authentication is a driver that is used to generate Oracle SQL statements that you configure when setting up the authenticator in Weblogic Admin Console:
- SQL List Groups: SELECT G_NAME FROM GROUPS WHERE UPPER(G_NAME) LIKE UPPER(?)
- SQL List Groups: SELECT G_NAME FROM GROUPS WHERE UPPER(G_NAME) LIKE UPPER(?)
- SQL Is Member: SELECT G_MEMBER FROM GROUPMEMBERS WHERE UPPER(G_NAME) = UPPER(?) AND UPPER(G_MEMBER) = UPPER(?)
- SQL List Member Groups SELECT G_NAME FROM GROUPMEMBERS WHERE UPPER(G_MEMBER) = UPPER(?)
- SQL Get Group Description (if description supported enabled) SELECT G_DESCRIPTION FROM GROUPS WHERE UPPER(G_NAME) = UPPER(?)
BITeamwork 1.8 (Nobel Release) – It’s Dynamite
BITeamwork has been out since Oracle Open World 2012 but since its involvement with several customers the product has evolved into a really amazing software offering. Clearly BITeamwork is the collaboration solution for Oracle BI which allows users to create dashboard comments, table and pivot table cell annotations, and then manage it all but it also has some amazing administration features. Actually, the roadmap for BITeamwork is to make up the gaps that customers see in the BI product such as custom metadata dictionaries, better management tools for ibots/agent, and a better caching system. Customers and implementers of Oracle BI should support the product now so that these items actually come to fruition.
The post BITeamwork 1.8 (Nobel Release) – It’s Dynamite appeared first on Art of Business Intelligence.