Tag Archives: Conferences
Previewing Four Sessions at the Atlanta Rittman Mead BI Forum 2015
In a post earlier this week I previewed three sessions at the upcoming Brighton Rittman Mead BI Forum 2015; in this post I’m going to look at four particularly interesting sessions at the Atlanta Rittman Mead BI Forum 2015 event running the week after Brighton, on May 13th-15th 2015 at the Renaissance Atlanta Midtown Hotel, Atlanta GA. As well as an optional one-day masterclass on big data development by myself and Jordan Meyer on the 13th, the main event itself has keynotes and product update sessions from Oracle’s BI product management team, a data visualisation challenge and a guest talk by John Foreman, author of the book “Data Smart” and Chief Data Scientist at Mailchimp; in terms of the main sessions though there are four that I’m particularly interested in, starting with one by a speaker new to the BI Forum, Qualogy’s Hasso Schaap, who’ll be talking to us about their use of Oracle’s new BI Cloud Service in his session “Developing strategic analytics applications on OBICS PaaS”
“In this session I’ll tell how we use the Oracle BI Cloud Service in our development plans for a strategic analytics application. Focussing on Strategic HR Planning there’s so much you can do with your data that we decided to put it in a packaged app. I will discuss the important parts of the development process and show how we fixed the issues we came up with. Developing in the BI Cloud is different and expectations are also different.
As an example there’s the part of prediction. How do we predict based on data in the BI Cloud and what are other possibilities. With prediction we were able to tell our customers a different story. A story that was different than before using old-school tools and techniques. In this session I will uncover some of the most appreciated functionality and will happily elaborate on the story behind ‘The present, the future, development and scenario planning’.”
My second featured session is by someone very-well known to previous BI Forum attendees, and to the wider Oracle BI+DW community: Stewart Bryson. Stewart of course used to head-up Rittman Mead in the US and then went-on to become our first Chief Innovation Officer, before leaving to start his own company Red Pill Analytics with Kevin McGinley, another old friend of Rittman Mead and the BI Forum. We’re very pleased to have both Stewart and Kevin delivering sessions at the Atlanta BI Forum, and for Stewart’s session he’s talking about something very close to his heart – “Supercharging BI Delivery with Continuous Integration”:
“One of the things I’ve never understood about the lifecycle features in most BI tools is why the designers feel the need to roll their own source control and DevOps features. Instead of focusing on deeper integration with tools and processes that exist in the other 90% of development paradigms, BI vendors instead start with a clean palette and create something completely siloed and desperately alone.
In this presentation, we’ll take a look at how some of these other development paradigms approach DevOps — paying perhaps the closest attention to the world of Java development and other JVM languages. We’ll see how approaches such as continuous integration and continuous delivery play a part in rapid, iterative delivery, and how we can apply some of those approaches to the world of OBIEE development.”
My third session is by another speaker new to the BI Forum, but someone who’s well-known in the BI and data warehousing world and who I met in-person for the first time at last year’s Oracle Openworld: Sumit Sarkar. Sumit works for Progress Software, makers of the DataDirect ODBC drivers that powers OBIEE’s connection to Hadoop, for example, as well as connectors to MongoDB, Salesforce, Oracle RightNow and Eloqua, and as he’ll explain in his session “Make sense of NoSQL data using OBIEE”, NoSQL databases :
“NoSQL databases have stormed the top 10 db-engines rankings with MongoDB at #4 and Cassandra at #8. It’s inevitable that these NoSQL databases, storing unstructured data without a standard query language, will have BI requirements for unarmed OBIEE teams. Not even a complete Oracle stack can save you with the release of Oracle NoSQL.This will be the first session of its kind to tackle standards based NoSQL connectivity.
So join me at BI Forum ’15 to take control of NoSQL data with your RPD and expand big data skills and thought leadership within your organization. Learn how organizations are using SQL access to NoSQL databases for integration across existing business intelligence platforms. We’ll talk about common challenges and gotchas that shops are facing when exposing unstructured NoSQL data to OBIEE. It can get out of hand pretty quickly otherwise …”
My final selection is from CERN, the European Organization for Nuclear Research and home of course of the Large Hadron Collider (and who announced on April 1st the first unequivocal evidence for The Force, almost upstaging our announcement of Oracle E-Business Suite being ported to Hadoop and MongoDB). There’s several session at both the Brighton and Atlanta BI Forums on Oracle’s new Big Data Discovery tool, and in this session CERN’s Manuel Martin Marquez will be talking about their work in this area, in his session “Governed Information Discovery: Data-driven decisions for more efficient operations at CERN”
“The European Centre for Nuclear Research, CERN, is running the world’s largest and more powerful particle accelerator complex in order to shed light on how the Universe works and which are its main building blocks. CERN’s particle accelerators and detectors infrastructure is comprehensively heterogeneous and complex. A number of critical subsystems, which represent cutting-edge technology in several engineering fields, need to be considered: cryogenics, power converters, magnet protection, etc. The historical monitoring and control data derived from these systems has persisted mainly using Oracle database technologies, but also other sorts of data formats such as JSOM, XML and plain text files. All of these must be integrated and combined in order to provide a full picture and better understanding of the overall status of the accelerator complex.
Therefore, a key challenge is to facilitate easy access to, flexible interaction with, and dynamic visualization of heterogeneous data from different sources and domains. In our session, we will share our experience with a potential solution for finding insights within our data, Oracle Endeca Data Discovery. In addition, we will feature practical examples relating to future possibilities for improving the control and monitoring of CERN’s accelerator complex, optimization results for accelerator operations and a demo of the implemented solution”
Full agenda details on the Atlanta Rittman Mead BI Forum 2015 can be found on the event homepage, along with details of the optional one-day masterclass on Delivering the Oracle Information Management and Big Data Reference Architecture, and our first-ever Data Visualisation Bake-Off, using the DonorsChoose.org dataset. Registration is now open and the event takes place between May 13th and 15th April 2015, at the Renaissance Atlanta Midtown Hotel, Atlanta GA.
Announcing the BI Forum 2015 Data Visualisation Challenge
The Rittman Mead BI Forum 2015 is running in Brighton from May 6th-8th 2015, and Atlanta from May 13th – 15th 2015. At this year’s events we’re introducing our first “data visualization challenge”, open to all attendees and with the dataset and scenario open from now until the start of each event. Using Oracle Business Intelligence 11g and any plugins or graphics libraries that embed and interact with OBIEE (full details and rules below), we challenge you to create the most effective dashboard or visualisation and bring it along to demo on the Friday of each event.
Help DonorsChoose.org Donors Use their Funds Most Effectively
This year’s inaugural data visualisation challenge is based around the DonorsChoose.org project and dataset, an online charity that makes it easy for anyone to help public school classroom projects that need funding (Rittman Mead will be making donations on behalf of the Brighton and Atlanta BI Forums to show our support for this great initiative). The Donorschoose.org project and dataset have been used in several hackathons and data crunching contests around the world, with analysis and visualisations helping to answer questions such as:
- Why do some projects get funded, while others don’t?
- Who donates to projects from different subjects?
- Does proximity to schools change donation behavior?
- What types of materials are teachers lacking the most? (eg chalk, paper, markers, etc)
- Do poorer schools ask for more or less money from their donors?
- If I need product x, what is the difference between projects asking for x that were successful vs those that aren’t.
More details on uses of the Donorschoose.org dataset can be found on the Donorschoose data blog, and example visualisations you could use to get some ideas and inspiration are on the Donorschoose.org Data Gallery showcase page.
Your challenge is to import this dataset into your analytical database of choice, and then create the best visualisation or dashboard in OBIEE to answer the following question: “Which project can I donate to, where my donation will have most impact?”
How Do I Take Part?
For more on the BI Forum 2015 Data Visualization Challenge including how to download the dataset and the rules of the challenge, take a look at the Rittman Mead BI Forum 2015 Data Visualisation Challenge web page where we’ve provided full details. You can either enter as an individual or as part of a team, but you must be registered for either the Brighton or Atlanta BI Forum events and come along in-person to demonstrate your solution – numbers at each event are strictly limited though, so make sure you register soon at the Rittman Mead BI Forum 2015 home page.
RMOUG15 Wrap-Up – Art of BI Software was There
We just wrapped up our 2nd conference event in Denver, Colorado at the Rocky Mountain Oracle User Group (RMOUG). This was the first time we sponsored the RMOUG event. There was a fantastic turnout with experts in all areas of Oracle technology. It was an excellent opportunity to expose many of the Oracle Database Administrators […]
The post RMOUG15 Wrap-Up – Art of BI Software was There appeared first on Art of Business Intelligence Blog.
Rittman Mead BI Forum 2015 Now Open for Registration!
I’m very pleased to announce that the Rittman Mead BI Forum 2015, running in Brighton and Atlanta in May 2015, is now open for registration.
Back for its seventh successful year, the Rittman Mead BI Forum once again will be showcasing the best speakers and presentations on topics around Oracle Business Intelligence and data warehousing, with two events running in Brighton, UK and Atlanta, USA in May 2015. The Rittman Mead BI Forum is different to other Oracle tech events in that we keep the numbers attending limited, topics are all at the intermediate-to-expert level, and we concentrate on just one topic – Oracle Business Intelligence Enterprise Edition, and the technologies and products that support it.
As in previous years, the BI Forum will run on two consecutive weeks, starting in Brighton and then moving over to Atlanta for the following week. Here’s the dates and venue locations:
- Rittman Mead BI Forum 2015 – Hotel Seattle, Brighton, May 6th – 8th 2015
- Rittman Mead BI Forum 2015 – Renaissance Atlanta Midtown Hotel, May 13th – 15th 2015
This year our optional one-day masterclass will be delivered by Jordan Meyer, our Head of R&D, and myself and will be on the topic of “Delivering the Oracle Big Data and Information Management Reference Architecture” that we launched last year at our Brighton event. Details of the masterclass, and the speaker and session line up at the two events are on the Rittman Mead BI Forum 2015 homepage.
Each event has its own agenda, but both will focus on the technology and implementation aspects of Oracle BI, DW, Big Data and Analytics. Most of the sessions run for 45 minutes, but on the first day we’ll be holding a debate and on the second we’ll be running a data visualization “bake-off” – details on this, the masterclass and the keynotes and our special guest speakers will be revealed on this blog over the next few weeks – watch this space!
Oracle BI Cloud Service for SaaS Application Reporting Part 1: Integrating BICS to Salesforce.com using REST APIs
Last month Mark Rittman covered a series of posts detailing the Oracle BI Cloud Service (BICS), aimed at departmental users who want the power of OBIEE 11g without the need to stand-up their own infrastructure. If you’re coming in late, here’s the link to the series.
Before the GA Rittman Mead participated in the beta program for release one of Oracle’s Business Intelligence Cloud Service (BICS), the global aim of the beta was both to understand the capabilities and to identify potential use cases of the new BICS platform. As Mark wrote, an excellent use case for BICS is to report on top of any SaaS application that expose the data (stored in the cloud) using REST APIs by taking advantage of the ApEx capabilities hosted in the Database Schema Service that comes with BICS. SaaS applications like Oracle’s Fusion CRM, Taleo or Salesforce.com – that was used during the beta program – can be easily integrated and queried with BICS.
The technical goal of our beta program has been to check the features, options and limitations of Oracle BICS by connecting it to a Salesforce.com instance, accessing the data exposed through REST APIs by using the ApEx native functions, storing the data in the Database Schema Service, creating the Repository using the new Model Editor and showing the data in dashboard by keeping the same data security settings configured in the source platform.
This series of post is going to explain all the details about our successful PoC, over the next few days we’ll be covering the following topics, and we’ll update the list with hyperlinks once the articles are published:
- Oracle BI Cloud Service for SaaS Application Reporting Part 1: Integrating BICS to Salesforce.com using REST APIs
- Oracle BI Cloud Service for SaaS Application Reporting Part 2: BICS ApEx components
- Oracle BI Cloud Service for SaaS Application Reporting Part 3: BICS Repository and Front-end configurations
In the first post of the series we’re going to define the steps required to setup a Salesforce.com demo environment that can be accessed with REST APIs. Later in the post we intend to analyse the most interesting Salesforce.com REST APIs that are used to extract the data from the platform. Before starting digging into the BICS-Salesforce.com integration, some definitions may be needed:
- RESTful APIs: A Web service API can be defined as RESTful if it conforms to the REST architectural constraints. Some more details about RESTful APIs can be found here.
- Salesforce.com: Salesforce.com is company specialised in software as a service (SaaS) and one of the major providers of CRM in the cloud. In addition to CRM Salesforce.com offers Force.com a cloud platform as a service (PaaS) that developers can use in order to build multitenant applications hosted on Salesforce.com servers. All the Force.com applications can be accessed by using RESTful APIs.
Environments Setup
In a new BI project it is never safe to directly extract data from a live (production) environment. Thus the first step needed in order to test BICS-Salesforce.com connection is to obtain a Salesforce.com environment that we can use without the risk of slowing it down, or even worse crashing and impacting users. A free demo Salesforce.com platform can be obtained by subscribing for the developers program and can be customised as needed. After requesting the Salesforce.com demo environment the creation of a security token is needed, the token is associated with the user invoking the RESTful API. To generate the token: login to Salesforce.com with the specific user -> click on the username -> My Settings -> Reset My Security Token. The new token should then be sent to the user’s email.
The third step in order to access Salesforce.com data with REST APIs is to define a Connected App. The creation of a Connected App generates a Consumer Key and a Consumer Secret that is going to be used later during the REST login calls. Once defined a Connected App, the last bit missing is to populate the Salesforce.com instance since it’s empty by default, it can be populated manually or by writing population scripts based on the create REST API statements explained here. An Oracle BICS instance is needed in order to analyse the Salesforce.com data, all the info regarding Oracle BICS and how to activate one instance can be found at this link. Having covered all the basics, it’s time to start analysing how the Salesforce.com data can be extracted.
Salesforce.com REST APIs
In the following sections we are going to analyse some of the REST APIs used in our process. The Salesforce.com REST APIs will be called with cURL commands, cURL is a command line tool and library for transferring data with URL syntax.
Salesforce.com Authentication
The first step in order to download the data from any Salesforce.com instance is to execute the authentication and retrieve the access token, the access token will then be used in all the following REST calls. There are various authentication mechanisms that can be used against Salesforce.com, for the aim of the beta program we used the one called “Username-Password OAuth Authentication Flow” that is described in detail here and in the image below.
The REST API authentication command is
curl -v https://login.salesforce.com/services/oauth2/token -d "grant_type=password" \
-d "client_id=CLIENT_ID" -d "client_secret=CLIENT_SECRET" \
-d "username=USERNAME" -d "password=PASSWORD_TOKEN"
where the following parameters must be assigned:
- CLIENT_ID: The client ID generated during the creation of the Connected App (step described in Basic Setup)
- CLIENT_SECRET: The client Secret generated during the creation of the Connected App (step described in Basic Setup)
- USERNAME: The username you want to use in order to access Salesforce.com data
- PASSWORD_TOKEN: The concatenation of password and user security token (e.g. if the password is ABC and the security token is 123 then PASSWORD_TOKEN is ABC123)
The response (if all the parameters are correct) should be like the following
{
"id":"https://login.salesforce.com/id/00Dx0000000BV7z/005x00000012Q9P",
"issued_at":"1278448832702",
"instance_url":"https://na1.salesforce.com",
"signature":"0CmxinZir53Yex7nE0TD+zMpvIWYGb/bdJh6XfOH6EQ=",
"access_token":"00Dx0000000BV7z!AR8AQAxo9UfVkh8AlV0Gomt9Czx9LjHnSSpwBMmbRcgKFmxOtvxjTrKW19ye6PE3Ds1eQz3z8jr3W7_VbWmEu4Q8TVGSTHxs"
}
where the most interesting parameters are:
- instance_url: defines the salesforce instance to use.
- access_token: defines the time-limited security token to use for all the following calls.
Salesforce.com List of Objects and Retrieve Metadata for an Object
Two of the Salesforce.com REST APIs were very useful when trying to build a general ETL that could be applied to any customised Salesforce.com instance:
- Get list of Objects: retrieves the list of all objects available in Salesforce.com (custom or not) with some additional information for each object.
- Retrieve Metadata for an Object: retrieves the list of columns for the selected object with related data types and additional information.
The Get list of Objects call is the following
curl INSTANCE/services/data/VERSION/sobjects/ -H "Authorization: Bearer TOKEN"
where
- INSTANCE is the instance_url parameter retrieved from the authentication call.
- TOKEN is the access_token parameter retrieved from the authentication call.
- VERSION is the REST APIs version used, in our beta test we used v29.0.
The response should be similar to the following in which you could see for each object (identified by the “name” field) all the metadata available.
{
"encoding":"UTF-8",
"maxBatchSize":200,
"sobjects":[
{
"name":"Account",
"label":"Account",
"keyPrefix":"001",
"labelPlural":"Accounts",
"custom":false,
"layoutable":true,
"activateable":false,
"urls":{
"sobject":"/services/data/v26.0/sobjects/Account",
"describe":"/services/data/v26.0/sobjects/Account/describe",
"rowTemplate":"/services/data/v26.0/sobjects/Account/{ID}"
},
"searchable":true,
"updateable":true,
"createable":true,
"deprecatedAndHidden":false,
"customSetting":false,
"deletable":true,
"feedEnabled":true,
"mergeable":true,
"queryable":true,
"replicateable":true,
"retrieveable":true,
"undeletable":true,
"triggerable":true
},
...
]
}
The Retrieve Metadata of an Object can be called for each object listed in Get list of Object response, the code is the following:
curl INSTANCE/services/data/VERSION/sobjects/OBJECT/ \
-H "Authorization: Bearer TOKEN"
where:
- INSTANCE and TOKEN and VERSION are the same parameters defined for Get list of objects call
- OBJECT is the Salesforce.com object to retrieve
The response for the Account object is similar to the following
{
"objectDescribe":{
"name":"Account",
"updateable":true,
"label":"Account",
"keyPrefix":"001",
...
"replicateable":true,
"retrieveable":true,
"undeletable":true,
"triggerable":true
},
"recentItems":[
{
"attributes":{
"type":"Account",
"url":"/services/data/v20.0/sobjects/Account/001D000000INjVeIAL"
},
"Id":"001D000000INjVeIAL",
"Name":"asdasdasd"
},
...
]
}
Interesting parameters for each column are:
- name: is the columns name.
- retrievable: if true it means that the particular column can be part of a query
- soapType and length or byteLength: these fields provide the field type information and the length of the field.
With the three REST APIs analysed we could recreate all the objects (custom or not) in any Salesforce.com (or Force.com) instance as tables in the Oracle Database Schema Service. In the next section we will see how to extract the data from the list of objects by using the Salesforce.com query capabilities.
Salesforce.com query
Salesforce provides two methods of querying the objects: including or excluding deleted object. The difference in the code is minimal, the first uses the /queryAll suffix while the second uses the /query. A Salesforce.com REST API query call is:
curl INSTANCE/services/data/VERSION/query/?q=SELECT+LIST_OF_COLUMNS+from+OBJECT \
-H "Authorization: Bearer TOKEN"
where:
- INSTANCE and TOKEN parameters are the ones retrieved from the Authentication call
- VERSION is the Salesforce.com REST API version used
- OBJECT is the Salesforce.com object to query
- LIST_OF_COLUMNS is the comma delimited list of columns to retrieve
The response of a query “select+Name+from+Account” is:
{
"done":true,
"totalSize":14,
"records":[
{
"attributes":{
"type":"Account",
"url":"/services/data/v20.0/sobjects/Account/001D000000IRFmaIAH"
},
"Name":"Test 1"
},
{
"attributes":{
"type":"Account",
"url":"/services/data/v20.0/sobjects/Account/001D000000IomazIAB"
},
"Name":"Test 2"
},
...
]
}
In the JSON result you can find:
- totalsize: the number of records retrieved
- Name: for each record the name of the Account
If the resultSet is too big, Salesforce.com will start paging results. If the initial query returns only part of the results, the end of the response will contain a field called nextRecordsUrl. For example:
"nextRecordsUrl" : "/services/data/v20.0/query/01gD0000002HU6KIAW-2000"
In order to get the next page of data a call like the following is needed passing the access token as a parameter.
curl INSTANCE/services/data/v20.0/query/01gD0000002HU6KIAW-2000 \
-H "Authorization: Bearer TOKEN"
Salesforce.com limits
It’s important to be aware of the limit set by Salesforce.com on the number of REST API calls per day. In order to check at any time the remaining number of calls available execute the following code.
curl INSTANCE/services/data/VERSION/limits/ -H "Authorization: Bearer TOKEN "X-PrettyPrint:1"
The DailyApiRequest -> Remaining field contained in the response shows the amount of calls still available.
{
"DailyApiRequests":
{
"Remaining":"4980",
"Max":"5000"
},
"DailyAsyncApexExecutions":
{
"Remaining":"250000",
"Max":"250000"
},
...
}
Resultset output format
Salesforce.com default output format is JSON, which is a common format for most of the cloud application. However there is the opportunity to also retrieve the result in XML format by using the HTTP ACCEPT header set to “application/xml”. This first release of the tool is based on Oracle DB11g which supports native XML and not JSON. For this reason during our beta program with Oracle BI Cloud Service we decided to use the XML output. Once BICS will be bundled with the Oracle DB 12c that supports JSON and XML native, the resultset output format could be kept as default in JSON. In this post we defined the steps required in order to setup a Salesforce.com demo environment that can be accessed with REST APIs and which are the most interesting Salesforce.com REST APIs that we used to extract the data in order to analyse it with Oracle BI Cloud Service platform. In the next post we will look more in detail at the BICS ApEx part of it by analysing how the Salesforce.com REST APIs can be called from ApEx.
