OBIEE.nl

Posts Tagged ‘Methodology’

After my last blog “Introducing the RittmanMead Delphi Methodology”, one of you posted the comment below. My answer ended up being so long that I decided to turn my response into my next blog. Here is the comment:

“What about iterations within the Delphi 18 stages? Do you have a structure for incorporating feedback from later stages into earlier stages i.e. learnings from the build phase being incorporated into and refining the design?”

What Damien was describing is really what governance is all about. Governance is how you go about improving your methodology.

Before you can get to that stage, you have to have a documented methodology and you have to use it. Sadly, not every project and/or IT shop can attest to talking the talk AND walking the walk.

Generally speaking, when an IT shop is operating without a well defined methodology, and it completes a project on time, it is due to heroic efforts and long hours on the part of the development team.

In this kind of environment, some of your best developers will get fed up leave you to work in better organized IT shops. Over time your development pool will degrade in quality and you will be less and less likely to bring projects in on time.

Sometimes, the business gets so fed up they outsource development and that’s not going to fix anything, if the team they outsource to doesn’t use a methodology either.

But let’s assume that you have a methodology and it is standard business process to use it on all your data warehouse projects.

The next step is for you to take measurements. This gets a bit tricky for us IT folks. It’s pretty easy to figure out that a widget factory should count how many widgets are produced/hour/day, but what should a data warehouse team be measured on?

Lines of code written/hour? Mappings deployed/week?

For data quality measurements – perhaps the number of reference data rows rejected/batch?

I was on a project once, where under pressure from the business a decision was made to skip the data profiling phase to “save time”. They ended up spending 18 months in UAT. There’s a new phrase for what happens to your project plan when you build a data warehouse without performing data profiling – they call it “Extract, Load and Explode”.

If the source data is really bad, IMHO it’s better for the company to spend this year’s budget fixing the data at source in the OLTP, and re-call the data warehouse team next year when the data is ready for prime time, the data stewards are empowered and a data governance policy is in place – but I digress.

Here are some industry standard warehouse measurements, but they are at a high level outside of the actual development processes:

• Do your data warehousing initiatives succeed more often than they fail?
• Is there a successful data warehouse implementation within your company?
• Is the successful data warehouse sustainable?
• Does your company know how much it is spending on data warehousing?
• Does your company have centralized IT groups?
• Does your company have an enterprise-wide meta data repository that supports the data warehouse and the operational systems?
• Has the quality of the data within your data warehouse improved over time?
• Do your programmers understand their role in helping the company reach its goals?

I think that a good DW development measurement might be hours in UAT per deliverable. If the design was really good, then theoretically the developers would build exactly the right thing, error free, the first time and the code and data should pass UAT and move into deployment quickly.

If the code is kicked back (fails UAT) that’s not a good thing. So the number of mappings that failed UAT or deployment would be good measurements as well.

Another measurement might be project plan accuracy. We all know how incredibly difficult it is to estimate how long it’s going to take to build these data warehouses and data marts. Good methodologies include processes to aid the project leader during the planning phase, but I have to say, having been in this position myself, if I have worked with the team previously, my plan is going to be much more accurate than if I am new to the team.

So now let’s assume you have finished your project, the code is deployed, the batch runs so flawlessly at night that the support & maintenance team is bored, your well trained business users are happily referring to their data mart documentation and you have gathered some meaningful measurements.

Additionally, you find yourself with time and resources available to support the governance phase.

The first step is to gather the lessons learned from the project. You need to document what went well and what didn’t and get everyone to agree. Next the team should brain storm what changes should be made to the methodology to improve things for next time.

These changes to the methodology have to be well documented and introduced carefully so that everyone understands the differences between the old way and the new way and why it changed. Sometimes the new way involves new tasks that aren’t much fun like going to meetings to gain consensus, documentation, change control procedures, coding standards, loosing access to production, business prioritization of tasks, issue log maintenance and time tracking. So your team has to buy into them.

Finally, you have to wait for the next project to complete with your changes in place and compare that project’s measurements to the first project’s to see if your methodology changes were effective.

Whew! Governance is hard. . .

That’s why so few IT shops are at that level of maturity. It is estimated that less than 2% of IT projects include a governance or methodology improvement phase.

To improve a business process such as widget production you can change the process and see results in hours or days. But changing IT methodology is harder. Each project takes longer, the team members are always changing and we never build the same thing twice.

Ideally, to be sure our methodology improvements worked, we should put a data warehouse team on a deserted island (well, somewhere isolated with really good connectivity) and make the team build a data mart. Then you’d have to improve the methodology, and have a similar team come along and build the exact same data mart. When they were finished you’d have to compare the measurement results.

That would be the only way to ensure that the process had indeed been improved, but even then I’m sure you’re thinking of how many parameters wouldn’t have been properly controlled the second time around.

Difficulties in measuring IT development efforts notwithstanding, for about 40 years now, businesses have wanted a way to determine if we IT folks are any good and figure out how to make us faster and cheaper. (I understand that this desire developed a few weeks after the first computer was installed.)

Back in 1989 Watts Humphrey first wrote about measuring how well IT projects were run in order to be able to assess the ability of IT government contractors. He called it the Process Maturity Framework. Then in 1995 Mark Paulk, Charles Weber, Bill Curtis, and Mary Beth Chrissis published their book on the Capability Maturity Model (CMM) which is based upon Humphrey’s work.

The book outlines exactly how to determine what level of maturity your IT organization is at, and how to improve things and move IT up from one level to the next.

There are five levels of maturity defined:

1. Initial (chaotic, ad hoc, heroic) the starting point for use of a new process.
2. Repeatable (project management, process discipline) the process is used repeatedly.
3. Defined (institutionalized) the process is defined/confirmed as a standard business process.
4. Managed (quantified) process management and measurement takes place.
5. Optimizing (process improvement) process management includes deliberate process optimization/improvement.

Jennifer’s easier to remember CMM definitions:

1. Dilbert cartoons are posted everywhere – (keep your CV/resume dusted off)
2. Talking the talk – (the methodology is written down)
3. Everyone is walking the walk – (using it)
4. Managers are measuring how far you can walk
5. Everyone’s getting predictably better at walking

The good news is that if you are at level 3, (everybody is walking the walk) you are working for a world class IT shop!

Figure 1 - Applying CMM Levels to Data Warehousing, David Marco

So to answer Damien’s comment, yes we have a structure for incorporating feedback from later stages into earlier stages. The last tasks on our project plan template are all about gathering the lessons learned and modifying the methodology for next time.

Sadly, so far I have always been asked to remove these tasks from my final signed off versions of the project plan in order to save money. – irony