Tim Peterson's BI Performance Blog: Local Cube Creation Issues

Creating very large local cubes (1GB+)

Thu, 13 Mar 2008 20:23:13 GMT

We have recently seen more organizations using very large local cube files. I used to suggest that it was best to keep local cube file size under 50MB. Now, however, I have seen local cubes used successfully that are 1 GB and larger in size. I have been re-thinking my ideas about the limitations of local cubes.

Here are some details:

1. Larger local cube files will be slower than small local cube files - but they can still have excellent performance. We have seen people very pleasantly surprised at the browsing speed of their 1GB local cubes.

2. You can create much larger local cube files using ASSL with a relational data source than you can using the Create Global Cube command or ASSL with an Analysis Server data source.

For example, using the relational ASSL on a Virtual PC with 2.4GB of RAM, I was able to create a 2GB local cube from a fact table with 56 million rows. On the same machine, local cube creation with an Analysis Server cube source was successful with 3 1/2 million fact table rows (332MB local cube), but failed with a memory error when attempting to create a local cube from 7 million rows.

I know of organizations using larger servers that were able to create 2GB local cubes using an Analysis Server cube source. The size of the local cube you can create is dependent on the resources (memory, in particular) of the computer you are using. But whatever hardware you are using, it appears that you can create larger local cubes using the relational source ASSL.

3. We are making some modifications in CubeSlice which will assist our customers in creating large local cube files. Our current build sets a time out for creating temp tables which is much lower than it should be. If you are having trouble creating large local cube files, let us know. We think we can help you be successful in doing so.

4. I am working on a white paper regarding the creation and use of large local cube files. Please let me know what you have experienced with them. Thanks!

Why local cube creation with ASSL is superior to local cube creation with Create Global Cube

Tue, 01 Jan 2008 01:29:25 GMT

I have written a white paper discussing seventeen reasons why the Analysis Services Scripting Language (ASSL) is better than the Create Global Cube command in creating local cubes in Analysis Services for SQL Server 2005.

ASSL can create local cubes that are smaller, faster, can be encrypted, and can include many more features.

You can read or download the paper here:

http://www.cubeslice.com/asslandcreateglobalcube.htm

And please give me your feedback. Thanks!

Excluding Unused Members to Reduce the Size of the Local Cube

Thu, 18 Oct 2007 01:49:03 GMT

We have just released an updated copy of my paper describing the use of local cubes in Analysis Services 2005. This update incorporates improvements we have made in the local cube creation abilities of CubeSlice. You can get the updated copy of the paper here:

Using Local Cubes wtih Microsoft SQL Server 2005 Analysis Services

www.cubeslice.com/UsingLocalCubesinMicrosoftSQLServer2005AnalysisServices.doc

The most important change in the document is in the following section, which describes a dramatic improvement in CubeSlice.

Excluding Unused Members to Reduce the Size of the Local Cube

One of the biggest advantages of CubeSlice Relational ASSL is that it is the only local cube creation option where you can exclude unused members from the local cube.

Cubes often have dimension members that are not being used - members that are not linked to any records in the fact table. This happens to a greater extent when a local cube is created with slicing - limited to one Sales Rep, for example.

By removing unused dimension members, the size of a local cube can often be reduced by 50% to 90%. We have even seen one situation where the size of the local cube was reduced by 99.6%.

If you choose to exclude unused members, you should also consider choosing the CubeSlice option to create the local cube using temporary tables. The use of temporary tables often dramatically speeds up the process of creating local cubes - and that's especially true when excluding unused members.

The Benefits of Using a Star Schema when Creating Cubes

Mon, 15 Oct 2007 22:46:09 GMT

Douglas McDowell has an article in the October 2007 issue of SQL Server Magazine where he argues that it's usually best to move transactional data into a star schema before creating Analysis Server cubes. You can read his article (with subscription) at http://www.sqlmag.com/Authors/AuthorID/1537/1537.html

He lists several reasons for his argument:

1. OLTP-based cubes are usually poorly designed for BI.

2. OLTP-based cubes usually are full of dirty data.

3. OLTP-based cubes don't handle history.

4. OLTP-based cubes still have to be processed.

It's a great article. I agree with his reasons and I'd like to add another one:

5. You can often create a better local cube if your Analysis Server cubes are based on a star schema.

When creating an AS2005 local cube, you can use either an Analysis Services cube or the source relational data as the data source for the local cube. We support both types of local cube creation in CubeSlice, but we can often do a better job when using the source relational data - creating smaller, better-performing cubes more quickly.

If you're creating a local cube from an Analysis Server cube, it really doesn't matter what you're using as the source data for that AS2005 server cube. But if you want the advantages of using the source relational data option, it's much easier with a star schema data structure. Analysis Services 2005 is extremely flexible in designing cubes. You can include information from many relational tables in each dimension. But this flexibility makes it very difficult (or very time-consuming or impossible) for CubeSlice to create a local cube using the relational source option. Use a star schema for your server cubes and CubeSlice will have no problem creating local cubes from that relational data source. CubeSlice can handle snowflaking in the dimension tables of the star schema, but often cannot handle dimensions that are created from fields in multiple transactional database tables.

How important is this choice? When using the relational data source, the user has the option to remove empty members from all the dimensions. This option often reduces the size of the local cubes by 50% to 90%, or even more. Smaller local cubes can be created more quickly and have better browsing performance. If you want the best possible local cubes, consider using a star schema as the source of your Analysis Server 2005 cubes.

Creating Local Cubes in SQL Server 2008 (Katmai)

Tue, 26 Jun 2007 16:03:08 GMT

We have begun testing the creation of local cube files using CubeSlice 9 and SQL Server 2008 (previously known as Katmai). We are using the June 7 Community Technology Preview of SQL Server 2008. Initial results are excellent.

The current build of CubeSlice 9 needs a slight modification in order to create local cubes with SQL Server 2008. If you would like to test local cube creation with CubeSlice and SQL Server 2008, please let me know and I will get you the modified build of CubeSlice. You can contact me at tepATsdgcomputingDOTcom. For information about SQL Server 2008 and to sign up for the Community Technology Preiview, see Microsoft's web site:

http://www.microsoft.com/sql/prodinfo/futureversion/default.mspx

Member Sorting in Analysis Services 2005 Local Cubes

Sat, 23 Jun 2007 14:30:47 GMT

There can be problems with member sorting in Analysis Services 2005 local cubes. Here is a description of the problem and how you can fix it.

In Analysis Services you can sort by the member name, the member key, or by a related attribute. When I use a related attribute for sorting, I normally set the AttributeHierarchyEnabled property for that attribute to False. If I'm only using the attribute for sorting there's no need to waste the processing time of building a separate attribute hierarchy for it.

But this common practice causes a problem. Local cubes do not support attributes with the AttributeHierarchyEnabled property set to False. Here is what happens to sorting when using the different ways of creating local cubes and what you can do about it:

1. CubeSlice Relational ASSL. The problem is completely fixed in our next build. Our testing shows proper sorting of members in all situations. The sorting is fixed automatically without the user being informed of any problem. (We hope to release this build by July 1st - if you need it sooner, let us know.)

2. CubeSlice OLAP ASSL. The problem can only be fixed by changing the properties in the Analysis Server cube. Set the AttributeHierarchyEnabled property to True for all attributes used in sorting. If you don't want the attribute to be visible in the cube, you can set the AttributeHierarchyVisible property to False. In the next build of CubeSlice, when this problem occurs, the local cube will be created, but the incorrect sorting will be reported in the Cube Creation Log in the Info field, with instructions as to how it can be fixed.

3. CREATE GLOBAL CUBE. The problem can only be fixed by changing the properties in the Analysis Server cube, as with CubeSlice OLAP ASSL.

4. Create Cube ASSL with a relational data source If you are directly editing ASSL, you can change the AttributeHierarchyEnabled element to True and the AttributeHierarchyVisible to False, and your local cube will be created with proper sorting.

5. Create Cube ASSL with an OLAP data source. If you reference any attribute that has the AttributeHierarchyEnabled property set to True, the creation of the local cube will fail. In our work on this probem, we have come to the conclusion that, when using an OLAP data source, the AttributeHierarchyEnabled property must be set to True in the source cube. (If any one knows a different way to handle this, please let us know.)