Preview the new community

We’re making it easier for you to find answers and network with others. Visit our preview site and let us know what you think.

Feedback Wanted - Cube Refresh Functionality

by Newcomer Wanting Newcomer on ‎03-03-2017 11:31 AM (2,217 Views)

We are working on the next generation of MicroStrategy tools that will provide users the ability to manage datasets including refreshing a dataset and schedule a dataset refresh.  To provide a better user experience, we would like to gather feedback from customers who have experience with these functionalities that we currently have on MicroStrategy Web.

 

Please share your use cases by answering the questions below in the comments:

 

Republish Cube

  • How often do you change the refresh policy settings when republishing a cube?
  • How often do you not select all data sources when republishing a cube?
  • Do you expect to refresh multiple cubes at the same time? (i.e. multi-select multiple cubes and refresh)

cuberepub1.png

 

Schedule Cube Refresh

  • How many schedules do you normally have for a cube?
  • If you create more than one schedule for a cube, how often do you do that?
  • What’s the use case of creating multiple schedules on a cube?

cuberepub2.png

 

  • If you have multiple schedules, do you expect to have an interface that allows you to manage all of the schedules, including view details, edit, and delete?
  • As an end user, you can create a schedule for a cube but cannot reuse the schedule for other cubes. To have a reusable schedule template, an Admin has to create one in the metadata. Do you think we should give end users the ability to create schedule templates so that they can be reused later for other cubes?

Comments
by Trusted Expert Trusted Expert
on ‎03-07-2017 10:10 AM

The case that I would see is that when you may want incremental refresh on a more frequent basis for performance, but there may be edge cases where the historical data changes, so that you would maybe want a full refresh of the data.

by Senior Guru Senior Guru
on ‎03-07-2017 11:20 AM

Presently what you present covert totaly are needs

by BrianFleenor Junior Seeker
on ‎03-07-2017 05:41 PM

Republish Cube

  • How often do you change the refresh policy settings when republishing a cube?

Refresh Policy is set once.  Shouldn't change (unless data availability changes and there is a benefit to do so).

  • How often do you not select all data sources when republishing a cube?

Very common.  I have a MTDI cube that I refresh the fact table daily and the dimensional table weekly.  The data's availability is at different intervals.  The fact table has new data available daily while a large dimensional table has new data available weekly.

  • Do you expect to refresh multiple cubes at the same time? (i.e. multi-select multiple cubes and refresh)

My cube refresh process is automated via System Manager using schedules.  I wouldn't expect to manually select multiple cubes and refresh.

 

Schedule Cube Refresh

  • How many schedules do you normally have for a cube?

Two.  With the MTDI cube, one for the daily fact table and a second for the weekly dimensional tables refresh.

  • If you create more than one schedule for a cube, how often do you do that?

Common for MTDI cube.

  • What’s the use case of creating multiple schedules on a cube?

MTDI cubes with varying data availability for each table.

  • If you have multiple schedules, do you expect to have an interface that allows you to manage all of the schedules, including view details, edit, and delete?

It would be helpful to be able to right-click and list all schedules used by the cube and then edit.

  • As an end user, you can create a schedule for a cube but cannot reuse the schedule for other cubes. To have a reusable schedule template, an Admin has to create one in the metadata. Do you think we should give end users the ability to create schedule templates so that they can be reused later for other cubes?

No.  Schedules can get out-of-hand rapidly and creation should remain with administrative oversight.

by Senior Guru Senior Guru
on ‎03-08-2017 06:36 AM

We are working for several customer who are replacing / removing cubes in their environments at the moment.

- Cubes are usually loaded when nightly database load has finished. Due to raising data volumes and complexity the ETL time is rising and time for a cube load is missing. End users complain, that their cubes are not ready in the morning. So reports are designed to use data base tables instead of cubes.

- Incremental refresh / update is most often not possible as small changes in facts or dimensions affect the majority rows of a cube. In complex scenarios that are quite common in finance, insurance, re-insurance, a cube usually need to be fully loaded.

- Data volume: A solution for analysis of debts and liabilities usually requires the analysis of an invoice at different time perspectives (key dates). This can hardly be designed as table, data aggregate or cube as the required size would be huge... instead smart views are used. So, several subjects / business units can not use cubes at all...

 

Republish Cube

  • How often do you change the refresh policy settings when republishing a cube?
    • Usually this is defined once by a developer. There is no need to change these settings.
  • How often do you not select all data sources when republishing a cube?
    • Never. Usually all required data source are required for a cube load / cube refresh.
  • Do you expect to refresh multiple cubes at the same time? (i.e. multi-select multiple cubes and refresh)
    • Yes! Almost always! Cubes are triggered when nightly database load for a certain subject has finished.

 

Schedule Cube Refresh

  • How many schedules do you normally have for a cube?
    • One
  • If you create more than one schedule for a cube, how often do you do that?
    • Never. We use triggered schedules
  • What’s the use case of creating multiple schedules on a cube?
    • Different business reasons / technical reasons that require different triggers? We could use one trigger, but maybe monitoring is easier with different trigger and differert schedules...

 

  • If you have multiple schedules, do you expect to have an interface that allows you to manage all of the schedules, including view details, edit, and delete?
    • Yes!
  • As an end user, you can create a schedule for a cube but cannot reuse the schedule for other cubes. To have a reusable schedule template, an Admin has to create one in the metadata. Do you think we should give end users the ability to create schedule templates so that they can be reused later for other cubes?

    • Tricky question! What about providing a privilege that can be assigned to different users, user groups like power users, developers and so on... This privilege should NOT be an administrative privilege. So it can be assigned on project level
by malikmud Helper
on ‎03-08-2017 09:33 AM

One of the more important features from an end user prespective is our ad-hoc users have the ability to create cubes against their Memory Allocation, We can individually alter a users designated memory limit to set exceptions.

 

We need the ability to unpublish a cube (remove from memory) but keep it available as something that can be republished at a later time if required from a web interface for our self-serve users.

 

Currently the only way to unpublish a cube is to delete the corrosponding files on the server or delete the cube itself.

These also needs to be a way to have cached cubes expire (Unpublish) based on a schedule.

 

by Nachow Pioneer
on ‎03-09-2017 03:48 PM

Hi, we have been struggling with the update functionality.

It would be great if you could specify the key by which you will update. Until this exists, we are forced to delete the part of the cube that we want to update and then run the update. This is because any attribute already existing in the cube that changes (for instance any status), actually that row doesn't exist anymore at the datawarehouse but MSTR will insert the new row, leaving the older one as well in the cube.

by Guru Guru
on ‎03-10-2017 09:09 AM

Am I the only one with a broken cube importation/refresh functionality in 10.4 ?

It's been broken for months, and support confirmed that currently, users can't import data (except if you give them write permissions on Shared folders Oo)

I was a bit annoyed to see a feedback request on a key feature that's been broken for a very long time, but I see a lot of people answering as if everything was working fine, so I'm wondering : am I the only one affected ?

by Enrique Lopez Junior Contributor
on ‎03-13-2017 10:14 AM

 Regarding cube refresh I would like to recommend to enhance MicroStrategy Analytics Enterprise in order to make the Migration more seamless. When i migrated from 10.3 to 10.4, i had to publish all the cubes again, and i find that fact inacceptable for large enterprises, the cube data should be migrated too. 

Also, when refreshing Facebook, twitter, and cloud sources in general after the migration, I found that most of them broke and couldn’t be refreshed anymore. Support kindly asked to re-do all the cubes and dashboards again. That’s inacceptable too, please fix that.

 

Thanks,

by AnthonyC Junior Contributor
on ‎03-15-2017 03:06 PM

Republish Cube

  • How often do you change the refresh policy settings when republishing a cube?

If I'm unlucky, I’ll have to do it once a year.

  • How often do you not select all data sources when republishing a cube?

Right now, all of our main cubes are from one data source so by default us admins select all. But there are numerous scenarios where users have created cubes from multiple data sources and they need to update only one of the data sources which is used as a lookup/reference type data source.

  • Do you expect to refresh multiple cubes at the same time? (i.e. multi-select multiple cubes and refresh)

Yes, this is an important in a particular scenario when a database fails. We have to republish a subset of cubes because of failure. We have a command script for all cubes but sometimes we need to update 5-8 at a time. For us we just copy out the command manager lines required, but in a user’s case they may want to refresh a bunch and clicking each one and sending to history is tedious.

 

Schedule Cube Refresh

  • How many schedules do you normally have for a cube?

Usually have one schedule

  • If you create more than one schedule for a cube, how often do you do that?

Once in a while, if database job runs late because of some sort of issue. We will change the schedule of a cube(s) to a later time until the database job performance is fixed.

  • What’s the use case of creating multiple schedules on a cube?

Cube updated frequently during peak work times, less during off peak times for up to date data.

  • If you have multiple schedules, do you expect to have an interface that allows you to manage all of the schedules, including view details, edit, and delete?

Yes, it would be nice if this can be managed in an easy to use interface.

  • As an end user, you can create a schedule for a cube but cannot reuse the schedule for other cubes. To have a reusable schedule template, an Admin has to create one in the metadata. Do you think we should give end users the ability to create schedule templates so that they can be reused later for other cubes?

Right now we don’t have a requirement to do this, but you never know when a request like this will come through from the users.

 

 

by AnthonyC Junior Contributor
on ‎03-15-2017 03:15 PM

Am I the only one with a broken cube importation/refresh functionality in 10.4 ?

I have users importing data to cubes in 10.4 HF1, no issues and they don't have write access to the shared folder. They actually do their cube work in their "My Reports" folder. I might be missing something or interpretting your issue incorrectly.  

 

 

 

by Guru Guru
on ‎03-15-2017 03:26 PM

@AnthonyC Well we were doing just like you : import in 'my reports' with no write access to shared folders. After migrating to 10.4 we started having an error saying that users don't have permission to create cubes. We contacted support, and they said it was a known bug, going to be fixed in 10.7. Foolishly, I believed them Oo

Can you PM me a screenshot of your users' privileges ? Maybe I missed a new one in 10.4 Oo

by Dirk1520666 Seeker
on ‎03-21-2017 09:42 AM

Looking for a faster way to republish cubes at destination after a project copy - something like 'Refresh all cubes' , queue them up and run one after another, or eliminate the need to republish by copying cubes from source (given a full meta-data copy was performed.

by Rogarcia Seeker
on ‎03-27-2017 09:39 AM

Republish Cube

  • How often do you change the refresh policy settings when republishing a cube?

Never, we use command manager to refresh after data load on the WH.

  • How often do you notselect all data sources when republishing a cube?

We do not need this feature now. All data is from the same data source.

  • Do you expect to refresh multiple cubes at the same time? (i.e. multi-select multiple cubes and refresh)

It would be a great feature. 

Schedule Cube Refresh

  • How many schedules do you normally have for a cube?

None, we refresh it on demand 3 times per day.

  • If you create more than one schedule for a cube, how often do you do that?

Not needed.

  • What’s the use case of creating multiple schedules on a cube?

Dont know

 

  • If you have multiple schedules, do you expect to have an interface that allows you to manage all of the schedules, including view details, edit, and delete?

No

  • As an end user, you can create a schedule for a cube but cannot reuse the schedule for other cubes. To have a reusable schedule template, an Admin has to create one in the metadata. Do you think we should give end users the ability to create schedule templates so that they can be reused later for other cubes?

No

 

What we think it will be a great feature is the possibility to delete and insert records with one refresh command. Now there is no option to delete and then insert, you need to do it in 2 steps
For example if some rows get deleted from the data source table there is no way to delete them from the cube and insert the new data in one step. 
The only way to achieve this is to run 2 incremental refresh reports: one to delete and one to insert the data. Using this method we will have the cube published with no data while the second refresh is made. Example: I have a cube with 3 regions. On the WH region 3 data has many changes, updates, inserts and deletes. So If I want the cube to have the same information as it is on the WH I need to run a refresh cube to delete Region = 3 and then run another refresh cube with insert Region = 3. Between these two refresh cubes there is no data for Region 3.

We did open an ER for this (104182) 

by U7lWRjFEDK Newcomer
on ‎03-30-2017 05:25 AM

Thanks for the good knowledge. All the experts give you information.

 

สโบเบ็ต

by Tim1220908 Helper
on ‎04-13-2017 11:59 AM

My comments:

 

I would like to queue refreshes to the same cube.  Curently, you can only run one cube incremental refresh at a time.  I would like the functionality to put the refresh job in queue, if there is another job currently executing against the cube.

Use Case:  We run multiple data loads nightly by state.  When a state completes its data load, then I want to kick off the refresh of the cube for that state.  My issue is that I often have multiple states completing at the same time and/or the cube might be in the middle of a refresh from another state and the jobs will error out.  

 

I ended up writing my own program to handle it, but it seems like it should be a function built in.

by Enrique Lopez Junior Contributor
on ‎04-13-2017 04:25 PM

Also when you refresh a cube in visual insight it hangs with a spinning wheel until it finish. That's kind of annoying. I think that it should be a button like "send to history", so that the dataset refresh in background and i can continue working with the dashboard using other datasets. 

by Pioneer cschneider Pioneer
4 weeks ago

I want to agree to the comment by malikmud.

A feature to delete cache for a user-published cube needs to be available from the Data Storage Page. If not the user has to delete everything that is based on the cube to free space or ask the administrator to delete cache. But that is too much overhead.

 

We need the ability to unpublish a cube (remove from memory) but keep it available as something that can be republished at a later time if required from a web interface for our self-serve users.

 

Currently the only way to unpublish a cube is to delete the corrosponding files on the server or delete the cube itself.

by Benn Active Contributor
4 weeks ago

Definately, Dynamic Sourcing for incremental refreshed cubes.

by Jason_Salemme Junior Contributor
Monday

I would like to add another use case here that would make the use of cube more viable at some of my clients who do not use cube today.  In the event there is a failure or a data issue with the last cube refresh (incremental or not), I need to quickly fall back to the previous version of the cube.  This allows for a level of fault tolerance needed for a highly available system where users cannot wait an additional N number of hours to have the cubes refresh. 

This use case is also important when a new project release is pushed to production, which requires the cubes to be rebuilt.  If there is a problem, I would need to run my Object Manager undo package to get back to the previous state and use the old cubes.  Again, in a highly available system I cannot way N number of hours for the entire set of cubes to be rebuilt.  I should be able to easily use the previous instance of the cubes.

by BrianFleenor Junior Seeker
Monday

Agree with Jason.  The observed behavior of an incremental failure is for the cube to republish itself.

This is totally not desirable for a large cube.