I am importing some very large tables into PowerBI. Each time the data refreshes, the process gets slowed down since all of the data is getting requested/refreshed.
Does Connect Cloud support updating/refreshing the data incrementally?
I am importing some very large tables into PowerBI. Each time the data refreshes, the process gets slowed down since all of the data is getting requested/refreshed.
Does Connect Cloud support updating/refreshing the data incrementally?
Incremental Refreshes are definitely supported by Connect Cloud for any source tables with a "datemodified" type of column. Please note that Microsoft only supports incremental refresh in Power BI Premium, Premium per user, Power BI Pro, and Power BI Embedded.
Since Connect Cloud is facilitating the data connectivity, implementing any incremental logic in a data tool is up to the user. (If more straightforward data replication is desired to a destination like SQL Server, please consider CData Sync, our dedicated ETL/ELT tool.)
The following steps are an overview for setting up incremental refreshes in PowerBI:
When importing your Connect Cloud data into PowerBI with Cloud's PowerBI Connector, select Transform Data (instead of Load):
In the Power Query Editor, select Manage Parameters and then New Parameter.
In the Name field, enter RangeStart. In Type field choose Date/Time. Enter a start date/time in the Current Value field.
Then click New to create a parameter named RangeEnd and click OK when finished.
In the PowerQuery Editor, select the "DateLastModified" column and then choose the dropdown > Date/Time Filters > Custom Filter. (Change type to Date/Time if it is another date type)
Specify the first and second conditions using the parameters you created. (Make sure that "equal to" is on either RangeStart or RangeEnd, but not both). Then click OK.
On Power Query Editor's Home ribbon, select Close & Apply
After you have defined RangeStart & RangeEnd parameters and filtered the data, you can define an incremental refresh policy. This policy is applied only after the model is published to PowerBI.com and a refresh is performed (manual or scheduled). Detailed instructions can be found in Microsoft's PowerBI documentation here.
Once your report is published, you must authenticate PowerBI.com with Connect Cloud to schedule a data refresh. More details can be found in this CData Community article.
This is amazing! Thanks for sharing
UPDATE: Due to limitations with PowerQuery’s ability to push filters to the native connector, it is suggested to use incremental refresh with Connect Cloud’s Virtual SQL Server (TDS) endpoint instead. Although the process can be set up with native Power BI Connector for Connect Cloud, requests will be missing the filters and result in all data being returned.
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.