How to wait for a Power Query refresh to finish?

Question

Setup:

• Windows 7 (at work)
• Windows 10 (at home)
• Excel 2016 (build 4627 at work)
• Excel 2016 (build 8730 at home)
• Power Query is setup to import, append, and transform a folder of Excel Files. This step works.

After using any technique to wait for a Power Query to finish refreshing as described in the "Things I have tried:" section shown below, a message box can be displayed and any other code can be executed before the Power Query tables have finished updating according to the refreshing indicator (spinner?) icon shown in the Queries and Connections pane.

The exception to the above statement is the OnTime method of the Application class, shown in the "Code" section below, which doesn't appear to interrupt the polling for a power query refresh. The problem is that it uses a hard coded amount of time to pause VBA code, and this will not always work since the size, amount, and duration of data being queried will change over time.

I tried:

• I have read all the StackOverflow (and other website resources) that state the use of the DoEvents, BackgrgoundQuery = False, and CalculateUntilAsyncQueriesDone methods and properties.
• I tried creating a class to Create Before/After Query Update Events as suggested at this link (not shown in the code example below).
• I tried using Do Until/While Loops with the .Refreshing = True/False property of the QueryTable method to wait for a refresh to finish.
• I tried setting the BackgroundQuery property in the Excel menu (menubar --> Data --> Connections --> Properties) to False as suggested by "subro" here: Wait until ActiveWorkbook.RefreshAll finishes - VBA, with an image of the Menu here:

.

Code:

Private Sub sht_sub_Refresh_AllConnections_dev()
    'Name: sht_sub_Refresh_AllConnections_dev
    'Purpose: An attempt at using VBA to wait for Queries to finish updating before displaying a message.
    'Description: Waits for a hard coded period of time before dislpaying the message box.
    'State: WIP.
    'Dev: Needs a way to look at the connection stream to somehow detect when its finished.
    
    'DECLARATIONS:
    '------------'
    Dim procName As String              'Stores this procedure's name.
    Dim qTblLst As QueryTables          'A query table collection object.
    Dim qTblObj As QueryTable           'A query table object.
    Dim conLst As Connections           'A connection collection object.
    Dim conObj As WorkbookConnection    'A connection object.
    Dim idx As Long                     'A loop counter.

    'INITIALIZATIONS:
    '---------------'
    procName = "sht_sub_Refresh_AllConnections_dev"    'Store this procedure's name.
    Linit.ini_Setup_Project                            'Setup the project if needed.
    Set conLst = ThisWorkbook.Connections              'Set the connections list object.
    Set conObj = conLst.Item(conLst.Count)             'Set an initial connection object.
    idx = 0                                            'As an exit if the do loop continues without end.
    
    'MAIN CODE BODY:
    '--------------'
    'Turn off backgroundquery for each connection type.
    For Each conObj In conLst                           'For each connection object,
        With conObj
            Select Case .Type                               'Check the connection type,
            Case 1                                        'If its an OLEDB connection then,
                .OLEDBConnection.BackgroundQuery = False    'Set it's backgroundquery property to false.
            Case 2                                        'If its an ODBC connection the,
                .ODBCConnection.BackgroundQuery = False     'Set it's backgroundquery property to false.
            End Select
        End With
    Next conObj
    
    ThisWorkbook.RefreshAll                             'Refresh all connections.
    
    'DEV: Using loops, DoEvents and a query name starting with the letters "zzzz" as suggsted here:
    'https://social.technet.microsoft.com/Forums/en-US/bc3f7748-8a52-498d-951c-4566b8adf45a/in-excel-2016-power-queries-dont-refresh-in-the-background-anymore?forum=powerquery
    'and here:
    'https://www.myonlinetraininghub.com/excel-forum/vba-macros/pause-macro-until-power-queries-finished-refreshing
    'Attempt to wait until the last connection has finished refreshing.
    Do Until Linit.gvTbl_ZZZZZ.QueryTable.Refreshing = True   'Wait until the last table starts refreshing,
        idx = idx + 1                                           'Icrement a loop count,
        If idx > 3000 Then Exit Do                              'If the loop goes longer then 3000 iterations exit,
    Loop                                                      'otherwise continue waiting.
    VBA.DoEvents                                              'Do events before continueing (doens't work).
    Do Until Linit.gvTbl_ZZZZZ.QueryTable.Refreshing = False  'Wait until the last table finishes refreshing,
        idx = idx + 1                                           'Icrement a loop count,
        If idx > 3000 Then Exit Do                              'If the loop goes longer then 3000 iterations exit,
    Loop                                                      'otherwise continue waiting.
    VBA.DoEvents                                              'Do events before continueing (doens't work).
    'DEV: The following is an attempt to get connections to
    '     finish refreshing before code continues as suggested here:
    'https://stackguides.com/questions/22083668/wait-until-activeworkbook-refreshall-finishes-vba
    Application.CalculateUntilAsyncQueriesDone         'This is placed here as well as after the refresh.
    VBA.DoEvents                                              'Do events before continueing (doens't work).
    Application.EnableEvents = False                          'Maybe turning off events helps? (nope...),
    Application.ScreenUpdating = False 'This is reset in the procedure called as an argument to the next line:
    Application.OnTime DateAdd("s", 3, Now), _
                       "Lwksh.sht_sub_Msg_RefreshDone"        'The called procedure just displays a message box.
    Application.EnableEvents = True                           'Restore events,
    Application.ScreenUpdating = True                         'Restore screen updating.
    
    'MEMORY CLEANUP:
    '--------------'
EXIT_CLEAN:
    procName = Empty                                     
    Set qTblLst = Nothing
    Set qTblObj = Nothing
    Set conLst = Nothing
    Set conObj = Nothing
    idx = 0
End Sub

Code notes:

• Anything in the code preceded by "Linit." is an object or variable that is set globally outside the procedure by the "Linit.ini_Setup_Project" procedure call in the "INITIALIZATIONS:" section of the code.
• For example "Linit.gvTbl_ZZZZZ" is an object variable that points to an empty one row Table which has a name that stars with the characters "zzzz" and is generated by Power Query and loaded to an Excel sheet. The code shows the link to the website where the suggestion for using an empty table like this was made.

Questions:

1. Is this a lost cause due to Power Query not having a built in callback to let Excel know that it has finished updating any refresh processes?
2. If this is not a lost cause, is there any other way, not described here, that could be used to trigger an error to occur in some way if a connection has not yet finished refreshing, or to trigger the error when a connection is finished? (The thinking here being that this error could be trapped as a possible way of detecting weather or not the refresh has completed provided the error doesn't stop the query from finishing).
3. Is there any way to probe the connection stream directly using VBA to look for the connection closed or finished state?
4. Is there any way to directly access the refresh process via some call to a program outside of Excel written in some other language such as C# or Python?
5. Can you think of anything else that might be tried or tested to make this work? I'll keep searching for an answer on my own, but after a full year of searching I'm feeling a bit out of luck.

Answer 1

I understand your pain @neurojelly. I have been there. But as it turns out the solution is quite simple and is not using VBA. In the Query properties window, you need to uncheck the "Enable background refresh" and then use DoEvents. I know for sure this works as I have been using this method for over a year now.

Please find the link to a sample file that has the code in it.
https://drive.google.com/uc?export=download&id=1ZLxSMEXPLda3QhaQoTyGGv3_sC-tpN-X

As for your second question, it is possible to use Iferror/OnEror method's to detect if a query returns an error, but it doesn't necessarily check for errors in the query. It identifies if the query itself is returning an error pop-up which is skipped by default while running the VBA code. This method works most of the time, but not always.

Answer 2

Here is a workaround

Sub MyProcedure()

    '
    ' Some procedures
    '
    Call ActiveWorkbook.RefreshAll
    Call NotifyWhenRefreshComplete
End Sub



Private Sub NotifyWhenRefreshComplete()
    Const PulseTimer As Currency = TimeValue("00:00:01")
    Dim b1 As Boolean, b2 As Boolean

    b1 = Sheet1.Range("ListObject1").ListObject.QueryTable.Refreshing
    b2 = Sheet1.Range("ListObject2").ListObject.QueryTable.Refreshing

    If b1 Or b2 Then
        Call Application.OnTime(Now + PulseTimer, "NotifyWhenRefreshComplete")
    Else
        Call MsgBox("Refresh Complete.", vbOKOnly)
    End If
End Sub

ListObject1 and ListObject2 are published tables. Only published tables are necessary to check if refresh is complete. You don't have to check unpublished tables.

However If you have a lot of published tables, then there is no wrong to iterate through all ActiveWorkbook.connections and check if each and every wbConn.OLEDBConnection.Refreshing status has return to false state, replacing the b1 and b2 boolean.

Note: For some reasons, I refuse to use the DoEvents, I want my users to be able to continue working with Excel while the connection is still running, and prompt them a message when the refresh is complete. But go ahead if you want to implement it with the Do... Loop iteration instead of the OnTime caller as shown above.

Lastly, there is a callback under Public WithEvents qt As QueryTable, look for qt.refreshing. You can use this method as well.

Hope this helps.

Answer 3

I'm calling the .Refresh method on ThisWorkbook.Connections.Ranges(1).ListObject.QueryTable, with BackgroundQuery:=False. It seems quite reliable - by now my Macro has probably called that method over 10,000 times with a very low error/freeze rate.

As I understand it, all those DoEvents workarounds etc became obsolete when PQ was properly exposed in the object model for Excel 2016.

Here's a rough code sample:

Dim cn As WorkbookConnection
For Each cn In ThisWorkbook.Connections
    cn.Ranges(1).ListObject.QueryTable.Refresh BackgroundQuery:=False
Next cn

Inside the For loop, you can examine cn.Name to control the execution of individual Queries. The Name property follows the Workbook Connections names, e.g. "Query - " & PQ Query Name.