Attempting to optimize a excel query by converting it to an ADODB connection

42 views Asked by At

I am trying to consolidate multiple hourly files all of which have a key column with a unique Identifier that will line them all up together. Each file is uniquely named in a single folder and has the common theme in the naming convention and also that inside the file the tab name starts with the number 1.

I am trying to extract the data in each file. Currently it is a string of merged queries which is extremely slow and is no longer useful with the progress of the files and larger data sets.

I combined the date and hour into decimal form as a unique identifier and was attempting to use that to line up the data sets as a type of index, but am struggling figuring out which Join function to use that will line the timelines up with the hourly data. Any help would be greatly appreciated and appreciate any insights or tips to encourage my development.

So far I have set up the loop to run through my select folder and set up the file name as string. I do want to add a selection portion as when I was going through the immediate window I noticed that I was gathering more files than what were necessary.

Sub QetHourlyGen(SQLQuery As String)

    Dim HrlyFileName As String
    Dim MyFilePath As String
    Dim cn As ADODB.Connection
    Dim rs As ADODB.Recordset
    Dim ws As Worksheet
    Dim i As Integer
    Dim RowCount As Long, ColCount As Long

    If SQLQuery = "" Then
        MsgBox _
            Prompt:="You missed the mark", _
            Buttons:=vbCritical, _
            Title:="Try again"
        Exit Sub
    End If
    Sheet1.Range("A1").CurrentArrayRegion.Offset(1, 0).Clear "I want to absorb the column names into the data set Not sure if this will accomplish that"
    MyFilePath = "File path that contains all of the required files"
    HrlyFileName = Dir(MyFilePath & "*Unique File name")
'Want to add a contingency to not select another uniquely named file that is similar(one is GN the one I do not want has GNw in it)
    Set cn = New ADODB.Connection
    Set rs = New ADODB.Recordset
    Do Until HrlyFileName = ""
         cn.ConnectionString = _
             "Provider=Microsoft.Ace.OLEDB.12.0;" & _
             "Data Source=" & MyFilePath & HrlyFileName & ";" & _
             "Extended Properties='Excel 12.0 Xml;HDR=Yes';"
         cn.Open
         rs.ActiveCommandConnection = cn
         rs.Source = "SELECT * FROM ['1 "Random Tab name but the 1 always starts the necessary tab"]"
         rs.Open
         '1 "Random Tab name but the 1 always starts the necessary tab".Range("").CopyFromRecordset rs ' this is where I would like to merge or join the file based off of the unique date format.
         rs.Close
         cn.Close
         HrlyFileName = Dir
    Loop
End Sub

In the end the hope is to have a "backbone" as the first 3 columns with the date hour and then the unique date format then to the right will be the rest of the hourly columns of each recordset found in the loop.

Date Hour Data (Blank Column by design) Primary Key

The 4 columns in the "Backbone" the Primary key is just date plus Time (hour),0,0) in decimal form this is the unique Identifier in all of the files. The backbone is just a timeline to tie in all of the files.

Second Image is of the "Data to add to line up"

The goal is to cycle through the files in the folder and extract and merge the contents with the "Backbone" keeping all of columns right of the primary key of the matched file.

0

There are 0 answers