MySqlBulkLoader error importing pipe delimited file

118 views Asked by At

I am using MySqlBulkLoader to import a pipe delimited file, with over 250K rows, to a MySQL database table using a vb.net application.

This will be a daily task so I am trying to get this as automated and fast as possible.

I have no control over the creation of the files.

All seems to work fine with a couple of exceptions. The one I am currently dealing with is an anomaly in a single row of the data where one of the pipes is preceded by a backslash |

The backslash is a genuine part of the data that needs to be imported, however, combined with the pipe seems to act as a control. The result is the error:

Row X doesn't contain data for all columns

I am currently trying various workarounds but wonder if anyone has experience with this, or knows a solution to this issue.

Using MySQLConnection
            bulk.TableName = "MyTable"
            Application.DoEvents()
            bulk.FieldTerminator = "|"
            bulk.LineTerminator = "\r\n"  
            bulk.FileName = SourceFile        
            bulk.NumberOfLinesToSkip = 1    

            bulk.Columns.Clear()
            For Each s In cols
                bulk.Columns.Add(s)         
            Next

            rows = bulk.Load()
        End Using

EDIT

I added the following to prep the file prior to import

My.Computer.FileSystem.WriteAllText(SourceFile, My.Computer.FileSystem.ReadAllText(SourceFile).Replace("\|", "|"), False)

This works and allows to import to complete, however the search and replace adds approximately 8 minutes on to the time, which is much longer than the time it takes to actually import the 250K+ rows, so not ideal.

EDIT

This is a sample of the data in the format received

Standard Row - This works

345634534556|1|100|0|NA|2|55402|1.2|NA|0|0|0|0|NEW||1.2|1||||178-|||The Green||||Manchester||M1 XXX|M1|XXX|Customer 1|DEV||2019-05-19|NEW|0|2021-12-02|0||02903428||90X590X473|

Anomalous Row - Causing the error

767655444567|1|100|0|NA|2|52270|2.2|NA|0|0|0|0|NEW||2.3|1||||154\|||The Green||||Manchester||M1 XXX|M1|XXX|Customer 2|DEV||2020-07-19|NEW|0|2021-11-25|0||09304415||9X059X6087|
0

There are 0 answers