Is it possible to avoid warning when using LOAD DATA LOCAL INFILE to import duplicates

184 views Asked by At

I have a table errors with the following columns : id_error, product_id, error_code which is already filled up with some errors. I am using id_error as a primary key and I added a UNIQUE index composed of columns product_id and error_code in order to ensure that there can't be two errors with the same error_code for the same product_id. For example :

+----------+------------+------------+
| error_id | product_id | error_code |
+----------+------------+------------+
|        1 |          4 |       1118 |
|        2 |          4 |       1119 |
|        3 |          4 |       1120 |
|        4 |          5 |       1121 |
+----------+------------+------------+

I want to import from a .csv file a list of errors, some of them possibly already be in the errors table. For example :

product_id, error_code
4,1120
4,1121
5,1121
5,1122

To do so, I am using the LOAD DATA statement and this works properly. For example :

LOAD DATA LOCAL INFILE 'C:/Users/Public/Documents/updated_errors.csv'
IGNORE INTO TABLE errors
FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'
IGNORE 1 LINES
(@col1,@col2) set product_id=@col1,error_code=@col2;

As a result, the errors table now looks like this, and this is the expected result :

+----------+------------+------------+
| error_id | product_id | error_code |
+----------+------------+------------+
|        1 |          4 |       1118 |
|        2 |          4 |       1119 |
|        3 |          4 |       1120 |
|        4 |          5 |       1121 |
|        5 |          4 |       1121 |
|        6 |          5 |       1122 |
+----------+------------+------------+

However, by doing so, I get a warning for each line that is already in the errors table to notify me that the UNIQUE key plays its role :

2 row(s) affected, 2 warning(s):
1062 Duplicate entry '4-1120' for key 'errors.UNIQUE'
1062 Duplicate entry '5-1121' for key 'errors.UNIQUE'
Records: 4  Deleted: 0  Skipped: 2  Warnings: 2

I want to automatise this process using Labview because it fits my workflow for this particular task. However it seems that the library I am using in Labview to access my database does not support warnings : it triggers an error saying that my query is wrong. I have double checked the query by running it directly in Workbench and there is no error, just the aforementionned warnings.

I also double checked everything on the Labview side and everything seems to work fine with other request. It just seems that this library consider warnings as errors.

I have tried to change the level of error verbosity with the following request (intending to change it back after the query), unfortunately as I am using a cloud DB, I do not think I can have a SUPER privilege nor a SYSTEM_VARIABLES_ADMIN privilege.

SET GLOBAL log_error_verbosiy = 1
Error Code: 1227. Access denied; you need (at least one of) the SUPER or SYSTEM_VARIABLES_ADMIN privilege(s) for this operation

I tried different combinations of primary keys and unique keys to avoid triggering warnings while keeping the security that prevents me from adding an already existing error but I have not been successful.

I am looking for a way to do one of the following :

  • Avoid warnings in case of duplicates with the LOAD DATA statement
  • Import a .csv file into the table with another statement that would not trigger any warning. I am thinking maybe a line-per-line import with a check for each line if the error is already in the table ?
  • Any other solution to achieve what I want to do ?

edit : I am using the Database Connectivity Toolkit for Big Data by Ovak Technologies in Labview

0

There are 0 answers