How to avoid PL/pgSQL loops? What are performance alternatives?

161 views Asked by At

I have close to 200M records in my tables. My migration script runs very slowly (for hours).

---My OLD Tables---

orders_old table 30M row count

id createdDate status price
1000376453 2021-10-14 completed 286
1000368038 2021-11-24 cancelled 334

order_status_history_old table 150M row count

modified_date order_id old_status new_status
2022-06-22 1000376453 created accepted
2022-06-23 1000376453 accepted completed
2022-07-16 1000368038 created accepted
2022-07-16 1000368038 accepted completed
2022-07-20 1000368038 completed cancelled

---My NEW Tables---

history_info table

id date updated_by

order_history table

id rev rev_type status

---My PlSql Code---

DO $$
DECLARE
ORDER_TABLE_RECORD                      RECORD;
HISTORY_INFO_ID                         INTEGER;
ORDER_STATUS_HISTORY_TABLE_RECORD       RECORD;
BEGIN

FOR ORDER_TABLE_RECORD IN SELECT * FROM ORDERS_OLD LOOP
    RAISE NOTICE 'ORDER STARTED %', ORDER_TABLE_RECORD.ID;

FOR ORDER_STATUS_HISTORY_TABLE_RECORD IN SELECT * FROM ORDER_STATUS_HISTORY_OLD OH WHERE OH.ORDER_ID = ORDER_TABLE_RECORD.ID LOOP

HISTORY_INFO_ID := NEXTVAL('HISTORY_INFO_SEQ');
INSERT INTO HISTORY_INFO(ID, DATE, UPDATED_BY)
VALUES(HISTORY_INFO_ID, ORDER_STATUS_HISTORY_TABLE_RECORD.last_modified_date, ORDER_STATUS_HISTORY_TABLE_RECORD.status_changer);

INSERT INTO ORDER_HISTORY(ID, REV, REVTYPE, STATUS)
VALUES(ORDER_TABLE_RECORD.ID :: BIGINT, HISTORY_INFO_ID, 1, 'NEW');

END LOOP;


END LOOP;
END $$;

Also, I'm trying alternative scripts like below script but I am getting an error.

DO $$
DECLARE
BEGIN

WITH OSHO_DATA AS (
    select osho.order_id, osho.new_order_status, OSHO.LAST_MODIFIED_DATE, OSHO.STATUS_CHANGER
    from orders_old oo
    left join order_status_history_old osho on osho.order_id = oo.id
),
HI_DATA AS (
INSERT INTO HISTORY_INFO(ID, DATE, UPDATED_BY)
SELECT NEXTVAL('HISTORY_INFO_SEQ'), OSHO_DATA.LAST_MODIFIED_DATE, OSHO_DATA.STATUS_CHANGER
FROM OSHO_DATA LIMIT 1
RETURNING *
)
INSERT INTO ORDER_HISTORY(ID, REV, REVTYPE, STATUS)
SELECT OSHO_DATA.order_id :: BIGINT, HI_DATA.ID, 1, GET_STATUS_CODE(OSHO_DATA.new_order_status)
FROM OSHO_DATA, HI_DATA;

END $$;

How do I migrate my old data to my new tables quickly with performance?

NOTE: The reason why I use loop instead of bulk insert is because I use the id after the first insert in the next insert process.

0

There are 0 answers