ValueError: structures must have the same size when inserting a strucutred np.array into another np.array using advanced indexing

Question

Using numpy 1.22

I have three 2d np.arrays. All of them are structured. I then want to combine them into one larger also structured np array. The issue arises with one structured array decoded_times with the following dtype: ('t1', '<M8[ns]'), ('t2', '<M8[ns]'), ('timedelta', '<m8[ns]') The array contains np.NaTs. The structured array i want to move the value into has this dtype: ('f1', '<f8'), ('o1', 'O'), ('t1', '<M8[ns]'), ('o2', 'O'), ('t2', '<M8[ns]'), ('timedelta', '<m8[ns]') The shapes are (602,3) / (602,6). I want to combine them using advanced indexing:

X_decoded[:, [2,4,5]] = X_decoded_times

The error i get is: ValueError: structures must have the same size

This is the code surrounding the issue:

    dtypes = [('f1', '<f8'), ('o1', 'O'), ('t1', '<M8[ns]'), ('o2', 'O'), ('t2', '<M8[ns]'), ('timedelta', '<m8[ns]')]
    X_decoded = np.empty((np.shape(encoded_X)[0],len(feature_names)), dtype=dtypes)

    X_decoded[:, [0]] = X_decoded_cont # shape (602,1)
    X_decoded[:, [2,4,5]] = X_decoded_times
    X_decoded[:, [1,3]] = X_decoded_cat # shape (602,2)

X_decoded_cont works fine, its a (602,1) np.float64 array.

When not trying to insert X_decoded_times, X_decoded_cat throws this issue ValueError: could not convert string to float: 'abcd'

I suspected, that combining X_decoded and X_decoded_cont somehow reset the dtypes but the dtypes are still the same when printing X_decoded.dytpe between lines 2/3 .

I can't find anything in the documentation on what the issue might be for my specific case.