How can I solve the atoti exception: "AtotiJavaException: Cannot make an array type non nullable"?

Question

I am trying to generate a table in atoti using the standard method:

table = session.read_pandas( DF, keys = ["col1",...."coln"], table_name = "My_Table")

However, it is throwing me this error related to reading one column which consists of numpy arrays where the values in the array are floats.

The full error reads as follows:

AtotiJavaException: Cannot make an array type non nullable. Field Type : FieldType(dataType=156032 -> Nullable double[], dataClass=class [D, defaultValueHolder=MultiTypeCell{codec=NULL, value=null})

Answer 1

Not 100% sure as your code is incomplete but you're probably getting this error because you're trying to add your array column to the keys of the table.

However:

• array columns are always nullable
• table keys cannot be nullable

Note: starting from atoti 0.7.0, a column is considered nullable if and only if column.default_value is None.

For example, this works:

import atoti as tt
import numpy as np
import pandas as pd

df = pd.DataFrame(
    columns=["id", "scalar", "array"],
    data=[
        ("abc", 1.0, np.array([1.0, 2.0])),
        ("def", 2.0, np.array([2.0, 3.0])),
    ],
)
session = tt.Session()
table = session.read_pandas(
    df,
    table_name="example",
    keys=["id"],
)

data_types = {name: table[name].data_type for name in table.columns}
assert data_types == {
    "id": "String",
    "scalar": "double",
    "array": "double[]",
}

rows = table.head()
pd.testing.assert_frame_equal(
    rows,
    pd.DataFrame(
        columns=["scalar", "array"],
        data=[
            (1.0, [1.0, 2.0]),
            (2.0, [2.0, 3.0]),
        ],
        index=pd.Index(["abc", "def"], name="id"),
    ),
)

but replacing keys=["id"] with keys=["id", "array"] will lead to the Cannot make an array type non nullable error.