Is there size limit on appending ORC data files to Vora tables

94 views Asked by At

I created a Vora table in Vora 1.3 and tried to append data to that table from ORC files that I got from SAP BW archiving process (NLS on Hadoop). I had 20 files, in total containing approx 50 Mio records.

When I tried to use the "files" setting in the APPEND statement as "/path/*", after approx 1 hour Vora returned this error message:

com.sap.spark.vora.client.VoraClientException: Could not load table F002_5F: [Vora [eba156.extendtec.com.au:42681.1640438]] java.lang.RuntimeException: Wrong magic number in response, expected: 0x56320170, actual: 0x00000000. An unsuccessful attempt to load a table might lead to an inconsistent table state. Please drop the table and re-create it if necessary. with error code 0, status ERROR_STATUS

Next thing I tried was appending data from each file using separate APPEND statements. On the 15th append (of 20) I've got the same error message.

1

There are 1 answers

0
Frank Legler On

The error indicates that the Vora engine on node eba156.extendtec.com.au is not available. I suspect it either crashed or ran into an out-of-memory situtation.

You can check the log directory for a crash dump. If you find one, please open a customer message for further investigation.

If you do not find a crash dump, it is likely a out-of-memory situation. You should find confirmation in either the engine log file or in /var/log/messages (if the oom killer ended the process). In that case, the available memory is not sufficient to load the data.