"gRPC message exceeds" Error when inserting large data

473 views Asked by At

I have a data.tql with a size of 4.4MB and having problem when inserting it to my graphdb.

1st Error: data.tql size is 4.4MB - “[CLI04] Client Error: Unable to connect to TypeDB server.”

2nd Error (I’m just using this for test, I’m still going to use the 1st one (data.tql 4.4MB)) data.tql size is 2.3MB (I reduced the original file size in half) - “gRPC message exceeds maximum size 4194304: 4834138”

I would like to seek assistance on how to fix this issue.

=====================

My another option is to segment the data per category.

attack-pattern.tql

========

$attack-pattern-1 isa sdo_attack_pattern, has stix_type “attack-pattern”, has stix_id “attack-pattern–b18eae87-b469-4e14-b454-b171b416bc18”, has created_by_ref “identity–c78cb6e5-0c4b-4611-8297-d1b8b55e40b5”, has created “2020-03-14T18:18:32.443Z”, has modified “2020-03-26T22:02:25.221Z”, has name “Non-Standard Port”, has description “My description”, has kill_chain_phases “[{‘kill_chain_name’: ‘mitre-attack’, ‘phase_name’: ‘command-and-control’}]”, has external_references “External References”, has object_marking_refs “[‘marking-definition–fa42a846-8d90-4e51-bc29-71d5b4802168’]”, has x_mitre_data_sources “[‘Network Traffic: Network Traffic Flow’, ‘Network Traffic: Network Connection Creation’, ‘Network Traffic: Network Traffic Content’]”, has x_mitre_detection “Some Detection”, has x_mitre_is_subtechnique “False”, has x_mitre_platforms “[‘Linux’, ‘macOS’, ‘Windows’]”, has x_mitre_version “1.0”;

=========

course-of-action.tql

=========

$course-of-action-1 isa sdo_course_of_action, has stix_type “course-of-action”, has stix_id “course-of-action–6e7db820-9735-4545-bc64-039bc4ce354b”, has created_by_ref “identity–c78cb6e5-0c4b-4611-8297-d1b8b55e40b5”, has created “2018-10-17T00:14:20.652Z”, has modified “2019-07-24T19:46:16.474Z”, has name “LC_MAIN Hijacking Mitigation”, has description “Enforce valid digital signatures for signed code on all applications and only trust applications with signatures from trusted parties.”, has external_references “[{‘source_name’: ‘mitre-attack’, ‘url’: ‘https://attack.mitre.org/mitigations/T1149’, ‘external_id’: ‘T1149’}]”, has object_marking_refs “[‘marking-definition–fa42a846-8d90-4e51-bc29-71d5b4802168’]”, has x_mitre_deprecated “True”, has x_mitre_version “1.0”;

=========

relation.tql

=========

$rel-1 (course_of_action: $course-of-action-1, attack_pattern: $attack-pattern-1) isa uses;

=========

Result: This is not possible because the $course-of-action-1 and $attack-pattern-1 is looking for a value that is on another file.

I hope you can help me fix this issue.

Thank you!

1

There are 1 answers

0
James Fletcher On

I think grpc can throw this error because many too errors are being thrown by the server, they are sent back to the client as a single message and then said message is too long. I'd recommend inserting a subset of the data to try to catch the error(s) that are being thrown many times and fix accordingly.