I'm currently syncing data from MongoDB to Elasticsearch via Kafka. When retrieving the data, I use the MongoDB Kafka Connector, and when sending it to Elasticsearch, I use kafka-connect-elasticsearch.
The data in Kafka has keys in the format
{"_id": {"$oid": "64e5cf730ed5822aa85b91ad"}}
When syncing from Kafka to Elasticsearch, I'm unable to retrieve the _id.$oid field to use it as the document_id key in Elasticsearch.
In the sink configuration, I currently have:
transforms=RenameField,ReplaceFieldKey,ExtractField
transforms.RenameField.type=org.apache.kafka.connect.transforms.ReplaceField$Value
transforms.RenameField.renames=_id:id
transforms.ReplaceFieldKey.type=org.apache.kafka.connect.transforms.ReplaceField$Key
transforms.ReplaceFieldKey.renames=_id:id
transforms.ExtractField.type=org.apache.kafka.connect.transforms.ExtractField$Key
transforms.ExtractField.field=id
When I set transforms.ExtractField.field=id, it becomes a map and doesn't work.
org.apache.kafka.connect.errors.DataException: MAP is not supported as the document id.
When I set it to id.$oid, it doesn't get recognized.
Key is used as document id and can not be null.
So is there any way to retrieve nested fields and use as key ? Thank you.