Using Neo4j Kafka Connect Pattern Strategy To Stream Data Into AuraDB

• as a Neo4j plugin (deprecating): As AuraDB is a database service running on the public cloud, users CAN NOT install this plugin into the AuraDB instance so they CAN NOT use this mode.

• as a Kafka-Connect Plugin: a plugin for the Confluent Platform that allows ingesting data into Neo4j from Kafka topics.

The below documentation provides an example of how to use the Neo4j Kafka Connect Sink Pattern Strategy to stream data from Kafka topics to Neo4j AuraDB instances on Linux (CentOS Linux release 7.9.2009)

1. Install Confluent Kafka and configure all required shell parameters. (This example is using Confluent 7.0.0). 

mkdir ~/kafka && cd ~/kafka

curl -O http://packages.confluent.io/archive/7.0/confluent-7.0.0.zip && unzip confluent-7.0.0.zip

# edit your shell profile add below lines:
export CONFLUENT_HOME=~/kafka/confluent-7.0.0
export KAFKA_HEAP_OPTS="-Xms2g -Xmx2g"
export PATH="${CONFLUENT_HOME}/bin:$PATH"
export $PATH

2. Install the Java runtime version 1.8+ (For example: JRE Install Example)

3. install kafka connect "datagen" and "neo4j" plugins. (We will use datagen source plugin as demo data generating tool).  

confluent-hub install --no-prompt confluentinc/kafka-connect-datagen:latest

confluent-hub install --no-prompt neo4j/kafka-connect-neo4j:2.0.2

4. start confluent/kafka services

confluent local services start

5. Open the Kafka admin console Web UI

http://<hostname>:9021

6. Create a "pageviews" topic with the default configuration

7. Create dategen source connector for "pageviews" with configuration and make sure it is running:

{
"name": "DatagenPageviews",
"config": {
"key.converter.schemas.enable": "false",
"value.converter.schemas.enable": "false",
"name": "DatagenPageviews",
"connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"kafka.topic": "pageviews",
"max.interval": "5000",
"quickstart": "pageviews"
}
}

8. Create a Neo4j Sink Connect with Pattern Strategy and make sure it is running:

{
"name": "Neo4jSinkConnectorPageview",
"config": {
"connector.class": "streams.kafka.connect.sink.Neo4jSinkConnector",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"topics": "pageviews",
"neo4j.authentication.basic.username": "neo4j",
"neo4j.authentication.basic.password": "M2ka2QejVTDAunY8Y9t1_8K-B2Kq8s-dqHJW34oabRw",
"neo4j.server.uri": "neo4j+s://d67f007f.databases.neo4j.io:7687",
"neo4j.connection.max.pool.size": "5",
"neo4j.encryption.enabled": "true",
"neo4j.database": "neo4j",
"neo4j.topic.pattern.relationship.pageviews": "(:User{!userid})-[:VIEWED{viewtime}]->(:Page{!pageid})",
"key.converter.schemas.enable": "false",
"value.converter.schemas.enable": "false"
}
}

9. Now, open Neo4j AuraDB browser to verify data has successfully streamed from Kafka to AuraDB.