To have a successful migration from your self-managed Neo4j database to Aura it is important you consider the following points:
Leverage the power of the causal cluster: designate read-only (RO) and read-write (RW) operations explicitly
AuraDB is set up as a 3 nodes causal cluster providing great benefits in terms of reliability and stability as well the possibility to run operations without downtime.
On a causal cluster, all write operations must be handled by the "leader", while read operations are preferably served by any of the 2 followers (the leader may still be solicited).
Whenever you submit queries, by default these are assumed to be read/write by default and as such are sent to the leader. As you know your workload you should find out which queries are read-only and make sure to explicitly mark them as such in your code.
See this depending on your driver language sued for your application: How to explicitly mark queries as READ while using the official Neo4j drivers?
Control the number of connections
With AuraDB the maximum number of connection a cluster node can accept is limited by the maximum of available server threads (setting dbms.connector.bolt.thread_pool_max_size = 400). This number has been selected for optimum stability and performance.
A cluster has 3 nodes, providing in total a maximum of (3x400) 1200 threads and this relates to the maximum number of connections (see this KB for further details). It is essential to keep in mind that write transactions are only performed by the leader.
It is important you in turn architect your application with that number in mind to avoid exhausting the cluster's connections resources. This consideration is typically important in environment where the scaling is achieved by the framework (kubernetes, lambda, openshift etc...). See further details here: How to operate with many applications connecting to my AuraDB Instance using the driver (maximum connections) ?
What you must consider is to limit the number of driver instances created to a low number and also adjust the setting of connection pool in your driver configuration.
Remember that you can scale with creating sessions within a driver as they are fast to create and manage and will only consume connections from the pool.
Ensure you have the store format for Aura
The datastore format evolves with Neo4j's core database version to deliver improvements and optimizations. For Aura we selected the best option for a cloud service (that makes best use of the IOPS resources) and as this is not the default you should consider checking it. See Changing the data store format when migrating to an AuraDB Instance
Drop your index when you import a Dump file
When you import a database Dump file to Aura, as part of the process, a consistency check is run to ensure the integrity and validity of the datastore. As most datastore would contain index (for best performance) the consistency check will take longer to complete. We advise you instead drop these index beforehand and rebuild them after loading the dump file as this is far more efficient.
See Indexes - Capture / Drop / Recreate existing
Update you driver version
AuraDB always runs the latest version of Neo4j. To get the best out of the database with your application the recommendation is to use the latest drivers. During the lifetime of your application you should also consider upgrading your drivers to catch up regularly with the latest improvements and benefit from the latest bug fixes. See What are the recommended driver versions to use with Aura ?
Safely preparing your Dump file to load to Aura
1 - Stop your database and produce the latest backup of your current on premise version >=3.5.x
a. neo4j-admin backup --backup-dir=/backup --name=mydb --from=backuphost:<<port_number>>
2 - (On your local 4.4.x - use latest version) Perform migration using neo4j-admin copy.
a. A sample command would be :./neo4j-admin copy --from-path=/backup --to-database=myDB --to-format=aligned
*Please note this will drop your indexes and constraints which is what you want to happen.
The --to-format=aligned option will force the copied database to be created using the aligned store format.
4 - Once the migration is complete you should be able to see a migrated db in the 4.4.4 databases directory.
Make sure the neo4j.conf is updated with the default database to the migrated database name in the config.
5 - Start the neo4j 4.4.x (use latest available version) instance.
Once the instance comes up check if migrated DB is available.
6 - Perform basic validations (node counts and relationships counts etc...)
7 - If all validations are good,stop the database and take the database dump of migrated DB.
8 - Upload the dump to the Aura instance. using push-to-cloud
9 - Apply the indexes to the database. These index statements are available in the migration log.
10 - Once the above steps are complete validate the data.
Comments
0 comments
Article is closed for comments.