Skip to content

Commit

Permalink
Release 1.1.0
Browse files Browse the repository at this point in the history
  • Loading branch information
jexp committed Aug 30, 2017
1 parent 060d30e commit aac22e2
Show file tree
Hide file tree
Showing 8 changed files with 1,825 additions and 126 deletions.
12 changes: 7 additions & 5 deletions README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -4,23 +4,25 @@ Data import from relational databases to Neo4j.

== Download & Install

Download & unzip the latest `neo4j-etl-components.zip` from https://github.com/neo4j-contrib/neo4j-etl-components/releases
Download & unzip the latest https://github.com/neo4j-contrib/neo4j-etl-components/releases/latest[neo4j-etl-components.zip].
Once unzipped, download the proper JDBC Driver xref:jdbc-drivers[here] and add it to the `lib` folder.

== Examples of command usage:

.Minimal command line
----
./bin/neo4j-etl mysql export --user <user> --password <pwd> --database northwind \
./bin/neo4j-etl { mysql | postgresql | oracle } export --url <url> --user <user> --password <pwd> \
--destination $NEO4J_HOME/data/databases/graph.db/ --import-tool $NEO4J_HOME/bin \
--csv-directory /tmp/northwind
----

.Full set of command line options
----
./bin/neo4j-etl mysql export --host 127.0.0.1 --port 3306 --user <user> --password <pwd> \
./bin/neo4j-etl { mysql | postgresql | oracle } export --url <url> --user <user> --password <pwd> \
--database <database> --destination $NEO4J_HOME/data/databases/graph.db/ \
--import-tool $NEO4J_HOME/bin --csv-directory /tmp \
--options-file ./import-tool-options.json --force --debug
--import-tool $NEO4J_HOME/bin --csv-directory /tmp \
--options-file ./import-tool-options.json
--force --debug
----

For detailed usage see also the: http://neo4j-contrib.github.io/neo4j-etl-components#neo4j-etl-cli[tool documentation].
Expand Down
7 changes: 2 additions & 5 deletions docs/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ include::../README.adoc[lines=2..-1,leveloffset=+1]

== Introduction


The Neo4j ETL components, especially the `neo4j-etl` tool, can be used to import well modeled (i.e. normalized) relational data into Neo4j.
It applies some simple rules for transforming the relational model.

Expand All @@ -18,8 +17,6 @@ The process as outlined below:
4. Generate Mapping Headers
5. Import into Neo4j using the bulk `neo4j-import` tool



== Architecture Diagram

image::neo4j-etl-architecture.png[]
Expand All @@ -37,7 +34,7 @@ image::neo4j-etl-architecture.png[]
* Build indexes
* Non-trivial datatypes (dates, binary)
* Support on Linux
* Support MySQL
* Support MySQL, PostgreSQL, Oracle
* Support user specified JDBC drivers

=== Plans for the Future
Expand Down Expand Up @@ -65,7 +62,7 @@ include::neo4j-etl.adoc[]

=== Inferring Schema with Mapping Rules (generate-metadata-mapping)

* Generic MySql database mapping based on the following rules
* Generic { MySql | PostgreSQL | Oracle } database mapping based on the following rules
* A _table_ with a foreign key is treated as a _Join_ and imported as a _node_ with a _relationship_
* Ex: `*Person* -> Address` is imported as `*(Person)-[:ADDRESS_ID]->*(Address)`
* A _table_ that has two foreign keys is imported as a _JoinTable_ and imported as a _relationship_
Expand Down
388 changes: 296 additions & 92 deletions docs/index.html

Large diffs are not rendered by default.

92 changes: 92 additions & 0 deletions docs/neo4j-etl-rdbms-export.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
NAME
neo4j-etl { mysql | postgresql | oracle } export - Export from MySQL, PostgreSQL, Oracle RDBMS.

SYNOPSIS
neo4j-etl { mysql | postgresql | oracle } export
[ --url <url> ] [ {-h | --host} <hostname> ] [ {-p | --port} <port #> ]
[ {-d | --database} <name> {-s | --schema} <schema>
{-u | --user} <username> [ --password <password> ]
[ --debug ] [ --delimiter <character> ] --destination <directory>
[ {--exclusion-mode | --exc} <exclude|include|none(default)> ]
[ --force ] --import-tool <directory>
[ --mapping-file <file|stdin> ] [ --options-file <file> ]
[ --quote <character> ] --csv-directory <directory>
[ {--relationship-name | --rel-name} <table(default)|column> ]
[ {--tiny-int | --tiny} <byte(default)|boolean> ]
[--] [ <table1 table2 ...>... ]

OPTIONS
--url <name>
Database connection URL.
This parameter is alternative to `--host`, `--port` and `--database`.

-h <hostname>, --host <hostname>
Host to use for connection to RDBMS.

-p <port #>, --port <port #>
Port number to use for connection to RDBMS.

-d <name>, --database <name>
Database name.

-u <username>, --user <username>
User for login to RDBMS.

--password <password>
Password for login to RDBMS.

-s <schema>, --schema <schema>
Schema to be exported.

--debug
Print detailed diagnostic output.

--csv-directory <directory>
Path to directory for intermediate CSV files.

--delimiter <character>
Delimiter to separate fields in CSV.

--destination <directory>
Path to destination store directory.

--exclusion-mode <exclude|include|none(default)>, --exc
<exclude|include|none(default)>
Specifies how to handle table exclusion. Options are mutually
exclusive.exclude: Excludes specified tables from the process. All
other tables will be included.include: Includes specified tables
only. All other tables will be excluded.none: All tables are
included in the process.

--force
Force delete destination store directory if it already exists.

--import-tool <directory>
Path to directory containing Neo4j import tool.

--mapping-file <file|stdin>
Path to an existing metadata mapping file. The name 'stdin' will
cause the CSV resources definitions to be read from standard input.

--options-file <file>
Path to file containing Neo4j import tool options.

--quote <character>
Character to treat as quotation character for values in CSV data.

--relationship-name <table(default)|column>, --rel-name
<table(default)|column>
Specifies whether to get the name for relationships from table names
or column names

--tiny-int <byte(default)|boolean>, --tiny <byte(default)|boolean>
Specifies whether to get the convert TinyInt to byte or boolean

--
This option can be used to separate command-line options from the
list of arguments (useful when arguments might be mistaken for
command-line options)

<table1 table2 ...>
Tables to be excluded/included

74 changes: 74 additions & 0 deletions docs/neo4j-etl-rdbms-generate-metadata-mapping.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
NAME
neo4j-etl { mysql | postgresql | oracle } generate-metadata-mapping - Create MySQL to Neo4j
metadata mapping Json.

SYNOPSIS
neo4j-etl { mysql | postgresql | oracle } generate-metadata-mapping
[ --url <url> ] [ {-h | --host} <hostname> ] [ {-p | --port} <port #> ]
[ {-d | --database} <name> {-s | --schema} <schema>
{-u | --user} <username> [ --password <password> ]
[ --debug ] [ --delimiter <delimiter> ]
[ {--exclusion-mode | --exc} <exclude|include|none(default)> ]
[ --options-file <file> ] [ --quote <quote> ]
[ {--relationship-name | --rel-name} <table(default)|column> ]
[ --tiny-int <byte(default)|boolean> ] [--] [ <table1 table2 ...>... ]

OPTIONS
--url <url>
Database connection URL.
This parameter is alternative to `--host`, `--port` and `--database`.

-h <name>, --host <name>
Host to use for connection to RDBMS.

-p <#>, --port <#>
Port number to use for connection to RDBMS.

-d <name>, --database <name>
Database name.

-u <name>, --user <name>
User for login to RDBMS.

--password <name>
Password for login to RDBMS.

-s <schema>, --schema <schema>
Schema to be exported.

--debug
Print detailed diagnostic output.

--delimiter <delimiter>
Delimiter to separate fields in CSV.

--exclusion-mode <exclude|include|none(default)>, --exc
<exclude|include|none(default)>
Specifies how to handle table exclusion. Options are mutually
exclusive.exclude: Excludes specified tables from the process. All
other tables will be included.include: Includes specified tables
only. All other tables will be excluded.none: All tables are
included in the process.

--options-file <file>
Path to file containing Neo4j import tool options.

--quote <quote>
Character to treat as quotation character for values in CSV data.

--relationship-name <table(default)|column>, --rel-name
<table(default)|column>
Specifies whether to get the name for relationships from table names
or column names.

--tiny-int <byte(default)|boolean>
Specifies whether to convert TinyInt to byte or boolean

--
This option can be used to separate command-line options from the
list of arguments (useful when arguments might be mistaken for
command-line options)

<table1 table2 ...>
Tables to be excluded/included

31 changes: 31 additions & 0 deletions docs/neo4j-etl-rdbms.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
NAME
neo4j-etl { mysql | postgresql | oracle } - Export tools for MySQL, PostgreSQL and Oracle RDBMS.

SYNOPSIS
neo4j-etl { mysql | postgresql | oracle } { export | generate-metadata-mapping | help* } [--]
[cmd-options] <cmd-args>

Where command-specific options [cmd-options] are:
export: [ --debug ] [ --url <url> ] [ {-h | --host} <hostname> ] [ {-p | --port} <port #> ]
{-d | --database} <name> [ {-s | --schema} <schema> ]
{-u | --user} <username> [ --password <password> ]
[ --options-file <file> ] --destination <directory> [ --delimiter <character> ]
[ --mapping-file <file|stdin> ] [ --quote <character> ]
[ {--exclusion-mode | --exc} <exclude|include|none(default)> ]
--import-tool <directory> [ {--relationship-name | --rel-name} <table(default)|column> ]
[ {--tiny-int | --tiny} <byte(default)|boolean> ] --csv-directory <directory> [ --force ]
generate-metadata-mapping: [ --debug ] [ {-url} <url> ] [ {-h | --host} <name> ] [ {-p | --port} <port #> ]
{-d | --database} <name> {-s | --schema} <schema>
{-u | --user} <name> [ --password <name> ]
[ --options-file <file> ] [ {--relationship-name | --rel-name} <table(default)|column> ]
[ --tiny-int <byte(default)|boolean> ] [ --delimiter <delimiter> ]
[ --quote <quote> ] [ {--exclusion-mode | --exc} <exclude|include|none(default)> ]
help:

Where command-specific arguments <cmd-args> are:
export: [ <table1 table2 ...>... ]
generate-metadata-mapping: [ <table1 table2 ...>... ]
help: [ <command>... ]

Where * indicates the default command(s)
See 'neo4j-etl help { mysql | postgresql | oracle } <command>' for more information on a specific command.
Loading

0 comments on commit aac22e2

Please sign in to comment.