VTD Processor for Apache Hive XML SerDe

Using VTD-XML based processor instead of default JDK DOM XPath processor can provide significant performance gains. See http://vtd-xml.sourceforge.net/

Use the following DDL to create Apache Hive table, note the "xml.processor.class" for VTD-XML processor,

CREATE [EXTERNAL] TABLE <table_name> (<column_specifications>)
ROW FORMAT SERDE "com.ibm.spss.hive.serde2.xml.XmlSerDe"
WITH SERDEPROPERTIES (
"xml.processor.class"="com.ximpleware.hive.serde2.xml.vtd.XmlProcessor",
"column.xpath.<column_name>"="<xpath_query>",
... ["xml.map.specification.<element_name>"="<map_specification>"
...
])
STORED AS
INPUTFORMAT "com.ibm.spss.hive.serde2.xml.XmlInputFormat"
OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat"
[LOCATION "<data_location>"]
TBLPROPERTIES (
"xmlinput.start"="<start_tag ",
"xmlinput.end"="<end_tag>"
);

See https://github.com/dvasilen/Hive-XML-SerDe for further details.

Dependency: vtd-xml-2.11.0.jar

http://search.maven.org/#search%7Cga%7C1%7Cvtd-xml

Important notice : License

VTD-XML use a dual GPLv2/commercial license.

This is the reason why Hive-XML-SerDe-VTD is a separate project licensed under GPLv2 instead of being integrated into core Hive-XML-SerDe, that is Apache 2 licensed.

So please be aware that if you use this module :

• either your whole Hive-XML-SerDe based project become GPLv2,
• or you have to purchase a commercial license from Ximpleware.