How to import an AvroKeyValueOutputFormat file into hive?

My mapreduce writes an avro file with the AvroKeyValueOutputFormat but I'm having some troubles to import this file into hive.

How I have to define my schema in hive to get it working?

Answers


you have to use the AvroSerDe described in

http://goo.gl/TwsRTd

or you have to transform your output to the RowFormat that you are using in your defined hive table (again using another mapreduce job)

Regards

Martin


The format of the AvroSerDe can be a little tricky. As long as you know the Avro schema though it tends to work wonders. Hopefully this example helps.

CREATE EXTERNAL TABLE HIVEDATA
ROW FORMAT
SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
WITH SERDEPROPERTIES ('avro.schema.literal'='
{
    "namespace": "originalname",
    "name": "feature_value",
    "type": "record",
    "fields": [
        {"name": "acct_id", "type": "long"},
        {"name": "feature_name", "type": ["null","string"], "default": null},
        {"name": "namespace", "type": ["null","string"], "default": null},
        {"name": "feature_value", "type": ["null","double"], "default": null}
    ]
}
')
STORED AS
INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
LOCATION '/hdfs/location'
;

Need Your Help

Modifying a Shader Storage Buffer Object from within a Vertex Shader

c++ opengl glsl opengl-4

In my base program (C++/OpenGL 4.5) I have copied the content of the Vertex Buffer to an Shader Storage Buffer (SSBO):