RuntimeException MetaException(message:org.apache.hadoop.hive.serde2.SerDeException org.apache.hadoop.hive.hbase.HBaseSerDe

On HDP cluster, I am trying create and integrate Hive tables with existing Hbase tables. It creates the hive table. But when I am try to query the hive table it throws the following exception especially when the number of columns exceeds 200.

I checked the # of columns in both hbase & hive is same. Not getting proper solution to debug it.

hive> select * from hbase_hive.lead;

FAILED: RuntimeException MetaException(message:org.apache.hadoop.hive.serde2.SerDeException 
org.apache.hadoop.hive.hbase.HBaseSerDe: 
columns has 273 elements while hbase.columns.mapping has 204 elements (counting the key if implicit))

Is there any column limitation in this case?

Please suggest me solution on this

Answers


This has fixed the issue.

https://hortonworks.my.salesforce.com/kA2E0000000LZQ5?srPos=0&srKp=ka2⟨=en_US

ROOT CAUSE:

A character limitation for PARAM_VALUE field in SERDE_PARAMS table in hive metastore for 4000 character is the root cause of this issue. This limitation prevents Hive from creating a table with high column numbers, eventually causing desc or select * from to fail with error above.

WORKAROUND: This issue can be worked around by doing the following in hive metastore
-- log into Hive Metastore DB -- >alter table SERDE_PARAMS MODIFY PARAM_VALUE VARCHAR(400000000);

Need Your Help

Replace values in matrix with other values

matlab matrix

I have a matrix with integers and I need to replace all appearances of 2 with -5. What is the most efficient way to do it? I made it the way below, but I am sure there is more elegant way.