/[Apache-SVN]
ViewVC logotype

Revision 1221830


Jump to revision: Previous Next
Author: heyongqiang
Date: Wed Dec 21 18:21:44 2011 UTC (12 years, 4 months ago)
Changed paths: 4
Log Message:
HIVE-2666 [jira] StackOverflowError when using custom UDF in map join
(Kevin Wilfong via Yongqiang He)

Summary:
Resource files are now added to the class path as soon as they are added via the
CLI.  This fixes the stack overflow error mentioned in the JIRA by ensuring a
consistent class loader between serializers and deserializers for the same
query.

Note that now serdes which contain a static block to register themselves are now
registered twice, once when adding the file to the class loader, and once when
an instance of the class is created.  Previously, registering a serde twice
resulted in an exception, to avoid this, I have downgraded it to a warning.

When a custom UDF is used as part of a join which is converted to a map join,
the XMLEncoder enters an infinite loop when serializing the map reduce task for
the second time, as part of sending it to be executed.  This results in a stack
overflow error.

Test Plan:
I ran the unit tests to verify nothing was broken.

I ran several queries which used custom UDFs and involved a join which was
converted to a map join.  I verified these completed successfully consistently

Reviewers: JIRA, heyongqiang

Reviewed By: heyongqiang

CC: heyongqiang, kevinwilfong

Differential Revision: 957

Changed paths

Path Details
Directoryhive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java modified , text changed
Directoryhive/trunk/ql/src/java/org/apache/hadoop/hive/ql/processors/AddResourceProcessor.java modified , text changed
Directoryhive/trunk/ql/src/java/org/apache/hadoop/hive/ql/processors/DeleteResourceProcessor.java modified , text changed
Directoryhive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java modified , text changed

infrastructure at apache.org
ViewVC Help
Powered by ViewVC 1.1.26