Error:"MetaStoreClient lost connection. Attempting to reconnect (1 of 24) after 5s. getCurrentNotifi...

SupportKB Problem Description:
After upgrading to HDP 3.0, the HiveServer2 fails to start and the following error is displayed in the HiveServer2 logs:

1. WARN [main]: metastore.RetryingMetaStoreClient (:()) -
2. MetaStoreClient lost connection. Attempting to reconnect (1 of 24) after 5s. getCurrentNotificationEventId
3. org.apache.thrift.TApplicationException: Internal error processing get_current_notificationEventId
4. at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
5. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
6. at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
7. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
8. at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_current_notificationEventId
9. (ThriftHiveMetastore.java:5575) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
10. at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_current_notificationEventId
11. (ThriftHiveMetastore.java:5563) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
12. at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getCurrentNotificationEventId
13. (HiveMetaStoreClient.java:2723) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
14. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
15. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
16. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
17. at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
18. at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
19. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
20. at com.sun.proxy.$Proxy40.getCurrentNotificationEventId(Unknown Source) ~[?:?]
21. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
22. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
23. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
24. at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
25. at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke
26. (HiveMetaStoreClient.java:2773) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
27. at com.sun.proxy.$Proxy40.getCurrentNotificationEventId(Unknown Source) ~[?:?]
28. at org.apache.hadoop.hive.metastore.messaging.EventUtils$MSClientNotificationFetcher.
29. getCurrentNotificationEventId(EventUtils.java:73)
30. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
31. at org.apache.hadoop.hive.ql.metadata.events.NotificationEventPoll.<init>
32. (NotificationEventPoll.java:103)
33. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
34. at org.apache.hadoop.hive.ql.metadata.events.NotificationEventPoll.initialize
35. (NotificationEventPoll.java:59)
36. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
37. at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:275)
38. ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
39. at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1063)
40. ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
41. at org.apache.hive.service.server.HiveServer2.access$1600(HiveServer2.java:140)
42. ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
43. at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1332)
44. ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
45. at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1176)
46. ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
47. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
48. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
49. ~[?:1.8.0_112]
50. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
51. at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
52. at org.apache.hadoop.util.RunJar.run(RunJar.java:318) ~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?]
53. at org.apache.hadoop.util.RunJar.main(RunJar.java:232) ~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?]In hive-metastore.log, the following message is displayed at the same timeline:
54. ERROR [pool-6-thread-200]: metastore.HiveMetaStore (HiveMetaStore.java:get_current_notificationEventId(7617))
55. - Not authorized to make the get_current_notificationEventId call.
56. You can try to disable metastore.metastore.event.db.notification.api.auth
57. org.apache.hadoop.hive.metastore.api.MetaException: User hive is not allowed to perform this API call
58. at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.authorizeProxyPrivilege
59. (HiveMetaStore.java:7655)
60. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
61. at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_current_notificationEventId(HiveMetaStore.java:7615)
62. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
63. at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) ~[?:?]
64. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
65. ~[?:1.8.0_112]
66. at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
67. at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
68. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
69. at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
70. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
71. at com.sun.proxy.$Proxy34.get_current_notificationEventId(Unknown Source) [?:?]
72. at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_current_notificationEventId.getResult
73. (ThriftHiveMetastore.java:18364)
74. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
75. at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_current_notificationEventId.getResult
76. (ThriftHiveMetastore.java:18349) [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
77. at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
78. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
79. at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
80. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
81. at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
82. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
83. at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_112]
84. at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_112]
85. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
86. [hadoop-common-3.1.0.3.0.0.0-1634.jar:?]
87. at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
88. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
89. at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
90. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
91. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
92. [?:1.8.0_112]
93. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
94. [?:1.8.0_112]
95. at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112] Cause:
    The connection to Metastore from HiveServer2 fails after upgrading the cluster when hive.metastore.event.db.notification.api.auth property is set to true. Solution:
    This is a known issue ( ). Workaround In HDFS > Configs > Custom core-site.xml, add the following:
96. hadoop.proxyuser.hive.hosts=HS2_HOST
97. hadoop.proxyuser.hive.groups=*Or
    In Hive >  Configs > Custom hive-site.xml, add the following:
98. hive.metastore.event.db.notification.api.auth=falseRestart the required services.
    For more details about this property, see About:
    This article created by Hortonworks Support (Article: 000007703) on 2018-08-06 07:32
    OS: All
    Type: Cluster_Administration
    Version: H3.0.0   Support ID: 000007703