Wednesday, June 9, 2021

Enabling PAM Authentication in Hive (HDP 2.6)

 

Overview

  • This article describes how to enable PAM authentication in Hive. 
  • By default, there's no authentication to Hive server. 
  • With PAM, authentication is performed against local OS user credentials.


Procedure

Step 1 - JPam Library

  • Download latest copy JPam library from http://jpam.sourceforge.net/
  • Latest version is 1.1 (JPam-Linux_amd64-1.1.tgz)
  • Once downloaded, create /usr/hdp/ext/lib folder in tbdrmnn1.
  • Copy JPam-1.1.jar and libjpam.so to the folder above


Step 2 - Hive Server Configuration

  • Add following to hive-env template in Ambari:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/hdp/ext/lib 
export JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/usr/hdp/ext/lib

  • add following to hive-site in Ambari:



hive.server2.authentication=pam
hive.server2.authentication.pam.services=passwd,login

  • Restart all affected services as prompted in Ambari.


Step 3 - Local System User

  • Run following command to allow root group read access:


# chmod 644 /etc/login.defs
# chmod 640 /etc/shadow


  • Add hive user to root group:

# usermod -a root hive


Verification

  • Create a local system user and assign it a password.
  • Use the following command to access hive:

$ beeline -u jdbc:hive2://hadoop1.mylocal.net:10000 -n <user> -p <password>


Saturday, March 21, 2020

Clearing table locks in MariaDB ColumnStore (MCS)

If you encounter the following error running MCS:

Internal error: CAL0009: Truncate table failed:  IDB-2009: Unable to perform the cpimport operation because 30084 with PID -1 is currently holding the table lock for session .  

It's due to locks placed on the objects.  Use the following command to view locked objects in MCS:

$ /usr/local/mariadb/columnstore/bin/viewtablelock

 There are 3 table locks

 Table                                   LockID  Process   PID    Session   Txn    CreationTime              State    DBRoots  
 db.table1  159107  DMLProc   43294  614       79082  Thu Mar 19 03:40:37 2020  LOADING  1        
 db.table2            159120  DMLProc   43294  602       79085  Thu Mar 19 05:00:43 2020  LOADING  1        
 db2.table3      159129  cpimport  30084  BulkLoad  n/a    Thu Mar 19 05:15:53 2020  LOADING  1        

Now that you have the list of locked objects, clear the table locks by LockID:

$ /usr/local/mariadb/columnstore/bin/cleartablelock 159129
Rolling back and clearing table lock for table db2.table3; table lock 159129

Sending rollback request to PM1...
Successful rollback response from PM1
Sending cleanup request to PM1...
Successful cleanup response from PM1

Table lock 159129 for table db2.table3 is cleared.



Thursday, March 9, 2017

Hive External Table

Script to create an external table in Hive to read records from a HDFS folder:

CREATE EXTERNAL TABLE Mytable (
    Id int,
    PlanID string,
    ServiceID string,
    SessionDuration int)
STORED AS TEXTFILE
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ","
LOCATION "/user/hive/staging/"
TBLPROPERTIES(
    "skip.header.line.count"="1"
);

Using the script above, it'll use the files in /user/hive/staging and skip the first line of each file.

Tuesday, August 23, 2016

Enabling client authentication in MongoDB

Before enabling authentication in MongoDB, we'll have to create a user and assign it a built-in role. We'll use the built-in "root" role that provides admin access to all databases.

I've done this in Ubuntu using MongoDB 3.2.

Sunday, August 7, 2016

Limiting grep output

Here's a quick way to limit output of the grep command.

Most of the time, we issue the following to find needle in the haystack:

grep needle file.txt

This prints out the matching pattern. If the output is way too long and we only need a section of it, we can use the extended grep (a.k.a. egrep) option.