Setting CPU min, max and governor

Use the script below to set the min, max and CPU governor in Linux:

#!/usr/bin/env bash

MIN_FREQ=800000
MAX_FREQ=1000000
GOVERNOR=powersave

for i in `seq 0 3`;
do
  echo ${GOVERNOR} > /sys/devices/system/cpu/cpu${i}/cpufreq/scaling_governor
  echo ${MAX_FREQ} > /sys/devices/system/cpu/cpu${i}/cpufreq/scaling_max_freq
  echo ${MIN_FREQ} > /sys/devices/system/cpu/cpu${i}/cpufreq/scaling_min_freq
done

Set the max number of cores accordingly.

Script requires the following packages to be installed:

• cpufrequtils
• cpufreqd

Run the command below to verify that the changes are applied:

$ cpufreq-info

cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009

Report errors and bugs to cpufreq@vger.kernel.org, please.

analyzing CPU 0:

  driver: intel_pstate

  CPUs which run at the same hardware frequency: 0

  CPUs which need to have their frequency coordinated by software: 0

  maximum transition latency: 4294.55 ms.

  hardware limits: 700 MHz - 3.40 GHz

  available cpufreq governors: performance, powersave

  current policy: frequency should be within 800 MHz and 1000 MHz.

                  The governor "powersave" may decide which speed to use

                  within this range.

  current CPU frequency is 754 MHz.

analyzing CPU 1:

  driver: intel_pstate

  CPUs which run at the same hardware frequency: 1

  CPUs which need to have their frequency coordinated by software: 1

  maximum transition latency: 4294.55 ms.

  hardware limits: 700 MHz - 3.40 GHz

  available cpufreq governors: performance, powersave

  current policy: frequency should be within 800 MHz and 1000 MHz.

                  The governor "powersave" may decide which speed to use

                  within this range.

  current CPU frequency is 800 MHz.

analyzing CPU 2:

  driver: intel_pstate

  CPUs which run at the same hardware frequency: 2

  CPUs which need to have their frequency coordinated by software: 2

  maximum transition latency: 4294.55 ms.

  hardware limits: 700 MHz - 3.40 GHz

  available cpufreq governors: performance, powersave

  current policy: frequency should be within 800 MHz and 1000 MHz.

                  The governor "powersave" may decide which speed to use

                  within this range.

  current CPU frequency is 800 MHz.

analyzing CPU 3:

  driver: intel_pstate

  CPUs which run at the same hardware frequency: 3

  CPUs which need to have their frequency coordinated by software: 3

  maximum transition latency: 4294.55 ms.

  hardware limits: 700 MHz - 3.40 GHz

  available cpufreq governors: performance, powersave

  current policy: frequency should be within 800 MHz and 1000 MHz.

                  The governor "powersave" may decide which speed to use

                  within this range.

  current CPU frequency is 795 MHz.

Enabling PAM Authentication in Hive (HDP 2.6)

 

Overview

• This article describes how to enable PAM authentication in Hive. 
• By default, there's no authentication to Hive server. 
• With PAM, authentication is performed against local OS user credentials.

Procedure

Step 1 - JPam Library

• Download latest copy JPam library from http://jpam.sourceforge.net/
• Latest version is 1.1 (JPam-Linux_amd64-1.1.tgz)
• Once downloaded, create /usr/hdp/ext/lib folder in tbdrmnn1.
• Copy JPam-1.1.jar and libjpam.so to the folder above

Step 2 - Hive Server Configuration

• Add following to hive-env template in Ambari:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/hdp/ext/lib 

export JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/usr/hdp/ext/lib

• add following to hive-site in Ambari:

hive.server2.authentication=pam

hive.server2.authentication.pam.services=passwd,login

• Restart all affected services as prompted in Ambari.

Step 3 - Local System User

• Run following command to allow root group read access:

# chmod 644 /etc/login.defs # chmod 640 /etc/shadow

• Add hive user to root group:

# usermod -a root hive

Verification

• Create a local system user and assign it a password.
• Use the following command to access hive:

$ beeline -u jdbc:hive2://hadoop1.mylocal.net:10000 -n <user> -p <password>

Clearing table locks in MariaDB ColumnStore (MCS)

If you encounter the following error running MCS:

Internal error: CAL0009: Truncate table failed:  IDB-2009: Unable to perform the cpimport operation because 30084 with PID -1 is currently holding the table lock for session .  

It's due to locks placed on the objects.  Use the following command to view locked objects in MCS:

$ /usr/local/mariadb/columnstore/bin/viewtablelock

 There are 3 table locks

 Table                                   LockID  Process   PID    Session   Txn    CreationTime              State    DBRoots  
 db.table1  159107  DMLProc   43294  614       79082  Thu Mar 19 03:40:37 2020  LOADING  1        
 db.table2            159120  DMLProc   43294  602       79085  Thu Mar 19 05:00:43 2020  LOADING  1        
 db2.table3      159129  cpimport  30084  BulkLoad  n/a    Thu Mar 19 05:15:53 2020  LOADING  1        
Now that you have the list of locked objects, clear the table locks by LockID:

$ /usr/local/mariadb/columnstore/bin/cleartablelock 159129
Rolling back and clearing table lock for table db2.table3; table lock 159129

Sending rollback request to PM1...
Successful rollback response from PM1
Sending cleanup request to PM1...
Successful cleanup response from PM1

Table lock 159129 for table db2.table3 is cleared.

Hive External Table

Script to create an external table in Hive to read records from a HDFS folder:

CREATE EXTERNAL TABLE Mytable (
    Id int,
    PlanID string,
    ServiceID string,
    SessionDuration int)
STORED AS TEXTFILE
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ","
LOCATION "/user/hive/staging/"
TBLPROPERTIES(
    "skip.header.line.count"="1"
);

Using the script above, it'll use the files in /user/hive/staging and skip the first line of each file.

Enabling client authentication in MongoDB

Before enabling authentication in MongoDB, we'll have to create a user and assign it a built-in role. We'll use the built-in "root" role that provides admin access to all databases.

I've done this in Ubuntu using MongoDB 3.2.