Setting CPU min, max and governor

Use the script below to set the min, max and CPU governor in Linux:

#!/usr/bin/env bash

MIN_FREQ=800000
MAX_FREQ=1000000
GOVERNOR=powersave

for i in `seq 0 3`;
do
  echo ${GOVERNOR} > /sys/devices/system/cpu/cpu${i}/cpufreq/scaling_governor
  echo ${MAX_FREQ} > /sys/devices/system/cpu/cpu${i}/cpufreq/scaling_max_freq
  echo ${MIN_FREQ} > /sys/devices/system/cpu/cpu${i}/cpufreq/scaling_min_freq
done

Set the max number of cores accordingly.

Script requires the following packages to be installed:

• cpufrequtils
• cpufreqd

Run the command below to verify that the changes are applied:

$ cpufreq-info

cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009

Report errors and bugs to cpufreq@vger.kernel.org, please.

analyzing CPU 0:

  driver: intel_pstate

  CPUs which run at the same hardware frequency: 0

  CPUs which need to have their frequency coordinated by software: 0

  maximum transition latency: 4294.55 ms.

  hardware limits: 700 MHz - 3.40 GHz

  available cpufreq governors: performance, powersave

  current policy: frequency should be within 800 MHz and 1000 MHz.

                  The governor "powersave" may decide which speed to use

                  within this range.

  current CPU frequency is 754 MHz.

analyzing CPU 1:

  driver: intel_pstate

  CPUs which run at the same hardware frequency: 1

  CPUs which need to have their frequency coordinated by software: 1

  maximum transition latency: 4294.55 ms.

  hardware limits: 700 MHz - 3.40 GHz

  available cpufreq governors: performance, powersave

  current policy: frequency should be within 800 MHz and 1000 MHz.

                  The governor "powersave" may decide which speed to use

                  within this range.

  current CPU frequency is 800 MHz.

analyzing CPU 2:

  driver: intel_pstate

  CPUs which run at the same hardware frequency: 2

  CPUs which need to have their frequency coordinated by software: 2

  maximum transition latency: 4294.55 ms.

  hardware limits: 700 MHz - 3.40 GHz

  available cpufreq governors: performance, powersave

  current policy: frequency should be within 800 MHz and 1000 MHz.

                  The governor "powersave" may decide which speed to use

                  within this range.

  current CPU frequency is 800 MHz.

analyzing CPU 3:

  driver: intel_pstate

  CPUs which run at the same hardware frequency: 3

  CPUs which need to have their frequency coordinated by software: 3

  maximum transition latency: 4294.55 ms.

  hardware limits: 700 MHz - 3.40 GHz

  available cpufreq governors: performance, powersave

  current policy: frequency should be within 800 MHz and 1000 MHz.

                  The governor "powersave" may decide which speed to use

                  within this range.

  current CPU frequency is 795 MHz.

Enabling PAM Authentication in Hive (HDP 2.6)

 

Overview

• This article describes how to enable PAM authentication in Hive. 
• By default, there's no authentication to Hive server. 
• With PAM, authentication is performed against local OS user credentials.

Procedure

Step 1 - JPam Library

• Download latest copy JPam library from http://jpam.sourceforge.net/
• Latest version is 1.1 (JPam-Linux_amd64-1.1.tgz)
• Once downloaded, create /usr/hdp/ext/lib folder in tbdrmnn1.
• Copy JPam-1.1.jar and libjpam.so to the folder above

Step 2 - Hive Server Configuration

• Add following to hive-env template in Ambari:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/hdp/ext/lib 

export JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/usr/hdp/ext/lib

• add following to hive-site in Ambari:

hive.server2.authentication=pam

hive.server2.authentication.pam.services=passwd,login

• Restart all affected services as prompted in Ambari.

Step 3 - Local System User

• Run following command to allow root group read access:

# chmod 644 /etc/login.defs # chmod 640 /etc/shadow

• Add hive user to root group:

# usermod -a root hive

Verification

• Create a local system user and assign it a password.
• Use the following command to access hive:

$ beeline -u jdbc:hive2://hadoop1.mylocal.net:10000 -n <user> -p <password>

Clearing table locks in MariaDB ColumnStore (MCS)

If you encounter the following error running MCS:

Internal error: CAL0009: Truncate table failed:  IDB-2009: Unable to perform the cpimport operation because 30084 with PID -1 is currently holding the table lock for session .  

It's due to locks placed on the objects.  Use the following command to view locked objects in MCS:

$ /usr/local/mariadb/columnstore/bin/viewtablelock

 There are 3 table locks

 Table                                   LockID  Process   PID    Session   Txn    CreationTime              State    DBRoots  
 db.table1  159107  DMLProc   43294  614       79082  Thu Mar 19 03:40:37 2020  LOADING  1        
 db.table2            159120  DMLProc   43294  602       79085  Thu Mar 19 05:00:43 2020  LOADING  1        
 db2.table3      159129  cpimport  30084  BulkLoad  n/a    Thu Mar 19 05:15:53 2020  LOADING  1        
Now that you have the list of locked objects, clear the table locks by LockID:

$ /usr/local/mariadb/columnstore/bin/cleartablelock 159129
Rolling back and clearing table lock for table db2.table3; table lock 159129

Sending rollback request to PM1...
Successful rollback response from PM1
Sending cleanup request to PM1...
Successful cleanup response from PM1

Table lock 159129 for table db2.table3 is cleared.

Hive External Table

Script to create an external table in Hive to read records from a HDFS folder:

CREATE EXTERNAL TABLE Mytable (
    Id int,
    PlanID string,
    ServiceID string,
    SessionDuration int)
STORED AS TEXTFILE
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ","
LOCATION "/user/hive/staging/"
TBLPROPERTIES(
    "skip.header.line.count"="1"
);

Using the script above, it'll use the files in /user/hive/staging and skip the first line of each file.

Enabling client authentication in MongoDB

Before enabling authentication in MongoDB, we'll have to create a user and assign it a built-in role. We'll use the built-in "root" role that provides admin access to all databases.

I've done this in Ubuntu using MongoDB 3.2.

Limiting grep output

Here's a quick way to limit output of the grep command.

Most of the time, we issue the following to find needle in the haystack:

grep needle file.txt

This prints out the matching pattern. If the output is way too long and we only need a section of it, we can use the extended grep (a.k.a. egrep) option.

Enabling WS-Security in Spring Boot using CXF, JAX-WS and JAXB

In my previous post, I've shown how to quickly create a WSDL/SOAP based web service. This post will build on top of that to include WS-Security. We'll be using simple username/password authentication.

Getting Caused by: com.sun.xml.bind.v2.runtime.IllegalAnnotationsException: 1 counts of IllegalAnnotationExceptions...

My previous post shows how we can easily create a SOAP based web service using Spring Boot, CXF, JAX-WS and JAXB.

There's a small matter to note when naming functions. The following function naming works:

package com.techtots.services;

import javax.jws.WebMethod;
import javax.jws.WebParam;
import javax.jws.WebResult;
import javax.jws.WebService;
import javax.xml.bind.annotation.XmlElement;

import com.techtots.contracts.UserRegisterRequest;
import com.techtots.contracts.UserRegisterResponse;

@WebService
public interface UserService {
    
    @WebMethod
    @WebResult(name = "userRegisterResponse")
    public @XmlElement(required = true, nillable = false) UserRegisterResponse registerUser(
            @XmlElement(required = true, nillable = false) 
            @WebParam(name = "userRegisterRequest")
            UserRegisterRequest userRegisterRequest);
}

Creating WSDL/SOAP web services in Spring Boot using CXF, JAX-WS and JAXB

Here's a quick way to use Spring Boot to expose web services via WSDL/SOAP using CXF, JAX-WS and JAXB.

Add the following artifacts into your Spring Boot pom.xml:

    org.apache.cxf
    cxf-rt-frontend-jaxws
    3.1.6



    org.apache.cxf
    cxf-rt-transports-http
    3.1.6

Configuring Apache to return CORS headers for Drupal Services

Here's what I did to configure Apache to return the proper CORS headers for my webapp consumption which is written in AngularJS:

Header always set Access-Control-Allow-Origin "*"
Header always set Access-Control-Allow-Methods "GET, POST, OPTIONS"
Header always set Access-Control-Allow-Headers "accept, content-type"
Header always set Access-Control-Allow-Credentials "true"

That can be set in the VirtualHost tags for your server instance. Please note that Access-Control-Allow-Origin value shouldn't be set to * in a production environment. This should only be done for testing environments. The mod_headers module must be enabled in Apache for this configuration to work.

Having these options should be sufficient. But since I'm using Drupal 7 Services, it doesn't play well with pre-flight call which uses HTTP OPTIONS method. Drupal services will return a 404 even if the correct endpoint is specified when OPTIONS method is used.

Here's the additional config using mod_rewrite to return HTTP 200 for all OPTIONS requests:

RewriteEngine On
RewriteCond %{REQUEST_METHOD} OPTIONS
RewriteRule ^(.*)$ $1 [R=200,L,E=HTTP_ORIGIN:%{HTTP:ORIGIN}]]

Resizing images in batch using mogrify

Here's a quick way to resize images via shell using mogrify. It's part of ImageMagick package.

mogrify -resize 60% *.jpg

This will resize all files ending with .jpg extension to 60% of their original size.

We're only scratching the surface of what mogrify can do. Have fun exploring :)

Default Parameter Values in PHP Functions

Here's a neat way to set some default parameter values in a PHP function. This only works if a function param is passed in as associative array.

function create_profile($values) {
  if (!is_array($values)) {
    $values = array();
  }
  
  // Default key/values
  $values += array(
      'uid' => NULL,
      'username' => '',
      'mobile' => '99999999',
      'location' => 'POINT(0 0)',
      'fb_id' => ''
  );
}

The most important line is at line 7; where both arrays are merged. If the key/value pair is in the $values array, it will be ignored; else the default values will be used.

Setting "placeholder" attribute in Drupal 7 form elements

Here's a quick way on how to use HTML5 placeholder attribute in Drupal forms. Example below is altering the login form in a custom theme:

function mytheme_form_alter(&$form, &$form_state, $form_id) {
  if ($form_id == 'user_login') {
    $form['name']['#attributes'] = array('placeholder' => $form['name']['#title']);
    
    $form['pass']['#attributes'] = array('placeholder' => $form['pass']['#title']);
  }
}

Using JMeter's Regular Expression Extractor

Here's how to use JMeter's Regular Expression Extractor post-processor to extract the response from a HTTP Request sampler and use it in the subsequent request.

In the screen above, I'm extracting the value of the hidden form element named "t".  If there's no match to the regular expression, the default value "NO TOKEN" will be returned.

To use the extracted value in subsequent HTTP requests, just use the usual JMeter parameter notation.  In this case, it's ${TOKEN}; as defined in the Reference Name value.

Using Drupal's hook_theme() for custom theming

This is a short post on how to use Drupal's hook_theme() to customize a layout within your own custom module.

Start off with implementing the hook_theme() function in your module:

/**
 * Implements hook_theme()
 * @param type $existing
 * @param type $type
 * @param type $theme
 * @param type $path
 * @return array
 */
function mypm_theme($existing, $type, $theme, $path) {
  $themes = array(
    'chat' => array(
      'variables' => array(
        'message' => NULL,
        'author' => NULL,
        'date_posted' => NULL,
        'is_sender' => NULL,
        'product_url' => NULL,
      )
    ),
  );

  return $themes;
}

What I've declared in the function is that I have a custom theme function called "chat". It takes in 5 variables defined in the variables array. As always, remember to clear Drupal's cache once you've defined a new theme function. Now that we have declared our theme function, we'll have to implement it:

function theme_chat($variables) {
  $main_style = 'chat-holder';

  if (isset($variables['is_sender'])) {
    if ($variables['is_sender'] == TRUE) {
      $main_style .= ' chat-holder-sent';
    }
    else {
      $main_style .= ' chat-holder-received';
    }
  }
  else {
    $main_style .= ' chat-holder-received';
  }

  $output = '
';

  $output .= '
';
  $output .= $variables['author'];
  $output .= '
';

  $output .= '
';
  $output .= format_date($variables['date_posted']);
  $output .= '
';

  $output .= '
';
  $output .= nl2br(check_plain($variables['message']));
  $output .= '
';

  $output .= $variables['product_url'];

  $output .= '
';

  return $output;
}

The theme_chat function builds the HTML output based on the variables passed in. You can now use this theme function in your form or normal page callback like below:

    $chat_output .= theme('chat',
            array(
                'message' => $chat->message,
                'author' => $chat->author_name,
                'date_posted' => $chat->created,
                'is_sender' => $chat->sender,
                'product_url' => $chat->product_url,
            ));

Spring 3 MVC + Hibernate 4 + JdbcTemplate + Partial REST controller (Part 3)

Last part of the article is to use either Hibernate or Spring's JdbcTemplate to retrieve data. The following class is the BaseDao where DataSource and Hibernate SessionFactory are injected and used by concrete DAO classes:

package com.techtots.services.api.dao;

import javax.sql.DataSource;

import org.hibernate.SessionFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate;

public abstract class BaseDao {

	@Autowired
	protected SessionFactory sessionFactory;
	
	protected DataSource dataSource;

	protected JdbcTemplate jdbcTemplate;
	protected NamedParameterJdbcTemplate namedParameterJdbcTemplate;

	public DataSource getDataSource() {
		return dataSource;
	}

	@Autowired
	public void setDataSource(DataSource dataSource) {
		this.dataSource = dataSource;
		this.jdbcTemplate = new JdbcTemplate(this.dataSource);
		this.namedParameterJdbcTemplate = new NamedParameterJdbcTemplate(this.dataSource);
		
	}

	public JdbcTemplate getJdbcTemplate() {
		return jdbcTemplate;
	}

	public NamedParameterJdbcTemplate getNamedParameterJdbcTemplate() {
		return namedParameterJdbcTemplate;
	}

}

Once we have this out of the way, we can now define a concrete DAO class. We'll have a simple UserDao for testing.

package com.techtots.services.api.dao;

import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.List;

import org.hibernate.Criteria;
import org.springframework.jdbc.core.RowMapper;
import org.springframework.stereotype.Repository;

import com.techtots.services.api.entity.User;
import com.techtots.services.common.vo.UserVo;

@Repository
public class UserDao extends BaseDao {

	public List<User> getList() {
		Criteria criteria = sessionFactory.getCurrentSession().createCriteria(User.class);
		
		return criteria.list();
	}
	
	public List<UserVo> list() {
		final List<UserVo> list = new ArrayList<UserVo>();
		
		getJdbcTemplate().query("select * from tbl_user", new RowMapper<Object>() {

			@Override
			public Object mapRow(ResultSet rs, int arg1) throws SQLException {
				UserVo vo = new UserVo();
				
				vo.setId(rs.getInt("id"));
				vo.setUsername(rs.getString("nickname"));
				
				list.add(vo);
				
				return null;
			}
			
		});
		
		return list;
	}
}

Since this is just a reference, I've implemented 2 different function to retrieve data. getList() uses Hibernate's SessionFactory while list() uses JdbcTemplate. Following classes are the UserVo and User entity classes which are used by the DAO respectively:

package com.techtots.services.common.vo;

public class UserVo {

	private int id;
	private String username;

	public int getId() {
		return id;
	}

	public void setId(int id) {
		this.id = id;
	}

	public String getUsername() {
		return username;
	}

	public void setUsername(String username) {
		this.username = username;
	}

}

package com.techtots.services.api.entity;

import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.persistence.Table;

@Entity
@Table(name = "tbl_user2")
public class User {

	@Id
	@GeneratedValue(strategy = GenerationType.AUTO)
	@Column(name = "id", nullable = false)
	private int id;

	@Column(name = "name", nullable = false, length = 200)
	private String name;

	@Column(name = "email", nullable = false, length = 250, unique = true)
	private String email;

	public int getId() {
		return id;
	}

	public void setId(int id) {
		this.id = id;
	}

	public String getName() {
		return name;
	}

	public void setName(String name) {
		this.name = name;
	}

	public String getEmail() {
		return email;
	}

	public void setEmail(String email) {
		this.email = email;
	}

}

To call this from your UserController, you'll need to add the @Transactional annotation at either the class or method level. You wouldn't usually add this at the controller level as it's better to have everything wrapped up in a service class.

Spring 3 MVC + Hibernate 4 + JdbcTemplate + Partial REST controller (Part 2)

Second part of the article will focus on the controllers which will process clients' requests as well as other helper classes.

We start off with a custom exception. We add on a HttpStatus code into the exception to return to clients:

package com.techtots.services.api.common.exceptions;

import org.springframework.http.HttpStatus;

public class RestException extends Exception {

 private static final long serialVersionUID = -6373811042517187537L;

 public static final HttpStatus DEFAULT_ERROR_STATUS = HttpStatus.BAD_REQUEST;
 
 private HttpStatus status;
  
 public HttpStatus getStatus() {
  return status;
 }

 public void setStatus(HttpStatus status) {
  this.status = status;
 }

 public RestException() {
  super();
  
  this.status = DEFAULT_ERROR_STATUS;
 }

 public RestException(String message) {
  super(message);
 
  this.status = DEFAULT_ERROR_STATUS;
 }
 
 public RestException(HttpStatus status, String message) {
  super(message);
  
  this.status = status;
 }
}

With Spring 3.2, we can use the @ControllerAdvice annotation to centralize validation and exception handling.

package com.techtots.services.api.common;

import org.apache.commons.lang3.exception.ExceptionUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.HttpHeaders;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.ControllerAdvice;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.servlet.mvc.method.annotation.ResponseEntityExceptionHandler;

import com.techtots.services.api.common.exceptions.RestException;

@ControllerAdvice
public class RestResponseExceptionHandler extends ResponseEntityExceptionHandler {

 Logger log = LoggerFactory.getLogger(RestResponseExceptionHandler.class);
 
 @ExceptionHandler(RestException.class)
 protected ResponseEntity handleBadRequest(RestException ex) {

  log.error(ExceptionUtils.getStackTrace(ex));
  
  HttpHeaders headers = new HttpHeaders();
  headers.setContentType(MediaType.TEXT_HTML);
  
  ResponseEntity entity = new ResponseEntity(ex.getMessage(), headers, ex.getStatus()); 
  
  return entity;
 }
}

Finally, a test controller to hook everything up:

package com.techtots.services.api.controller;

import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.ResponseBody;

import com.techtots.services.api.common.exceptions.RestException;

@Controller
@RequestMapping("/auth")
public class AuthController extends BaseController {
 @RequestMapping(value = "/test", method = RequestMethod.GET, produces = "application/json")
 @ResponseBody
 public String test() throws Exception {

  return "ok";
 }

 @RequestMapping(value = "/test2", method = RequestMethod.POST, consumes = "application/json", produces = "application/json")
 @ResponseBody
 public void test2() throws Exception {

  throw new RestException("something's missing...");
 }

}

Calling the test method will return a JSON encoded message. While calling the test2 method will throw a HTTP exception with the message "something's missing" with HTTP error code 400 (the default HTTP error code returned in RestResponseExceptionHandler).

Spring 3 MVC + Hibernate 4 + JdbcTemplate + Partial REST controller (Part 1)

This is post serves as a reference point to quickly setup a Java web application with Spring 3 MVC, Hibernate 4, JdbcTemplate and a partial REST controller. We'll be consuming and responding JSON content via the REST controllers.

I'm using Eclipse as the IDE and will be connecting to MySQL database.

Let's start off with the libraries:

• antlr-2.7.7.jar
• aopalliance.jar
• bonecp-0.7.1.RELEASE.jar
• commons-lang3-3.1.jar
• commons-logging-1.1.3.jar
• dom4j-1.6.1.jar
• guava-14.0.1.jar
• hibernate-commons-annotations-4.0.2.Final.jar
• hibernate-core-4.2.4.Final.jar
• hibernate-jpa-2.0-api-1.0.1.Final.jar
• jackson-annotations-2.2.0.jar
• jackson-core-2.2.0.jar
• jackson-databind-2.2.0.jar
• javassist-3.15.0-GA.jar
• jboss-logging-3.1.0.GA.jar
• jboss-transaction-api_1.1_spec-1.0.1.Final.jar
• mysql-connector-java-5.1.26-bin.jar
• slf4j-api-1.7.5.jar
• slf4j-simple-1.7.5.jar
• spring-aop-3.2.3.RELEASE.jar
• spring-beans-3.2.3.RELEASE.jar
• spring-context-3.2.3.RELEASE.jar
• spring-core-3.2.3.RELEASE.jar
• spring-expression-3.2.3.RELEASE.jar
• spring-jdbc-3.2.3.RELEASE.jar
• spring-orm-3.2.3.RELEASE.jar
• spring-tx-3.2.3.RELEASE.jar
• spring-web-3.2.3.RELEASE.jar
• spring-webmvc-3.2.3.RELEASE.jar

We'll continue to configure the web app by specifying the web.xml:

  
    contextConfigLocation    /WEB-INF/*ctx.xml  

  
    org.springframework.web.context.ContextLoaderListener
  

  
    api
    org.springframework.web.servlet.DispatcherServlet
    1
  

  
    api
    /api/*
  

  
    /errors/error.jsp
  

Next is to create the application context XML. We'll only have 3 bean definitions in the app-ctx:

 
  
  
  
  
  
  
  
  
  
  
  
  
 

 
  

  
    
      org.hibernate.dialect.MySQL5Dialect
      true
    
  
  
 
 
 
  
 
 

Now for the api-servlet.xml. No bean definition in this file as everything is annotation based.

We also have a single JSP to handle errors in errors/error.jsp. This is as bare as it gets:

<%@ page language="java" contentType="text/html; charset=UTF-8"
    pageEncoding="UTF-8"%>








<%
out.println(request.getAttribute("javax.servlet.error.message"));
out.println(request.getAttribute("javax.servlet.error.status_code"));

%>

This concludes basic setup for the webapp. Next part will include Java classes for the controller and error handling.

Two things to get Kohana up and running

Here are 2 things to do right after meeting all requirements dictated by Kohana's install script:

1. 
   Set Cookie::$salt value in application/bootstrap.php:
   
   

   Cookie::$salt = 'my*salt*value*';
   

2. 
   Change base_url value in application/bootstrap.php to match the server configuration:
   
   

   Kohana::init(array(
    'base_url'   => '/my_app/',
   ));
   

   
   In this example, you should be calling http://myserver.com/my_app/
   
   

Java Message Format Using Named Placeholder

The Java MessageFormat class allows user to pre-define a string with placeholders and then fill the placeholders with actual strings later to construct a proper message.

It's all fine if you're used to numbered placeholders e.g. {0} and {1}. Since I'm used to Drupal's format_string() function, here's a better alternative. Apache Commons has a StrSubstitutor class which allows use of named placeholders. Instead of using:

String template = "Welcome {0}!  Your last login was {1}";
String output = MessageFormat.format(template1, new Object[] { "gabe", new Date().toString() });

You can now do:

String template = "Welcome ${username}!  Your last login was ${lastlogin}";

Map data = new HashMap();
data.put("username", "gabe");
data.put("lastlogin", new Date().toString());
  
String output2 = StrSubstitutor.replace(template2, data);

Although StrSubstitutor is a bit more verbose, but it helps when you're handling lots of key/value pairs.