Run MySQL Cluster Multi Masters For High Availability

Hello Folks, it’s has been awhile I didn’t write new articles,

it’s has been awhile I didn’t write new articles, so time to give back to the community, I will describe how to implement MySQL cluster for high-availability and disturbed workload

MySQL Cluster Architect comes with new process  ndbd and ndb_mgmd

ndb is used to handle all data and table using the NDB Cluster Engine

ndbmtd multi threaded data  handler in NDB Cluster Engine

ndb_mgmd is the Cluster Management Server Daemon responsible for distribute the configuration and log around the cluster

in this setup we will use 4 server to distribute MySQL service cluster process

Group Beta

  •  ndbd
  • ndbd

Group Alpha

  • mysqld & ndb_mgmd
  • mysql & ndb_mgmd

our applications will communicate to load balance that distribute the workload  to Alpha Group

to begin setup we have to download mysql cluster manager package from Oracle website feel free to create your account

pickup your platform and download the package for i use the Cluster+Generic Linux x86 (64bit) version

download and extract the package  in all nodes

inside the mcm1.4.3 folder  is a bin folder  has 2 files mcm and mcmd the client and the demon for the cluster manager

we need to run mcmd in all nodes so they can communicate to each others

lets setup out cluster first we need create site ( all nodes we need to group it in a site )

now lets run the client add the site  MySQL Cluster Manager Interface

mcm client


second we need to load the cluster package in the site we created


now lets define the roles for the node who plays what  & plays data role &  mysqld & cluster management

now run the cluster and check the services status

confirm the roles for each host upon your design

now we want to communicate with out lovely cluster

as we built 2 servers with mysqld they are up and running under node id 51,52

by default mysql cluster will not sync the mysql.user table as it running in MyISM and for that we need to enable a mysql routine that sync the users over the cluster

to do that first login to mysqld nodes as a root  and import distribute mysql privileges routine .sql

now create your remote account and it will be synced over the cluster

Now feel free to scale up alpha or beta to any number you want also you can create nodes mixed of alpha and beta.

and keep your eyes on the nodes

to add a new node to our cluster we load the mcm package in the node and the demon

mcm> add package -b /home/ec2-user/mcm/cluster -h 7_6_8;


as it clear our new node is in the sites but didn’t joined the cluster yet because he has no rule to add it  and after that we need to start the role inside the cluster so nodes can collaborate and sync

now adding extra node and load more database data store on

checking our cluster status

to run added processes  on target cluster we use start process -a mycluster

now everything should be steady for our cluster

now lets change all ndbd to multi threaded version

all our data engine now running multi threaded version


Happy Hits 😀

Linux Performance Co-Pilot with WebUI

Performance Co-Pilot allow sysadmins to collect and measure data from various systems , it comes in RPM packages for Red Hat 6 to 7


installing Performance CO-Pilot

pcp  packages comes with different services pmcd,pmlogger 

pmcd  : performance metrics collector daemon

pcp packages come with many commands to gather information about the machine like pmatop, pmstat, pminfo,  pmval

pmatop one of my favorite tool as it give you a big picture

it shows information about disk , memory, cpu , network , process , swap , lvm


Screen Shot 2016-07-23 at 1.23.04 PM

pmstat   it show loadavg, memory ,  swap , io , system , cpu come with -s ( sample counter [how many times it should collect this data ] ) and -t for time interval

pminfo command to list all available metrics

lets use metrics called to see how many bytes we receive in our interface

let’s start pmlogger

the pmlogger service will save the log archievs into /var/log/pcp/pmlogger/[hostname]/date-day

we can use pmval with parameter -a to tell it to use this archive and set the matrix

we can assign specified start and end time for pmval

-S start time , -T end time

with  ISO-formatted date example -S  ‘@ Wed Feb 25 05:01:00 2016’ -T ‘@ Wed Feb 25 06:01:00 2016’

this will query 2 hours from 5am Feb 25 2016 TO 7am Feb 25 2016 with the metrics  you like

not the fun part pcp offer a web real time monitor in different flavours and styles one of my best is called vector.

first we need to install and run the pcp web service

check which port this service use

access to localhost:44323/vector

it shows nice metrics of disk IOPS , Throughputs ,Network Packets and more


happy debugging folks

IPtables PREROUTING, POSTROUTING for mixed interfaces via DNAT & SNAT

hello world,

let’s hit the point directly

1 – we have traffic coming from Source IP to our box and we need to Route it to another destination ( traffic forwarding )

2- we have traffic coming from Source IP to our box and we need to Route it to another destination ( traffic forwarding )  through specified interface Continue reading IPtables PREROUTING, POSTROUTING for mixed interfaces via DNAT & SNAT

phpMyAdmin no password dev env

if you setup your development environment with no mysql root password

after you setup phpmyadmin package it will land you with this error

Login without a password is forbidden by configuration (see AllowNoPassword)

to fix this error  vi  /etc/phpmyadmin/

find line : 96 ”    /* $cfg[‘Servers’][$i][‘AllowNoPassword’] = TRUE;”

remove the comment /*

$cfg[‘Servers’][$i][‘AllowNoPassword’] = TRUE;

now you will be able to login without password

Fix Mcrypt WARNING Ubuntu Server

today i faced problem in setup a mcrypt module

i did install it via

when i did try to enable it via

i got this error

root@ubuntu:/etc/php5/apache2/conf.d# php5enmod mcrypt
WARNING: Not enabling the mcrypt module for apache2 SAPI since module symlink
WARNING: already exists in /etc/php5/apache2/conf.d with different content.
WARNING: Not enabling the mcrypt module for cli SAPI since module symlink
WARNING: already exists in /etc/php5/cli/conf.d with different content.

to solve this unlink the 20-mcrypt.ini inside this folders


Docker Containers Crash Course

Docker Crash Course

As Hypervisor is slow to boot and use a lot of resources and need full installation

the Container Technology not that old we used to use LXC – openVZ extra

but what a cool about Docker is it really lightweight with awesome images build  and we can ship many services in one machine

it come into two parts [DockerClient,  DockerServer]

and today i will write the best quick intro i could tell

1 – Introduction [how it will work]

docker run the process inside container and when it done it EXIT the container ( by EXIT i mean it STOP the container )

docker has a official images [distros] called  saved in ( registry ) there is public registry also you can have a private registry, example

Continue reading Docker Containers Crash Course

audit keystrokes with pam

The pam_tty_audit PAM module is used to enable or disable TTY auditing. By default, the kernel does
not audit input on any TTY

this modules is part of auditd and it takes  3 parameters

1 – disable  is pattern to disable the module in specified users u can use =* to disable it globaly

2 – enable  is pattern to enable for specified users

3 – open_only to monitor fork apps

lets assume we want to monitor keystrokes via incoming ssh connection

this is helpful because some bad users could remove their history or use screen command

we will use the pam_tty_audit inside the ssh pam file /etc/pam.d/sshd


add this the module to end of the file



to view users log

here is output example

tty report provides u with the userid example 0 ? 18 bash user id 0 for root

500 ? 28 bash user id 500 for tester account

Protect Boot & Single user mode

as a physical security is main factor in our security prospective

we all need to to protect unauthorised access to our linux box after we protect bios

and we all know that anyone can rest the root password via accessing the single mode

so we have 3 ways 1st thing to disable single use mode entirely  2nd adding a password 3rd encrypt the disk with luks

single use mode configuration located under /etc/sysconfig/init

the last line of the init configuration instruct the user shell for single user mode

sushell  this shell allow access with full root privilege  we can change the shell type to control the single user mode

if we sit it /sbin/nologin no single user mode will be activated on the boot and the machine will continue booting to default run level 😉

we can set it to sulogin to make boot asks for root password before it continue to give a full root access


we can add more password layer for grub configuration via adding password –encrypt HASH from grub-crypt command

one important thing attacker can manipulate boot start services by pressing (i) in the boot sequence

attacker can disable any running service  example i disabled iptables in the boot 😀

Screen Shot 2015-08-17 at 3.32.33 AM

we can protect from this disaster by disable hot keys in /etc/sysconfig/init

protect console from reboot via ctrl-alt-delete

attacker can press ctrl-alt-delete to your machine to make it reboot

to disable it we need to change behaviour of this intercept in /etc/init/control-alt-delete.conf

by add comment to the exec line to disable reboot

SSH Tunnelling

the most famous method is using D parameter in ssh connection to bind a port local in your machine and this port tunnel back to our remotebox
to send our traffic to this server


then you can configure your application and browser to use the your local ip with the port 1337 to send traffic to the remote server

this is the traditional tunnelling way

lets make bigger scenario

lets assume that we have access to box with 2 interfaces
first interface with public ip and second one with internal private lan

the public ip 41.x.x.x
the private lan ip

inside the private lan machine with ip and running ssh service and we want to connect to this machine
its impossible to connect to it from outside without tunnelling

lets do some tunnel magic

from our box to the remote box we will do ssh
OURBOX ==SSH==> 41.x.x.x
inside the remote box we will tunnel back to our machine

this will open port 1337 in the OURBOX this port redirect to machine in port 22


this ssh connection will lead u to the

sometimes you may need to skip ssh host verification as you connect to your local machine via this ssh option parameters UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no

also this method could be use to bind to your internal ip to send ssh server back to better administration with vim also it possible to forward X via this tunnelling method


example scenario

our client don’t have public ip and writing commands in teamviewer is very silly thing

so we ask our client to connect back to our machine

after client log in inside our machine we can connect to our client ssh via


Happy Tunnelling


Secure/Lock accounts with PAM tally2

pam_tally2 is a PAM module to allow interaction in users interface on numbers of failed login attempt it can can reset count on success, can deny access if too many attempts fail.

this module is unique because it  not just reflect remote connection but also reflect the ttys and any system login method as it use PAM

example from tty:


some parameters

  1. deny used to block access of numbers of failed attempts
  2. unlock_time used to to set a time duration for blocked access in seconds
  3. even_deny_root root is excluded by default, you set this parameter to tell tally2 count for root too
  4. root_unlock_time same as unlock_time but  for root only


example PAM config:


to reflect the tty access we have to configure our tally2 module in /etc/pam.d/system-auth


here is our final layout for system auth

to reflect the  remote connections  that use password example sshd

we config our /etc/pam.d/password-auth with tally


notice that we have done 2 things  one in auth interface that verify the account and 2nd one in the account interface to reflect the permissions of the account


here is the  some output of /var/log/secure


as you see tally2 kills the connection 🙂

for manual interaction with tally2 counter

there is a command called pam_tally2

to remove a counter failures