Tag Archives: CentOS

OpenSSL on CentOS

How to install latest version of OpenSSL on CentOS?

Hi there, today I would like to show you how to install latest version of OpenSSL (1.1.0g) on CentOS 7

Do I need latest version of OpenSSL?

In general - you don't. Default version is doing great job and it's secure. I needed it for compiling Apache HTTP with HTTP/2 support. Default version of OpenSSL installed on CentOS (1.0.1e) does not support it yet, hence the need of using latest version instead.

If you need it for any other reason, this tutorial is for you:)

How to check current version of OpenSSL?

In order to check current version of installed package you need to execute following command:

openssl version

It will print out version of installed package like OpenSSL 1.0.1e-fips 11 Feb 2013

How to install latest version of OpenSSL?

I compile OpenSSL from source code. In order to compile it successfully you need to install some tools that will help you compile it:

sudo yum install libtool perl-core zlib-devel -y

It will install compiler and few other libraries that are required to compile OpenSSL.

Next download latest version of OpenSSL source code. I like to use releases page on GitHub. I choose the version without FIPS simply because I don't need compatibility with it. And I think that it's a bit more secure to have OpenSSL without FIPS, as fixes are usually included much faster in regular version than in FIPS version. If you want to read more about it, use this link.

In order to download source code, use following command:

curl -O -L https://github.com/openssl/openssl/archive/OpenSSL_1_1_0g.tar.gz

Source code comes in compressed package. In order to decompress it use following command:

tar -zxvf OpenSSL_1_1_0g.tar.gz
cd openssl-OpenSSL_1_1_0g

Now it's time to configure and compile OpenSSL:

./config --prefix=/usr/local/openssl --openssldir=/usr/local/openssl shared zlib
make
make test

prefix and openssldir sets the output paths for OpenSSL. shared will force crating shared libraries and zlib means that compression will be performed by using zlib library

It is worth to run the tests to see if there are any unexpected errors. If there are any, you need to fix them before installing library.

In order to install library you need to execute:

sudo make install

I usually delete all source files to keep system clean after installation. However sources of OpenSSL are required to compile other tools such us Apache, Nginx etc., so I don't remove them.

Add new version to PATH

After the installation you will probably want to check the version of OpenSSL but it will print out old version. Why? Because it's also installed on your server. I rarely override packages installed via yum. The reason is that when there is new version of OpenSSL and you will install it via yum, it will simply override compiled version, and you will have to recompile it again.

Instead of overriding files I personally like to create new profile entry and force the system to use compiled version of OpenSSL.

In order to do that, create following file:

sudo vi /etc/profile.d/openssl.sh

and paste there following content:

# /etc/profile.d/openssl.sh
pathmunge /usr/local/openssl/bin

Save the file and reload your shell, for instance log out and log in again. Then you can check the version of your OpenSSL client. Or maybe...

Link libraries

Or maybe you will get an error with loading shared libraries? In order to fix that problem we need to create an entry in ldconfig.

Create following file:

sudo vi /etc/ld.so.conf.d/openssl-1.1.0g.conf

And paste there following contents:

# /etc/ld.so/conf.d/openssl-1.1.0g.conf
/usr/local/openssl/lib

We simply told the dynamic linker to include new libraries. After creating the file you need to reload linker by using following command:

sudo ldconfig -v

And volia! Check the version of your OpenSSL now. It should print out OpenSSL 1.1.0g 2 Nov 2017

How to disable Transparent Huge Pages on CentOS

How to disable Transparent Huge Pages on CentOS

Hi there! I'd like to show you how to disable Transparent Huge Pages on CentOS 7. First question you might ask - why do you even want to disable Transparent Huge Pages (THP)? Well, when you are using your server for database purposes and you have tools like MySQL, Oracle or MongoDB, the recommendation is to disable THP for performance purposes.

How to check if THP is enabled or not?

In order to test THP you need to check two files:

cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag

Value in square brackets [] is currently active value. If you have [never] it means that transparent huge pages are disabled. If you have [always] - THP are enabled.

How to disable THP manually?

If you want to disable THP you can do it manually by typing these two commands:

echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled
echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag

You can cat the files one more time to see if value is changed to [never]

How to disable THP automatically?

Method above works fine until you restart the server. It will turn on on system restart. In order to disable them on system startup, you need to add Unit file with script that will disable THP.

Create following file:

sudo vi /etc/systemd/system/disable-thp.service

and paste there following content:

[Unit]
Description=Disable Transparent Huge Pages (THP)

[Service]
Type=simple
ExecStart=/bin/sh -c "echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled && echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag"

[Install]
WantedBy=multi-user.target

Save the file and reload SystemD daemon:

sudo systemctl daemon-reload

Than you can start the script and enable it on boot level:

sudo systemctl start disable-thp
sudo systemctl enable disable-thp
Git on CentOS

How to install GIT on CentOS

Today I would like to show you how I install GIT on CentOS servers. It might be super easy or slightly harder. It depends if you want to have the latest version (2.15.1) or you don't care:)

Easy way to install git on CentOS with yum

The easiest way is to use yum of course. Simply type

sudo yum install git -y

and it will install git on your server. That's it 😀 However if you will check your git version:

git --version

you will note that it will install 1.8.3.1 version (or higher). But wait! Isn't there git v2.0? Yes, there is. And I will show you how to install it.

How to compile git from source on CentOS?

I show you two approaches. The only difference is with using OpenSSL version. By default CentOS is shipped with older version. I like to have most recent version so I compile OpenSSL from source. It is not mandatory, since you can use default version.  If you prefer to upgrade it  follow this tutorial first and get back here once you will have latest version. If not just follow along.

There are 5 mandatory and 2 optional steps to have latest Git version on CentOS.

First you need to install some tools that are required to compile git. You will need some libraries and compiler. Install them by executing following command:

sudo yum install autoconf libcurl-devel expat-devel gcc gettext-devel kernel-headers openssl-devel perl-devel zlib-devel -y

Here is the fun part. Even if you don't have git installed, after executing command above you will actually have git on your server. It is because one of the packages has git as a dependency. But you shouldn't worry about that too much now.

If you didn't compile latest OpenSSL like I did you must install one more package:

sudo yum install openssl-devel -y

Step two is to download git source code. The easiest option is to download it from GitHub. Visit git release page - https://github.com/git/git/releases and pick desired version. I'm usually using latest, non-rc version. In my case it's 2.15.1. Right click on tar.gz icon and copy the link.

Now execute following command on your server (link you see is copied from GitHub):

curl -O -L https://github.com/git/git/archive/v2.15.1.tar.gz

It will download tarball with git source files to your server. -O flag will save the file instead of redirecting curl output to stdout. -L will follow redirects if any (usually there is one when using GitHub releases).

Step three is obviously unpacking downloaded source code:

tar -zxvf v2.15.1.tar.gz

Now is the fun part - compiling. Very important is to know, that you should never ever use root account for compiling. This might lead to serious security issues. Imagine that you will compile code that contains malicious parts as root. This code can modify your server and cut off your root access. Google about it, there were such cases in the past. So if you use root account, think about creating separate account to manage your server. Read this tutorial to learn how to setup additional account with root privileges. I'm not saying that you can't compile it as root, because obviously you can. But you just shouldn't do that.

Execute following command as non-root account:

cd git-v2.15.1
make clean
make configure

Depending on which OpenSSL you are using...

If you compiled latest OpenSSL use following command:

./configure CFLAGS='-I/usr/local/openssl/include' LDFLAGS='-L/usr/local/openssl/lib' --prefix=/usr/local/git --with-openssl=/usr/local/openssl

If you are using regular version:

./configure --prefix=/usr/local/git

Note parameter --prefix in configure command above. It will prevent overriding Git installed via yum. It will also help with overriding compiled version with packages installed from yum.  Once package is configure run:

make

For installing you need to have root account or be in sudoers group. So last step is installation:

sudo make install

Now you have latest version of git in your system!

Optionally you can remove downloaded package and uncompressed directory. To cleanup execute:

cd ..
rm -rf git-v2.15.1 v2.15.1.tar.gz

Adding GIT to $PATH

Here is the important part of this tutorial. To force using GIT that was just installed we need to add it to $PATH. It will prevent running git installed with yum.

Create file git.sh in this location:

sudo vi /etc/profile.d/git.sh

and paste there following contents:

pathmunge /usr/local/git/bin

After exiting and logging again to your SSH session you will always call GIT compiled from source:)

How to update GIT to newer version?

When new version of GIT will be released, simply repeat the steps in this tutorial, but with newer version. You don't need to remove GIT that is currently installed. New build will override the one that you already have.

Drawbacks of compiling git from source

You must know that there are certain drawbacks of compiling git from source. There are not really painful to be honest.

  1. You must manually check if there is new version of git. When you try to do yum update, you won't see it on the list
  2. Theoretically you will have two GITs installed - one via yum (old version), and latest version compiled from source. In practice, you will only use the one that was compiled.

If you don't want to install git manually you can use our Lamp On Steroids project based on Ansible. We have git role that automates the whole process.

How to configure NTP and timezone on CentOS

How to configure NTP on CentOS 7

This will be pretty short tutorial about NTP. I will show you how to install and configure NTP on your server.

What is NTP?

NTP stands for Network Time Protocol and it keeps the server synchronised with correct time. Moreover you can synchronise all servers to one of your server to make sure that all have the same time and there are no time-shifts between them. It's nice to have such tool installed on your server.

How to set system timezone on CentOS?

First thing before that you need to do before installing and configuring NTP server is to set correct timezone. By using timezonectl command you can get information about your current timezone:

timedatectl | grep "Time zone"

Now, you need to decide which timezone you want to use. I'm strongly recommend using UTC as a timezone. Some servers are configured to particular timezone like Pacific/Honolulu for instance. But you should use UTC. If you are curious why, I highly recommend reading this article about UTC.

It will have some downsides, especially when you will have to translate output of date command to your local timezone. Most probably, there will be a time-shift.

In order to set timezone as UTC execute following command:

sudo timedatectl set-timezone UTC

However if you want to stick to your local timezone you can use replace UTC in command above to one of the options from this command:

timedatectl set-timezone UTC

For instance:

sudo timedatectl set-timezone Europe/Kiev

Now once you have your correct timezone set, you can proceed to setting up NTP server

How to install and configure Chrony as NTP server?

There are couple of options when you want to choose tool for NTP server. Two most popular are:

  • NTP
  • Chrony

In this tutorial I will use Chrony. It comes out of the box with CentOS and it's a bit faster than NTP. There is nice comparison of these two tools here.

Before you will install chrony make sure that you don't have ntp installed. Having two servers on one machine can cause some issues.

Execute following command to remove ntp and ntpdate packages:

sudo yum remove ntp ntpdate -y

Once you are sure that you don't have ntp installed, make sure that chrony is installed. It should be installed by default, but it is nice to check it:

sudo yum install chrony -y

Now you need to configure pooling from external NTP server. I highly recommend commonly used pool.ntp.org for that purpose.

In order to use the servers for pooling edit chrony configuration file:

sudo vi /etc/chrony.conf

At very top of the file there should be lines that starts with server. Remove them all and use the one from pool.ntp.org. So the beginning of your file after changes should look like that:

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server 0.pool.ntp.org iburst
server 1.pool.ntp.org iburst
server 2.pool.ntp.org iburst
server 3.pool.ntp.org iburst

# Ignore stratum in source selection.
stratumweight 0
...

Save the changes. Now it's time to start NTP service and enable it on boot, so it will start automatically after system reboot:

sudo systemctl start chronyd.service
sudo systemctl enable chronyd.service

That's it! You configured NTP server!

You can use following commands here to check synchronisation status:

chronyc tracking
chronyc sources -v

You can use our Ansible LAMP on Steroids project to make configuration of your server easier!

It is based on Ansible. If you don't know what Ansible is, check our tutorial first.

Clone our repository and setup your server faster with LAMP on steroids.

Yum - package manager

Yum package manager – tips and tricks

Yum is a package manager on CentOS. You will probably use it pretty often to install some software. In this article I will show you how to configure it, how to speed up packages installation and how to use something more than just yum install package-name.

Create list of installed packages

One of the first thing that I do on new server is making a list of installed packages. When you have fresh system everything works out of the box. But after some time you installed so many packages etc. and you might wonder - was this package installed at the beginning? Can I safely remove it? Such list comes handy then:)

sudo yum list installed > /etc/yum/initial-packages-list

Enable GPG check before you will install anything

GPG checks are really important. They tell yum to verify signature of downloaded package. It lower the chance of installing suspicious packages from weird sources. Usually they are enabled, but it's worth to check that.

You need to look for following line:

gpgcheck=1

in few files. The most important file is /etc/yum.conf Make sure that line above is present in [main] section. If it's set to 0 or line is missing, add it. If there are any other sections than [main], check them too.

In addition to that you need to check all .repo files in /etc/yum.repos.d directory. Most of them will have gpgcheck enabled. If they have gpgcheck=0 you can either remove the line, or change it to gpgcheck=1. If you will remove the line, default from /etc/yum.conf will be used.

Install packages with yum and speed it up with delta RPM

If you want to install additional package you need to execute following command:

sudo yum install PACKAGE_NAME -y

You can install multiple packages at once by separating them with space. -y modifier will skip prompt with confirmation.

You can speed up packages installation with Delta RPMs. Delta RPM contains the difference between old and new package. It means that you don't have to download entire package, just the difference between the versions. It results in smaller size of downloaded file and generally makes installing packages faster. It doesn't work for all packages, but it's nice to have it installed anyway:

sudo yum install deltarpm -y

That's it, it will be enabled from now on, you don't need to do anything else!

Remove unnecessary packages with yum

There is an unwritten rule - the less packages the better. It's important from security and performance side. If you have less stuff installed you will be able to keep everything updated. You won't have to fix new security issue in package you don't need etc.

There are couple of packages that can be safely removed from servers with CentOS.

To remove the package use command:

sudo yum remove PACKAGE_NAME_HERE -y

Use -y argument with caution, it will immediately remove package from your system without the need for confirmation from you side:)

Command here will remove unnecessary packages from CentOS. If you plan to use MariaDB instead of MySQL, don't remove mariadb-libs!

It's also good to use yum info command to check information about given package if you are in doubt that you need to delete it.

Here is the list of packages that I usually delete and system is still working fine (CentOS 7.4)

sudo yum remove btrfs-progs gssproxy hyperv-daemons-license libgudev1 mariadb-libs NetworkManager NetworkManager-libnm NetworkManager-tui NetworkManager-wifi postfix -y

Get information about given package

If you are not sure about given package, you can always get information about it by executing following command:

yum info PACKAGE_NAME_HERE

Update core packages with yum

There are some packages that can't be (easily) removed from CentOS. They are usually safe to update and it's worth to keep the up to date.

In order to update given package you need to execute following command:

sudo yum update PACKAGE_NAME -y

You can also update all packages by executing following command:

sudo yum update -y

But I prefer to update packages in few steps, so I can control everything.

Update all core packages with following command. These are the packages that comes with CentOS and must be installed for proper system functioning. You can safely update them:

sudo yum update bash binutils ca-certificates centos-release chkconfig coreutils cpio cryptsetup-libs cyrus-sasl-lib curl dbus dbus-libs dracut elfutils-libelf expat filesystem gawk glib2 glibc glibc-common gmp gnupg2 grep gzip kmod kmod-libs kpartx krb5-libs libblkid libcap libdb libffi libgcc libgcrypt libmount libuuid libsemanage libstdc++ libssh2 libtasn1 libxml2 lua ncurses nspr nss nss-softokn nss-softokn-freebl nss-sysinit nss-tools nss-util openldap openssl-libs pam pcre pinentry procps-ng python python-libs python-pycurl readline rpm setup shadow-utils shared-mime-info systemd systemd-libs systemd-sysv util-linux tar tzdata xz yum yum-plugin-fastestmirror zlib -y

Update other packages with yum

Apart from core packages there are some tools added to CentOS minimal installation. They can be removed but it's good to keep them installed as they add neat features to OS. Let's update them too:

sudo yum update audit authconfig bind-libs-lite bind-license biosdevname cronie cronie-anacron crontabs device-mapper-persistent-data dhclient dmidecode dnsmasq dracut-network dracut-config-rescue e2fsprogs epel-release ethtool file fipscheck freetype gettext gnutls gobject-introspection grub2 grubby hwdata iproute iprutils iputils initscripts irqbalance kbd kernel kernel-tools kernel-tools-libs kexec-tools libcroco libdrm libgomp libnetfilter_conntrack libpciaccess lsscsi lvm2 make microcode_ctl mozjs17 nettle openssh openssh-clients openssh-server openssl os-prober parted pciutils-libs plymouth plymouth-scripts policycoreutils polkit python-gobject-base python-perf python-pyudev rdma-core rsync rsyslog selinux-policy selinux-policy-targeted sudo tuned vim-minimal virt-what xfsprogs yum-utils -y

Remove not used packages with yum

Yum comes with nice feature - it can removes packages that are not used on the system. You can remove them by executing

sudo yum autoremove -y

Block certain package updates

Sometimes there is an need to block certain packages from being updated or installed. You can block them in two ways.

First is to --exclude option that you can use on any yum command. For instance:

sudo yum update --exclude="git*"

* means that anything that will be after git will also be excluded. Command above will update all packages, but git won't be updated or installed.

Second option is to add same rule to /etc/yum.conf file under [main] section. It will affect all yum commands and you will not have to use --exclude.

Take a look at the example below:

[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=0
exclude=git*
...

One thing that you should remember is that it might also affect other packages. For instance if you try to install/update package that has git as a dependency, you won't be able to do that.

Check if there are any other packages to update?

After I remove unwanted stuff, updates core and other packages and making autoremove I check if there are any other packages marked for update.

You can check it with

yum list updates

If there are any other packages left for update, you should get info about them and update them or remove them from your system.

Power yum with cron

If you have slower connection on your server, sometimes making yum update takes a lot of time. You can speed it up with yum-cron. It adds a cronjob that periodically checks for packages to update. It can also download them and install them automatically.

First, lets install yum-cron:

sudo yum install yum-cron -y

Before running it I strongly suggest to remove automatic updates. It's OK to check for updates and download them, but you should NEVER install them automatically. They can really screw up your server (been there, done that (unfortunately...)).

If packages are broken or have configuration changes they will break your server anyway, but it's way better to fix it right away, than break it with auto update at 2 AM.

So make sure that auto installing is disabled. Edit /etc/yum/yum-cron.conf and make sure that apply-updates option is set to no. If it is, you can safely start yum-cron with

sudo systemctl start yum-cron.service

You should also add it to system startup, so when you reboot your machine, it will start automatically:

sudo systemctl enable yum-cron.service

Searching for packages

Sometimes you need to install package name, but you are not sure about the name. You can easily search available packages with yum search command:

yum search PACKAGE_NAME

Additional tools

There are some nice packages that are not installed by default on minimal installation, like net-tools (provides netstat command). You can install it on your system, it is pretty handy:)

sudo yum install net-tools -y

Easier way?

You can use our Ansible LAMP on Steroids project to make configuration of your server easier!

If you don't know what Ansible is, check our tutorial first.

Clone our repository and setup your server faster with LAMP on steroids.

Ansible templates and handlers

Ansible tutorial – part 3

In this part we will use templates and handlers in Ansible.

A short word about Ansible tutorial

This is multipart series about Ansible, great tool for provisioning and configuration management. You will learn how to create useful roles and use Ansible in efficient way. You will also pick up some good practices:)

If you like tl;dr take a look at shorter veresion of this tutorial on GitHub: https://github.com/blacksaildivision/ansible-tutorial
There are also some examples!

Part 1: Intro to Ansible and Inventory files
Part 2: Playbooks and tasks
Part 3: Templates and handlers 
Part 4: Variables

 

Ansible templates

Templates are simple text files that we can use in Ansible. Most of the time you will use them to replace configuration files or place some documents on the server.

Start from creating new directory inside our nginx role and name it templates . It should be on the same level as tasks directory. This it the place where we will store our templates.

Let's say that we want to change the configuration of nginx. We will use simplest way and we will just replace whole nginx.conf file. Inside template directory create a file and name it for instance nginx.conf.j2  J2 is extension of Jinja2 templating language that Ansible is using.

Copy+paste following content into that file:

# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _;
        root         /usr/share/nginx/html;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }
}

We just logged in into the server and copied the contents of /etc/nginx/nginx.conf file. This is most common way to obtain configs for template files. Copy their contents from server after installation of the given tool. We can be sure that this is valid config and it should work out of the box.

Task for template

We need the task that will replace current file on the server with the template.

Just before the task that checks if nginx is working add another task:

- name: copy nginx configuration file
  template:
    src: nginx.conf.j2
    dest: /etc/nginx/nginx.conf
    owner: root
    group: root
    mode: u=rw,g=r,o=r
  tags: [nginx, config]

We need to name the task first. After that we are using template module from Ansible. It takes couple of parameters. Two mandatory and couple of optional parameters.

  • src  is required parameter. You need to specify name of the template file from templates directory. In our case it's the file we just created - nginx.conf.j2
  • dest  is another required parameter. It's the path where the file should be placed on the server. Nginx in default installation keeps it config files in /etc/nginx directory. It's important to use filename as well, not only directory!

Rest of the parameters are optional, but when you are using template module, good practice is to specify the ownership and file permissions.

  • owner  is username that should own the file, like in chown command
  • group  is the group that should own the file, like in chown command
  • mode  is set of permissions for the file, like from chmod command. There are couple of ways to set the permissions. We prefer the syntax like we used in the task above. But you can also specify numeric version like that:
mode: 0644

Handlers

As you might now, changing something in configuration file won't give any results without restarting nginx. One way to solve that problem, would be to edit service task in our role and set the state to restarted  instead of started . It would result in restarting nginx. But if you would run this playbook over and over again, on each run it would restart nginx, which should not happen.

It would be the best if we could restart nginx automatically, but only when there are changes in configuration file.

Handlers to the rescue! Ansible comes with awesome feature called handlers. You can execute handler, only when given task will have status changed.

Edit the template task above and add notifier:

- name: copy nginx configuration file
  template:
    src: nginx.conf.j2
    dest: /etc/nginx/nginx.conf
    owner: root
    group: root
    mode: u=rw,g=r,o=r
  notify: restart nginx
  tags: [nginx, config]

You can add notifier to any task. You must specify the name of the notifier. There are almost no restrictions, it's just a string.

Next, we need to add yet another directory to our role and call it handlers  (directory should be on the same level as tasks and templates).

Inside that directory create text file and name it main.yml  (All main.yml files are automatically included by Ansible).

Place following content in handlers/main.yml  file, you just created:

- name: restart nginx
  service:
    name: nginx
    state: restarted

Each handler should have a name. You need to pass the name to notify  in task like we did in example above. Next you can pick any module you want from Ansible and use it. Most of the time it'll be service module. Here we can use restarted state.

When something will change in the template, task will result in changed status. Handler will be notified, will execute and it will restart nginx.

Run your playbook couple of times, and make sure that you will have everything to OK in recap. Change the template and add epoll support for nginx in templates/nginx.conf.j2  file:

events {
    worker_connections 1024;
    use epoll;
}

Run playbook. You will notice that the handler will execute and will restart nginx.

Ansible tutorial – part 2

This is part 2 of Ansible tutorial. In this part you will learn how to create playbooks, what are roles and tasks and how to use privileges escalation in Ansible.

A short word about Ansible tutorial

This is multipart series about Ansible, great tool for provisioning and configuration management. You will learn how to create useful roles and use Ansible in efficient way. You will also pick up some good practices:)

If you like tl;dr take a look at shorter veresion of this tutorial on GitHub: https://github.com/blacksaildivision/ansible-tutorial
There are also some examples!

Part 1: Intro to Ansible and Inventory files
Part 2: Playbooks and tasks
Part 3: Templates and handlers 
Part 4: Variables

 

Playbooks

Playbooks are essential feature of Ansible. In Playbooks we specify what and how our server will be configured and provisioned.

Go to our directory that we created in first part -  vagrant_ansible_tutorial . Create a file inside and name it playbook.yml . Name is irrelevant here, you can use anything you want. Open this file in editor and paste following content:

- hosts: ansible_tutorial
  tasks:
   - debug:
      msg: Test of Ansible Playbook!

First line contains host(s) that we want to provision. You must specify group or single host from Inventory file  hosts  that we created in part 1. Good practice is to specify group instead of single host. In our case it is  ansible_tutorial , so we will provision our Vagrant box.

You can also specify multiple hosts here and set different tasks or roles:

- hosts: webservers
  tasks:
   - debug:
      msg: Installing nginx
   - debug:
      msg: Installing PHP

- hosts: databases
  tasks:
   - debug:
      msg: Installing MongoDb

 

Next, we need to specify tasks . Task in Ansible is like single command in bash on steroids. With tasks you can install software, add users, remove files etc. Our task will just print out the message in command line, so no harm will be done if it won't work as expected.

We will get more into the tasks later on. For now just use the playbook above.

If you are familiar with YAML or Ansible you might notice that there are three dashes ---  missing. If you will browser through the example playbooks on GitHub, you might note that all files starts with --- . This is coming from YAML and it is optional in Ansible. You might or might now use it, it doesn't make any difference. We never use it on our side and we won't use it in this tutorial as well. But if you prefer, you can add ---  to start of each file like this:

---
- hosts: ansible_tutorial

Go to your terminal and execute following command:

ansible-playbook -i hosts playbook.yml

 We need to pass two arguments. First one is an inventory file. Second argument is playbook file that we want to run.

Your output should look similar to:

PLAY [ansible_tutorial] ********************************************************

TASK [setup] *******************************************************************
ok: [192.168.60.70]

TASK [debug] *******************************************************************
ok: [192.168.60.70] => {
    "msg": "Test of Ansible Playbook!"
}

PLAY RECAP *********************************************************************
192.168.60.70              : ok=2    changed=0    unreachable=0    failed=0

You might note that there is also task named setup . This is Ansible internal task for gathering information about the server. We will use them later on. For now, just know, that such additional task exists.

Next lines contains our debug task. It got status ok  and it was executed correctly.

Probably the most important part of this output is recap . It contains information about current run of playbook. There are 4 metrics that are worth checking:

  • ok  - tasks that were executed correctly or didn't change anything on provisioned server.
  • changed  - things that were modified, like you just add new user. If you run the same playbook twice it will have ok status.
  • unreachable - usually should not happen, you will get it when host becomes unreachable, connection was dropped or you are out of internet etc
  • failed  - when tasks were unable to execute correctly

In the end, perfect playbook should not contain any failed and unreachable errors. On first run you should get only ok and changed tasks. Second execution of the same playbook should result only in ok statuses.

Roles in Ansible

Role, apart from Playbook, is another essential feature of Ansible. In previous example we created tasks directly in playbook file. You must know that this is wrong approach. 99% of the time you will use roles instead of using tasks directly in playbooks. We used it only to show you how to execute working playbook.

Think of how do you use a server. For sure you need to install some kind of web server like apache or nginx. There will be couple of things to do (tasks) in order to install nginx. We can group them into single role. Roles will help you organise your structure.

Role should have single responsibility like installing and configuring nginx. It should also be reusable. So you can for instance use it in different group of servers or even use it in separate project.

Let's create our first test role that will install and configure nginx. In the project directory create directory and name it roles . In this directory you should keep all roles you create.

Inside there you should create another directory and name it nginx . Name of the directory is the same as name of your role. Each role directory, like nginx, usually contains couple of directories inside like tasks, defaults, handlers etc. For now we will just create directory named tasks . Inside tasks directory, create file and name it main.yml  In the end your directory structure should look like this:

Screen Shot 2016-06-22 at 13.02.08

At the very beginning it might look messy, but after a while you will see that it's quite clever way of organisation.

Just for the start, we will use our debug message as simple start. Your main.yml file should look like this:

- debug:
    msg: Test of Ansible Playbook!

This is just copy+paste from playbook.yml  Now we need to include our role into the playbook itself. Edit playbook.yml  so it will look like that:

- hosts: ansible_tutorial
  roles:
   - nginx

As you can see, we just removed tasks  from there and added roles  instead. Each hosts group can contain multiple roles. You can also reuse given role in different hosts.

Now let's run our playbook again:

ansible-playbook -i hosts playbook.yml

Output will be very similar as before and the end result should be the same.

Tasks in Ansible

Let's write some some real tasks that will help us install nginx. Our OS is CentOS 7 as we specified it in Vagrantfile . In order to get nginx up we need to install epel repostitory and nginx itself. Start by removing everything from main.yml  in tasks directory. We don't need debug message there, it was only for test purposes. Instead create following task:

- name: install epel
  yum:
    name: epel-release
    state: present
  tags: [nginx]

Each task should start with name. It's not necessary but it's very informative and easier to read in ansible command output.

Next line contains module name that we will use. In CentOS we are using yum for installing files. If the base images was Ubuntu for instance, we would use apt module. Yum/Apt module needs to know two things. First is the name of the package we would like to install. In our case it's epel-release . Second thing is state - whether application should be installed or removed. We want to install it, so we should use present  state.

Last thing are tags. They are optional in each task but usually it's good to use them. When you use ansible-playbook command, you can run tasks that contains only given tag. We basically add tag(s) for each task we create. Sometimes it's not necessary, but it's just convenient to have them there.

Run your playbook with ansible-playbook command like before. If you are using the same image as we in our tutorial, you might wonder that task resulted in ok  status, instead of changed . It should be changed when we install new packages, right? Yes, it should, but the vagrant image has epel-release already installed. So Ansible is checking if it's installed, and if so, it won't do anything. Clever, isn't it?

Once we have epel repository we can install nginx. Let's add another task to our role:

- name: install epel
  yum:
    name: epel-release
    state: present
  tags: [nginx]
  
- name: install nginx
  yum:
    name: nginx
    state: present
  tags: [nginx]

As you can see, we are just installing nginx. Let's run the playbook. Whooops! You should get a failure - You need to be root to perform this command.

Privileges

In hosts file we specified that the user that should connect to the server is vagrant. But it does not have enough permissions to install things. We need to use privileges escalation.

Let's edit our playbook file and add become  lines:

- hosts: ansible_tutorial
  become: yes
  become_user: root
  roles:
   - nginx

What we just did is we enabled become so now we can allow Ansible to execute command as different user. As we need to install some packages we need root user. Ansible will login as vagrant user, but each command will be executed as it would be root account.

You might ask - why not to edit hosts file and use root as a user. Answer is simple - the secure way is to disable ssh for root account to prevent harm to your server. root should not have the option to direct login via ssh.

You can also specify these values per task. But usually most of the tasks we do require root power, so we are defining it for entire connection to given host.

Try to run the command again. It should succeed this time. You should get changed=1 in output. It means that nginx has been installed. Try to run the playbook again. It will get ok  status, so nothing was changed.

Finish installing nginx with Ansible

Last missing task is to start nginx. Add following task inside main.yml

- name: enable nginx
  service:
    name: nginx
    state: started
  tags: [nginx, status]

We are using Ansible module called service. It is responsible for managing services. In general you can find all possible modules for Ansible on this page. It's super useful and it contains lot of examples for each command. It's also well written. If you don't know something, just take a look at documentation of given module and you should get an answer to your question:)

Service module requires two things - name of the service, which in our case is nginx, and state of the service. We want to start it, so state should be set to started .

Run your playbook. After thatm open your web browser and type the IP of your server (you can find it in Vagrant file). If you are following the tutorial, simply visit this URL: http://192.168.60.70/

You should see nginx welcome page!

Tags

Now it's the time to use some tags power. Note that we added another tag named status  to our task. Imagine the situation when you just want to check if nginx is working. You don't want to execute other tasks that are installing things. We should only execute tasks that have given tag:

ansible-playbook -i hosts playbook.yml --tags="status"

It should execute only two tasks. One is default setup task, and second one is the task with status tag. If you have multiple roles, like nginx, php, mysql and you will add status tag to all tasks connected to service module, command above will look through all roles. In one command you are able to check if all services are working fine:)

You can also specify multiple tags using comma separated list:

ansible-playbook -i hosts playbook.yml --tags="status,nginx"
Varnish on CentOS - full guide

Varnish – how to make websites fly?

In this episode we will make our websites fly with Varnish on CentOS. What exactly is Varnish? Varnish is an HTTP accelerator designed for websites with high traffic. It can speed up you website significantly, especially when you have more reads than writes. It's designed to handle heavily loaded websites as well as APIs. It's used by many high-traffic websites like The Guardian, New York Times etc. In this episode we will learn how to install Varnish on CentOS.

LAMP on steroids

This is part of our series LAMP on steroids. Check the links below to learn how to setup awesome webserver!

  1. Choosing VPS
  2. Install EPEL
  3. Install and configure Apache HTTPD server
  4. Harden Apache with ModSecurity and OWASP Core Rule Set
  5. Install and configure PHP
  6. Install and configure MySQL server
  7. Configure firewall based on iptables
  8. Create developer user and setup SSH key-pair
  9. Configure SSH
  10. Install and configure Varnish to speed up websites
  11. More to come...

How to install Varnish on CentOS?

Installation is pretty straightforward:

rpm --nosignature -i https://repo.varnish-cache.org/redhat/varnish-4.0.el6.rpm
yum install varnish -y
varnishd -V

Varnish comes in two versions. Widely used version 3 and version 4 which is recommended at the moment. Version 3 is generally deprecated  since the end of April 2015, so we will install latest version of Varnish 4. After installation  varnishd -V  shows the version that is inside our system. In our case it's 4.0.3

Now we can try to run Varnish by executing following command:

sudo service varnish start

...and you should see the error - varnish failed to start. Why's that? Read next section about SElinux policy. If you don't see the error you can move forward to next part of this tutorial.

SElinux policy for Varnish on CentOS

The problem is that Varnish in version 4 requires some additional permissions in order to run under CentOS. We have SElinux enabled by default. We shouldn't not disable that (which is sometimes common practice in such situations). We will add the rule to the policy, so Varnish can start without any problems.

First of all we need the tool called audit2allow. Let's install it:

sudo yum install policycoreutils-python -y

Now, try to start Varnish one more time. It should give you fail error. After that execute following command:

cd ~/sources
mkdir varnish
cd varnish
sudo grep varnishd /var/log/audit/audit.log | audit2allow -M varnishd2
sudo semodule -i varnishd2.pp

We will use our sources directory that we created at the beginning of our series. You can of course use any directory you want.

Next we look through audit.log on CentOS and try to find varnishd occurrences. audit.log file contains blocked activities by SELinux. From these blocked activities we will generate module -M for SELinux that will grant access to required permissions.

Note 2 at the end of varnishd2  name. We need to add new policy. varnishd  policy already exists in SELinux policies.

Enable the module and try to start Varnish service again. In our case it didn't work out, and we still got fail error. So we did it again:

sudo grep varnishd /var/log/audit/audit.log | audit2allow -M varnishd3
sudo semodule -i varnishd3.pp
sudo service varnish start

and Varnish started without any errors!

Setup firewall for Varnish

So now it's the time to play around with Varnish. The installation only is not enough to be able to have Varnish fully working. Let's start by adding some firewall rules in order to be able to test Varnish on CentOS.

By default Varnish is running on port 6081 , but we don't have rules in our firewall, so you won't be able to see the actual result. Open firewall rules file and add these rules:

-A INPUT -p tcp --dport 6081 -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp --sport 6081 -m state --state ESTABLISHED -j ACCEPT
-A INPUT -p tcp --sport 6081 -m state --state ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp --dport 6081 -m state --state NEW,ESTABLISHED -j ACCEPT

It's the same rules like for instance for port 80 for Apache. For testing purposes we will enable to access our website by two different ports - 80 (by Apache) and 6081 (by Varnish). Port 80 will work without caching, port 6081 will serve cached content.  At the end we will change our setup, so only port 80 will be available and it'll be Varnish port, not Apache like for now.

Restart the firewall to apply the rules:

sudo service firewall restart

And now you can try to test it by calling http://example.com:6081

You should see the error, and that is perfectly fine.

Test with microtime()

Now it's time to test Varnish cache. The easiest way is to create PHP script with following content in /var/www/example/com/htdocs :

<?php
#/var/www/example.com/htdocs/varnishtest.php
echo microtime();

then try to run it in your browser: http://example.com/varnishtest.php and refresh couple of time. On each refresh you should see different contents.  Now it's time to make http://example.com:6081/varnishtest.php  working.

First of all we need to configure backend for Varnish. Edit file /etc/varnish/default.vcl

You should see backend default there. By default port is set to 8080, but our Apache is working on port 80, so we need to change that:

vcl 4.0;

# Default backend definition. Set this to point to your content server.
backend default {
    .host = "127.0.0.1";
    .port = "80";
}

Exit editor and reload Varnish:

sudo service varnish reload

Try to reload the page couple of times. At second refresh you should see exact same thing as before. So result from microtime()  will not change. If you achieve such effect, you can be happy! Varnish is working correctly:D

If you will see at response headers you will get additional headers. Most important is Age  header. It should have value > 0. By that you can see that Varnish is caching your website correctly. But if you still can see that it's not working (content is still changing) you should see next section of this article.

Troubleshooting Varnish Cache

There are plenty of reasons why Varnish might not cache your website. You must know that it's not easy to diagnose what's wrong with your cache/website. Here are few tips that might be helfpul

ModSecurity is running

If you have ModSecurity enabled there might be an issue with caching. Or you might see 403 pages sometimes. There's an easy fix for that. You need to disable 960020 rule. In order to do that you must edit the httpd-security.conf  file that we created during Apache setup and add line  so the file will look like that:

# File location: /usr/local/apache2/httpd/conf/extra/httpd-security.conf

#...
<IfModule security2_module>
      Include conf/crs/modsecurity_crs_10_setup.conf
      Include conf/crs/base_rules/*.conf
      # Include conf/crs/experimental_rules/*.conf
      # Include conf/crs/optional_rules/*.conf


      #.... configure ModSecurity here
      
      #Add this line: 
        SecRuleRemoveById 960020
</IfModule>
#... Rest of the file

Remember that you can't do it in .htaccess. It's important to remove the rule after you include rules from OWASP!

After that restart HTTP daemon and check if page is caching or not:

sudo service httpd restart

Missing caching header

Another issue might be connected to missing caching header. You can set how long each page can be cached by setting the cache file. Try to change varnishtest.php file like that:

<?php
header( 'Cache-Control: max-age=60' );
echo microtime();

It means that this file should be keep in cache no longer than 60 seconds. Try to refresh the page and see if it brings any help or not?

Double caching header

If You are using mod_expire for Apache, there might be situation where you have two caching headers. One is set from PHP script, second one by Apache. The easiest way to check it is to see header in Chrome for instances. If You will see two caching headers like that:

Cache-Control:max-age=10
Cache-Control:max-age=0

You need to adjust your Vhost file for file caching. For instance sometimes you can find such settings in your httpd.conf or Virtual host file:

<IfModule mod_expires.c>
    #If you are using caching for static contents this line must be present, don't remove it!
    ExpiresActive on

    #Default cache is set to 0, comment it out
     ExpiresDefault 0
   
     #Disable cache per content type, comment it out
     ExpiresByType application/json                      "access plus 0 seconds"
     ExpiresByType application/xml                       "access plus 0 seconds"     ExpiresByType text/html                             "access plus 0 seconds"
</IfModule>

If you plan to use Varnish cache, you should control your cache inside your application, and don't rely on Apache cache when it comes to website content, API endpoints etc. You can still cache static assets like images, scripts or styles with Apache. Double Cache-Control header will prevent Varnish from caching PHP scripts.

Those cookies...

This is really important. By default Varnish will NOT cache any output if there are any cookies. That's how usually authentication mechanisms works for logged in users. If you are logged in you want to see most recent content instead of cached one. Make sure that you are not sending any Cookies. If you have some problems with Tracking cookies like from Optimizely or Google Analytics  you an always unset them in Varnish. Try this approach. Open default.vcl  file where you set the backend port. Change sub vcl_recv section to something like this:

sub vcl_recv {
	 if(req.http.Cookie){
    set req.http.Cookie = ";" + req.http.Cookie;
    set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
    set req.http.Cookie = regsuball(req.http.Cookie, ";(PHPSESSID)=", "; \1=");
    	set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");

    if (req.http.Cookie == "") {
        	unset req.http.Cookie;
    }
  	}
}

This is example taken from Varnish webiste. It will unset all cookies but not the cookie contains PHPSESSID. Save changes in file and restart varnish:

sudo service varnish restart

Check your page now.

Nothing helps 🙁

If nothing from above helped, you need to find the issue. There is also great documentation that comes from Varnish and you can see Varnish troubleshooting guide. If still nothing helps you can post comments here below, use StackOverflow or some forums. We learned that Varnish sometimes can be pain in the a** and it's hard to debug why something doesn't work...

Purging Varnish Cache

Great! Now we have our cache up and running! We change something in our script, refresh the page, but hey! There is still cached content! So now the question is, how to clear Varnish cache?

Method 1 - varnishadm

Let's introduce first way to purge Varnish cache. First method is with usage of Varnish tools. One of them is varnishadm which will help us to clear the cache. Let's try it:

sudo varnishadm ban 'req.http.host == "lamp.com:6081" && req.url ~ "/varnishtest.php"'

This is very basic usage. We will use ban command from varnishadm to add ban in Varnish settings. In ban command we can use any parameters that are available in Varnish settings, but here are two basic things that you will need.

First we need to check host req.http.host == "lamp.com:6081" . This is very useful if we have multiple domains on our server. We need to be sure that we are clearing the cache only for one, particular website, not for all websites. Port is necessary for now, because we don't have Varnish on port 80 now.

Next is the most important thing - URL. req.url ~ "/varnishtest.php"  We can use here any regex expression, which is really helpful. This command will clear only one particular URL. You can use req.url ~ "^/article" to ban all URLs that starts with /article phrase. Sometimes helpful command is req.url ~ ".*"  that will clear cache for every page and file.

Short note about clearing the cache - if you have really high traffic on your website you need to be very careful when it comes to clearing Varnish cache. You don't want to kill your website with really high traffic that will goes straight to Apache for instance. So sometimes it's better to wait for Varnish cache to expire by itself instead of clearing the cache manually. It all depends on your website and architecture.

Method 2 - PURGE request

OK, but how to clear the cache from PHP script for instance? It's not safe to execute shell commands from script directly. Therefore Varnish comes with another way of clearing cache - PURGE request.

You can send PURGE command with cURL, for instance:

curl -X PURGE http://example.com:6081/varnishtest.php

ModSecurity fix

If you have ModSecurity enabled, you will get 403 error. That's because PURGE request method is not allowed. We need to edit OWASP settings file located in /usr/local/apache2/conf/crs/modsecurity_crs_10_setup.conf  and we need to find the line with tx.allowed_methods . Then we need to add PURGE request, so it will look like that:

setvar: 'tx.allowed_methods=GET HEAD POST OPTIONS PURGE', \

Restart Apache to load new policy:

sudo service httpd restart

Then try to execute curl command from above. You should get contents of our test file instead of an error.

 

As you can see, you will get our test file contents, but it won't clear Varnish cache for this URL. We need to prepare Varnish to enable PURGE requests. Edit VCL file located in /etc/varnish/default.vcl  We need to add following section:

acl purge {
        "localhost";
        "192.168.77.1"/24;
}

sub vcl_recv {
        #....

        if (req.method == "PURGE") {
                if (!client.ip ~ purge) {
                        return(synth(405,"Not allowed."));
                }
                return (purge);
        }
        #.....

}

In acl purge  we define addresses that can send PURGE requests. Localhost is obvious, if we want to send requests from PHP scripts. But sometimes we need to clear the cache from external IP address. So it's good to add it here. If you need to clear the cache only from PHP script, and do not send it from outside, leave localhost  only.

In sub vcl_recv  section we check if request method is PURGE, if so, we need to check if client.ip  is within allowed list for PURGE requests.

Try to send request again. It should clear the cache.

VCL

So what is this magic VCL file? In short words it a file designed to define request handling and caching policies for Varnish. We can specify additional rules, like removing request cookies, or disable the cache if user is logged in. Configuration strongly depends on your website or CMS you are using. You can use ready VCL files like this one - mattiasgeniar/varnis-4.0-configration-templates. This is really cool and well commented example VCL file. You can find multiple tricks for Varnish there, and it's really worth checking. It has configuration for WordPress and Drupal, but it should also work well with Symfony2 or Zend Framework. You can also find more details about Varnish and Symfony2 on their website.

It's all really depends on your website, used CMS or framework, installed plugins etc. So it's really hard write one VCL for everyone. You need to do it by trial and error or googling. But really good start is to check VCL file mentioned above.

Varnish settings per domain/Virtual Host

What if we have multiple websites and we want to have separate configuration per website? For instance one website is based on Drupal, second one is just pure PHP. Varnish settings for global is not possible due to some conflicts. How to deal with such issue?

VCL file and language is pretty powerful and we can resolve such conflict It's not that easy, but doable. Edit  default.vcl  file - /etc/varnish . Take a look on example file below:

vcl 4.0;

backend default {
    .host = "127.0.0.1";
    .port = "80";
}

# ...

sub vcl_recv {

    # ... global rules for all websites here

    # Return 404 error error in requests without Host. It's optional but helpful
    if(! req.http.Host){
        return (synth(404, "Host header missing"));
    }

    if(req.http.Host == "example.com"){
        include "/etc/varnish/example.com.vcl";
    }else if (req.http.Host == "secondexample.com"){
        # you can also place normal rules here, no need to include files
    }
}

Varnish has include command. With this command you can include pieces of code inside different files. So you can have kind of per domain settings. It's not perfect, because you need do it inside sub directive. It's not possible to include whole sub vcl_recv sections  etc.

With this there comes second disadvantage. If you have larger pieces of instructions per domain and you have to modify another sub  like vcl_backend_response , you need to make similar if statement inside each section.

You can modify the if as you wish. But remember, that you can also use ~  to mark domains with subdomains + www etc.

In this example there's no port. If you are testing Varnish on different port, so it's not running on port 80, you need to specify the port as well. if(req.http.Host == "example.com:6081")

This is not perfect solution, but it's working. In real life you probably won't have to create statement like that. Most of the rules can be applied globally. But if you need to make an exception for particular domain, now you know how to do that;)

Varnish useful tools

Here are couple of really useful tools that comes with Varnish.

varnishstat

This command will give you an access to Varnish statistics. You can see lot of data there, but really important are cache_hit  and cache_miss . If you have lot of cache_miss , that means, that your content is not served from Varnish, but from backend like Apache. It also means that something is not OK and you should really look through your configuration and caching.

If you have great number of cache_hit  and not that many cache_miss  it's perfectly alright. When Varnish cache expire for particular URL it goes directly to backed and cache_miss  value is incremented.

varnishtop

varnishtop is yet another useful piece of software you can find in varnish toolbox. It shows the URLs that have most hits to your backend. It is also great debugging tool. The cool thing is that you are able to filter the logs. There are so many filters that it's not possible to list all of them here. But there is great documentation for this tool, that you can find here.

Here are two examples, that are really useful

  • varnishtop -i ReqURL  it will show list of URLs that comes from Varnish most frequently 
  • varnishtop -i BareqURL  it will show you list of URLs that goes to your backend the most. This is really great command, because you can easily catch most frequent misses.

varnishlog

This is another great tool. It logs all requests in real-time with tons of data. It's great for debugging why particular URL doesn't work (ie. it's not cached, but it should). It has very similar syntax to varnishtop, so if you master varnishtop, you can use varnishlog without any problems.

By deafult, Varnish doesn't log to file. This is really important. Why's that? Think about such situation. You have an API with some endpoint or one particular page in WordPress. This endpoint/page has heavy load, let's say 1000 requests per second. Apache can't handle it, so you install Varnish and suddenly everything is OK. But you don't know what's going on in logs. In Apache access.log you have only few requests which has cache_miss in Varnish. But Varnish doesn't log anything to file. And to be honest - it's correct behavior. First of all, it will speed up entire service, because you don't need to waste the time for I/O operations on your hard-drive every requests. Second of all, digging in really big access.log is pain in the a** and it's difficult to find anything there.

But if you need logging to file, because let's say, you don't have some really high traffic you can enable logging to file by executing:

sudo service varnishlog start

and if you want to run it on system start

sudo chkconfig varnishlog on

Varnish logs can be found by default in /var/log/varnish . But please, be aware, that it will generate lot of writes to your hard-drive and log file will increase it's size pretty fast on heavy load websites. So this solution is not recommended.

varnishncsa

It's another useful tool for logging. It's different output from varnishlog. varnishlog will log every detail into your log about the query. If you need logs similar to for instance Apache logs, you should use varnishncsa command. By default it's disabled just like varnishlog. If you want to log to file you should start the service varnishncsa and add it to autostart with chkconfig.

varnishncsa

Performance

OK, awesome, Varnish is super cool , my websites goes blazing fast! But hey! Is it possible to speed it up even more? Well, theoretically - yes. There are few tips that can increase your throughput. But you must remember one thing - the MOST important part is to have high hit rate. If you have low hit rate and high miss rate, this tips won't help you that much. So remember, achieve high hit rate first. Then take care of Varnish tuning:)

Few tips before you start tuning up Varnish:

  • There is no one common ctrl+c ctrl+v method that will improve results on all machine
  • Change one thing at the time and then WAIT for results if it helps or not
  • Monitor statistics with Varnish tools. They will tell you whether the change will work or not
  • Remember that Varnish is blazing fast by default. If you don't have enormous traffic, you'll probably don't need these tips.
  • Increasing particular values too much can't make your hardware die. It will eat up all your CPU or RAM resources, so please be careful.
  • Monitor, monitor and monitor your system:)

OK, let's start tuning. First, i suggest to read Varnish book and chapter about tuning. It will help better understanding of what's going on.

Varnish has configuration file that contains important options that will help you increase Varnish power. Edit the file by sudo vi /etc/sysconfig/varnish

Now Varnish supports couple of ways how to configure Varnish daemon. They are all present in this file so figure out which one are you using. By default they are called Alternative 1, Alternative 2 and Alternative 3.

In my opinion Alternative 3 is the best, because you only need to change variable value instead of whole DEAMON_OPTS  string. In Varnish 4 it's enabled by default, rest of alternatives should be commented out. If not, comment out Alternative 1 and Alternative 2, and uncomment Alternative 3. But if you want you can stick to any Alternative you wish. It's just the matter of passing configuration to Varnish daemon.

You can set Varnish port there, secret file etc. But among the settings you will find threads section, and storage type. Let's start with storage type.

Files or RAM for cache? 

# # Cache file size: in bytes, optionally using k / M / G / T suffix,
# # or in percentage of available disk space using the % suffix.
VARNISH_STORAGE_SIZE=256M
#
# # Backend storage specification
VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}"

Now it's up to You and your server resources how to set these values. By default Varnish will use your RAM as a storage. It also can use the file method, but keeping things in RAM is lot faster than reading them from hard-drive, especially if it's not SSD. But if you have little RAM on your server and lot of URLs to cache, use file , instead of malloc .

So this is one thing that will speed up your Varnish - using RAM. How many? Larger value is definitely better, but keep in mind, that it's not strict value. Varnish will add overhead per each object in cache (around 1kb). So if you have large amount of objects, it will increase storage size. You need to play around with this value a bit and check how much is enough. Keep in mind that you need to have enough free RAM for other processes like httpd or php-fpm.

If you will use RAM check if the storage size is big enough to handle all your requests. Use this command:

varnishstat -1 | grep n_lru_nuked

If the number of LRU nuked objects is greater than 0, then it means that or your storage size is not big enough, or you have something wrong with caching. For instance you try to cache assets or images for very long period of time.

Threads settings

Second thing that will increase Varnish performance are threads. In the same file you will find settings for them:

# # The minimum number of worker threads to start
VARNISH_MIN_THREADS=50
#
# # The Maximum number of worker threads to start
VARNISH_MAX_THREADS=1000

There are multiple options for threading in Varnish, but you should mostly play around with these two values. You can increase both values here, but you should keep withing the Varnish guidelines. Here is explanation how Varnish threads works from Varnish documentation.

When a connection is accepted, the connection is delegated to one of these thread pools. The thread pool will further delegate the connection to available thread if one is available, put the connection on a queue if there are no available threads or drop the connection if the queue is full.

Read here how to calculate how many threads you should run to not to kill your server.

Varnish and security

There are various of methods how to secure more your Varnish. We will not cover them here. But to make things clear - most important thing is to secure your purging requests. Do not allow anyone to purge your cache from outside. He can purge entire domain, and if you have huge traffic - it can kill your website when all requests will come to your backend server.

You can install also firewall for Varnish. It's really similar to ModSecurity for Apache. You have basically two options:

They are just additional rules in VCL that will block unwanted requests.

Our opinion about that? They are OK, but we like to keep Varnish VCL as simple as possible and do not add additional overhead to parsing requests. In addition we have ModSecurity for Apache installed, and it does basically the same job. So is it worth to have two security layers and add exception rules to both of them? We will just stick to Apache ModSecurity:)

Varnish on port 80

Now it's the most important part. For now we use Varnish on "test" port 6081. Before you will switch Varnish to port 80, make sure, that IT WORKS. If requests are cached correctly etc. It's important to put working Varnish as an entry point to your server, instead of just additional layer that will forward all your requests to Apache.

First you need to pick the new port for Apache. Common practice is to use 8080, but this port is often used for different tools like SOLR. We usually set port 81 for Apache. After you will pick a port number, you need to edit few files.

Note that if you are not following our tutorial from start, your files might be located elsewhere. 

httpd.conf

We need to change main port for Apache in httpd.conf file

# File located in /usr/local/apache2/conf/httpd.conf
Listen 81

All VirtualHosts files

All Virtual Hosts files also contain port. You need to change every single one file with new port.

#All files with .conf extension located in /usr/local/apache2/conf/vhosts
<VirtualHost *:81>
        ServerName example.com

Firewall rules for new Apache port (optional step)

This is optional step. We have working firewall on our server. If you will need to access website directly (not through Varnish cache) you can open new port. It's usually useful when it comes to debugging. If this is your app issue or is it Varnish cache fault. If You need to enable new port, add appropriate lines inside firewall rules.

# Firewall rules are in /etc/iptables.firewall.rules
-A INPUT -p tcp --dport 81 -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp --sport 81 -m state --state ESTABLISHED -j ACCEPT
-A INPUT -p tcp --sport 81 -m state --state ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp --dport 81 -m state --state NEW,ESTABLISHED -j ACCEPT

Restart firewall to load new rules

sudo service firewall restart

Varnish daemon settings

Next thing is that we need to change Varnish daemon settings to listen on port 80.

# File is located in /etc/sysconfig/varnish
VARNISH_LISTEN_PORT=80

Varnish VCL file

Last thing is to change VCL file backend port, and update all rules to remove previous port 6081

# File is located in /etc/varnish/default.vcl
backend default {
    .host = "127.0.0.1";
    .port = "81";
}

# Remove :6081 if present in host
if(req.http.Host == "example.com"){

Remove old Varnish port from firewall

Remember when you added port 6081 to firewall in order to test some stuff? You will not need that port to be open.You can remove these lines from your firewall rules, and restart the firewall.

Restart httpd and Varnish

Once you change all files it's time to restart all daemons and check if everything works as expected.

sudo service httpd restart
sudo service varnish restart

Now check your website. It should be powered by Varnish now!

What's next?

Lengthy tutorial, wasn't it? But now your websites should be blazing fast! Remember to check it from time to time, if you have good hit/miss ratio:)

As always you can do everything here with Ansible and our repo, which you can find on GitHub.

Now you should be in place where you have secure CentOS, working LAMP and Varnish on top of everything. What's next? I believe that we should add nginx to our setup to keep everything nice and slim, and after that start installing some real useful things like Redis, RabbitMQ and add some good tools to PHP etc. See you next time!

How to configure SSH on CentOS

How to configure secure SSH on CentOS

Hello everyone! In this tutorial I will show you how to increase server security by tuning up configuration of SSH.

Before you begin

There are basically two requirements for this tutorial:

  1. You need to have working SSH keys. You need to be able to login to your server by using them. After completing this tutorial, SSH keys will be the only way to access server. If you won't be able to login by using them, well, you will lose access to your server. In order to add user and configure keys you can follow this tutorial.
  2. Make sure that at least one of the users is in wheel group (has access to sudo). Root should not have access to login via SSH. So if you will block this option and you won't have any sudo user, you won't be able to do much on the server. Follow this tutorial in order to configure sudo.

Disable password authentication for SSH on CentOS

Login to Your server/Vagrant Box and open SSH daemon configuration file:

sudo vi /etc/ssh/sshd_config

Now we need to find the line for password authentication and change it to:

PasswordAuthentication no

Unfortunately, disabling this option can still lead to password authentication by using PAM-based authentication. In order to fully disable authentication with password, make sure that PAM is also disabled:

ChallengeResponseAuthentication no

Also we need to make sure that this line is uncommented. It will  enable SSH login by using public key:

PubkeyAuthentication yes

Save the file and exit from the editor. In order to apply changes, you need to restart SSH daemon:

service sshd restart

After that, try to open new SSH session in new window. Do not logout from your current session! If you won't be able to login with new session, you can undo the changes with existing session. If you will be able to successfully login, you can proceed.

How to secure SSH on CentOS even more?

There are still some things that will help you improve SSH security. Edit the same configuration file as before. Below You will find the configuration options that I usually use for SSH.

Disable root login

PermitRootLogin no

This option will disable root login via ssh. So it means that from now on you won't be able to login to your server as root via ssh.

Allow only specific users to be able to login via SSH

AllowUsers developer

By default you are able to login as any user that is created inside the system. It can be easily limited to particular users. Just give space separated list after AllowUsers. It might not be present in your config, so you need to add this line (for instance at the end of the file).

AllowUsers developer vagrant

Enable protocol 2 for ssh

Protocol 2

This option is set by default in most CentOS installation, but just make sure that there's no version 1 instead. It's less secure protocol.

Ignore rhost

IgnoreRhosts yes

It will disable insecure access via RSH.

Disable login for users with empty passwords

PermitEmptyPasswords no

This line will disable login for users that have empty passwords. Make sure that your account has password set, before changing that!

Enable strict mode for ssh

StrictModes yes

SSH will check users's permission in their home directory before accepting login. It should be set to yes because users may leave their directory or files world-writable. Again, this might be tricky. It's the best to change that, restart SSHD daemon and try to login from new session. If you have any problems, you can undo this change with existing session. If you have any issues with that, try to set valid permissions for your .ssh directory and files inside. Also set valid username and group for your files:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/*
chown -R YOUR_USERNAME:YOUR_USERNAME ~/.ssh

Disable other authentication methods

GSSAPIAuthentication no
KerberosAuthentication no

If you don't plan to login with GSS API, or Kerberos you can disable them as well.

Disable X11 Forwarding

X11Forwarding no

If you don't use X11 you can safely disable it as well.

Show last login

PrintLastLog yes

Nice feature is to show last successful login after you will login via SSH.

Restart SSH daemon

Remember that after any changes inside the file You need to restart sshd daemon:

sudo service sshd restart

SSH crypto

In addition to changes above that should be applied, you can increase SSH security even more by configuring ciphers and available algorithms (thanks to @Amar for the suggestion:)

This is usually safe to execute, but you must remember that not all algorithms are supported by various tool. Here you can find great chart showing, which tools support given algorithms. But let's be honest, most of you is probably using OpenSSH which supports all the changes I will present here. However if you are using different tool and you won't be able to login to your server, check with the page and enable additional algorithms.

These config options will probably not be listed in your config file. You need to just add them somewhere, like at the end of the file.

Configure server authentication

HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key

Server must confirm the identity to the client. There are bunch of algorithms available, but this is the list of most secure.

This might be present in your configuration file, also there might be more not commented lines with HostKeys. Leave only these two enabled and comment out the rest.

Configure key exchange

KexAlgorithms curve25519-sha256@libssh.org

There are many more key exchange algorithms, but this is probably the most secure.

Configure ciphers

Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com

Ciphers are used to encrypt the data. As with key exchange, there are multiple algorithms. These are the safest.

MACs - Message Authentication Codes

MACs hmac-ripemd160,hmac-ripemd160-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,umac-128@openssh.com,umac-128-etm@openssh.com

MACs are used for data integrity. Again, line above contains the safest algorithms only.

After these changes, don't forget to restart sshd daemon.

Easier way?

You can use our Ansible LAMP on Steroids project to make configuration of your server easier!

If you don't know what Ansible is, check our tutorial first.

Clone our repository and setup your server faster with LAMP on steroids.

CentOS, users, groups, sudo and SSH keys

Create sudo users and groups in CentOS

I will show you how to organise groups and users in CentOS.

As an example we will create account for user named developer. The purpose of this account is:

  • logging via SSH instead of using root account
  • access to sudo command for management tasks
  • write access to website files
  • read access to logs

Create list with available groups

If you are on fresh system it's handy to create a list with all available groups. It's also handy to check which users belongs to that groups. It might be helpful after some time when you need to decide if given group was at the beginning or can it be safely removed. getent command can help you with that.

getent group > /etc/initial-group-list
cat /etc/initial-group-list

Remove group from the system

If you need to remove group from CentOS simply use following command:

sudo groupdel NAME_OF_THE_GROUP_TO_DELETE

Create new group

In order to create group you need to use groupadd command:

sudo groupadd NAME_OF_THE_NEW_GROUP

I usually add group named www (or www-data, whatever works for you). To this group I add php daemons, nginx workers etc. It makes life easier with writing to files. In order to create such group execute following command:

sudo groupadd www

Create list with available users

Same like with group, I like to have list of initial users. In order to create such list you can use getent too:

getent passwd > /etc/initial-users-list
cat /etc/initial-users-list

Delete user from CentOS

In case you would like to remove any user from the system, use following command:

sudo userdel -r USERNAME_TO_REMOVE

-r flag will remove also his home directory. If you wish to delete the user, but to keep his files, omit this flag.

Create new user in CentOS

Let's create new user developer that we mentioned at the beginning:

sudo adduser developer

and create the password for his account:

sudo passwd developer

If you want to add developer user to www group created before use usermod command:

sudo usermod -g www developer

If you want to add this user sudo powers (and you should if you want to use this user instead of root), add it to wheel group. wheel group is special group in CentOS configured in sudoers file. Whoever belongs to this group can have sudo powers.

sudo usermod -g wheel developer

Optional parameters to useradd command

There are lot of additional parameters for useradd command but there are two especially useful.

First one is helpful when you don't want to create user home directory. It means that user will not have it's own place under /home directory to store it's files. This option is helpful when you are creating user for system service like Apache httpd for instance. So in order to create user with no home directory use --no-create-home:

sudo useradd httpd --no-create-home

Another useful feature is to specify shell of given user. It's nice if you want to cut of possibility to login to the system via SSH for instance. Add --shell /sbin/nologin to disable login for given user, like so:

sudo useradd httpd --shell /sbin/nologin --no-create-home

You can use --shell and --no-create-home parameters separately:)

 

How to setup SSH keys for new created user?

Each user should have RSA key-pair. It makes life easier and you should use it if you want to login to different servers, use GIT etc. In order to create such user key-pair you first need to login to user you created. Most probably you are using root account to execute all commands, but you should never ever login via SSH as a root.

It's much better to create separate user for system management and use only this account. Login via SSH to your server to account your created. In my case it's developer user so my command looks like this:

ssh developer@IP_OF_THE_SERVER_HERE

Once you'll be logged in (after providing the password), you can create RSA key pair. Execute following command:

ssh-keygen -t rsa -b 4096

-t rsa means that it will be RSA key, but this is standard for creating SSH keys. Fun part is with strength of the key -b 4096. By default it's 1024 bits, but to make it harder to break I usually provides 4096. It's not necessary, but you should do that. Some services requires key length to be minimum 2048, but it's better to create even longer one.

Generator will ask you some questions, but you should generally confirm them with enter and leave the defaults. When it comes to SSH on the server, I usually don't set the password. It makes life easier in automated scripts etc.

After that private and public key should be generated as expected. You can find them in ~/.ssh directory.

Add authorized key to user

In order to login with SSH keys to the server, instead of using password you need to add authorized key to developer user. In my opinion it's must have feature as using password login is super risky. Again, been there, done that, I was hacked, even when my password was strong. With SSH logging even strongest bruteforce attack will fail:)

You need to add your key to ~/.ssh/authorized_keys on the server. If You have ssh-copy-id command available just execute:

ssh-copy-id developer@IP_OF_YOUR_SERVER

Make sure that you are executing this command from your computer, not from the server. If you don't have SSH key created locally, you can generate it in the same way as on the server, by using ssh-keygen command.

If uou don't have ssh-copy-id available (for instance from Windows), you can do it manually.

ssh developer@IP_OF_YOUR_SERVER
cd ~/.ssh
vi authorized_keys
//Press "i" to enter in input mode, paste there your code (usually it's right click of the mouse) and :wq (colon, w, q) it will save and quit from vi 
chmod 600 authorized_keys

So here how it goes:

  1. ssh to the server as usual with password.
  2. Change location to .ssh directory.
  3. Create authorized_keys file with vi
  4. Paste there your local public key, save the file and quit
  5. Set permissions on authorized_keys.

Test ssh login with keys

Now You can try to log in with Your key.

ssh developer@IP_OF_YOUR_SERVER -i path/to/your/PRIVATE/key/file

You shouldn't be prompted for your account password!

Easier way?

If you don't want to spend your precious time executing each of these commands by hand, you can use Ansible and our LAMP on steroids project to speed things up!

If you don't know what Ansible is - you can read our tutorial about it here.

LAMP on steroids project is available on GitHub here.