Tag Archives: Vagrant

Ansible tutorial – part 2

This is part 2 of Ansible tutorial. In this part you will learn how to create playbooks, what are roles and tasks and how to use privileges escalation in Ansible.

A short word about Ansible tutorial

This is multipart series about Ansible, great tool for provisioning and configuration management. You will learn how to create useful roles and use Ansible in efficient way. You will also pick up some good practices:)

If you like tl;dr take a look at shorter veresion of this tutorial on GitHub: https://github.com/blacksaildivision/ansible-tutorial
There are also some examples!

Part 1: Intro to Ansible and Inventory files
Part 2: Playbooks and tasks
Part 3: Templates and handlers 
Part 4: Variables

 

Playbooks

Playbooks are essential feature of Ansible. In Playbooks we specify what and how our server will be configured and provisioned.

Go to our directory that we created in first part -  vagrant_ansible_tutorial . Create a file inside and name it playbook.yml . Name is irrelevant here, you can use anything you want. Open this file in editor and paste following content:

- hosts: ansible_tutorial
  tasks:
   - debug:
      msg: Test of Ansible Playbook!

First line contains host(s) that we want to provision. You must specify group or single host from Inventory file  hosts  that we created in part 1. Good practice is to specify group instead of single host. In our case it is  ansible_tutorial , so we will provision our Vagrant box.

You can also specify multiple hosts here and set different tasks or roles:

- hosts: webservers
  tasks:
   - debug:
      msg: Installing nginx
   - debug:
      msg: Installing PHP

- hosts: databases
  tasks:
   - debug:
      msg: Installing MongoDb

 

Next, we need to specify tasks . Task in Ansible is like single command in bash on steroids. With tasks you can install software, add users, remove files etc. Our task will just print out the message in command line, so no harm will be done if it won't work as expected.

We will get more into the tasks later on. For now just use the playbook above.

If you are familiar with YAML or Ansible you might notice that there are three dashes ---  missing. If you will browser through the example playbooks on GitHub, you might note that all files starts with --- . This is coming from YAML and it is optional in Ansible. You might or might now use it, it doesn't make any difference. We never use it on our side and we won't use it in this tutorial as well. But if you prefer, you can add ---  to start of each file like this:

---
- hosts: ansible_tutorial

Go to your terminal and execute following command:

ansible-playbook -i hosts playbook.yml

 We need to pass two arguments. First one is an inventory file. Second argument is playbook file that we want to run.

Your output should look similar to:

PLAY [ansible_tutorial] ********************************************************

TASK [setup] *******************************************************************
ok: [192.168.60.70]

TASK [debug] *******************************************************************
ok: [192.168.60.70] => {
    "msg": "Test of Ansible Playbook!"
}

PLAY RECAP *********************************************************************
192.168.60.70              : ok=2    changed=0    unreachable=0    failed=0

You might note that there is also task named setup . This is Ansible internal task for gathering information about the server. We will use them later on. For now, just know, that such additional task exists.

Next lines contains our debug task. It got status ok  and it was executed correctly.

Probably the most important part of this output is recap . It contains information about current run of playbook. There are 4 metrics that are worth checking:

  • ok  - tasks that were executed correctly or didn't change anything on provisioned server.
  • changed  - things that were modified, like you just add new user. If you run the same playbook twice it will have ok status.
  • unreachable - usually should not happen, you will get it when host becomes unreachable, connection was dropped or you are out of internet etc
  • failed  - when tasks were unable to execute correctly

In the end, perfect playbook should not contain any failed and unreachable errors. On first run you should get only ok and changed tasks. Second execution of the same playbook should result only in ok statuses.

Roles in Ansible

Role, apart from Playbook, is another essential feature of Ansible. In previous example we created tasks directly in playbook file. You must know that this is wrong approach. 99% of the time you will use roles instead of using tasks directly in playbooks. We used it only to show you how to execute working playbook.

Think of how do you use a server. For sure you need to install some kind of web server like apache or nginx. There will be couple of things to do (tasks) in order to install nginx. We can group them into single role. Roles will help you organise your structure.

Role should have single responsibility like installing and configuring nginx. It should also be reusable. So you can for instance use it in different group of servers or even use it in separate project.

Let's create our first test role that will install and configure nginx. In the project directory create directory and name it roles . In this directory you should keep all roles you create.

Inside there you should create another directory and name it nginx . Name of the directory is the same as name of your role. Each role directory, like nginx, usually contains couple of directories inside like tasks, defaults, handlers etc. For now we will just create directory named tasks . Inside tasks directory, create file and name it main.yml  In the end your directory structure should look like this:

Screen Shot 2016-06-22 at 13.02.08

At the very beginning it might look messy, but after a while you will see that it's quite clever way of organisation.

Just for the start, we will use our debug message as simple start. Your main.yml file should look like this:

- debug:
    msg: Test of Ansible Playbook!

This is just copy+paste from playbook.yml  Now we need to include our role into the playbook itself. Edit playbook.yml  so it will look like that:

- hosts: ansible_tutorial
  roles:
   - nginx

As you can see, we just removed tasks  from there and added roles  instead. Each hosts group can contain multiple roles. You can also reuse given role in different hosts.

Now let's run our playbook again:

ansible-playbook -i hosts playbook.yml

Output will be very similar as before and the end result should be the same.

Tasks in Ansible

Let's write some some real tasks that will help us install nginx. Our OS is CentOS 7 as we specified it in Vagrantfile . In order to get nginx up we need to install epel repostitory and nginx itself. Start by removing everything from main.yml  in tasks directory. We don't need debug message there, it was only for test purposes. Instead create following task:

- name: install epel
  yum:
    name: epel-release
    state: present
  tags: [nginx]

Each task should start with name. It's not necessary but it's very informative and easier to read in ansible command output.

Next line contains module name that we will use. In CentOS we are using yum for installing files. If the base images was Ubuntu for instance, we would use apt module. Yum/Apt module needs to know two things. First is the name of the package we would like to install. In our case it's epel-release . Second thing is state - whether application should be installed or removed. We want to install it, so we should use present  state.

Last thing are tags. They are optional in each task but usually it's good to use them. When you use ansible-playbook command, you can run tasks that contains only given tag. We basically add tag(s) for each task we create. Sometimes it's not necessary, but it's just convenient to have them there.

Run your playbook with ansible-playbook command like before. If you are using the same image as we in our tutorial, you might wonder that task resulted in ok  status, instead of changed . It should be changed when we install new packages, right? Yes, it should, but the vagrant image has epel-release already installed. So Ansible is checking if it's installed, and if so, it won't do anything. Clever, isn't it?

Once we have epel repository we can install nginx. Let's add another task to our role:

- name: install epel
  yum:
    name: epel-release
    state: present
  tags: [nginx]
  
- name: install nginx
  yum:
    name: nginx
    state: present
  tags: [nginx]

As you can see, we are just installing nginx. Let's run the playbook. Whooops! You should get a failure - You need to be root to perform this command.

Privileges

In hosts file we specified that the user that should connect to the server is vagrant. But it does not have enough permissions to install things. We need to use privileges escalation.

Let's edit our playbook file and add become  lines:

- hosts: ansible_tutorial
  become: yes
  become_user: root
  roles:
   - nginx

What we just did is we enabled become so now we can allow Ansible to execute command as different user. As we need to install some packages we need root user. Ansible will login as vagrant user, but each command will be executed as it would be root account.

You might ask - why not to edit hosts file and use root as a user. Answer is simple - the secure way is to disable ssh for root account to prevent harm to your server. root should not have the option to direct login via ssh.

You can also specify these values per task. But usually most of the tasks we do require root power, so we are defining it for entire connection to given host.

Try to run the command again. It should succeed this time. You should get changed=1 in output. It means that nginx has been installed. Try to run the playbook again. It will get ok  status, so nothing was changed.

Finish installing nginx with Ansible

Last missing task is to start nginx. Add following task inside main.yml

- name: enable nginx
  service:
    name: nginx
    state: started
  tags: [nginx, status]

We are using Ansible module called service. It is responsible for managing services. In general you can find all possible modules for Ansible on this page. It's super useful and it contains lot of examples for each command. It's also well written. If you don't know something, just take a look at documentation of given module and you should get an answer to your question:)

Service module requires two things - name of the service, which in our case is nginx, and state of the service. We want to start it, so state should be set to started .

Run your playbook. After thatm open your web browser and type the IP of your server (you can find it in Vagrant file). If you are following the tutorial, simply visit this URL: http://192.168.60.70/

You should see nginx welcome page!

Tags

Now it's the time to use some tags power. Note that we added another tag named status  to our task. Imagine the situation when you just want to check if nginx is working. You don't want to execute other tasks that are installing things. We should only execute tasks that have given tag:

ansible-playbook -i hosts playbook.yml --tags="status"

It should execute only two tasks. One is default setup task, and second one is the task with status tag. If you have multiple roles, like nginx, php, mysql and you will add status tag to all tasks connected to service module, command above will look through all roles. In one command you are able to check if all services are working fine:)

You can also specify multiple tags using comma separated list:

ansible-playbook -i hosts playbook.yml --tags="status,nginx"
Ansible tutorial - part 1

Ansible tutorial – part 1

What you will learn in part 1 of this tutorial? What is Ansible? How to install Ansible on Windows/Linux/MacOS? How to prepare test environment for Ansible? Intro to Inventory files. First connection to the server.

A short word about Ansible tutorial

This is multipart series about Ansible, great tool for provisioning and configuration management. You will learn how to create useful roles and use Ansible in efficient way. You will also pick up some good practices:)

If you like tl;dr take a look at shorter veresion of this tutorial on GitHub: https://github.com/blacksaildivision/ansible-tutorial
There are also some examples!

Part 1: Intro to Ansible and Inventory files
Part 2: Playbooks and tasks
Part 3: Templates and handlers 
Part 4: Variables

 

First thing first - what is Ansible?

Ansible is a simple yet powerful tool for configuration management and orchestration of your infrastructure. It speeds up installing software, configuring servers and most important - it reduces manual steps you need to make, to setup your server(s). It is also great alternative to Puppet and Chef. Both are similar tools to Ansible, but in my opinion Ansible is much easier to learn and master.

How do WE use Ansible (possible use cases)?

We have lot of servers that we need to take care of. At the beginning we were just logging into our servers via SSH and fire all commands in terminal. It was painful job, so we decided that we need to automate things. We checked Puppet, Chef and Ansible and we decided to stick with the last one. It was really good choice. Now we use Ansible for:

  • Installing our entire LAMP stack on new server
  • Configuration of most of the software on our servers
  • Testing if servers are configured properly
  • Deploying our applications
  • Keeping Infrastructure As Code (IAC) - all infrastructure is in single Git repository

Most of these tasks are done with single command. For instance we can update nginx on all servers with bumping the version in our variable file and execute one command in terminal. It will be updated on all servers at once.  No more logging it via SSH and executing commands one by one:)

 

What do I need to know about Ansible?

Ansible is using files for configuration and provisioning. By using Ansible you can go with IAC - Infrastructure As Code. It means that you can keep you entire infrastructure in GIT repository. You can track changes and use all goodies that comes with source version control.

Ansible is based on YAML files. If you don't know it, you will learn it in minutes. Really easy and simple.

For templates it uses Jinja2 engine. Powerful and easy to learn.

It does not require additional software to be installed on the server. It uses old-school ssh protocol for connecting and configuring the server. One requirements is to have Python installed on the server, but these days, all common Linux distributions have it out of the box.

Let's get started - how to install Ansible?

No matter what is you operating system, when you will be installing Ansible, make sure that you will install version 2.0  or higher. It has lot more features than 1.x versions. It is stable and commonly used these days.

If you are using OS X or Linux, it is not a big deal for you. Here are some tutorials from Ansible documentation - how to install Ansible on OS X or Linux? The easiest way is to use APT, Yum for Linux and Pip or Homebrew.  Please make sure that you are installing Ansible in version 2.0 or higher:)

If you are using Windows, it's a bit more hassle, unless you already have the Windows with Bash terminal! If you do, simply use APT for installing Ansible. However if you don't have it yet, you need to install more things. On our Windows machines we use Babun (much better version of Cygwin, if you didn't hear about it, check it out right now!). To install Ansible with Babun, please use this tutorial.

 

Prepare test environment

You will need a server or virtual machine where you can test Ansible. If you are just starting with I highly recommend to use empty and fresh server. You don't want to mess up production server while learning it!

If you have some clean server on DigitalOcean for example, you can use it. However easier (and free) way is to install Vagrant and use dummy box. If something will go wrong, you can always remove everything and start fresh. For the purpose of this tutorial, we will use Vagrant.

Start with a new directory for example: vagrant_ansible_tutorial . In this directory create file named Vagrantfile and paste following content into that file:

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure(2) do |config|
  config.vm.box = "geerlingguy/centos7"
  config.vm.network "forwarded_port", guest: 80, host: 8080
  config.vm.network "private_network", ip: "192.168.60.70"
  config.vm.provider "virtualbox" do |vb|
     vb.memory = "2048"
  end
end

You can adjust some parameters if you like. Save the changes in file and execute vagrant up in your terminal. You must be in vagrant_ansible_tutorial directory. Wait until your box will be up and running.

Inventory file

First thing that we need to do before we can actually start using Ansible is to create an Inventory file. Inventory files are the way to specify connection to the servers for Ansible. You can treat is as an alternative to SSH command. So instead of executing:

ssh -i my_private_key_file_path username@server_ip_or_domain.com

you need to add following line to inventory file:

server_ip_or_domain.com ansible_user=username ansible_private_key_file=my_private_key_file_path

Let's create a file in our directory project and name it hosts .  Paste following code into that file:

[ansible_tutorial]
192.168.60.70 ansible_user=vagrant ansible_private_key_file="./.vagrant/machines/default/virtualbox/private_key"

Let's start from 2nd line. We specified IP address to our server. This is the same IP as in Vagrantfile . Instead of using the IP you can use domain address like blacksaildivision.com.

Next is the user that we will use to connect to the server. Vagrant default user is vagrant . Last thing is private key that we are using for accessing the server. After starting the box, Vagrant will create new key in the .vagrant directory.

Alternatively you can specify the password instead of SSH key. But if you are using passwords instead of keys for accessing your server, you should change it ASAP, because you are doing it wrong. Passwords are easy to hack. Trust me, at the beginning our server was hacked twice, even though we had pretty strong password. We didn't know about such thing as SSH back then:) If you will use password, Ansible will also give you a warning.

If you need to alter the port or add additional arguments for your connection, check Ansible documentation here.

Groups in Inventory files

In first line we have [ansible_tutorial]  and it's name of the group of the servers.  You can omit it, but the good practice is to specify the group for server(s). Take a look on following file:

[web]
147.120.24.3 ansible_user=developer ansible_private_key_file="~/.ssh/web"
147.120.24.4 ansible_user=developer ansible_private_key_file="~/.ssh/web"
147.120.24.5 ansible_user=developer ansible_private_key_file="~/.ssh/web"
 	 	
[database]
11.60.215.157 ansible_user=db ansible_private_key_file="~/.ssh/db"
11.60.215.158 ansible_user=db ansible_private_key_file="~/.ssh/db"

There are two groups of the servers - web and database. Now we can execute different commands on each group. For instance on database group we can install MongoDB related software and on servers that belongs to web group we can install nginx+php.

It's good practice to specify a group, even if you have single server. It's more descriptive way than using IP or host name.

Testing connection to server

Last thing in this part is to test connection between Ansible and the server. Execute following command:

ansible ansible_tutorial -m ping -i hosts

If you didn't connect to this server via ssh before, you will get message about authentication for new host. Just type yes.

You should get following input:

192.168.60.70 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

What we just did is we checked if Ansible is able to connect to the server. If you get SUCCESS message, you are ready to start provisioning with Ansible!

Explaining the command you just executed. There are 3 parameters. First one is the group from inventory file. You can also specify particular host or IP from inventory file or you can use all  to test all servers within given inventory file:

ansible all -m ping -i hosts

Second thing is module name. We need to use ping module to check the connection. Last thing is inventory file, which in our case is created hosts  file.

If you have the problems with getting SUCCESS message, you need to debug the connection. Use -vvvv to get debug output. Usually you are able to figure out what is wrong with the connection from debug output:

ansible all -m ping -i hosts -vvvv

In most cases it is wrong private key path or some typos.

Varnish on CentOS - full guide

Varnish – how to make websites fly?

In this episode we will make our websites fly with Varnish on CentOS. What exactly is Varnish? Varnish is an HTTP accelerator designed for websites with high traffic. It can speed up you website significantly, especially when you have more reads than writes. It's designed to handle heavily loaded websites as well as APIs. It's used by many high-traffic websites like The Guardian, New York Times etc. In this episode we will learn how to install Varnish on CentOS.

[sc:lamp_series ]

How to install Varnish on CentOS?

Installation is pretty straightforward:

rpm --nosignature -i https://repo.varnish-cache.org/redhat/varnish-4.0.el6.rpm
yum install varnish -y
varnishd -V

Varnish comes in two versions. Widely used version 3 and version 4 which is recommended at the moment. Version 3 is generally deprecated  since the end of April 2015, so we will install latest version of Varnish 4. After installation  varnishd -V  shows the version that is inside our system. In our case it's 4.0.3

Now we can try to run Varnish by executing following command:

sudo service varnish start

...and you should see the error - varnish failed to start. Why's that? Read next section about SElinux policy. If you don't see the error you can move forward to next part of this tutorial.

SElinux policy for Varnish on CentOS

The problem is that Varnish in version 4 requires some additional permissions in order to run under CentOS. We have SElinux enabled by default. We shouldn't not disable that (which is sometimes common practice in such situations). We will add the rule to the policy, so Varnish can start without any problems.

First of all we need the tool called audit2allow. Let's install it:

sudo yum install policycoreutils-python -y

Now, try to start Varnish one more time. It should give you fail error. After that execute following command:

cd ~/sources
mkdir varnish
cd varnish
sudo grep varnishd /var/log/audit/audit.log | audit2allow -M varnishd2
sudo semodule -i varnishd2.pp

We will use our sources directory that we created at the beginning of our series. You can of course use any directory you want.

Next we look through audit.log on CentOS and try to find varnishd occurrences. audit.log file contains blocked activities by SELinux. From these blocked activities we will generate module -M for SELinux that will grant access to required permissions.

Note 2 at the end of varnishd2  name. We need to add new policy. varnishd  policy already exists in SELinux policies.

Enable the module and try to start Varnish service again. In our case it didn't work out, and we still got fail error. So we did it again:

sudo grep varnishd /var/log/audit/audit.log | audit2allow -M varnishd3
sudo semodule -i varnishd3.pp
sudo service varnish start

and Varnish started without any errors!

Setup firewall for Varnish

So now it's the time to play around with Varnish. The installation only is not enough to be able to have Varnish fully working. Let's start by adding some firewall rules in order to be able to test Varnish on CentOS.

By default Varnish is running on port 6081 , but we don't have rules in our firewall, so you won't be able to see the actual result. Open firewall rules file and add these rules:

-A INPUT -p tcp --dport 6081 -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp --sport 6081 -m state --state ESTABLISHED -j ACCEPT
-A INPUT -p tcp --sport 6081 -m state --state ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp --dport 6081 -m state --state NEW,ESTABLISHED -j ACCEPT

It's the same rules like for instance for port 80 for Apache. For testing purposes we will enable to access our website by two different ports - 80 (by Apache) and 6081 (by Varnish). Port 80 will work without caching, port 6081 will serve cached content.  At the end we will change our setup, so only port 80 will be available and it'll be Varnish port, not Apache like for now.

Restart the firewall to apply the rules:

sudo service firewall restart

And now you can try to test it by calling http://example.com:6081

You should see the error, and that is perfectly fine.

Test with microtime()

Now it's time to test Varnish cache. The easiest way is to create PHP script with following content in /var/www/example/com/htdocs :

<?php
#/var/www/example.com/htdocs/varnishtest.php
echo microtime();

then try to run it in your browser: http://example.com/varnishtest.php and refresh couple of time. On each refresh you should see different contents.  Now it's time to make http://example.com:6081/varnishtest.php  working.

First of all we need to configure backend for Varnish. Edit file /etc/varnish/default.vcl

You should see backend default there. By default port is set to 8080, but our Apache is working on port 80, so we need to change that:

vcl 4.0;

# Default backend definition. Set this to point to your content server.
backend default {
    .host = "127.0.0.1";
    .port = "80";
}

Exit editor and reload Varnish:

sudo service varnish reload

Try to reload the page couple of times. At second refresh you should see exact same thing as before. So result from microtime()  will not change. If you achieve such effect, you can be happy! Varnish is working correctly:D

If you will see at response headers you will get additional headers. Most important is Age  header. It should have value > 0. By that you can see that Varnish is caching your website correctly. But if you still can see that it's not working (content is still changing) you should see next section of this article.

Troubleshooting Varnish Cache

There are plenty of reasons why Varnish might not cache your website. You must know that it's not easy to diagnose what's wrong with your cache/website. Here are few tips that might be helfpul

ModSecurity is running

If you have ModSecurity enabled there might be an issue with caching. Or you might see 403 pages sometimes. There's an easy fix for that. You need to disable 960020 rule. In order to do that you must edit the httpd-security.conf  file that we created during Apache setup and add line  so the file will look like that:

# File location: /usr/local/apache2/httpd/conf/extra/httpd-security.conf

#...
<IfModule security2_module>
      Include conf/crs/modsecurity_crs_10_setup.conf
      Include conf/crs/base_rules/*.conf
      # Include conf/crs/experimental_rules/*.conf
      # Include conf/crs/optional_rules/*.conf


      #.... configure ModSecurity here
      
      #Add this line: 
        SecRuleRemoveById 960020
</IfModule>
#... Rest of the file

Remember that you can't do it in .htaccess. It's important to remove the rule after you include rules from OWASP!

After that restart HTTP daemon and check if page is caching or not:

sudo service httpd restart

Missing caching header

Another issue might be connected to missing caching header. You can set how long each page can be cached by setting the cache file. Try to change varnishtest.php file like that:

<?php
header( 'Cache-Control: max-age=60' );
echo microtime();

It means that this file should be keep in cache no longer than 60 seconds. Try to refresh the page and see if it brings any help or not?

Double caching header

If You are using mod_expire for Apache, there might be situation where you have two caching headers. One is set from PHP script, second one by Apache. The easiest way to check it is to see header in Chrome for instances. If You will see two caching headers like that:

Cache-Control:max-age=10
Cache-Control:max-age=0

You need to adjust your Vhost file for file caching. For instance sometimes you can find such settings in your httpd.conf or Virtual host file:

<IfModule mod_expires.c>
    #If you are using caching for static contents this line must be present, don't remove it!
    ExpiresActive on

    #Default cache is set to 0, comment it out
     ExpiresDefault 0
   
     #Disable cache per content type, comment it out
     ExpiresByType application/json                      "access plus 0 seconds"
     ExpiresByType application/xml                       "access plus 0 seconds"     ExpiresByType text/html                             "access plus 0 seconds"
</IfModule>

If you plan to use Varnish cache, you should control your cache inside your application, and don't rely on Apache cache when it comes to website content, API endpoints etc. You can still cache static assets like images, scripts or styles with Apache. Double Cache-Control header will prevent Varnish from caching PHP scripts.

Those cookies...

This is really important. By default Varnish will NOT cache any output if there are any cookies. That's how usually authentication mechanisms works for logged in users. If you are logged in you want to see most recent content instead of cached one. Make sure that you are not sending any Cookies. If you have some problems with Tracking cookies like from Optimizely or Google Analytics  you an always unset them in Varnish. Try this approach. Open default.vcl  file where you set the backend port. Change sub vcl_recv section to something like this:

sub vcl_recv {
	 if(req.http.Cookie){
    set req.http.Cookie = ";" + req.http.Cookie;
    set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
    set req.http.Cookie = regsuball(req.http.Cookie, ";(PHPSESSID)=", "; \1=");
    	set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");

    if (req.http.Cookie == "") {
        	unset req.http.Cookie;
    }
  	}
}

This is example taken from Varnish webiste. It will unset all cookies but not the cookie contains PHPSESSID. Save changes in file and restart varnish:

sudo service varnish restart

Check your page now.

Nothing helps 🙁

If nothing from above helped, you need to find the issue. There is also great documentation that comes from Varnish and you can see Varnish troubleshooting guide. If still nothing helps you can post comments here below, use StackOverflow or some forums. We learned that Varnish sometimes can be pain in the a** and it's hard to debug why something doesn't work...

Purging Varnish Cache

Great! Now we have our cache up and running! We change something in our script, refresh the page, but hey! There is still cached content! So now the question is, how to clear Varnish cache?

Method 1 - varnishadm

Let's introduce first way to purge Varnish cache. First method is with usage of Varnish tools. One of them is varnishadm which will help us to clear the cache. Let's try it:

sudo varnishadm ban 'req.http.host == "lamp.com:6081" && req.url ~ "/varnishtest.php"'

This is very basic usage. We will use ban command from varnishadm to add ban in Varnish settings. In ban command we can use any parameters that are available in Varnish settings, but here are two basic things that you will need.

First we need to check host req.http.host == "lamp.com:6081" . This is very useful if we have multiple domains on our server. We need to be sure that we are clearing the cache only for one, particular website, not for all websites. Port is necessary for now, because we don't have Varnish on port 80 now.

Next is the most important thing - URL. req.url ~ "/varnishtest.php"  We can use here any regex expression, which is really helpful. This command will clear only one particular URL. You can use req.url ~ "^/article" to ban all URLs that starts with /article phrase. Sometimes helpful command is req.url ~ ".*"  that will clear cache for every page and file.

Short note about clearing the cache - if you have really high traffic on your website you need to be very careful when it comes to clearing Varnish cache. You don't want to kill your website with really high traffic that will goes straight to Apache for instance. So sometimes it's better to wait for Varnish cache to expire by itself instead of clearing the cache manually. It all depends on your website and architecture.

Method 2 - PURGE request

OK, but how to clear the cache from PHP script for instance? It's not safe to execute shell commands from script directly. Therefore Varnish comes with another way of clearing cache - PURGE request.

You can send PURGE command with cURL, for instance:

curl -X PURGE http://example.com:6081/varnishtest.php

ModSecurity fix

If you have ModSecurity enabled, you will get 403 error. That's because PURGE request method is not allowed. We need to edit OWASP settings file located in /usr/local/apache2/conf/crs/modsecurity_crs_10_setup.conf  and we need to find the line with tx.allowed_methods . Then we need to add PURGE request, so it will look like that:

setvar: 'tx.allowed_methods=GET HEAD POST OPTIONS PURGE', \

Restart Apache to load new policy:

sudo service httpd restart

Then try to execute curl command from above. You should get contents of our test file instead of an error.

 

As you can see, you will get our test file contents, but it won't clear Varnish cache for this URL. We need to prepare Varnish to enable PURGE requests. Edit VCL file located in /etc/varnish/default.vcl  We need to add following section:

acl purge {
        "localhost";
        "192.168.77.1"/24;
}

sub vcl_recv {
        #....

        if (req.method == "PURGE") {
                if (!client.ip ~ purge) {
                        return(synth(405,"Not allowed."));
                }
                return (purge);
        }
        #.....

}

In acl purge  we define addresses that can send PURGE requests. Localhost is obvious, if we want to send requests from PHP scripts. But sometimes we need to clear the cache from external IP address. So it's good to add it here. If you need to clear the cache only from PHP script, and do not send it from outside, leave localhost  only.

In sub vcl_recv  section we check if request method is PURGE, if so, we need to check if client.ip  is within allowed list for PURGE requests.

Try to send request again. It should clear the cache.

VCL

So what is this magic VCL file? In short words it a file designed to define request handling and caching policies for Varnish. We can specify additional rules, like removing request cookies, or disable the cache if user is logged in. Configuration strongly depends on your website or CMS you are using. You can use ready VCL files like this one - mattiasgeniar/varnis-4.0-configration-templates. This is really cool and well commented example VCL file. You can find multiple tricks for Varnish there, and it's really worth checking. It has configuration for WordPress and Drupal, but it should also work well with Symfony2 or Zend Framework. You can also find more details about Varnish and Symfony2 on their website.

It's all really depends on your website, used CMS or framework, installed plugins etc. So it's really hard write one VCL for everyone. You need to do it by trial and error or googling. But really good start is to check VCL file mentioned above.

Varnish settings per domain/Virtual Host

What if we have multiple websites and we want to have separate configuration per website? For instance one website is based on Drupal, second one is just pure PHP. Varnish settings for global is not possible due to some conflicts. How to deal with such issue?

VCL file and language is pretty powerful and we can resolve such conflict It's not that easy, but doable. Edit  default.vcl  file - /etc/varnish . Take a look on example file below:

vcl 4.0;

backend default {
    .host = "127.0.0.1";
    .port = "80";
}

# ...

sub vcl_recv {

    # ... global rules for all websites here

    # Return 404 error error in requests without Host. It's optional but helpful
    if(! req.http.Host){
        return (synth(404, "Host header missing"));
    }

    if(req.http.Host == "example.com"){
        include "/etc/varnish/example.com.vcl";
    }else if (req.http.Host == "secondexample.com"){
        # you can also place normal rules here, no need to include files
    }
}

Varnish has include command. With this command you can include pieces of code inside different files. So you can have kind of per domain settings. It's not perfect, because you need do it inside sub directive. It's not possible to include whole sub vcl_recv sections  etc.

With this there comes second disadvantage. If you have larger pieces of instructions per domain and you have to modify another sub  like vcl_backend_response , you need to make similar if statement inside each section.

You can modify the if as you wish. But remember, that you can also use ~  to mark domains with subdomains + www etc.

In this example there's no port. If you are testing Varnish on different port, so it's not running on port 80, you need to specify the port as well. if(req.http.Host == "example.com:6081")

This is not perfect solution, but it's working. In real life you probably won't have to create statement like that. Most of the rules can be applied globally. But if you need to make an exception for particular domain, now you know how to do that;)

Varnish useful tools

Here are couple of really useful tools that comes with Varnish.

varnishstat

This command will give you an access to Varnish statistics. You can see lot of data there, but really important are cache_hit  and cache_miss . If you have lot of cache_miss , that means, that your content is not served from Varnish, but from backend like Apache. It also means that something is not OK and you should really look through your configuration and caching.

If you have great number of cache_hit  and not that many cache_miss  it's perfectly alright. When Varnish cache expire for particular URL it goes directly to backed and cache_miss  value is incremented.

varnishtop

varnishtop is yet another useful piece of software you can find in varnish toolbox. It shows the URLs that have most hits to your backend. It is also great debugging tool. The cool thing is that you are able to filter the logs. There are so many filters that it's not possible to list all of them here. But there is great documentation for this tool, that you can find here.

Here are two examples, that are really useful

  • varnishtop -i ReqURL  it will show list of URLs that comes from Varnish most frequently 
  • varnishtop -i BareqURL  it will show you list of URLs that goes to your backend the most. This is really great command, because you can easily catch most frequent misses.

varnishlog

This is another great tool. It logs all requests in real-time with tons of data. It's great for debugging why particular URL doesn't work (ie. it's not cached, but it should). It has very similar syntax to varnishtop, so if you master varnishtop, you can use varnishlog without any problems.

By deafult, Varnish doesn't log to file. This is really important. Why's that? Think about such situation. You have an API with some endpoint or one particular page in WordPress. This endpoint/page has heavy load, let's say 1000 requests per second. Apache can't handle it, so you install Varnish and suddenly everything is OK. But you don't know what's going on in logs. In Apache access.log you have only few requests which has cache_miss in Varnish. But Varnish doesn't log anything to file. And to be honest - it's correct behavior. First of all, it will speed up entire service, because you don't need to waste the time for I/O operations on your hard-drive every requests. Second of all, digging in really big access.log is pain in the a** and it's difficult to find anything there.

But if you need logging to file, because let's say, you don't have some really high traffic you can enable logging to file by executing:

sudo service varnishlog start

and if you want to run it on system start

sudo chkconfig varnishlog on

Varnish logs can be found by default in /var/log/varnish . But please, be aware, that it will generate lot of writes to your hard-drive and log file will increase it's size pretty fast on heavy load websites. So this solution is not recommended.

varnishncsa

It's another useful tool for logging. It's different output from varnishlog. varnishlog will log every detail into your log about the query. If you need logs similar to for instance Apache logs, you should use varnishncsa command. By default it's disabled just like varnishlog. If you want to log to file you should start the service varnishncsa and add it to autostart with chkconfig.

varnishncsa

Performance

OK, awesome, Varnish is super cool , my websites goes blazing fast! But hey! Is it possible to speed it up even more? Well, theoretically - yes. There are few tips that can increase your throughput. But you must remember one thing - the MOST important part is to have high hit rate. If you have low hit rate and high miss rate, this tips won't help you that much. So remember, achieve high hit rate first. Then take care of Varnish tuning:)

Few tips before you start tuning up Varnish:

  • There is no one common ctrl+c ctrl+v method that will improve results on all machine
  • Change one thing at the time and then WAIT for results if it helps or not
  • Monitor statistics with Varnish tools. They will tell you whether the change will work or not
  • Remember that Varnish is blazing fast by default. If you don't have enormous traffic, you'll probably don't need these tips.
  • Increasing particular values too much can't make your hardware die. It will eat up all your CPU or RAM resources, so please be careful.
  • Monitor, monitor and monitor your system:)

OK, let's start tuning. First, i suggest to read Varnish book and chapter about tuning. It will help better understanding of what's going on.

Varnish has configuration file that contains important options that will help you increase Varnish power. Edit the file by sudo vi /etc/sysconfig/varnish

Now Varnish supports couple of ways how to configure Varnish daemon. They are all present in this file so figure out which one are you using. By default they are called Alternative 1, Alternative 2 and Alternative 3.

In my opinion Alternative 3 is the best, because you only need to change variable value instead of whole DEAMON_OPTS  string. In Varnish 4 it's enabled by default, rest of alternatives should be commented out. If not, comment out Alternative 1 and Alternative 2, and uncomment Alternative 3. But if you want you can stick to any Alternative you wish. It's just the matter of passing configuration to Varnish daemon.

You can set Varnish port there, secret file etc. But among the settings you will find threads section, and storage type. Let's start with storage type.

Files or RAM for cache? 

# # Cache file size: in bytes, optionally using k / M / G / T suffix,
# # or in percentage of available disk space using the % suffix.
VARNISH_STORAGE_SIZE=256M
#
# # Backend storage specification
VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}"

Now it's up to You and your server resources how to set these values. By default Varnish will use your RAM as a storage. It also can use the file method, but keeping things in RAM is lot faster than reading them from hard-drive, especially if it's not SSD. But if you have little RAM on your server and lot of URLs to cache, use file , instead of malloc .

So this is one thing that will speed up your Varnish - using RAM. How many? Larger value is definitely better, but keep in mind, that it's not strict value. Varnish will add overhead per each object in cache (around 1kb). So if you have large amount of objects, it will increase storage size. You need to play around with this value a bit and check how much is enough. Keep in mind that you need to have enough free RAM for other processes like httpd or php-fpm.

If you will use RAM check if the storage size is big enough to handle all your requests. Use this command:

varnishstat -1 | grep n_lru_nuked

If the number of LRU nuked objects is greater than 0, then it means that or your storage size is not big enough, or you have something wrong with caching. For instance you try to cache assets or images for very long period of time.

Threads settings

Second thing that will increase Varnish performance are threads. In the same file you will find settings for them:

# # The minimum number of worker threads to start
VARNISH_MIN_THREADS=50
#
# # The Maximum number of worker threads to start
VARNISH_MAX_THREADS=1000

There are multiple options for threading in Varnish, but you should mostly play around with these two values. You can increase both values here, but you should keep withing the Varnish guidelines. Here is explanation how Varnish threads works from Varnish documentation.

When a connection is accepted, the connection is delegated to one of these thread pools. The thread pool will further delegate the connection to available thread if one is available, put the connection on a queue if there are no available threads or drop the connection if the queue is full.

Read here how to calculate how many threads you should run to not to kill your server.

Varnish and security

There are various of methods how to secure more your Varnish. We will not cover them here. But to make things clear - most important thing is to secure your purging requests. Do not allow anyone to purge your cache from outside. He can purge entire domain, and if you have huge traffic - it can kill your website when all requests will come to your backend server.

You can install also firewall for Varnish. It's really similar to ModSecurity for Apache. You have basically two options:

They are just additional rules in VCL that will block unwanted requests.

Our opinion about that? They are OK, but we like to keep Varnish VCL as simple as possible and do not add additional overhead to parsing requests. In addition we have ModSecurity for Apache installed, and it does basically the same job. So is it worth to have two security layers and add exception rules to both of them? We will just stick to Apache ModSecurity:)

Varnish on port 80

Now it's the most important part. For now we use Varnish on "test" port 6081. Before you will switch Varnish to port 80, make sure, that IT WORKS. If requests are cached correctly etc. It's important to put working Varnish as an entry point to your server, instead of just additional layer that will forward all your requests to Apache.

First you need to pick the new port for Apache. Common practice is to use 8080, but this port is often used for different tools like SOLR. We usually set port 81 for Apache. After you will pick a port number, you need to edit few files.

Note that if you are not following our tutorial from start, your files might be located elsewhere. 

httpd.conf

We need to change main port for Apache in httpd.conf file

# File located in /usr/local/apache2/conf/httpd.conf
Listen 81

All VirtualHosts files

All Virtual Hosts files also contain port. You need to change every single one file with new port.

#All files with .conf extension located in /usr/local/apache2/conf/vhosts
<VirtualHost *:81>
        ServerName example.com

Firewall rules for new Apache port (optional step)

This is optional step. We have working firewall on our server. If you will need to access website directly (not through Varnish cache) you can open new port. It's usually useful when it comes to debugging. If this is your app issue or is it Varnish cache fault. If You need to enable new port, add appropriate lines inside firewall rules.

# Firewall rules are in /etc/iptables.firewall.rules
-A INPUT -p tcp --dport 81 -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp --sport 81 -m state --state ESTABLISHED -j ACCEPT
-A INPUT -p tcp --sport 81 -m state --state ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp --dport 81 -m state --state NEW,ESTABLISHED -j ACCEPT

Restart firewall to load new rules

sudo service firewall restart

Varnish daemon settings

Next thing is that we need to change Varnish daemon settings to listen on port 80.

# File is located in /etc/sysconfig/varnish
VARNISH_LISTEN_PORT=80

Varnish VCL file

Last thing is to change VCL file backend port, and update all rules to remove previous port 6081

# File is located in /etc/varnish/default.vcl
backend default {
    .host = "127.0.0.1";
    .port = "81";
}

# Remove :6081 if present in host
if(req.http.Host == "example.com"){

Remove old Varnish port from firewall

Remember when you added port 6081 to firewall in order to test some stuff? You will not need that port to be open.You can remove these lines from your firewall rules, and restart the firewall.

Restart httpd and Varnish

Once you change all files it's time to restart all daemons and check if everything works as expected.

sudo service httpd restart
sudo service varnish restart

Now check your website. It should be powered by Varnish now!

What's next?

Lengthy tutorial, wasn't it? But now your websites should be blazing fast! Remember to check it from time to time, if you have good hit/miss ratio:)

As always you can do everything here with Ansible and our repo, which you can find on GitHub.

Now you should be in place where you have secure CentOS, working LAMP and Varnish on top of everything. What's next? I believe that we should add nginx to our setup to keep everything nice and slim, and after that start installing some real useful things like Redis, RabbitMQ and add some good tools to PHP etc. See you next time!

How to configure SSH on CentOS

How to configure secure SSH on CentOS

Hello everyone! In this tutorial I will show you how to increase server security by tuning up configuration of SSH.

Before you begin

There are basically two requirements for this tutorial:

  1. You need to have working SSH keys. You need to be able to login to your server by using them. After completing this tutorial, SSH keys will be the only way to access server. If you won't be able to login by using them, well, you will lose access to your server. In order to add user and configure keys you can follow this tutorial.
  2. Make sure that at least one of the users is in wheel group (has access to sudo). Root should not have access to login via SSH. So if you will block this option and you won't have any sudo user, you won't be able to do much on the server. Follow this tutorial in order to configure sudo.

Disable password authentication for SSH on CentOS

Login to Your server/Vagrant Box and open SSH daemon configuration file:

sudo vi /etc/ssh/sshd_config

Now we need to find the line for password authentication and change it to:

PasswordAuthentication no

Unfortunately, disabling this option can still lead to password authentication by using PAM-based authentication. In order to fully disable authentication with password, make sure that PAM is also disabled:

ChallengeResponseAuthentication no

Also we need to make sure that this line is uncommented. It will  enable SSH login by using public key:

PubkeyAuthentication yes

Save the file and exit from the editor. In order to apply changes, you need to restart SSH daemon:

service sshd restart

After that, try to open new SSH session in new window. Do not logout from your current session! If you won't be able to login with new session, you can undo the changes with existing session. If you will be able to successfully login, you can proceed.

How to secure SSH on CentOS even more?

There are still some things that will help you improve SSH security. Edit the same configuration file as before. Below You will find the configuration options that I usually use for SSH.

Disable root login

PermitRootLogin no

This option will disable root login via ssh. So it means that from now on you won't be able to login to your server as root via ssh.

Allow only specific users to be able to login via SSH

AllowUsers developer

By default you are able to login as any user that is created inside the system. It can be easily limited to particular users. Just give space separated list after AllowUsers. It might not be present in your config, so you need to add this line (for instance at the end of the file).

AllowUsers developer vagrant

Enable protocol 2 for ssh

Protocol 2

This option is set by default in most CentOS installation, but just make sure that there's no version 1 instead. It's less secure protocol.

Ignore rhost

IgnoreRhosts yes

It will disable insecure access via RSH.

Disable login for users with empty passwords

PermitEmptyPasswords no

This line will disable login for users that have empty passwords. Make sure that your account has password set, before changing that!

Enable strict mode for ssh

StrictModes yes

SSH will check users's permission in their home directory before accepting login. It should be set to yes because users may leave their directory or files world-writable. Again, this might be tricky. It's the best to change that, restart SSHD daemon and try to login from new session. If you have any problems, you can undo this change with existing session. If you have any issues with that, try to set valid permissions for your .ssh directory and files inside. Also set valid username and group for your files:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/*
chown -R YOUR_USERNAME:YOUR_USERNAME ~/.ssh

Disable other authentication methods

GSSAPIAuthentication no
KerberosAuthentication no

If you don't plan to login with GSS API, or Kerberos you can disable them as well.

Disable X11 Forwarding

X11Forwarding no

If you don't use X11 you can safely disable it as well.

Show last login

PrintLastLog yes

Nice feature is to show last successful login after you will login via SSH.

Restart SSH daemon

Remember that after any changes inside the file You need to restart sshd daemon:

sudo service sshd restart

SSH crypto

In addition to changes above that should be applied, you can increase SSH security even more by configuring ciphers and available algorithms (thanks to @Amar for the suggestion:)

This is usually safe to execute, but you must remember that not all algorithms are supported by various tool. Here you can find great chart showing, which tools support given algorithms. But let's be honest, most of you is probably using OpenSSH which supports all the changes I will present here. However if you are using different tool and you won't be able to login to your server, check with the page and enable additional algorithms.

These config options will probably not be listed in your config file. You need to just add them somewhere, like at the end of the file.

Configure server authentication

HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key

Server must confirm the identity to the client. There are bunch of algorithms available, but this is the list of most secure.

This might be present in your configuration file, also there might be more not commented lines with HostKeys. Leave only these two enabled and comment out the rest.

Configure key exchange

KexAlgorithms curve25519-sha256@libssh.org

There are many more key exchange algorithms, but this is probably the most secure.

Configure ciphers

Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com

Ciphers are used to encrypt the data. As with key exchange, there are multiple algorithms. These are the safest.

MACs - Message Authentication Codes

MACs hmac-ripemd160,hmac-ripemd160-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,umac-128@openssh.com,umac-128-etm@openssh.com

MACs are used for data integrity. Again, line above contains the safest algorithms only.

After these changes, don't forget to restart sshd daemon.

Easier way?

You can use our Ansible LAMP on Steroids project to make configuration of your server easier!

If you don't know what Ansible is, check our tutorial first.

Clone our repository and setup your server faster with LAMP on steroids.

CentOS, users, groups, sudo and SSH keys

Create sudo users and groups in CentOS

I will show you how to organise groups and users in CentOS.

As an example we will create account for user named developer. The purpose of this account is:

  • logging via SSH instead of using root account
  • access to sudo command for management tasks
  • write access to website files
  • read access to logs

Create list with available groups

If you are on fresh system it's handy to create a list with all available groups. It's also handy to check which users belongs to that groups. It might be helpful after some time when you need to decide if given group was at the beginning or can it be safely removed. getent command can help you with that.

getent group > /etc/initial-group-list
cat /etc/initial-group-list

Remove group from the system

If you need to remove group from CentOS simply use following command:

sudo groupdel NAME_OF_THE_GROUP_TO_DELETE

Create new group

In order to create group you need to use groupadd command:

sudo groupadd NAME_OF_THE_NEW_GROUP

I usually add group named www (or www-data, whatever works for you). To this group I add php daemons, nginx workers etc. It makes life easier with writing to files. In order to create such group execute following command:

sudo groupadd www

Create list with available users

Same like with group, I like to have list of initial users. In order to create such list you can use getent too:

getent passwd > /etc/initial-users-list
cat /etc/initial-users-list

Delete user from CentOS

In case you would like to remove any user from the system, use following command:

sudo userdel -r USERNAME_TO_REMOVE

-r flag will remove also his home directory. If you wish to delete the user, but to keep his files, omit this flag.

Create new user in CentOS

Let's create new user developer that we mentioned at the beginning:

sudo adduser developer

and create the password for his account:

sudo passwd developer

If you want to add developer user to www group created before use usermod command:

sudo usermod -g www developer

If you want to add this user sudo powers (and you should if you want to use this user instead of root), add it to wheel group. wheel group is special group in CentOS configured in sudoers file. Whoever belongs to this group can have sudo powers.

sudo usermod -g wheel developer

Optional parameters to useradd command

There are lot of additional parameters for useradd command but there are two especially useful.

First one is helpful when you don't want to create user home directory. It means that user will not have it's own place under /home directory to store it's files. This option is helpful when you are creating user for system service like Apache httpd for instance. So in order to create user with no home directory use --no-create-home:

sudo useradd httpd --no-create-home

Another useful feature is to specify shell of given user. It's nice if you want to cut of possibility to login to the system via SSH for instance. Add --shell /sbin/nologin to disable login for given user, like so:

sudo useradd httpd --shell /sbin/nologin --no-create-home

You can use --shell and --no-create-home parameters separately:)

 

How to setup SSH keys for new created user?

Each user should have RSA key-pair. It makes life easier and you should use it if you want to login to different servers, use GIT etc. In order to create such user key-pair you first need to login to user you created. Most probably you are using root account to execute all commands, but you should never ever login via SSH as a root.

It's much better to create separate user for system management and use only this account. Login via SSH to your server to account your created. In my case it's developer user so my command looks like this:

ssh developer@IP_OF_THE_SERVER_HERE

Once you'll be logged in (after providing the password), you can create RSA key pair. Execute following command:

ssh-keygen -t rsa -b 4096

-t rsa means that it will be RSA key, but this is standard for creating SSH keys. Fun part is with strength of the key -b 4096. By default it's 1024 bits, but to make it harder to break I usually provides 4096. It's not necessary, but you should do that. Some services requires key length to be minimum 2048, but it's better to create even longer one.

Generator will ask you some questions, but you should generally confirm them with enter and leave the defaults. When it comes to SSH on the server, I usually don't set the password. It makes life easier in automated scripts etc.

After that private and public key should be generated as expected. You can find them in ~/.ssh directory.

Add authorized key to user

In order to login with SSH keys to the server, instead of using password you need to add authorized key to developer user. In my opinion it's must have feature as using password login is super risky. Again, been there, done that, I was hacked, even when my password was strong. With SSH logging even strongest bruteforce attack will fail:)

You need to add your key to ~/.ssh/authorized_keys on the server. If You have ssh-copy-id command available just execute:

ssh-copy-id developer@IP_OF_YOUR_SERVER

Make sure that you are executing this command from your computer, not from the server. If you don't have SSH key created locally, you can generate it in the same way as on the server, by using ssh-keygen command.

If uou don't have ssh-copy-id available (for instance from Windows), you can do it manually.

ssh developer@IP_OF_YOUR_SERVER
cd ~/.ssh
vi authorized_keys
//Press "i" to enter in input mode, paste there your code (usually it's right click of the mouse) and :wq (colon, w, q) it will save and quit from vi 
chmod 600 authorized_keys

So here how it goes:

  1. ssh to the server as usual with password.
  2. Change location to .ssh directory.
  3. Create authorized_keys file with vi
  4. Paste there your local public key, save the file and quit
  5. Set permissions on authorized_keys.

Test ssh login with keys

Now You can try to log in with Your key.

ssh developer@IP_OF_YOUR_SERVER -i path/to/your/PRIVATE/key/file

You shouldn't be prompted for your account password!

Easier way?

If you don't want to spend your precious time executing each of these commands by hand, you can use Ansible and our LAMP on steroids project to speed things up!

If you don't know what Ansible is - you can read our tutorial about it here.

LAMP on steroids project is available on GitHub here.

Iptables for CentOS

How to secure server with iptables?

Hi there! In this tutorial I would like to show you how to increase server security by using iptables as a firewall. To be honest, not many people are actually using iptables or any firewall. I think that this is bad practice, because you they allow all traffic to go in and out. You should always limit the possible entry points to your server.

Firewalld vs iptables

Since CentOS 7, we have new tool called firewalld. This is not actually an alternative to iptables. firewalld is a wrapper for iptables. Many people say, that it's easier to use than iptables, but to be honest I believe that it's not flexible enough. Maybe I'm wrong, but I'd love to see some advanced example, how to transform iptables rules below to firewalld 🙂  If you want to use firewalld instead of iptables, unfortunately you need to read different tutorial. Here is great article about firewalld from DigitalOcean.

How to install iptables on CentOS7?

Before we will install iptables, we need to get rid of firewalld first :

sudo yum remove firewalld -y

Next, we can install iptables:

sudo yum install iptables iptables-services -y

iptables-services is simple script that will help us save and restore firewall rules.

Secure iptables rules for CentOS

First, let's check if there are any rules by executing following command:

sudo iptables -S

If you will get following output:

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

It means that you allow all traffic, both incoming and outgoing to your server. However if you have anything more than output above, copy it to separate file as a backup.

The easiest way of adding rules is by editing iptables rules file. Open the file, or create one if it doesn't exists:

sudo vi /etc/sysconfig/iptables

I will describe whole file line by line, but at the bottom of this post you can find whole content that I'm using for iptables.

Opening and closing tags

*filter

File must contains two indicators:

  • start of the ruleset *filter
  • end of the ruleset COMMIT

You need to have both in order to get iptables configured properly. Between these two lines, you can add iptables rules.

Clear all existing rules

-X
-F
-Z

At the very beginning I'd like to clear whole rules. In other words - enable all traffic. The reason is that I want to be able to execute that file over and over again, and I will always set the rules that I have in file. No other rules will be applied (for instance rules added by command line).

Allowing loopback

-A INPUT -i lo -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
-A INPUT -d 127.0.0.0/8 -j REJECT
-A OUTPUT -d 127.0.0.0/8 -j REJECT

Next thing is to allow all loopbacks. Those are local connection and blocking them might cause errors in some connections. In addition we will block those which doesn't use lo0.

Keep established connections

-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

All connections that are active now, should remain untouched. It will prevent from interruption of services.

PING command

-A OUTPUT -p icmp --icmp-type echo-request -j ACCEPT
-A INPUT -p icmp --icmp-type echo-reply -j ACCEPT
-A INPUT -p icmp --icmp-type echo-request -j ACCEPT
-A OUTPUT -p icmp --icmp-type echo-reply -j ACCEPT

In most cases you will need to be able to ping server. These rules will allow two things. First - you'll be able to ping your own server. Second - you will be able to execute ping from your server. Both are usually needed and quite useful.

Protection from PING of Death attack

-N PING_OF_DEATH
-A PING_OF_DEATH -p icmp --icmp-type echo-request -m hashlimit --hashlimit 1/s --hashlimit-burst 10 --hashlimit-htable-expire 300000 --hashlimit-mode srcip --hashlimit-name t_PING_OF_DEATH -j RETURN
-A PING_OF_DEATH -j DROP
-A INPUT -p icmp --icmp-type echo-request -j PING_OF_DEATH

Ping is cool, however you might get attacked with Ping of Death attack. Here is simple protection.

Prevent some nasty attacks

-N PORTSCAN
-A PORTSCAN -p tcp --tcp-flags ACK,FIN FIN -j DROP
-A PORTSCAN -p tcp --tcp-flags ACK,PSH PSH -j DROP
-A PORTSCAN -p tcp --tcp-flags ACK,URG URG -j DROP
-A PORTSCAN -p tcp --tcp-flags FIN,RST FIN,RST -j DROP
-A PORTSCAN -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP
-A PORTSCAN -p tcp --tcp-flags SYN,RST SYN,RST -j DROP
-A PORTSCAN -p tcp --tcp-flags ALL ALL -j DROP
-A PORTSCAN -p tcp --tcp-flags ALL NONE -j DROP
-A PORTSCAN -p tcp --tcp-flags ALL FIN,PSH,URG -j DROP
-A PORTSCAN -p tcp --tcp-flags ALL SYN,FIN,PSH,URG -j DROP
-A PORTSCAN -p tcp --tcp-flags ALL SYN,RST,ACK,FIN,URG -j DROP
-A INPUT -f -j DROP
-A INPUT -p tcp ! --syn -m state --state NEW -j DROP

This is really nice piece of rules that will prevent port scanning, SYN flood attacks, invalid packages, malformed XMAS packets, NULL packets, etc.

UDP traffic

-A INPUT -p udp --sport 53 -j ACCEPT
-A OUTPUT -p udp --dport 53 -j ACCEPT
-A INPUT -p udp --sport 123 -j ACCEPT
-A OUTPUT -p udp --dport 123 -j ACCEPT

I enable usually only ports for outgoing traffic (from our server to outside world). There are two ports that I'd like to open:

  • 53 - DNS port. It's a must if you want to use curl or yum. If you will have it closed, you will not resolve any domain name.
  • 123 - NTP port. If you are using chrony or ntpd, you need to enable that port to allow NTP deamon synchronisation.

TCP traffic

# Open TCP ports for incoming traffic
-A INPUT -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp --sport 22 -m state --state ESTABLISHED -j ACCEPT
-A INPUT -p tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp --sport 80 -m state --state ESTABLISHED -j ACCEPT
-A INPUT -p tcp --dport 443 -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp --sport 443 -m state --state ESTABLISHED -j ACCEPT

# Open TCP ports for outgoing traffic
-A INPUT -p tcp --sport 22 -m state --state ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -p tcp --sport 80 -m state --state ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -p tcp --sport 443 -m state --state ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp --dport 443 -m state --state NEW,ESTABLISHED -j ACCEPT

With TCP it's more complicated, but it's not that hard. First, you need to think what traffic you need to access from your server (outgoing traffic). I usually allow only SSH, HTTP and HTTPS traffic. Yum requires HTTP and HTTPS ports for pulling new packages. You will need it also for wget or curl. SSH is not mandatory, but if you want to pull packages from git via ssh protocol, you will need it as well.

I usually enable the same for incoming traffic. If you have httpd or nginx installed, you need to enable port 80. If you are using SSL for HTTPS, you need to enable 443 also. In addition to these two ports you must enable port 22 for SSH. If you will block this, you won't be able to get access to your server!

Block everything else

-A INPUT -j DROP
-A FORWARD -j DROP
-A OUTPUT -j DROP

At the very end, before closing COMMIT tag I add these three rules. So everything that was not specified above will be dropped. Both incoming and outgoing traffic.

How to apply rules?

There are two ways how you can apply the rules. First, save the changes in iptables file. First method is not permanent method. It's good way of testing your firewall before saving them permanently. If anything will go wrong, you can just restart the server and you will have all traffic open. Make sure that you check SSH access with these rules. Log out and try to login after applying rules.

So non permanent way of applying rules is:

sudo iptables-restore < /etc/sysconfig/iptables

Try to check rules with iptables -S to see the difference:) Check if everything is working fine. If so, you can set them permanently. After each server restart, rules will be applied automatically.

sudo systemctl start iptables.service
sudo systemctl enable iptables.service

If you want to reload rules, simply edit the file, add what you need and restart iptables service:

sudo systemctl restart iptables.service

You can use our Ansible LAMP on Steroids project to make configuration of your server easier!

It is based on Ansible. If you don't know what Ansible is, check our tutorial first.

Clone our repository and setup your server faster with LAMP on steroids.

Whole content of iptables rules

*filter

# Clear all iptables rules (everything is open)
-X
-F
-Z

# Allow loopback interface (lo0) and drop all traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
-A INPUT -d 127.0.0.0/8 -j REJECT
-A OUTPUT -d 127.0.0.0/8 -j REJECT

# Keep all established connections
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

# Allow ping
-A OUTPUT -p icmp --icmp-type echo-request -j ACCEPT
-A INPUT -p icmp --icmp-type echo-reply -j ACCEPT
-A INPUT -p icmp --icmp-type echo-request -j ACCEPT
-A OUTPUT -p icmp --icmp-type echo-reply -j ACCEPT

# Protect from ping of death
-N PING_OF_DEATH
-A PING_OF_DEATH -p icmp --icmp-type echo-request -m hashlimit --hashlimit 1/s --hashlimit-burst 10 --hashlimit-htable-expire 300000 --hashlimit-mode srcip --hashlimit-name t_PING_OF_DEATH -j RETURN
-A PING_OF_DEATH -j DROP
-A INPUT -p icmp --icmp-type echo-request -j PING_OF_DEATH

# Prevent port scanning
-N PORTSCAN
-A PORTSCAN -p tcp --tcp-flags ACK,FIN FIN -j DROP
-A PORTSCAN -p tcp --tcp-flags ACK,PSH PSH -j DROP
-A PORTSCAN -p tcp --tcp-flags ACK,URG URG -j DROP
-A PORTSCAN -p tcp --tcp-flags FIN,RST FIN,RST -j DROP
-A PORTSCAN -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP
-A PORTSCAN -p tcp --tcp-flags SYN,RST SYN,RST -j DROP
-A PORTSCAN -p tcp --tcp-flags ALL ALL -j DROP
-A PORTSCAN -p tcp --tcp-flags ALL NONE -j DROP
-A PORTSCAN -p tcp --tcp-flags ALL FIN,PSH,URG -j DROP
-A PORTSCAN -p tcp --tcp-flags ALL SYN,FIN,PSH,URG -j DROP
-A PORTSCAN -p tcp --tcp-flags ALL SYN,RST,ACK,FIN,URG -j DROP

# Drop fragmented packages
-A INPUT -f -j DROP

# SYN packets check
-A INPUT -p tcp ! --syn -m state --state NEW -j DROP

# Open ports for outgoing UDP traffic
-A INPUT -p udp --sport 53 -j ACCEPT
-A OUTPUT -p udp --dport 53 -j ACCEPT
-A INPUT -p udp --sport 123 -j ACCEPT
-A OUTPUT -p udp --dport 123 -j ACCEPT


# Open TCP ports for incoming traffic
-A INPUT -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp --sport 22 -m state --state ESTABLISHED -j ACCEPT
-A INPUT -p tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp --sport 80 -m state --state ESTABLISHED -j ACCEPT
-A INPUT -p tcp --dport 443 -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp --sport 443 -m state --state ESTABLISHED -j ACCEPT

# Open TCP ports for outgoing traffic
-A INPUT -p tcp --sport 80 -m state --state ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -p tcp --sport 443 -m state --state ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp --dport 443 -m state --state NEW,ESTABLISHED -j ACCEPT


# Drop all other traffic
-A INPUT -j DROP
-A FORWARD -j DROP
-A OUTPUT -j DROP

COMMIT

What's next?

We secured our system with basic firewall. That will increase the security of our server. In one of the next episodes we will change the configuration of our SSH and therefore make it more secure.

As always You can use our Ansible playbook for faster provisioning of our server. You can find it on GitHub.

Install MySQL community server on CentOS

How to install and configure latest version of MySQL on CentOS?

Hi there! Today I want to show you how to install latest version of MySQL Community server (5.7.16) on CentOS 7. I will show you how to install it, set root password, configure server and optimize it for performance. Also I will show you how to create databases and assign users to them.

How to install MySQL on CentOS  in latest version

When you try to install MySQL on bare CentOS, you will actually install MariaDB (fork of MySQL) instead. It comes with version 5.5.* It's up to you what you want to use, but I prefer MySQL. Especially 5.7 version.  There is great performance boost and lot of new features added, comparing to 5.5 or 5.6.

If you have any data on MySQL, it's best to create a backupof data, before you will change anything. Here is a tutorial how to perform backup of MySQL databases.

Before you will actually install MySQL, make sure that MariaDB is not installed. You can just remove it. TIP: removing MariaDB will not remove any databases. It will remain on your hard drive.

sudo yum remove mariadb -y

So let's get to installation. First you need to enable MySQL repository. It's easy by adding following file to yum repositories:

sudo vi /etc/yum.repos.d/mysql-community.repo

and place there following content:

[mysql57-community]
name=MySQL 5.7 Community Server
baseurl=http://repo.mysql.com/yum/mysql-5.7-community/el/7/$basearch/
enabled=1
gpgcheck=1
gpgkey=http://dev.mysql.com/doc/refman/5.7/en/checking-gpg-signature.html

Save the changes. It will enable MySQL repository where from you can install latest version.

Installation is pretty easy:

sudo yum install mysql-community-server -y

That's it! You have MySQL installed in latest version!

Setting root password in MySQL

Setting root password for MySQL is important. You should not leave your database server unprotected, even if you only plan to use localhost connection.

Before you will start anything, make sure that MySQL is up and running:

sudo systemctl start mysqld

If this is fresh installation of MySQL (MariaDB or older version of MySQL was never installed) , MySQL will generate temporary password.

You can find it by using following command:

grep 'temporary password' /var/log/mysqld.log

You will get random password, for instance  something like wG]_xj8tus. This is current password to root account. Keep in mind that this is only temporary password. It will expire, so you need to change it as soon as possible.

In order to change root password I like to use mysqladmin tool. So execute following command:

mysqladmin -u root -pwG]_xj8tus password NEW_STRONG_ROOT_PASSWORD

You need to replace wG]_xj8tus with the password from grepping log file.  Note that there is no space after -p argument. Also replace NEW_STRONG_ROOT_PASSWORD with your new root password. Keep in mind, that it needs to be strong password, containing lower and upper case characters, numbers and special characters. MySQL 5.7 comes with plugin that validates password strength. It won't allow you changing password to something that is easy to guess or brute force.

If you just upgrading MySQL/MariaDB, your root password will be the same. You can of course change it with mysqladmin if you wish.

However if you are upgrading from MySQL/MariaDB version prior to 5.7 it's nice to execute following command:

mysql_upgrade -u root -p

Type your root password. It will check all the data and alter tables for 5.7 rules.

You might ask - how about mysql_secure_installation? You can also use this command for setting root password after installation. However when you install fresh version of MySQL 5.7 there is no test database, anonymous users etc. Everything is clean and secured.

Configuration of MySQL 5.7

I like to tune up default MySQL configuration. With 5.7 version, lot of defaults are set in order to provide high security and performance. However there are few things I like to change. Most of the stuff is stored under /etc/my.cnf file. MySQL is reading configuration from there during startup. You can also change it dynamically, but here I will focus only on this file.

Edit the file first:

sudo vi /etc/my.cnf

There will be some defaults, you can leave them as they are. Most probably, all options here will be missing, so you need to add them under [mysqld] section.

Temporary in-memory tables

tmp-table-size=32M
tmp-heap-table-size=32M

I increase the size of temporary in-memory tables created by MySQL. If you are doing lot of advanced GROUP BY queries, you will probably have to increase it. Make sure that you are setting both to the same value. If one will be lower, second one will be limited to lower value as well.

Query caching

Query caching is disabled by default. It caches result sets from most frequent queries to database. It usually give nice performance boost to MySQL server:

query-cache-type=1
query-cache-size=32M

InnoDB buffer

Also another performance boost is creating buffer for InnoDB pools. It will keep the data in RAM, so reads will be much faster. Make sure to not to set these values higher than your RAM limits. It should be large enough to store as much data as possible.

innodb-buffer-pool-instances=1
innodb-buffer-pool-size=128M

Logging slow queries

Slow queries can kill your server. It's nice to know earlier that something is going on. I usually turn this on and check it frequently for such queries. If there are some complicated / not optimal queries, you should fix them as soon as possible.

slow-query-log=1
slow-query-log-file=/var/lib/mysql/localhost-slow-query.log<

TIMESTAMP fields behavior

If you would check MySQL log during startup, you can note that there is a warning regarding TIMESTAMP fields. You can read more about these changes here. If you want to suppress that warning, use following option

explicit-defaults-for-timestamp=1

In order to apply changes, you need to restart MySQL daemon:

sudo systemctl restart mysqld

 

Few notes about playing with configuration. You must know that it's not easy and it's highly depends on number of tables, queries, inserts and lot of other factors. I usually use two tools. First is MySQLTuner that can get you some insights about what to change in order to gain performance. Usually it's just tuning query cache, and InnoDB buffer size. However practice shows that it doesn't always show correct values. Sometimes you can get slower performance instead.

So I use other tool - Datadog. You can also use New Relic, or any other monitoring tool that generates charts. So I usually change one thing at the time. I'm restarting MySQL server and waiting around 48h to see if it's actually performing better or not. This might seem to be slow approach but it's definitely safe:)

But before you start to playing with your configuration, make sure that you don't have slow queries or large tables that you scan without indexes. Sometimes just adding index will reduce query time. No need to change MySQL configuration files:)

MySQL group

MySQL during installation create it's own group - mysql. Most of the data files, and logs have mysql:mysql ownership. It makes it hard to read the logs when you are non-root user. You need to use sudo to get access to logs. You can make it simpler by adding user to mysql group. When user will be in such group, he can easily read logs. In my case, user is named developer.

sudo usermod -a -G mysql developer

Create databases and users

Probably the most important part from MySQL user is how to create database and users? First login to MySQL with your root account:

mysql -u root -p

And then start with creating databases. You need to enter following query:

CREATE DATABASE database_name_here;

Replace database_name_here with your database name. I like to create name for database from domain. So for instance for blacksaildivision.com, database name would be blacksaildivision. Remember that you can't use the . in database name.

Once you have your database, you should create dedicated user that only has access to this database. Having separate user per database is good approach from security perspective.

In order to create user in database execute following query:

CREATE USER 'blacksaildivision'@'localhost' IDENTIFIED BY 'NotEasyToGuessPassword123^#';

It will create blacksaildivision user that can connect to MySQL server only via localhost. It means that he won't have access from outside. After IDENTIFIED BY you need to type password for user you want to create. Same rules as for root password, it can't be easy to guess.

After user is created it's time to give him access to database you created before. Query goes like this:

GRANT ALL PRIVILEGES ON blacksaildivision.* TO 'blacksaildivision'@'localhost';

So you give user full access to blacksaildivision database. .* means that he should have access to all tables in blacksaildivision database. After TO you need to specify user that you created in step before.

To apply privileges, you need to reload them. Fire following query:

FLUSH PRIVILEGES;

And that's it! Create as many databases and users as you wish:) You can test connection to database with new credentials from command line. First exit from current MySQL session and than use following command:

mysql -u blacksaildivision -p blacksaildivision

-u stands for user, -p means that user will connect by using password. Last thing is database name you want to connect to.

Start MySQL on system boot

Last thing is to add MySQL to boot list. So after CentOS will start, MySQL will start as well:

sudo systemctl enable mysqld

Remember that you can use our lamponsteroids project based on Ansible that will automate whole server setup:)

Install PHP from source on CentOS

PHP – how to install from source on CentOS

In this tutorial I would like to show you how to install latest version of PHP on CentOS 7. If you are using PHP you most probably will want to have latest version of PHP7. PHP 5 support officially ends this year. Version 7 is now commonly used. It gives lot of performance boost and new features.

Unfortunately default version that comes from repo in CentOS 7 is PHP 5.4, so you can't use yum command without any custom repo like remi. Including custom repository is one way of installing desired PHP version. Another option is to compile it from source code. This tutorial will show you how to do that. It's not as hard as it might sounds:)

Install required tools for compilation

In order to compile PHP from source you need to install few tools and libraries. First you need EPEL repository to be enabled. This repository contains more recent version of packages. Most probably you have it installed already, but just to be sure, execute following command:

sudo yum install epel-release -y

Once you have it installed execute following command to install required packages:

sudo yum install autoconf libtool re2c bison libxml2-devel bzip2-devel libcurl-devel libpng-devel libicu-devel gcc-c++ libmcrypt-devel libwebp-devel libjpeg-devel openssl-devel -y

Download and unpack PHP Source code

Next step is downloading PHP source code. Easiest option is to download it from GitHub PHP releases. Choose the version you would like to install. In my case it's 7.2.3. Copy link to tar.gz archive and execute following commands:

curl -O -L https://github.com/php/php-src/archive/php-7.2.3.tar.gz
tar -zxvf php-7.2.3.tar.gz
cd php-src-php-7.2.3

It will download the archive from GitHub, unpack the sources and change working directory to unpacked sources.

Compile PHP

Now it's time to compile PHP. First we need to build configure command. In order to do that execute following command:

./buildconf --force

Once configure command is created we can use it to configure PHP installation. This process will enable certain PHP extensions such as PDO, FPM, OPCache, GD library etc. If you need any libraries that are not provided here, you can execute ./configure --help option and check if there is something you need. Following command will install PHP with most common extensions:

./configure --prefix=/usr/local/php --enable-fpm --disable-short-tags --with-openssl --with-pcre-regex --with-pcre-jit --with-zlib --enable-bcmath --with-bz2 --enable-calendar --with-curl --enable-exif --with-gd --enable-intl --enable-mbstring --with-mysqli --enable-pcntl --with-pdo-mysql --enable-soap --enable-sockets --with-xmlrpc --enable-zip --with-webp-dir --with-jpeg-dir --with-png-dir

Apart from enabling extensions command above will also set where PHP will be installed. In my case it's /usr/local/php location. If you will want to remove compiled PHP you will simply have to remove entire directory given under --prefix option.

Next it's time to compile PHP. Please be aware that it takes few minutes:

make clean
make

Install compiled PHP

Once PHP is compiled it is time to install it. Simply execute following command:

sudo make install

PHP Configuration

PHP-FPM setup

Before we will be able to run PHP from Apache we need to setup PHP-FPM worker. After installation there should be PHP-FPM default configuration file in installation directory. We will alter the file and then change it a bit.

cd /usr/local/php/etc
mkdir fpm.d
cp php-fpm.conf.default php-fpm.conf
vi php-fpm.conf

We need to  uncomment/change these lines:

include=etc/fpm.d/*.conf
pid = /var/run/php-fpm.pid
error_log = log/php-fpm.log

COPY EVERYTHING UNDER Pool Definitions TO CLIPBOARD AND REMOVE IT FROM php-fpm.conf FILE
;;;;;;;;;;;;;;;;;;;;
; Pool Definitions ;
;;;;;;;;;;;;;;;;;;;;

include=/etc/fpm.d/*.conf - by default there is one pool defined inside php-fpm.conf file. The best way to solve it is the same way as we solved Apache vhosts. We will include each pool in separate directory. In php-fpm.conf file one pool is already defined. We need to delete it from this file and put it inside fpm.d directory. We will have better control over the pools. The easiest way is just to Cut it from this file and paste it into new one.

Now let's create the file inside fpm.d for our example.com domain:

cd fpm.d
vi example.com.conf

PASTE TEXT FROM CLIPBOARD HERE AND CHANGE THESE LINES:

[www] -> [example_com] //Must be unique per file
user = apache
group = www
listen = 127.0.0.1:9000 //Port must be unique per file
catch_workers_output = yes
slowlog = /var/www/example.com/logs/php-fpm.slow.log
request_slowlog_timeout = 30s
php_flag[display_errors] = off
php_admin_value[error_log] = /var/www/example.com/logs/php-fpm.error.log
php_admin_flag[log_errors] = on
php_admin_value[memory_limit] = 64M
php_admin_value[open_basedir] = /var/www/example.com/htdocs

Each pool must have different name. So we need to change it from [www] to something else, for instance to domain name. It'll be easier to find the issues inside log files.

We set user and group to the same user as apache to have access to files.

Port will be different per pool. Standard way is to start from port 9000. Next will be 9001 etc.

We will catch errors and log them to file. In addition we set logging for  slow requests.

Nice part is that we can overwrite the settings from php.ini here. So we can overwrite error_log or memory_limit for instance. We should also set open_basedir so PHP will have access only to files inside our htdocs directory. Our server will be more secure with this setting.

php.ini and OPCache configuration

Second thing is php.ini file. After installation  php.ini file should located in /usr/local/php/lib. This is only the location. After compiling from source You won't anything there so we need to copy it from uncompressed sources.

cd /usr/local/php/lib
cp ~/sources/php-5.6.6/php.ini-development ./php.ini
vi php.ini

This is pretty large file with lot of configuration settings. Fortunately we only need to change some of the options:

short_open_tag = On
open_basedir = /var/www
disable_functions = exec,passthru,shell_exec,system,proc_open,popen
expose_php = Off
max_execution_time = 30
memory_limit = 64M
date.timezone = Europe/Warsaw
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
display_errors = Off
display_startup_errors = Off
log_errors = On
post_max_size = 5M
upload_max_filesize = 4M

opcache.enable=1
opcache.memory_consumption=64
opcache.interned_strings_buffer=16
opcache.max_accelerated_files=7000
opcache.validate_timestamps=0 ;set this to 1 on production server
opcache.fast_shutdown=1

So we set few things here, enable <? tag, limit access to files from PHP level, disabled dangerous functions, adjusts timezone, security, max execution times, errors etc. In addition we have enable OPCache for PHP.

Each one of these options are well commented inside php.ini file. If You don't like the settings here or You need something else, feel free to change it for Your purposes.

Useful shell scripts for PHP

/etc/init.d/php-fpm

As You probably remember during Apache setup we create script so we can use service command to start / stop Apache process. Now we will do the same for PHP-FPM

With PHP source code there comes ready script for that purpose.

cd /etc/init.d
cp ~/sources/php-5.6.6/sapi/fpm/init.d.php-fpm php-fpm
vi php-fpm

Now we need to setup configuration for the file:

prefix=/usr/local/php
exec_prefix=${prefix}

php_fpm_BIN=${exec_prefix}/sbin/php-fpm
php_fpm_CONF=${prefix}/etc/php-fpm.conf
php_fpm_PID=/var/run/php-fpm.pid

Save the file and add executable permission.

chmod +x php-fpm
servcie php-fpm status
service php-fpm start
servcie php-fpm status

After that we should have php-fpm process up and running!

Add PHP to $PATH

We can do one more thing to make our life easier:) Add PHP executable to PATH, so we'll be able to call php command from every directory.

echo 'pathmunge /usr/local/php/bin' > /etc/profile.d/php.sh

Execute such command, log out, log in and You'll be able to execute:

php -v

Setup Apache for PHP-FPM

Now is the time to finally setup Apache for .php files. Let's edit one of the Virtual Hosts now.

vi /usr/local/apache2/conf/vhosts/example.com.conf

<VirtualHost *80>
    ServerName example.com

    <LocationMatch "^/(.*\.php(/.*)?)$">
        ProxyPass fcgi://127.0.0.1:9000/var/www/example.com/htdocs/$1
    </LocationMatch>

////Rest of the file below

So basically we need to proxy all files with .php extension to our PHP-FPM process.  Also we need to restart Apache and make sure PHP-FPM is running httpd server:

service php-fpm start
service httpd restart

How to test if PHP is working?

We need to test if our PHP installation works. The easiest way to debug and check what's going on would be to create test.php file inside our /var/www directory.

vi /var/www/example.com/htdocs/test.php

and paste phpinfo() function there:

<?php

phpinfo();

Save the file and open the file in Your browser, assuming that your vagrant setup is correct. For instance http://example.com/test.php or 192.168.99.99/test.php

If everything is OK you should get information about PHP installation. Well done!

What's next

We are one step closer to our LAMP server. The only thing we are missing now is MySQL which we will install in upcoming episodes.

If You are running Ansible for provisioning You can find everything from this series inside my GitHub.

Hardening Apache with Mod Security

Apache hardening with mod_security

In this part of setup complete webserver we will harden Apache with popular mod_security. Mod_security is a small module that works like application firewall. It protect the app before most common attacks and vulnerabilities. It's good to have such thing on the webserver.

I assume that You have Apache already installed, if not check out previous part - How to install Apache on CentOS?

[sc:lamp_series]

How to install mod_security on Apache httpd server

First thing is installation of required tools. We need them to compile mod_security.

yum install automake libtool libxml2-devel

Next thing that we need is to download and decompress mod_security. Download links can be found on official mod_security page.

cd ~/sources
wget https://www.modsecurity.org/tarball/2.9.0/modsecurity-2.9.0.tar.gz
tar -zxvf modsecurity-2.9.0.tar.gz

Now it's time to compile mod_security. While ./confgure we need to pass paths to axps and apr binaries. All binaries should be inside bin directory in apache installation path.

cd modsecurity-2.9.0
./autogen.sh
./configure --with-apxs=/usr/local/apache2/bin/apxs --with-apr=/usr/local/apache2/bin/apr-1-config --with-apu=/usr/local/apache2/bin/apu-1-config
make
make install
cp /usr/local/modsecurity/lib/mod_security2.so /usr/local/apache2/modules

If there was no error mod_security is ready to use. It can be found in /usr/local/modsecurity We need to copy generated .so file to apache extension directory.

ModSecurity and OWASP rules

mod_security is nothing without the rules that tells what attacks should be blocked. Fortunately there is a great package with lot of rules provided by OWASP. We will use such rules package to harden Apache HTTPD. So let's download, unzip and copy the rules to apache configuration directory.

cd ~/sources
wget -O owasp.tar.gz https://github.com/SpiderLabs/owasp-modsecurity-crs/tarball/master
mkdir /usr/local/apache2/conf/crs
tar -zxvf owasp.tar.gz -C /usr/local/apache2/conf/crs --strip 1
cd /usr/local/apache2/conf/crs
cp modsecurity_crs_10_setup.conf.example modsecurity_crs_10_setup.conf

Now we are ready to use ModSecurity!

ModSecurity configuration

In previous article we added httpd-security.conf file with some basic rules that improves Apache security.  We will modify this file to load mod_security with OWASP rules and add some basic configuration. You need to know that mod_security is pretty large module with tons of configuration option. You can find them in ModSecurity reference manual.

vi /usr/local/apache2/conf/extra/httpd-security.conf

Once You open the file add these lines somewhere in the file:

LoadModule security2_module modules/mod_security2.so

<IfModule security2_module>
      Include conf/crs/modsecurity_crs_10_setup.conf
      Include conf/crs/base_rules/*.conf
      # Include conf/crs/experimental_rules/*.conf
      # Include conf/crs/optional_rules/*.conf

      SecRuleEngine On
      SecRequestBodyAccess On
      SecResponseBodyAccess On 
      SecResponseBodyMimeType text/plain text/html text/xml application/octet-stream
      SecDataDir /tmp

      # Debug log
      SecDebugLog /usr/local/apache2/logs/modsec_debug.log
      SecDebugLogLevel 3

      SecAuditEngine RelevantOnly
      SecAuditLogRelevantStatus ^2-5
      SecAuditLogParts ABCIFHZ
      SecAuditLogType Serial
      SecAuditLog /usr/local/apache2/logs/modsec_audit.log
</IfModule>

So from the top:

  • First we need to load mod_security module.
  • Next are rules from OWASP that we will include to ModSecurity. We need to include the setup and base rules. OWASP core rule set comes with lot more features that are marked as optional or experimental. We can enable those rules, but we also need to remember that it might not play well with our website. It's rather testing by trial and error then one rule will work well on every website. But in general including base_rules is OK.
  • SecRuleEngine enables detection and blocking of malicious attacks.
  • SecRequestBodyAccess enable inspection of data transported request  bodies
  • SeResponseBodyAccess buffer response bodies matched by SecResponseBodyMimeType
  • SecDataDir working directory for ModSecurity temporary purposes
  • Next thing is Debug log. By default all error logs goes to apache error log, but we can set different path to debug log. Best practice would be to change it per domain inside particular VirtualHost file. In previous article we setup directory structure and we have logs directory there. It would be wise to used it for debug log as well.
  • Audit Log is complementary log for Debug log. It has detail information about every error. It's disabled by default so we need to enable it and turn on logging relevant (warnings and errors) issues. Next options are for configuration the audit log. In general there are lot more of discussing at this topic.

If You want to learn more about how to setup and read mod_security logs, here is really great article about mod_security logging by Infosec Institue.

Now we just need to save the file restart apache and our httpd server has better security.

service httpd restart

What's next?

If You are following our series, You should have now part of LAMP stack (Linux Apache MySQL PHP). Apache is secured with mod_security.

Small note to those who would like to install mod_evasive as well to increase the security. To be honest, it's really not worth to install it on Apache. Why? Because when You run multiple instances via MPM mod_evasive doesn't share the info between the MPM instances. It means that one instance of apache can block the attacker but others wont. So if You have many MPM workers mod_evasive is just useless.

As always, if You are using Ansible for server provisioning You can use ready playbook, that will cover everything in this series. You can find it on GitHub.

In next episode we will add P to our LAMP server.

How to install apache from source on CentOS

How to install latest Apache HTTPD on CentOS

Hi there! Today I'd like to show you how do I install and configure Apache HTTPD on CentOS 7. I like to have it installed in minimal and secure way. Beware! This is pretty long tutorial covering lot's of aspects from compilation, through configuration, SSL/HTTPS/HTTP2, basic hardening etc.

How to install Apache HTTPD on CentOS using yum - easy way

There are two ways to install Apache HTTPD on CentOS. First is with yum and it is the simplest version:

sudo yum install httpd -y

Volia! You have httpd installed. However, if you check the version:

httpd -v

You will most probably get 2.4.6 version or slightly newer. If you check Apache website, you will note, that they have 2.4.33 version available. So, if you want to have access to latest features such as HTTP/2 support or latest bugfixes, you will have to try more difficult method which is compiling Apache from source. It might seem complicated but it's really not.

OpenSSL - do you have latest version?

If you want to enable HTTP/2 in Apache HTTPD which I strongly recommend for increased performance you need to have latest version of OpenSSL installed in your system. Older version does not support it, so you need to compile new version from source. I have separate tutorial how to install latest OpenSSL - make sure that you follow that first and then get back to this tutorial.

Remove old HTTPD first

Make sure that you don't have httpd installed. On some machines it comes by default, or you might be using older version. In order to avoid complications later I advise you to remove it first. However you must know that if you have some websites online that are using Apache, they will have some downtime before you setup new Apache. Execute following command to remove current Apache httpd from your system:

sudo yum remove httpd -y

Install EPEL

There are lots of different libraries in EPEL, but for compiling Apache HTTPD with HTTP/2 support we need one thing that EPEL provides - libnghttp2

In order to install EPEL repository execute following command:

sudo yum install epel-release -y

Install required tools for compilation

You need to install some tools that will help us compile Apache. It's basic stuff like compiler, required libraries etc:

sudo yum install autoconf expat-devel libtool libnghttp2-devel pcre-devel -y

Download and unpack source code

Next thing that you need are packages with source files. For compiling Apache, you will need 3 different packages - httpd itself, apr and apr-util. Last two are Apache Runtime libraries. They are required for Apache HTTPD.

I like to download packages from GitHub releases. Here are the links to the packages:

Click on tar.gz icon, copy the link to package and download them with curl or wget. Or simply copy commands below:

curl -O -L https://github.com/apache/httpd/archive/2.4.33.tar.gz
curl -O -L https://github.com/apache/apr/archive/1.6.3.tar.gz
curl -O -L https://github.com/apache/apr-util/archive/1.6.1.tar.gz

Unpack downloaded sources:

tar -zxvf 2.4.33.tar.gz
tar -zxvf 1.6.3.tar.gz
tar -zxvf 1.6.1.tar.gz

APR and APR-Util

Apache requires APR library for compilation. You need to copy the source codes to correct directory:

cp -r apr-1.6.3 httpd-2.4.33/srclib/apr
cp -r apr-util-1.6.1 httpd-2.4.33/srclib/apr-util

It's important to not to include version number in APR directories. If you just copy apr-1.6.3 without changing the name, it will give you a warning about missing apr directory.

Compile source code

Now you are ready to compile Apache httpd. It's important that you should not use root user for compilation. It can lead to serious security issues. I described it more on my other tutorial about installing GIT. In short words, imagine that you downloaded package from wrong source with malicious code. If you would compile it as root user, anything can happen to your server. Including cutting of your root access. I'm not saying that it's not possible to compile packages as root, because it is. It's just not safe. If you want to create separate user with sudo powers, you can read this tutorial.

So get inside httpd directory and compile your Apache version:

cd httpd-2.4.33
./buildconf
./configure --enable-ssl --enable-so --enable-http2 --with-mpm=event --with-included-apr --with-ssl=/usr/local/openssl --prefix=/usr/local/apache2
make

First command ./buildconf will build ./configure file required for configuration of the build.

./configure command will setup everything for compilation of Apache HTTPD. Here are the options that I use:

  • --enable-ssl will build Apache with SSL support, so you can enable HTTPS on your websites.
  • --enable-so will enable dynamically loaded modules. So you can enable and disable modules without recompilation (I will describe modules in configuration part)
  • --enable-http2 will enable HTTP/2 support.
  • --with-mpm will set multiprocessing modules for Apache. I'm using event, but you can use worker or prefork instead. event works best for me and I think that it is mpm that will give you most performance.
  • --with-included-apr It will use APR library that you copied to srclib directory
  • --with-ssl will point compiler to newer version of OpenSSL. Make sure that you compiled it first!
  • --prefix is the installation path for Apache httpd compiled package

Whole process might take a while. It depends how fast your server is.

Install HTTPD

After it's compiled you can install it. For that you need sudo or root account:

sudo make install

Apache should be installed in the directory you specified with --prefix option.

Cleanup

Last thing you can do now is to remove downloaded files. You won't need them now. It's not mandatory, but it's nice to keep server clean.

cd ..
rm -rf 1.6.3.tar.gz 1.6.1.tar.gz 2.4.33.tar.gz apr-1.6.3 apr-util-1.6.1 httpd-2.4.33

Add Apache executables to PATH

If you try to type httpd -v in your command line, it will result in command not found. That's because httpd is not on your $PATH. I'd like to have all executables from Apache available from everywhere. In order to achieve that, create file

sudo vi /etc/profile.d/httpd.sh

and paste there following contents:

pathmunge /usr/local/apache2/bin

Save the file, log out and log in from your current session to reload your profile. After that you should be able to use httpd -v command:)

Add Systemd entry

Starting, restarting, and enabling Apache on server start via systemctl command is very important thing.  You need to create another file:

sudo vi /etc/systemd/system/httpd.service

and paste there following contents:

[Unit]
Description=The Apache HTTP Server
After=network.target

[Service]
Type=forking
ExecStart=/usr/local/apache2/bin/apachectl -k start
ExecReload=/usr/local/apache2/bin/apachectl -k graceful
ExecStop=/usr/local/apache2/bin/apachectl -k graceful-stop
PIDFile=/usr/local/apache2/logs/httpd.pid
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Save the file and reload the systemctl daemon

sudo systemctl daemon-reload

Now you can try to start your Apache httpd server with following command:

sudo systemctl start httpd

It should start properly. If you will have any warnings, don't bother with them now. I will show you proper configuration in next step.

Once it's up and running you can try to type your server IP address in your browser like http://43.184.89.190/ and check if you see It works! message:) If so, you have Apache httpd running fine!

Create dedicated user and group for Apache

I usually create additional user and group for httpd daemon. It's good practice from security side. Each service should operate as separate user. It limits possible damage during attacks, httpd exploitation etc.

If you want to learn more about creating user and groups I recommend reading this tutorial. Here I'll just simply create group and user without shell.

sudo groupadd www
sudo useradd httpd -g www --no-create-home --shell /sbin/nologin

You can change the names as you wish. I like to use www group instead of httpd group for example. I usually add there other services as well, like nginx or php-fpm.

Adjust main config file httpd.conf

httpd.conf is main Apache httpd configuration file. You should start by editing this file with editor of your choice. I like to use vi, but you can use nano etc...:

sudo vi /usr/local/apache2/conf/httpd.conf

I usually don't remove any of the configuration directives from that file. If something needs to be deleted I comment it out with # at the beginning of new line. It's easier to revert changes.

Now I scroll through the file from top to bottom looking for particular lines and check if it has correct value. If I don't find given line I just add it usually at the bottom of the file.

Here are the important values:

# Make sure that ServerRoot is set to the same value as --prefix during ./configure
ServerRoot /usr/local/apache2

# Set ServerName to prevent warning on Apache start
ServerName localhost

# Default port set to 80 - HTTP protocol
Listen 80

# Set user and group
User httpd
Group www

# Configure entry file for your application. If you plan to use PHP make sure that it's as first possible file
DirectoryIndex index.php index.html

# Hide Apache version from header and from error files
ServerTokens prod
ServerSignature off

# Disable ETag to prevent disposing sensitive values like iNode
FileETag none

Save the file. After each change to configuration file you must restart HTTPD in order to apply changes. In order to do so execute following command:

sudo systemctl restart httpd

After restarting, make sure that Apache is working fine!

Configure loaded modules

During compilation we set --enable-so modules which means that we can disable and enable modules in configuration files.

List of modules is pretty long. Some of the are disabled (they have # at the beginning of the line). You should know, that the more modules are enabled the  "slower" Apache httpd is. I'm not saying that it's super slow, but you can google for some benchmarks showing different configurations.

Here is what I like to do. First of all, I comment out all modules = everything is disabled. I enable only that modules that I really use + the modules that are required for proper functioning of Apache httpd. It has few benefits - Apache is faster, eats less resources (CPU and RAM) and it's more resistant for given attacks. Usually when new security issue pops out, it's rather connected to one of the modules, than whole httpd. So you can have more chances to avoid potential security risk with having some stuff disabled.

So edit main configuration file one more time:

sudo vi /usr/local/apache2/conf/httpd.conf

And set modules that you use. Here is my go-to list of enabled modules:

# These modules must be enabled if you want Apache to start
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule mime_module modules/mod_mime.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule unixd_module modules/mod_unixd.so
LoadModule dir_module modules/mod_dir.so

# If you are using PHP with PHP-FPM which I highly suggest enable proxy modules 
LoadModule proxy_module modules/mod_proxy.so 
LoadModule proxy_fcgi_module modules/mod_proxy_fcgi.so 

# Enable pretty links and mod_rewrite that is highly used in all frameworks and CMSes 
LoadModule rewrite_module modules/mod_rewrite.so 

# Useful for WordPress sites - enables Require for setting up access to given resources. 
LoadModule access_compat_module modules/mod_access_compat.so 

# One more useful thing for WordPress and Let's Encrypt - enables Alias. If you are using composer and wpackagist it's a must, otherwise if you don't plan to use aliases, leave that disabled 
LoadModule alias_module modules/mod_alias.so 

# Enable gzip extension for compressing static files 
LoadModule deflate_module modules/mod_deflate.so 
LoadModule filter_module modules/mod_filter.so 

# Enable expires header for caching assets on browser side 
LoadModule expires_module modules/mod_expires.so 

# Enable SSL 
LoadModule http2_module modules/mod_http2.so 
LoadModule socache_shmcb_module modules/mod_socache_shmcb.so 
LoadModule ssl_module modules/mod_ssl.so 

# Enable status module for monitoring Apache. If you don't plan to use that, leave that commented out 
LoadModule authz_host_module modules/mod_authz_host.so 
LoadModule status_module modules/mod_status.so

Save the changes and restart Apache. Make sure that it works after changes!

Configure MPM

Configure Multi-Processing modules contains two parts. First I override default Apache configuration for event mode that I picked during compilation. In that case you will know what is actually configured. So let's start with that first.

At the bottom of httpd.conf you need to uncomment the line:

Include conf/extra/httpd-mpm.conf

It will enable advanced Apache httpd MPM configuration and it will override the defaults.

Now edit enabled file:

sudo vi /usr/local/apache2/conf/extra/httpd-mpm.conf

This file contains configuration of all MPM There are configuration for each MPM module, so make sure that you are setting correct values. I enabled event mode, so this is the section I care about:

<IfModule mpm_event_module>
    StartServers 5
    MinSpareThreads 75
    MaxSpareThreads 250
    ThreadsPerChild 25
    MaxRequestWorkers 400
    MaxConnectionsPerChild 0
</IfModule>

Save the file and restart Apache. This was the easiest part of MPM configuration.

The real part of MPM configuration is later on when you start running your production website. When you feel that your website/application start running low on higher traffic you can start playing around with that values.

Important thing that you need to know - there is no one config to rule them all. What I want to say is that one config that works fine on one server might not work as fine as on second server. There are dozens of factors like application specific, CPU and RAM, traffic etc. I encourage you to play around with these values to find the optimal settings for your server.

Here is great post that explains in depth MPM configuration. I have two advice's about performance tuning.

First one, that you probably don't even need to change anything if you have regular website without huge amount of traffic. It's just good to know what settings are applied by mpm configuration. When httpd-mpm.conf file was commented out, you didn't have any idea what the settings are. Once it's enabled, you at least know what is configured.

Second advice is - do it slowly. Performance tuning is lengthy process, and as I said, it highly depends on various factors. I like to change one settings, like StartServers for instance and I wait day or two and monitor response times, CPU and RAM usage etc. Sometimes even if you increase something you won't see a difference in response time, but you will get higher CPU usage. Then you just can rollback the changes. If you modify 3 or 4 values at one time, it's hard to say which comes with best (or any) result.

Setup GZIP compression

Using GZIP compression has serious impact on performance and your website loading time. Let's turn it on in few simple steps. In order to use it make sure that mod_deflate and mod_filter are enabled.

First create new file that will contain GZIP settings for Apache:

sudo vi /usr/local/apache2/conf/extra/httpd-deflate.conf

Paste there following content and save the file:

<IfModule mod_deflate.c>
    <IfModule mod_filter.c>
        AddOutputFilterByType DEFLATE application/ecmascript
        AddOutputFilterByType DEFLATE application/javascript
        AddOutputFilterByType DEFLATE application/rss+xml
        AddOutputFilterByType DEFLATE application/xml
        AddOutputFilterByType DEFLATE application/x-javascript
        AddOutputFilterByType DEFLATE text/css
        AddOutputFilterByType DEFLATE text/html
        AddOutputFilterByType DEFLATE text/plain
        AddOutputFilterByType DEFLATE text/xml
    </IfModule>
</IfModule>

It will add GZIP compression to most popular file types such as HTML, CSS, JS etc. If you need anything else that requires compressing, simply add more MIME types.

Once you save the file you need to include it to main Apache configuration. So open httpd.conf one more time and on the bottom of the page there is section where you include different things, such as MPM. In this section add following line:

Include conf/extra/httpd-deflate.conf

Save changes and restart HTTPD. Gzip compression will be enabled!

Setup assets caching with expires headers

Another important thing that has serious impact on performance is caching. If you want to use cache headers make sure that you enabled mod_expires for that purpose. Rest of procedure is similar to enabling GZIP compression.

Start with creating file that will contain configuration for caching:

sudo vi /usr/local/apache2/conf/extra/httpd-expires.conf

Paste following configuration and add/remove mime types or adjust time of expiration it if you need:

<IfModule mod_expires.c>
    # Enable expirations
    ExpiresActive On

    # Expirations for given mime type
    ExpiresByType image/gif "access plus 1 month"
    ExpiresByType image/ico "access plus 1 month"
    ExpiresByType image/jpg "access plus 1 month"
    ExpiresByType image/jpeg "access plus 1 month"
    ExpiresByType image/png "access plus 1 month"
    ExpiresByType text/css "access plus 1 month"
    ExpiresByType text/javascript "access plus 1 month"
</IfModule>

Same as before, you need to add entry to httpd.conf file and after that, restart the server:

Include conf/extra/httpd-expires.conf

Enable SSL configuration

If you plan to use HTTPS which I highly recommend you need to include SSL configuration. By default it is commented out from main configuration file. Before including this file, let's edit it first:

sudo vi /usr/local/apache2/conf/extra/httpd-ssl.conf

File contains secure configuration and sensible defaults that you don't need to change. It will provide secure connection that targets most browsers. However this file also contains default VirtualHost for HTTPS which I always remove. The reason is - I like to know what entry points to the server are available. Since this page doesn't display anything useful we can simply remove it.

So scroll to the point where VirtualHost section starts and remove whole section. It's pretty long so make sure that you remove it completely including <VirutalHost...></VirtualHost> lines :

## Remove this part:
##
## SSL Virtual Host Context
##
<VirtualHost _default_:443>
#...
</VirtualHost>

Save changes to the file. Now it's time to include this file into httpd.conf file. Edit it and scroll to the bottom. SSL is commented out by default so all you need to do is to uncomment it like so:

# Secure (SSL/TLS) connections
Include conf/extra/httpd-ssl.conf

Save the changes and restart HTTPD service. This is only initial part of adding HTTPS to your website. To be able to successfully connect to your server with SSL there are few more steps.

Add Let's Encrypt integration

If you are not using Let's Encrypt free SSL certificates you can skip that part.

When you install certbot for obtaining Let's Encrypt certificates you must somehow validate the domain. I use webroot as authenticator. It means that inside your website directory certbot will create .well-known directory and will try to put some files there. It works in most cases unless you start playing around with root path to your website, like placing it in subdirectory. Sometimes there is also permission issue.

There is neat trick to solve all this issues. When you create new certificate, point webroot to /var/lib/letsencrypt
directory instead to your domain directory like /var/www/blacksaildivision.com/htdocs

Next, create new file for Apache configuration:

sudo vi /usr/local/apache2/conf/extra/httpd-acme.conf

and paste there following contents:

Alias /.well-known/acme-challenge/ "/var/lib/letsencrypt/.well-known/acme-challenge/"
<Directory "/var/lib/letsencrypt/">
    AllowOverride None
    Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec
    Require method GET POST OPTIONS
</Directory>

Please note that mod_alias must be enabled. What it will do on each request to .well-known/acme-challenge that certbot uses it will point it to /var/lib/letsencrypt directory. It solves the problem for all domains globally.

Last thing is to enable this feature in main configuration of Apache. Edit the file and add following section near the bottom of the file where all other Includes are placed:

# Let's Encrypt integration
Include conf/extra/httpd-acme.conf

Restart the Apache to apply new changes.

Enable extended status

If you want to get more insights about Apache or you plan to use monitoring tools like Datadog it's wise to enable extended version of status. It will give you basic info about workers and httpd itself. In order to enable, edit main configuration file and somewhere and following code (I like to place it just before all Include sections):

# Enable extended status
<IfModule status_module>
    ExtendedStatus On
</IfModule>

Remember that you need to enable mod_status and mod_authz_host in order to get status working. Mod_authz_host will be required later on when I show you how to configure localhost. Mod_status response should be restricted to avoid exposing sensible information.

In order to apply changes, please restart httpd.

Setting up root directory

Let's start with final part - setting up Virtual Hosts. There are couple of approaches. It depends how many websites you plan to have on single server, does all your apps have domains or are they just microservices etc. I'll show you two of my approaches. Regardless of any approach you need a directory where you will store you app/website files and logs.

On CentOS there should be directory /var/www which is recommended place to store website files. If you don't have such directory create one first:

sudo mkdir /var/www

Make sure that this directory has valid permission and valid owner which should be root:

sudo chmod 755 /var/www
sudo chown root:root /var/www

Setting up website directories for files and logs

Once root directory is ready it's time to setup directories for websites. Let's assume that I want to create folder structure for blacksaildivision.com website. So first I create directory inside /var/www and name it as domain:

sudo mkdir /var/www/blacksaildivision.com

Inside this directory I need two separate directories. One for my website files like index.html etc and one for logs. There are many ways of keeping logs on the server. Some people uses native directories like /usr/local/apache2/logs or /var/log but I prefer to keep everything in one place. So all nginx/httpd/php logs will go to logs directory inside my domain directory.

sudo mkdir /var/www/blacksaildivision.com/htdocs
sudo mkdir /var/www/blacksaildivision.com/logs

Once directories are created it's time to set valid owner and permissions. Let's start with ownership:

sudo chown root:root /var/www/blacksaildivision.com
sudo chown developer:www /var/www/blacksaildivision.com/htdocs
sudo chown developer:www /var/www/blacksaildivision.com/logs

So main directory for domain should be owned by root. However two directories inside should be owned by custom user. In that case main user for the server is called developer. He has sudo powers and has access to website files and can perform commands like git pull to download latest code.

Why not httpd user that we created for Apache? The reasons are permissions. When you download files with git pull as developer user, but directory is owned by httpd user it most often leads to unexpected errors and mismatch in owners.

If you need ( and you should ) create separate account for developer/server owner read this tutorial.

Group is set to www which I usually use as generic group for all website related applications such as httpd, php-fpm etc. Please note that developer user also belongs to that group.

Now it's time to set directory permissions:

sudo chmod 755 /var/www/blacksaildivision.com
sudo chmod 2775 /var/www/blacksaildivision.com/htdocs
sudo chmod 2775 /var/www/blacksaildivision.com/logs

2 at the beginning means that every new subdirectory created will have the same ownership as parent directory (developer:www)

So what if I need to store more than one domain? Simply create new directory inside /var/www like mywebsite.com and repeat the steps with htdocs, logs directory, set ownership and permissions.

Do I need to create domain directory? It depends. I usually create such directory even if I host single website only. There are some cases like microservices that does not have a domain and are run behind load-balancer. In that case you can skip creating domain directory and use /var/www/htdocs and /var/www/logs. Just remember to change paths accordingly in next steps.

Create basic Virtual Host file for HTTP website

Directories were created so now it's the most important part - setting up VirtualHost. Again, there are many different approaches how to store the virtual hosts etc. I'll show you mine. Let's create basic Virtual Host file that will only load static files from given domain. As example I use blacksaildivision.com domain.

First I create a file that will contain configuration for HTTP version of the website. I always add domain to the filename so it's easier to find given configuration if you have multiple domains:

sudo vi /usr/local/apache2/conf/extra/httpd-vhost-blacksaildivision.com.conf

Paste Virtual Host config there and adjust your domain and directory structure:

<VirtualHost *:80>

ServerName blacksaildivision.com

# Directory settings
DocumentRoot /var/www/blacksaildivision.com/htdocs
<Directory /var/www/blacksaildivision.com/htdocs>
    AllowOverride All
    Require all granted
    Options +FollowSymLinks -Indexes -Includes
</Directory>

# Logging
ErrorLog "/var/www/blacksaildivision.com/logs/httpd-error.log"
CustomLog "/var/www/blacksaildivision.com/logs/httpd-access.log" common

</VirtualHost>

File starts and ends with block defining VirtualHost. Inside this blocks you need to place your domain configuration. First one is ServerName which is your domain.

Next part is directory settings. You need to point httpd to read files from correct directory which is defined by DocumentRoot. Just below that line there are settings for root directory. AllowOverride will allow .htaccess files to override all default instructions in this block. Require all granted adds general access to this directory. It's a must if you want to read files from given location. Lastly I allow following symlinks which are often used by various frameworks and CMSes. I disable Indexes and Includes because of security reasons.

Last parts are logs. I define two logs. One is error log which will contain all invalid requests details etc. CustomLog is simply access log that will contain all requests.

So in general if you want to add different or another domain, simply create file with configuration like above and update ServerName, DocumentRoot and paths to logs.

Last part is to include this configuration file in main httpd configuration file:

sudo vi /usr/local/apache2/conf/httpd.conf

and at the very bottom of the file Include configuration:

Include conf/extra/httpd-vhost-blacksaildivision.conf

Save changes and restart Apache. Now try to visit your website! In case of any issues like 4xx/5xx errors simply check error log and see what the problem is. Usually there are typos or incorrect configuration of your domain causing the issues.

Adding PHP support

Most of you will probably use Apache with PHP. I use PHP in PHP-FPM mode. In my opinion it gives greater control over mod_php when it comes to performance tuning, configuration etc. It also has one more benefit - if you will ever need to switch from Apache to Nginx it's much easier as preferred way of using nginx + php is PHP-FPM.

If you want to read more about how to install and configure PHP - please follow this tutorial.

Edit the VirtualHost file where you want to add PHP support and start adding there following lines ( I prefer to add it just before logs section):

# PHP-FPM settings
<Proxy "fcgi://127.0.0.1:9001" timeout=30>
</Proxy>

It will define Proxy for PHP-FPM daemon. In my case FPM process is running on localhost (127.0.0.1) and on port 9001. Optional parameter is timeout. It's not required but sometimes it's helpful. Imagine situation where one of your scripts has bug like infinite loop or it's execution time takes way more than 30s. Apache will have to wait until proxy will return something. Now imagine that someone called this script 10000 times. It will kill your httpd daemon. It's safe to add timeout as additional guard. You can experiment with timeoutvalue for your application.

Proxy is defined, not it's time to use it. Just after section add  following lines:

<FilesMatch "\.php$">
    <If "-f %{SCRIPT_FILENAME}">
        SetHandler "proxy:fcgi://127.0.0.1:9001"
    </If>
</FilesMatch>

First I use FilesMatch to detect if requested file has .php extension. If it has I'm checking if the file actually exists. If you won't check it, request will be passed to FPM process anyway which can result in unexpected results and unnecessary lines in log files.

If everything is OK I'm setting Handler to defined proxy and PHP will handle rest of the request.

You can save the file and restart Apache and check if everything works fine:)

If you are using status or ping from PHP-FPM for performance debugging or tools such as Datadog and you don't want to expose additional port in your firewall, you can achieve it with HTTPD as well. Simply add following lines before <FilesMatch> section:

<LocationMatch "/(status|ping)">
    RewriteEngine Off
    Require all denied
    Require ip 127.0.0.1 60.120.72.81
    SetHandler "proxy:fcgi://127.0.0.1:9001"
</LocationMatch>

I'm checking if requested path matches /status or /ping. If match is found I'm turning of RewriteEngine in case of redirect in .htaccess etc. Than I'm turning off access for everyone but I allow access from list of IPs. I allow access from localhost, but I also allow access from my server IP address. So for instance If my server has IP 60.120.72.81 I add it as well.

Last thing is to set handler to use to proxy for PHP-FPM daemon.

Enable HTTPS inside VirtualHost

It's time to enable HTTPS for our website! I assume that you already have SSL certificates purchased and downloaded or generated via Let's Encrypt.

Edit the file first:

sudo vi /usr/local/apache2/conf/extra/httpd-vhost-blacksaildivision.com.conf

Just for now you can leave the regular HTTP version and add following contents below VirtualHost section. In such way you will have both HTTP and HTTPS enabled for single website. Later on I'll show you how to redirect HTTP to HTTPS. So just paste the following contents at the end of edited file:

<IfModule mod_ssl.c>
<VirtualHost *:443>

ServerName blacksaildivision.com
ServerAlias www.blacksaildivision.com

# Redirect www to non-www
RedirectMatch permanent "^www\.(.*)$" "https://blacksaildivision.com/"

# Directory settings
DocumentRoot /var/www/blacksaildivision.com/htdocs
<Directory /var/www/blacksaildivision.com/htdocs>
    AllowOverride All
    Require all granted
    Options +FollowSymLinks -Indexes -Includes
</Directory>

# PHP-FPM settings
<Proxy "fcgi://127.0.0.1:9001" timeout=30>
</Proxy>

<LocationMatch "/(status|ping)">
    RewriteEngine Off
    Require all denied
    Require ip 127.0.0.1
    SetHandler "proxy:fcgi://127.0.0.1:9001"
</LocationMatch>

<FilesMatch "\.php$">
    <If "-f %{SCRIPT_FILENAME}">
        SetHandler "proxy:fcgi://127.0.0.1:9001"
    </If>
</FilesMatch>

# Logging
ErrorLog "/var/www/blacksaildivision.com/logs/httpd-error.log"
CustomLog "/var/www/blacksaildivision.com/logs/httpd-access.log" common

# SSL configuration
SSLEngine on
SSLCertificateFile "/etc/letsencrypt/live/blacksaildivision.com/cert.pem"
SSLCertificateKeyFile "/etc/letsencrypt/live/blacksaildivision.com/privkey.pem"
SSLCertificateChainFile "/etc/letsencrypt/live/blacksaildivision.com/chain.pem"

</VirtualHost>
</IfModule>

Everything is wrapped with IfModule section. I usually check if mod_ssl is enabled. In case someone by accident would remove that module from list of enabled modules, section with VirutalHost simply won't be loaded and will not throw an error.

This is new VirtualHost so it must be wrapped in  <VirtualHost *:443>  section. Please note that port is now set to 443 instead of 80. It's default port for HTTPS. You might see some similarities between HTTP and HTTPS version. Most of things are just copied from HTTP version, so I won't be covering this parts here. Instead I will focus on new things.

You already know ServerName directive. But below there is another one - ServerAlias. You can basically access your website in two ways - with www. (https://www.blacksaildivision.com) and without www. (https://blacksaildivision.com). I prefer to use version without www. Less typing and URL looks more friendly IMHO:) So I want to redirect all www traffic to non-www. We don't have such redirect in HTTP version, but we will add it later on when I'll show you how to redirect from HTTP to HTTPS.

ServerAlias create an alias to given VirtualHost identified by ServerName. So no matter which URL you will use - the one from ServerName or from ServerAlias(es), Apache will use configuration from given VirtualHost. For instance we can add another alias like - ServerAlias blacksaildivisioncopy.com and it can point to same configuration. So your website can be accessible from two different URLs. It's very rare case but this is just an example.

What is important that your website should have redirect, so there always will be only single copy of your website. For instance if you have both www and non-www versions available without any redirect, your content will be indexed twice and will be seen as two different websites. It's bad for the SEO and your ranking in search engines. In order to redirect from non-www to www you should use RedirectMatch directive. This is permanent redirect, so it won't (and should not) change in the future. Also please make sure that you have your SSL cert valid for www. domain as well. If you won't have it, you will get an error when visiting version with www. prefix.

Last new part is SSL configuration at the bottom of the file. First we need to tell Apache that we will use SSL with SSLEngine on. Next are the paths to Certificate, Key and Chain. Chain is sometimes not mandatory in some SSL providers. So if you don't have Chain file, skip this directive. I'm using Let's Encrypt so paths to SSL are set to /etc/letsencrypt/live/blacksaildivision.com/ If you just have certificates bought from SSL provider, simply create new directory inside your domain like so:

sudo mkdir /var/www/blacksaildivision.com/ssl

and place all cert files inside this directory. Remember to update paths inside your VirtualHosts!

Once it's done simply save the changes and restart Apache. Try to visit your website with HTTPS and HTTP. Both version should work just fine. You can also check your server with https://www.ssllabs.com/ssltest/ If you configured everything correctly, you should get A score. You can aim for A+, but in my opinion there is too much disadvantages like limited access to your website from older devices etc.

Redirect HTTP to HTTPS

As we want to serve traffic over HTTPS only, we need to redirect all HTTP requests to HTTPS. Edit the file with your domain configuration and replace <VirtualHost *:80> to following code:

<VirtualHost *:80>

ServerName blacksaildivision.com
ServerAlias www.blacksaildivision.com

# Redirect everything permanently to https://blacksaildivision.com/
Redirect permanent "/" "https://blacksaildivision.com/"

# Turn off logging
ErrorLog /dev/null
CustomLog /dev/null common

</VirtualHost>

It's much shorter than previous version. I added ServerAlias with www. prefix. Using Redirect directive I'm redirecting all traffic to https:// version. And last thing is turning off logging. It's not necessary to log redirects from HTTP to HTTPS. If you have logging in HTTPS VirutalHost, logs will be there anyway. There's no need to duplicate code.

Save changes, restart Apache and check if redirect is working fine:)

Run Apache httpd on system start

Last thing is to add Apache httpd daemon to start with system boot. So after server start/restart  httpd will run automatically:

sudo systemctl enable httpd

So that's it. You have fully working Apache httpd in latest version installed on your system 🙂 This process might take some time, but you will have full control over httpd.

As always you can use our LampOnSteroids project (based on Ansible) to speed and automate everything up!

Our services: