Add swap file to Linux

Some VPS providers provision their VPS without any swap space.
Given that these servers often have a small amount of physical memory even on a moderately used server it is likely that you will see processes being killed to recover memory, e.g.

Sep 12 15:01:30 xxxxxx.xxx.xxx kernel: mysqld invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0

When this happens to the mysqld process that is a bad thing as it can potentially lead to data loss, as it may have done in this example:

160912 15:01:45  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 120271507105
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 366 row operations to undo
...
160912 15:02:08  InnoDB: Rolling back trx with id 1531B60, 366 rows to undo
160912 15:02:08  InnoDB: Waiting for the background threads to start
InnoDB: Rolling back of trx id 1531B60 completed

One possible way to alleviate the problem is to add swap space and monitor it’s usage.

Add swapfile

This example adds a 1GB (1048576 * 1KiB blocks) swap file.

# dd if=/dev/zero of=/swapfile1 bs=1024 count=1048576
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB) copied, 10.2767 s, 104 MB/s
# chown root:root /swapfile1
# chmod 0600 /swapfile1
# ls -l /swapfile1
-rw------- 1 root root 1073741824 Sep 13 08:01 /swapfile1
# mkswap /swapfile1
Setting up swapspace version 1, size = 1048572 KiB
no label, UUID=407c4f8a-354a-45f6-8791-3b817e22eb5d
# swapon /swapfile1
# vi /etc/fstab # Add the following line to fstab
/swapfile1 none swap sw 0 0

# free
             total       used       free     shared    buffers     cached
Mem:       2009540    1936264      73276      81772       8964     679132
-/+ buffers/cache:    1248168     761372
Swap:      1048572          0    1048572

Monitor swap usage

By running free or looking at /proc/meminfo.

# free
             total       used       free     shared    buffers     cached
Mem:       3962700    3838140     124560     148016      85728    2325164
-/+ buffers/cache:    1427248    2535452
Swap:      1048572          0    1048572

# grep -i --color swap /proc/meminfo
SwapCached:            0 kB
SwapTotal:       1048572 kB
SwapFree:        1048572 kB

If you want to move the swap file then turn swap off first by running swapoff /swapfile1.

Memory Overcommit

From a forum thread on StackExchange:

Linux does memory overcommit. That means it allows process to request more memory than really available on the system. When a program tries to malloc(), the kernel says “OK you got the memory”, but don’t reserve it. The memory will only be reserved when the process will write something in this space.
To see the difference, you have 2 indicators: Virtual Memory and Resident Memory. Virtual is the memory requested by the process, Resident is the memory really used by the process.
With this system, you may go into “overbooking”, kernel grants more memory than available. Then, when your system goes on 0 byte of free memory and Swap, he must sacrifice (kill) a process to gain free memory.
That’s when OOM Killer goes into action. The OOM selects a process based on his memory consumption, and many other elements (parent gains ¹⁄₂ of the score of his children; if it’s a root owned process, score is divided by 4, etc.. Have a look on Linux-MM.org/OOM_Killer
You can influence on the OOM scoring by tunning the /proc/MySQL_PID/oom_adj file. By setting it to -17, your process will never be killed. But before doing that, you should tweak your MySQL configuration file in order to limit MySQL memory usage. Otherwise, the OOM Killer will kill other system process (like SSH, crontab, etc…) and your server will be in a very unstable state, maybe leading to data corruption which is worse than anything.
Also, you may consider using more swap.