内存不足

发生情况：多个服务部署在用一服务器，几个java程序，还有redis，jar目录出现文件【hs_err_pid16493.log】!

#  
# There is insufficient memory for the Java Runtime Environment to continue.  
# Native memory allocation (mmap) failed to map 149946368 bytes for committing reserved memory.  
# Possible reasons:  
#   The system is out of physical RAM or swap space  
#   In 32 bit mode, the process size limit was hit  
# Possible solutions:  
#   Reduce memory load on the system  
#   Increase physical memory or swap space  
#   Check if swap backing store is full  
#   Use 64 bit Java on a 64 bit OS  
#   Decrease Java heap size (-Xmx/-Xms)  
#   Decrease number of Java threads  
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2640), pid=16493, tid=0x00007f2500bfb700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_171-b11) (build 1.8.0_171-b11)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.171-b11 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
	
……
	
Memory: 4k page, physical 8010180k(123544k free), swap 2097148k(0k free)
	
vm_info: Java HotSpot(TM) 64-Bit Server VM (25.171-b11) for linux-amd64 JRE (1.8.0_171-b11), built on Mar 28 2018 17:07:08 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
	
time: Wed Dec 12 17:41:54 2018
elapsed time: 93 seconds (0d 0h 1m 33s)![file](/upload/2019/1/image-155003468825320190213131130891.png)

第一次见到这种东西很懵逼，百度查了下类似文件，翻译开头说内存不足引起的。

查看top，刚开始只注意到Mem 可以空间还有将近200M，就没忘内存方面想，以为是高大上的jvm问题!

后发现swap分区free为0. use100%!

然后就各种查资料，有可以查看swap分区占用程序pid的脚本，便开始解决问题!

vim swap.sh

#!/bin/bash
# Check who used swap

SUM=0
OVERALL=0
for DIR in `find /proc/ -maxdepth 1 -type d | egrep "^/proc/[0-9]"` ; do
PID=`echo $DIR | cut -d / -f 3`
PROGNAME=`ps -p $PID -o comm --no-headers`
for SWAP in `grep Swap $DIR/smaps 2>/dev/null| awk '{ print $2 }'`
do
let SUM=$SUM+$SWAP
done
echo "PID=$PID - Swap used: $SUM - ($PROGNAME )"
let OVERALL=$OVERALL+$SUM
SUM=0
	
	
done
echo "Overall swap used: $OVERALL"

运行命令