内存不足

  • 发生情况: 多个服务部署在用一服务器,几个java程序,还有redis,jar目录出现文件【hs_err_pid16493.log】!
#  
# There is insufficient memory for the Java Runtime Environment to continue.  
# Native memory allocation (mmap) failed to map 149946368 bytes for committing reserved memory.  
# Possible reasons:  
#   The system is out of physical RAM or swap space  
#   In 32 bit mode, the process size limit was hit  
# Possible solutions:  
#   Reduce memory load on the system  
#   Increase physical memory or swap space  
#   Check if swap backing store is full  
#   Use 64 bit Java on a 64 bit OS  
#   Decrease Java heap size (-Xmx/-Xms)  
#   Decrease number of Java threads  
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2640), pid=16493, tid=0x00007f2500bfb700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_171-b11) (build 1.8.0_171-b11)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.171-b11 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
	
……
	
Memory: 4k page, physical 8010180k(123544k free), swap 2097148k(0k free)
	
vm_info: Java HotSpot(TM) 64-Bit Server VM (25.171-b11) for linux-amd64 JRE (1.8.0_171-b11), built on Mar 28 2018 17:07:08 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
	
time: Wed Dec 12 17:41:54 2018
elapsed time: 93 seconds (0d 0h 1m 33s)![file](/upload/2019/1/image-155003468825320190213131130891.png)

第一次见到这种东西很懵逼,百度查了下类似文件,翻译开头说内存不足引起的。

查看top,刚开始只注意到Mem 可以空间还有将近200M,就没忘内存方面想,以为是高大上的jvm问题!

后发现swap分区free为0. use100%!

然后就各种查资料,有可以查看swap分区占用程序pid的脚本,便开始解决问题!

vim swap.sh

#!/bin/bash
# Check who used swap

SUM=0
OVERALL=0
for DIR in `find /proc/ -maxdepth 1 -type d | egrep "^/proc/[0-9]"` ; do
PID=`echo $DIR | cut -d / -f 3`
PROGNAME=`ps -p $PID -o comm --no-headers`
for SWAP in `grep Swap $DIR/smaps 2>/dev/null| awk '{ print $2 }'`
do
let SUM=$SUM+$SWAP
done
echo "PID=$PID - Swap used: $SUM - ($PROGNAME )"
let OVERALL=$OVERALL+$SUM
SUM=0
	
	
done
echo "Overall swap used: $OVERALL"

运行命令

./swap.sh |sort -nr -k5 |head

file

发现有许多java程序pid并不存在但是占用swap很大,故将这些pid kill掉即可。

查看TOP

file

swap分区恢复正常

996.icu

最近的文章

硬盘空间不足

硬盘空间不足某服务器没隔一段时间经常连接不上登陆服务器查看输入命令cd/u按tab键自动补全,发现会…

继续阅读
更早的文章

Hello Motor!

坐标:廊坊岗位:Java程序员座驾:春分NK250!微信:欢迎各位前来骚扰!…

继续阅读
0评论