Too many open files 的解决方法

>由来

gogs 打开一个android app库的资源文件夹(/coding/Coding-Android/src/master/app/src/main/res/drawable-xxhdpi) 返回 500

>源头

[git@linux gogs]$ ./gogs web
2016/01/12 15:35:31 [T] Custom path: /home/git/opt/gogs/custom
2016/01/12 15:35:31 [T] Log path: /home/git/opt/gogs/log
2016/01/12 15:35:31 [I] Gogs 0.8.19.0111
2016/01/12 15:35:31 [I] Log Mode: File(Info)
2016/01/12 15:35:31 [I] Cache Service Enabled
2016/01/12 15:35:31 [I] Session Service Enabled
2016/01/12 15:35:31 [I] Git Version: 2.5.0
2016/01/12 15:35:31 [I] SQLite3 Supported
2016/01/12 15:35:31 [I] Run Mode: Production
2016/01/12 15:35:31 [T] Doing: CheckRepoStats
2016/01/12 15:35:31 [I] Listen: http://0.0.0.0:2048
[Macaron] Started GET /coding/Coding-Android/src/master/app/src/main/res/drawable-xxhdpi for 127.0.0.1
2016/01/12 15:35:36 [D] Session ID: bd615aa3d0616740
2016/01/12 15:35:36 [D] CSRF Token: Ff7aL72oaVjpxMgcOS0fqcVokWE6MTQ1MjU2NzkxMjM0NTE3Njg2MA==
2016/01/12 15:35:37 [...routers/repo/view.go:134 Home()] [E] GetCommitsInfo: GetCommitByPath (app/src/main/res/drawable-xxhdpi//ic_location_list_check.png): pipe2: too many open files

• 原来是pipe2: too many open files

>ulimit

[git@linux gogs]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31850
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 8182
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

• 嗯，(open files)=4096，那也不小啊，gogs君，这是Bug吗？好吧，我改…

>解决方案

[root@linux gogs]# cat /proc/sys/fs/file-max
813396

[root@linux gogs]# ulimit -n 32768

[root@linux gogs]# ulimit -Sn
32768

[root@linux gogs]# ulimit -Hn
65536

[root@linux gogs]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31850
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 32768
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 32768
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

>终极方案

[root@linux gogs]# cat /etc/security/limits.conf
# /etc/security/limits.conf
#
#This file sets the resource limits for the users logged in via PAM.
#It does not affect resource limits of the system services.
#
#Also note that configuration files in /etc/security/limits.d directory,
#which are read in alphabetical order, override the settings in this
#file in case the domain is the same or more specific.
#That means for example that setting a limit for wildcard domain here
#can be overriden with a wildcard setting in a config file in the
#subdirectory, but a user specific setting here can be overriden only
#with a user specific setting in the subdirectory.
#
#Each line describes a limit for a user in the form:
#
#<domain>        <type>  <item>  <value>
#
#Where:
#<domain> can be:
#        - a user name
#        - a group name, with @group syntax
#        - the wildcard *, for default entry
#        - the wildcard %, can be also used with %group syntax,
#                 for maxlogin limit
#
#<type> can have the two values:
#        - "soft" for enforcing the soft limits
#        - "hard" for enforcing hard limits
#
#<item> can be one of the following:
#        - core - limits the core file size (KB)
#        - data - max data size (KB)
#        - fsize - maximum filesize (KB)
#        - memlock - max locked-in-memory address space (KB)
#        - nofile - max number of open file descriptors
#        - rss - max resident set size (KB)
#        - stack - max stack size (KB)
#        - cpu - max CPU time (MIN)
#        - nproc - max number of processes
#        - as - address space limit (KB)
#        - maxlogins - max number of logins for this user
#        - maxsyslogins - max number of logins on the system
#        - priority - the priority to run user process with
#        - locks - max number of file locks the user can hold
#        - sigpending - max number of pending signals
#        - msgqueue - max memory used by POSIX message queues (bytes)
#        - nice - max nice priority allowed to raise to values: [-20, 19]
#        - rtprio - max realtime priority
#
#<domain>      <type>  <item>         <value>
#

#*               soft    core            0
#*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
#@student        -       maxlogins       4

# End of file

• 编辑这个配置文件修改limits

[root@linux gogs]# vi /etc/security/limits.conf

[root@linux gogs]# cat /etc/security/limits.conf
# /etc/security/limits.conf
#
#<domain>      <type>  <item>         <value>
*   soft    nproc   32768
*   hard    nproc   65536
*   soft    nofile  32768
*   hard    nofile  65536
# End of file

• 系统重启后生效。

>limits.conf的工作原理

limits.conf是pam_limits.so的配置文件
/etc/pam.d/下的应用程序调用pam_***.so模块
当用户访问服务器，服务程序将请求发送到PAM模块，PAM模块根据服务名称在/etc/pam.d 目录下选择一个对应的服务文件，然后根据服务文件的内容选择具体的PAM模块进行处理。
例：限制admin用户登录到sshd的服务不能超过2个在/etc/pam.d/sshd 中添加 session required pam_limits.so 在/etc/security/limits.conf中添加 admin - maxlogins 2
查看应用程序能否被PAM支持，用ldd

>limits.conf文件中的参数配置方式

暂时地，登录 shell 会话期间使用ulimit 命令配置
永久地，通过将一个相应的 ulimit 语句添加到由登录 shell 读取的文件之一（例如 ~/.profile 或者 ~/.bashrc），即特定于 shell 的用户资源文件；或者通过编辑 /etc/security/limits.conf.

>后记

以上方法并没有完全解决gogs:too many files -> return 500的问题,如果以服务的模式运行 [root@linux gogs]# systemctl start gogs.service问题依然存在，但是在Shell 会话中[git@linux gogs]$ ./gogs web 却没有问题，不再出现too many files 的错误。 Google了一下还是没有什么好的思路，就去github.com/gogits/gogs的Issue中找答案，那里本来就有一个关于too manyfiles的讨论-[GitHub Issue]
原来是sysemd在搞怪。systemd对limits的配置是在相应的服务定义文件中（如gogs的服务定义是在 /usr/lib/systemd/system/gogs.service），如果需要配置相应的limits，systemd 提供了对应的directive(eg: LimitNOFILE LimitMEMLOCK …)， systemd忽略通过ulimit配置的limits（是否理会/etc/security/limits.conf中的配置以及如何响应就不得而知了，没有深究）对systemd不是特别了解，有时间再深入研究。
下面是我的gogs.service

[git@linux gogs]$ cat /usr/lib/systemd/system/gogs.service
[Unit]
Description=Gogs (Go Git Service)
After=syslog.target
After=network.target

[Service]
LimitMEMLOCK=infinity
LimitNOFILE=65535
Type=simple
User=git
Group=git
WorkingDirectory=/home/git/opt/gogs
ExecStart=/home/git/opt/gogs/gogs web
Restart=always
Environment=USER=git HOME=/home/git

[Install]
WantedBy=multi-user.target

北野