How to run large mongodb database SQL over 1GB

Most UNIX-like operating systems, including Linux and OS X, provide ways to limit and control the usage of system resources such as threads, files, and network connections on a per-process and per-user basis. These “ulimits” prevent single users from using too many system resources. Sometimes, these limits have low default values that can cause a number of issues in the course of normal MongoDB operation.

The open-file-limit exists to prevent users/process to use up all resources on a machine. Every file-descriptor uses a certain amount of RAM and a malicious or malfunctioning program could bring down the whole server. An open file may be a regular file, a directory, a block special file, a character special file, an executing text reference, a library, a stream or a network file (Internet socket, NFS file or UNIX domain socket.).

I used the below command to check the Max open file hard and soft limit,

cat /proc/<pid>/limits

I tried ulimit command, but that is not helpful to set limit of specific process.

# ulimit -n 2048

But still i am seeing 1024 in

# cat /proc/<pid>/limits
..
Max open files 1024 4096 files
..

The initial soft and hard limits for open files are set in /etc/security/limits.conf and enforced at login through the PAM-module pam_limits.so.

To change the system-wide open files limit, Append or change the following line in /etc/sysctl.conf

fs.file-max = 100000

(replace 100000 with the desired number). Then apply the changes to the running system with:

$> sysctl -p

To change the ulimit for users, edit the file /etc/security/limits.conf or append:

www-data soft nofile 8192
www-data hard nofile 8192

Set the soft to the hard-limit, so you don’t have to raise it manually, as user. It is also possible to set a wildcard:

* soft nofile 8192
* hard nofile 8192

For root the wildcard will not work and extra lines have to be added:

root soft nofile 8192
root hard nofile 8192

The ulimit shell built-in can show current limits or set limits for the current process (shell) and all subsequent child processes only. It can’t be used to change limits for already running processes (e.g. system daemons, programs started previously from the current shell or at any time from other shells or started by cron or inetd etc).

prlimit(1) allows to get or set one or more process resource limits for given PID.

prlimit –pid 13134 –nofile=1024:64000

Set the soft and hard limits for the number of open files to 1024 and 4095, respectively for process ID 13134.

Leave a Reply