A few notes relating to working with the linux memory.

Understanding Linux Memory

See: http://www.kernel.org/doc/gorman/

This is: a mirror of Mel Gorman's book "Understanding the Linux Virtual Memory Manager".

Clearing caches : /proc/sys/vm/drop_caches

It's possible to directly clear caches (pagecache, dentries, inodes) by using /proc:


Writing to this will cause the kernel to drop clean caches, dentries and
inodes from memory, causing that memory to become free.

To free pagecache:
        echo 1 > /proc/sys/vm/drop_caches
To free dentries and inodes:
        echo 2 > /proc/sys/vm/drop_caches
To free pagecache, dentries and inodes:
        echo 3 > /proc/sys/vm/drop_caches

As this is a non-destructive operation and dirty objects are not freeable, the
user should run `sync' first.

See: http://www.kernel.org/doc/Documentation/sysctl/vm.txt

I've found this useful for testing file IO intensive applications (where I want to be sure that the application is reading from disk rather than from pagecache). It's also potentially useful for clearing out the dentries and inodes after (or during) running an application that opens many many files (because the kernel caches these in case they are used again and releases them back from memory gradually). See also my comments on vfs_cache_pressure in this page).

Reducing inode/dentry time in the cache: /proc/sys/vm/vfs_cache_pressure

Controls the tendency of the kernel to reclaim the memory which is used for
caching of directory and inode objects.

At the default value of vfs_cache_pressure=100 the kernel will attempt to
reclaim dentries and inodes at a "fair" rate with respect to pagecache and
swapcache reclaim.  Decreasing vfs_cache_pressure causes the kernel to prefer
to retain dentry and inode caches. When vfs_cache_pressure=0, the kernel will
never reclaim dentries and inodes due to memory pressure and this can easily
lead to out-of-memory conditions. Increasing vfs_cache_pressure beyond 100
causes the kernel to prefer to reclaim dentries and inodes.

See: http://www.kernel.org/doc/Documentation/sysctl/vm.txt

I've seen this discussed relating to slocate (https://lkml.org/lkml/2006/8/3/173, and see some useful pointers later in the thread: https://lkml.org/lkml/2006/8/4/111) but it could be effective on any server where an application which reads over a vast number of files only once causes slab_cache to grow rapidly and consume a large amount of memory that you know can be safely released. This could cause performance problems if other applications are actively using contents of the cache that now need to be reallocated.

What's in the slab?

Frequently used objects are cached in the slab cache (such as the inodes and dentries mentioned above) - detailed information can be found in: /proc/slabinfo

See documentation: http://www.kernel.org/doc/man-pages/online/pages/man5/slabinfo.5.html

pdflush and the linux page cache

The problem: A system is running perfectly well much of the time but has periods of absolute lock-up with messages like "INFO: task cron:4741 blocked for more than 120 seconds.". During these times activity on the system grinds to a halt and activities like ssh logins fail/time-out.

Worth a try is to check and adjust the LinuxIoScheduler - though the effectiveness of this approach is difficult to find and seems to provide a more subtle performance change.

Looking at the system during problem periods we have the pdflush appearing quite often, and looking at system monitoring tools (for example http://munin-monitoring.org/) the size of the buffered/dirty cached data is large and/or fluctuating.

Another place to look is to adjust the behaviour of pdflush - the kernel daemon responsible for pushing cached disk writes back to disk. Here Gregory Smith's article "The Linux Page Cache and pdflush: Theory of Operation and Tuning for Write-Heavy Loads" has a lot of good ideas. Primarily the points on tuning recommendations for write-heavy operations. Of these points the main one is that dirty_background_ratio should be the focus - a setting that allows the amount of dirty cached data kept in memory to be tuned.

Main references:

Other External References

BradsWiki: LinuxMemoryManagement (last edited 2013-11-24 10:38:56 by BradleyDean)