Sungju's Slow Life

Personal journal


  • Find binary path of a process in vmcore

    In general, vmcore does not containing user space data which is useful to check process related information. If the vmcore is collected with user space data as well, you can use ps -a <pid> to get some process details like below. It still doesn’t show binary path, though. To get binary details, you can use… Continue reading

  • How to find the actual memory usage by each process

    Checking RSS gives you an idea how much physical memory is consumed at the moment. However, there are shared area such as shared object (.so), shared memory (shmget), etc. To get real amount consumed by application side, those shared area should be divided by the number of processes which is done by kernel in /proc/*/smaps… Continue reading

  • How to check memory usage in Linux – part 1.2 – Per-process level memory usage check

    ps This is the basic command to see process list with memory usage. Here the memory related columns is %MEM, VSZ and RSS. pmap pmap shows the memory allocation layout in a process. You can check what shared object (.so) files are loaded, how much is allocated in heap, how much is used for stack,… Continue reading

  • How to check memory usage in Linux – part 1.1 – System level memory usage check

    free It is showing the overall memory usage in the system and how much is available to use. We are usually focusing on used and free column to see how much memory was consumed and how much is available to use. There are some monitoring tools focus on those two columns only and alarming if… Continue reading

  • How to check memory usage in Linux – part 1. User space

    Here I am explaining how to check memory usage in the user space, especially from the command line. There are two perspective to see memory usage – System level and per-process level. System level memory usage check Per-process level memory usage check System level memory usage check free top /proc/meminfo /proc/buddyinfo /proc/pagetypeinfo vmstat /var/log/sa/ You… Continue reading

  • How to track SLAB usage using slub_debug=U

    When there is high usage in kernel side, most common reason is high usage in SLAB. As the SLAB memory blocks are all data, it is hard to tell which parts of code was consuming this memory blocks. Fortunately, there is this kernel option slub_debug=U which is saving backtrace of the allocation and freeing calls.… Continue reading

  • Difference between ‘pgscank/s’ and ‘pgscand/s’

    In the sar output like below, there are pgscank and pgscand. In the man page, it says. This description is not much helpful to understand the difference between those two columns. These data are from kernel code which is showing in vmstat. It is basically updated when there’s memory reclaiming. If the reclaiming was happened… Continue reading

  • 5-level paging in KVM

    Latest kernels such as RHEL8 and above are supporting 5-level page table. You can check if this is available/enabled by looking at /boot/config-$(uname -r). If you don’t want to use 5-level page table, you can disable that by adding ‘no5lvl’ as a kernel parameter in grub file (/boot/grub2/grubenv) and reboot the system. This 5 level… Continue reading

  • Install mpykdump on local machine

    Here is the sequence of commands to build mpykdump on your system. This is based on crash version 8.0.0. You may want to check what version of crash your system has to match it. Below is the sequence on RHEL8. You can ignore the error from ‘crash’ compile as we are not going to use… Continue reading

  • How to check page caches in the kernel memory.

    If you would like to know what files were occupying the page caches which you can see with ‘free’ in command line or ‘kmem -I’ in kernel memory, you can start with ‘/proc/sys/vm/drop_caches’. I wrote a script that is extracting the page caches from the vmcore based on the above information. You can find the… Continue reading

  • How intel_idle works

    When the system is in IDLE state which means nothing to run and swapper is running, it calls cpuidle_idle_call() like shown in the below. This cpuidle_idle_call() is called from arch_cpu_idle(). cpuidle_idle_call() is the main idle loop which is checking idle driver and do further steps if the driver is installed and active. It can change… Continue reading

  • SystemTap to monitor SLAB usage

    Checking memory leak in SLAB is not an easy task. It is happening in kernel side and it can go through several different route. Here I want to show how we can use SystemTap to monitor those SLAB alloc/free activities. As a first step you need to find out the monitoring points. SLAB can be… Continue reading