Manjusaka

Manjusaka

HomeArchives

Notes on inotify on Linux

#Linux#编程#随笔#总结143
AI Translation
This post is translated from Chinese into English through AI.View Original
AI-generated summary
The author discusses their experience with inotify, a Linux kernel parameter used for monitoring file system events. They explain that inotify can be used to monitor individual files or directories and provides events for various file system changes such as file creation, deletion, and modification. However, inotify has limitations, such as the lack of support for recursive monitoring and restrictions on the number of events and watchers that can be created. The author also mentions a warning they received in their IDE regarding the low inotify watch limit and suggests modifying the corresponding parameters to resolve the issue. They conclude by expressing their appreciation for the Linux system and the need to document their findings for future reference.

Recently, I haven't been in the mood to write any articles. Even the planned papers on Raft have been delayed due to some issues. But I still wanted to write something, so I decided to write a short article to record a problem I encountered today regarding a Linux kernel parameter, and also take some notes.

Getting Started#

I'm not a big fan of Mac, so I use Manjaro as my development environment at home (here's an advertisement, it's a great distribution, ready to use out of the box, only 50 cents for the ad). And my code tools are the JetBrains suite and VSCode.

Today, when I opened Goland, I noticed that the IDE gave me this warning: External file changes sync may be slow: The current inotify(7) watch limit is too low.

So, as you all know, I have a compulsion to address these warnings, so I decided to look it up.

A Brief Discussion#

We often have the need to monitor changes in a file or a directory, such as creating or deleting files. Our usual approach may be to use a brute-force polling method.

However, this approach has poor performance. So, is there any way to handle this?

Yes! Linux provides the corresponding API to handle this, and that's what we're going to talk about today: inotify.

According to the official description, inotify is actually quite simple:

The inotify API provides a mechanism for monitoring file system events. Inotify can be used to monitor individual files, or to monitor directories. When a directory is monitored, inotify will return events for the directory itself, and for files inside the directory.

In other words, inotify is used to monitor file system events. It can be used on individual files or directories. Changes to the monitored file or the files inside the monitored directory are within the scope of monitoring.

After monitoring the corresponding file, inotify will return the following events:

  1. IN_ACCESS: File is readable.

  2. IN_ATTRIB: Metadata has changed.

  3. IN_CLOSE_WRITE: File opened for writing was closed.

  4. IN_CLOSE_NOWRITE: File not opened for writing was closed.

  5. IN_CREATE: File/directory was created in the monitored directory.

  6. IN_DELETE: File/directory was deleted in the monitored directory.

  7. IN_DELETE_SELF: File/directory being monitored was deleted.

  8. IN_MODIFY: File was modified.

  9. IN_MOVE_SELF: File/directory being monitored was moved.

  10. IN_MOVED_FROM: File/directory was moved out of the monitored directory.

  11. IN_MOVED_TO: File/directory was moved into the monitored directory.

  12. IN_OPEN: File was opened.

There are a total of 12 event types, which cover our common needs. However, inotify also has its own drawbacks.

  1. It does not support recursive monitoring. For example, if I monitor directory A, I can capture the event of creating directory B under A. But we cannot capture events under directory B unless we add directory B to the monitoring queue as well.

  2. There are few available inotify libraries for Python.

For the first drawback, the common solution is to implement recursive monitoring ourselves. When there is a file/directory creation event in the main directory, we add the corresponding file/directory to the monitoring queue.

But this brings a new problem. If we do it this way for a very large project, the resulting memory consumption is alarming. So, when inotify was designed, some kernel parameters were set to limit this.

The two most common ones are:

  1. /proc/sys/fs/inotify/max_queued_events: Limits the length of the event queue. Once events start to accumulate, new events will be discarded.

  2. /proc/sys/fs/inotify/max_user_watches: Limits the number of watchers that can be created per User ID, to prevent excessive memory usage due to too many monitors.

By default, the value of max_user_watches depends on the Linux distribution. For most distributions, its value is relatively small. In other words, once the limit is reached, it will not be possible to add new watchers. This is also the reason why the IDE prompts External file changes sync may be slow: The current inotify(7) watch limit is too low.

You can modify the corresponding parameters by editing /etc/sysctl.conf to solve this problem.

Finally#

Linux is truly a treasure trove. It seems like every now and then I come across something I haven't encountered before. So, I'll record it as an article, a random thought, and also for my own reference.

Loading...
Ownership of this post data is guaranteed by blockchain and smart contracts to the creator alone.