Process mount is read only

by

I have a rw file system /myraid0.

# cat /proc/mounts | grep myraid0
/dev/mapper/isw_cfdbejjgdi_myraid0p1 /myraid0 ext4 rw,relatime,stripe=8,data=ordered 0 0

I have no problem writing to /myraid0 as any user.

redis@host:~$ echo hi > /myraid0/tmp/redis/test
redis@host:~$ cat /myraid0/tmp/redis/test
hi

However, my process can’t write a file on /myraid0, due to EROFS (Read-only file system).

open("temp-4036.rdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EROFS (Read-only file system)

This shows the process views the mount as ro.

# cat /proc/15920/mounts | grep myraid0
/dev/mapper/isw_cfdbejjgdi_myraid0p1 /myraid0 ext4 ro,relatime,stripe=8,data=ordered 0 0

Why does the process only have read only view of the mount?

Thanks!

Additional details

  • Ubuntu 16.04
  • Linux jeff-apartment-2015 4.4.0-36-generic #55-Ubuntu SMP Thu Aug 11 18:01:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
  • App is redis-server. It’s failing to save the snapshots to a non-root device
  • Redis error is

Failed opening .rdb for saving: Read-only file system

  • Installed version 3.0.6 via apt-get
  • Launched by systemd
  • Listing of /proc/15920/ns (requested by @VenkatC):
    lrwxrwxrwx 1 redis redis 0 Sep 16 05:39 cgroup -> cgroup:[4026531835]
    lrwxrwxrwx 1 redis redis 0 Sep 15 18:03 ipc -> ipc:[4026531839]
    lrwxrwxrwx 1 redis redis 0 Sep 15 18:03 mnt -> mnt:[4026532343]
    lrwxrwxrwx 1 redis redis 0 Sep 15 18:03 net -> net:[4026531957]
    lrwxrwxrwx 1 redis redis 0 Sep 15 18:03 pid -> pid:[4026531836]
    lrwxrwxrwx 1 redis redis 0 Sep 15 18:03 user -> user:[4026531837]
    lrwxrwxrwx 1 redis redis 0 Sep 15 18:03 uts -> uts:[4026531838]
  • apparmor_status | grep redis returns nothing (requested by @Gilles)
  • Nothing else in the logs more informative than the strace output.

Process namespaces

Working bash shell for redis user

ls -l /proc/7359/ns/mnt
lrwxrwxrwx 1 redis redis 0 Sep 15 18:03 /proc/7359/ns/mnt -> mnt:[4026531840]

Not working redis process

ls -l /proc/15920/ns/mnt
lrwxrwxrwx 1 redis redis 0 Sep 15 18:03 /proc/15920/ns/mnt -> mnt:[4026532343]

Working bash shell for my user

ls -atlrh /proc/7138/ns/mnt
lrwxrwxrwx 1 jeff jeff 0 Sep 15 18:03 /proc/7138/ns/mnt -> mnt:[4026531840]

Answer

As you can see, redis process is running in it’s own ‘mount namespace’, with readonly option.

Redis proc started by Systemd [/proc/15920/ns/mnt] -> mnt:[4026532343]

your shell [/proc/7138/ns/mnt] -> mnt:[4026531840]

Look at systemd unit for the redis-server startup and update options related to the mountflags to suit your needs

Looking at redis-server systemd unit file, I see below settings

# grep -i readwrite /etc/systemd/system/redis.service 
ReadWriteDirectories=-/var/lib/redis
ReadWriteDirectories=-/var/log/redis
ReadWriteDirectories=-/var/run/redis
ReadWriteDirectories=-/etc/redis

so you could add /myraid0 as an additional ReadWriteDirectories and restart redis service

Attribution
Source : Link , Question Author : Jeff , Answer Author : VenkatC

Leave a Comment