mdadm: Cannot get exclusive access to /dev/md127

[Rich Emberson]

I've got a two disk raid and I want to it and convert the
disks to some non-raid usage. The raid was never used and has no data.
Also, the raid disks are encrypted along with the rest of the disk
in the machine.

When I try to stop raid I get the following:

# mdadm --stop /dev/md127
mdadm: Cannot get exclusive access to /dev/md127:Perhaps a running process,
mounted filesystem or active volume group?

Googling about this there are a number of folks hitting this problem
over a number of fedora releases. Sadly, I could not find any fix.
For some the solution (nuclear solutions) was to use a different linux
distro or reinstall. I've got a lot of data on other disks on this system
and
really don't want to use a nuclear solution.


Background information:

$ uname -a
Linux medusa 3.17.7-200.fc20.x86_64 #1 SMP Wed Dec 17 03:35:33 UTC 2014
x86_64 x86_64 x86_64 GNU/Linux

MDADM
# mdadm --misc --detail /dev/md127
Version : 1.2
Creation Time : Fri Apr 25 17:19:52 2014
Raid Level : raid1
Array Size : 3906885632 (3725.90 GiB 4000.65 GB)
Used Dev Size : 3906885632 (3725.90 GiB 4000.65 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent

Intent Bitmap : Internal

Update Time : Sun Feb 1 09:37:32 2015
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Name : localhost.localdomain:raid1
UUID : be78cb55:f26bbf2b:7ad9b344:a2427875
Events : 8527

Number Major Minor RaidDevice State
0 8 33 0 active sync /dev/sdc1
1 8 49 1 active sync /dev/sdd1

MDADM.CONF
# cat /etc/mdadm.conf
# mdadm.conf written out by anaconda
MAILADDR root
AUTO +imsm +1.x -all
ARRAY /dev/md/raid1 level=raid1 num-devices=2
UUID=be78cb55:f26bbf2b:7ad9b344:a2427875

LSOF
# lsof | grep md127
md127_rai 1044 root cwd DIR 253,1
4096 2 /
md127_rai 1044 root rtd DIR 253,1
4096 2 /
md127_rai 1044 root txt
unknown /proc/1044/exe
udisksd 2120 root 13r REG 0,15
4096 30332 /sys/devices/virtual/block/md127/md/sync_action
udisksd 2120 root 14r REG 0,15
4096 30345 /sys/devices/virtual/block/md127/md/degraded
gmain 2120 2121 root 13r REG 0,15
4096 30332 /sys/devices/virtual/block/md127/md/sync_action
gmain 2120 2121 root 14r REG 0,15
4096 30345 /sys/devices/virtual/block/md127/md/degraded
gdbus 2120 2123 root 13r REG 0,15
4096 30332 /sys/devices/virtual/block/md127/md/sync_action
gdbus 2120 2123 root 14r REG 0,15
4096 30345 /sys/devices/virtual/block/md127/md/degraded
probing-t 2120 2124 root 13r REG 0,15
4096 30332 /sys/devices/virtual/block/md127/md/sync_action
probing-t 2120 2124 root 14r REG 0,15
4096 30345 /sys/devices/virtual/block/md127/md/degraded
cleanup 2120 2125 root 13r REG 0,15
4096 30332 /sys/devices/virtual/block/md127/md/sync_action
cleanup 2120 2125 root 14r REG 0,15
4096 30345 /sys/devices/virtual/block/md127/md/degraded

FSTAB ENTRY
/dev/mapper/luks-ab8911f9-b3a4-4f33-8b10-7e52c7217ae9
/raid1 ext4 defaults,x-systemd.device-timeout=0 1 2

BY-UUID
# ls -l /dev/disk/by-uuid/ | grep md127
lrwxrwxrwx. 1 root root 11 Jan 10 19:32
ab8911f9-b3a4-4f33-8b10-7e52c7217ae9 -> ../../md127

Thanks
Richard

[Chris Murphy]

What do you get for:

dmsetup ls
cryptsetup status

/dev/mapper/luks-ab8911f9-b3a4-4f33-8b10-7e52c7217ae9 /raid1

This suggests an encrypted device mapper device created from the raid1
device, and it's active. So that has to be closed, or wipefs before
you can stop the raid and then wipe the raid, i.e. you need to tear it
down in the reverse order it was created.

You can try this: DOUBLE CHECK DEVICE VALUES
wipefs --backup -a /dev/md127
mdadm -S /dev/md127
wipefs --backup -a /dev/sd[cd]1

That will backup the luks signature, then wipe it, stop the array,
then backup and wipe the mdadm superblock signature. That ought to fix
the problem unless there's also LVM stuff going on here too.

The --backup isn't strictly necessary but will write out a file and
instructions on how to restore what it wipes in case you get the
command wrong and wipe the wrong device.
--
Chris Murphy

[Rich Emberson]

Thanks - wipefs fails see below

# dmsetup ls
luks-ab8911f9-b3a4-4f33-8b10-7e52c7217ae9 (253:9)
luks-35719a97-5898-4420-9a56-1576ffdc6db3 (253:1)
luks-c80d9988-30f6-42cf-a9b7-89ee3408eccb (253:6)
luks-03c06df8-f9b9-4f0d-847e-79a7ed527888 (253:0)
luks-a2ebb2b0-527d-47f3-83ef-e5908805f31d (253:3)
luks-5ee2ed8e-4bdf-43e1-adb0-34a70610a77f (253:2)
luks-d348150c-3a5d-456e-bedd-1cfaa51839f9 (253:7)
luks-24ef3c76-0b18-45e3-a532-5983f77be5e3 (253:4)
luks-02fdb3bb-bd65-4d50-a586-63bc13e0dc30 (253:5)
luks-2871ae45-e563-4ae4-991d-d71693c749bf (253:8)
# cryptsetup status luks-ab8911f9-b3a4-4f33-8b10-7e52c7217ae9
/dev/mapper/luks-ab8911f9-b3a4-4f33-8b10-7e52c7217ae9 is active.
type: LUKS1
cipher: aes-xts-plain64
keysize: 512 bits
device: /dev/md127
offset: 4096 sectors
size: 7813767168 sectors
mode: read/write

# wipefs --backup -a /dev/md127
wipefs: error: /dev/md127: probing initialization failed: Device or
resource busy

# mdadm -S /dev/md127
mdadm: Cannot get exclusive access to /dev/md127:Perhaps a running process,
mounted filesystem or active volume group?


On 02/01/2015 08:19 PM, Chris Murphy wrote:

What do you get for:

dmsetup ls
cryptsetup status

I see this:

/dev/mapper/luks-ab8911f9-b3a4-4f33-8b10-7e52c7217ae9 /raid1


This suggests an encrypted device mapper device created from the raid1
device, and it's active. So that has to be closed, or wipefs before
you can stop the raid and then wipe the raid, i.e. you need to tear it
down in the reverse order it was created.

You can try this: DOUBLE CHECK DEVICE VALUES
wipefs --backup -a /dev/md127
mdadm -S /dev/md127
wipefs --backup -a /dev/sd[cd]1

That will backup the luks signature, then wipe it, stop the array,
then backup and wipe the mdadm superblock signature. That ought to fix
the problem unless there's also LVM stuff going on here too.

The --backup isn't strictly necessary but will write out a file and
instructions on how to restore what it wipes in case you get the
command wrong and wipe the wrong device.

[Chris Murphy]

Post by Rich Emberson
# cryptsetup status luks-ab8911f9-b3a4-4f33-8b10-7e52c7217ae9
/dev/mapper/luks-ab8911f9-b3a4-4f33-8b10-7e52c7217ae9 is active.

That's the problem. First use df or mount to see if this device has a
mounted filesystem, and if so unmount it. Then you need to close it.

cryptsetup luksClose luks-ab8911f9-b3a4-4f33-8b10-7e52c7217ae9

Now you can wipefs the md127 device, then stop the array, then wipe
the members of the array.
--
Chris Murphy

[Chris Murphy]

Also, you probably want to check crypttab to make sure this device
entry is removed, and also from fstab to make sure whatever file
system was on it isn't going to be automounted anymore. Otherwise good
chance you'll get a failed boot while a non-existent fs is being
looked for (and it'll take a while to time out to get to this failure
also).

If you still get a boot delay and ultimately a failure looking for
this device, rebuild the initramfs with dracut -f.

Chris Murphy

[Rich Emberson]

Tried:

# wipefs -a /dev/md127

which did not word, so I next tried:

# wipefs -f -a /dev/md127
/dev/md127: 6 bytes were erased at offset 0x00000000 (crypto_LUKS): 4c 55
4b 53 ba be

and then

# mdadm -S /dev/md127
mdadm: Cannot get exclusive access to /dev/md127:Perhaps a running process,
mounted filesystem or active volume group?

but the filesystem is still mounted.

curious, that creating an encrypted raid system on fedora seems to be an
irreversible action.

Now, with part of the raid file system "wipefs"-ed, above, I fear rebooting
and hanging when it does not come up.

Richard

[Rick Stevens]

Post by Rich Emberson
# wipefs -a /dev/md127
# wipefs -f -a /dev/md127
/dev/md127: 6 bytes were erased at offset 0x00000000 (crypto_LUKS): 4c
55 4b 53 ba be
and then
# mdadm -S /dev/md127
mdadm: Cannot get exclusive access to /dev/md127:Perhaps a running
process, mounted filesystem or active volume group?
but the filesystem is still mounted.

Wiping a filesystem (and certainly using a force option) is a bad idea
if the filesystem is mounted. You should have unmounted it before you
did the wipefs, but I'm sure you knew that.

"lsof +d /mountpoint" or "lsof +D /mountpoint" should show you what is
using the filesystem if a "umount" doesn't work. In dire straits, you
could try "umount -f" to force an unmount, but again if something is
using the filesystem, that could be bad.

Also, just in case you're using LVM, make sure you disable/delete any
logical volumes in the volume group ("lvchange" and "lvremove") and
disable/delete the volume group as well ("vgchange" and "vgremove").
I'd also recommend destroying the PVs associated with the deleted VGs
before you try to delete the RAID.

Post by Rich Emberson
curious, that creating an encrypted raid system on fedora seems to be an
irreversible action.

IIRC, it's a one-way encryption of the data. If that's the case, of
course it's irreversible. If you're talking about not being able to get
rid of an encrypted filesystem, you're just not doing it correctly. All
filesystems can be deleted unless there's something specific in the
hardware that prevents it (protected partitions of FLASH drives, for
example).

Post by Rich Emberson
Now, with part of the raid file system "wipefs"-ed, above, I fear
rebooting and hanging when it does not come up.

Make sure the filesystem is not in your /etc/fstab file (or has the
mount option "noauto") so the system doesn't try to mount it on the
reboot.
----------------------------------------------------------------------
- Rick Stevens, Systems Engineer, AllDigital ***@alldigital.com -
- AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 -
- -
- You know the old saying--any technology sufficiently advanced is -
- indistinguishable from a Perl script -
- --Programming Perl, 2nd Edition -
----------------------------------------------------------------------