Drbd Forcing a Full Resync in a Split Brain Situation

I have a DRBD setup similar to an old post that’s being used between two Ubuntu servers hosting MySQL. Every few months though the pair goes into a split-brain situation where the secondary can’t see the primary and refuses to reconnect. Users are unaffected as the primary is still working fine, but the HA is lost.

After trying a few different combinations of commands this is what seems to work best for me and cause the quickest recovery. I’m only dealing with a 10GB device so a full sync takes about 10min. If you’re using DRBD for a much larger device, make sure you consider the sync time before doing this.

On the secondary node, enter the following commands.

Note: After the disconnect, the status will change to “Secondary/Unknown”.

drbdadm secondary all
drbdadm disconnect all
drbdadm invalidate all
drbdadm connect all

On the functioning primary node:

drbdadm connect all

Remember, it’s your data you’re dealing with so make sure you’re responsible before you run commands like this.

Update – no sign of the root cause of the issue either. After a system update that included the drbd package, things seem to have settled down.