dfmicro

Run MicroShift clusters inside rootful Podman containers. Each cluster gets its own network, loop-device backed LVM storage, and a kubeconfig. Workers can be added when multi-node operation is needed.

Read the dfmicro documentation for versioned guides and examples.

Verified on Linux (Fedora / RHEL). Best-effort support on macOS via rootful Podman machine.

Installation

The installer detects Linux or macOS and the amd64 or arm64 architecture, then downloads and verifies the matching release archive:

curl -fsSL https://raw.githubusercontent.com/leelavg/dfmicro/main/install.sh | sh

Set VERSION to install a specific release, or INSTALL_DIR to change the destination directory:

curl -fsSL https://raw.githubusercontent.com/leelavg/dfmicro/main/install.sh |
  VERSION=0.0.2 INSTALL_DIR="$HOME/bin" sh

List available releases without installing:

curl -fsSL https://raw.githubusercontent.com/leelavg/dfmicro/main/install.sh | sh -s -- --list

Or build from source:

git clone -b main https://github.com/leelavg/dfmicro
cd dfmicro
make build

Quick start

One-time setup

Linux: Configure the required privileged commands once:

dfmicro ops sudoers create

macOS: Initialize a rootful Podman machine once. If you do not already have one, run:

podman machine init --rootful

Start the machine whenever you use dfmicro:

podman machine start

Create a cluster

dfmicro cluster create              # create cluster with default name
dfmicro cluster kubeconfig          # outputs the cluster kubeconfig to stdout
kubectl get nodes                   # save above config and run cluster operations

dfmicro docs                        # for all the cli commands and args
dfmicro docs --examples             # samples for specific scenarios

Testing

Run the suite on a clean Linux host with rootful Podman and sudoers created against the built binary:

DFMICRO_BIN=./bin/dfmicro perl tests.pl --upto 8

Levels 4-7 require a local tests.conf, copied from tests.conf.example.

Run one level from a clean slate, including only that level's setup:

DFMICRO_BIN=./bin/dfmicro perl tests.pl --level 6

See all test options with:

./tests.pl -h

Usage:	perl tests.pl [options]

  --upto N        	Run through level N (0-8) (default: 0)
  --level N       	Run only level N (0-8) from a clean slate
  --etcd          	Create clusters with etcd
  --list [MODE]   	List "tests" or "cmds" (default: tests)
  --keep          	Leave resources for inspection
  --pause         	Wait for Enter before cleanup
  --fail-fast     	Stop after the first failed check
  --cleanup       	Remove suite resources without testing
  --timeout DURATION	Limit total runtime, for example 15m
  --help          	Show this help

Test output and executed commands are written to /tmp/dfmicro-test/tests.log and /tmp/dfmicro-test/cmds.log.

Command reference

See internal/docs/cli.md for the full command reference, or run dfmicro docs to print it.

For development setup and design notes, see dev.md. For what is planned and what is done, see the devlog.

FAQ

How do I access cluster routes from my host?

IPs and Domain names mentioned are for default values, edit as per your cluster settings.

Routes use a cluster-specific base domain, such as metrics.apps.demo.dfmicro.io, and need DNS resolution to the cluster node IP. Find your cluster's node IP:

kubectl get nodes -o wide | grep 172.20

Choose your setup method (replace 172.20.0.11 with your actual node IP):

Linux with NetworkManager + dnsmasq:

echo 'address=/apps.demo.dfmicro.io/172.20.0.11' | sudo tee /etc/NetworkManager/dnsmasq.d/dfmicro.conf

sudo nmcli general reload dns-full
curl -k https://metrics.apps.demo.dfmicro.io/metrics

Linux with standalone dnsmasq:

echo 'address=/apps.demo.dfmicro.io/172.20.0.11' | sudo tee /etc/dnsmasq.d/dfmicro.conf

sudo systemctl reload dnsmasq
curl -k https://metrics.apps.demo.dfmicro.io/metrics

Linux with systemd-resolved:

echo 'DNS=172.20.0.11' | sudo tee /etc/systemd/resolved.conf.d/dfmicro.conf
echo 'Domains=~apps.demo.dfmicro.io' | sudo tee -a /etc/systemd/resolved.conf.d/dfmicro.conf
sudo systemctl restart systemd-resolved
curl -k https://metrics.apps.demo.dfmicro.io/metrics

Linux / macOS:

echo '172.20.0.11 metrics.apps.demo.dfmicro.io' | sudo tee -a /etc/hosts
curl -k https://metrics.apps.demo.dfmicro.io/metrics

Or use curl --resolve (no /etc/hosts edit):

curl -k --resolve metrics.apps.demo.dfmicro.io:443:172.20.0.11 https://metrics.apps.demo.dfmicro.io/metrics

Cleanup:

sudo rm -f /etc/NetworkManager/dnsmasq.d/dfmicro.conf /etc/dnsmasq.d/dfmicro.conf
sudo nmcli general reload dns-full || sudo systemctl reload dnsmasq

Why dnsmasq instead of in-cluster DNS?

Podman's aardvark-dns only resolves container names, not in-cluster routes. It has no upstream config to forward to CoreDNS. The router on node IP `172.20.0.11` handles SNI dispatch, so dnsmasq just needs to answer `*.apps.demo.dfmicro.io` with that IP.

System reboots with "watchdog did not stop!" in journal

Symptom: Machine reboots hard during cluster creation, especially with multiple clusters. Journal shows:

kernel: watchdog: watchdog0: watchdog did not stop!
kernel: watchdog0: pretimeout event

Cause: Cluster startup causes heavy disk I/O. Real-time antivirus/EDR scanning processes (e.g., wdavdaemon from Microsoft Defender) scan this I/O, causing kernel stalls. On encrypted drives, the encryption/decryption overhead compounds the scanning load. Hardware watchdog (iTCO_wdt, 30-second heartbeat) assumes OS is frozen and forces hard reboot.

Fix: Exclude container storage and microshift processes from real-time scanning:

# Exclude container storage (all clusters)
sudo mdatp exclusion folder add --path "/var/lib/containers"

# Exclude cluster LVM backing files (optional, for extra safety)
dfmicro cluster config --name cluster-name | grep lvm-disk
# Example: lvm-disk: /home/user/.config/dfmicro/cluster-name/cluster-name.img
sudo mdatp exclusion folder add --path "/home/user/.config/dfmicro"

# Exclude microshift processes
sudo mdatp exclusion process add --name microshift
sudo mdatp exclusion process add --name microshift-etcd

# Restart mdatp
sudo systemctl restart mdatp

If using different scanning software (ClamAV, Sophos, etc.), add equivalent exclusions for /var/lib/containers and the cluster config directory.

dfmicro also mounts etcd to tmpfs to minimize disk I/O during startup.

Last resort: If exclusions don't work and reboots persist, blacklist the hardware watchdog entirely:

echo "blacklist iTCO_wdt" | sudo tee /etc/modprobe.d/disable-watchdog.conf
echo "blacklist intel_oc_wdt" | sudo tee -a /etc/modprobe.d/disable-watchdog.conf
sudo modprobe -r iTCO_wdt
sudo modprobe -r intel_oc_wdt

Disables hardware watchdog completely. wdavdaemon falls back to software watchdog. Safe for development.

Why not fixed in dfmicro: Real-time scanning is a host-level security feature. The exclusion must be configured on the host.

Contributing

Bug reports and suggestions are welcome via issues. This is a personal project with a focused scope and occasional history rewrites for now, so pull requests are not being accepted at this time.

Acknowledgements

None of this would exist without the incredible work at MicroShift.

Thanks to Anika and Tara for sourcing df shims and carrying out the proof of concept under guidance.

Thanks to all the folks whose shared knowledge is archived in OpenAI Luna, Claude Sonnet/Haiku and IBM Bob agents which made the timeline shorter.

License

Apache 2.0. See LICENSE.

Keyboard shortcuts

dfmicro

dfmicro