Problem

systemctl stop datadog-agent-installer consistently hung for the full TimeoutStopSec (90 seconds) before systemd sent SIGKILL. This affected every restart triggered by the fleet setup script.

Root cause investigation

Root cause 1 (fixed): fxutil.OneShot swallowed SIGTERM

The daemon used fxutil.OneShot instead of fxutil.Run. fx.New() registers a SIGTERM handler internally via signalReceivers.Start() which populates app.Done(). OneShot never reads app.Done(), so SIGTERM was consumed silently and the process only died on SIGKILL.

Fix: switched runFxWrapper to fxutil.Run, which blocks on <-app.Done() and triggers the full fx graceful shutdown on SIGTERM.

Root cause 2 (fixed): get-states subprocess blocked on packages.db file lock

Even after the fxutil.Run fix, the daemon's systemd unit still hung for 90s. Inspecting the cgroup during the hang revealed the culprit:

CGroup: /system.slice/datadog-agent-installer.service
        └─1028888 .../installer get-states

The main daemon process had already exited, but an orphaned installer get-states subprocess remained alive in the service cgroup. Systemd waits for all cgroup members to exit before declaring the unit stopped.

The get-states subprocess calls installer.NewInstaller() → db.New() → bbolt.Open() to acquire an exclusive file lock on packages.db. If another process (e.g., the fleet setup script installing packages concurrently) holds that lock, db.New blocks for up to 5 minutes with no way to interrupt it — bbolt's Timeout option is not context-aware.

When systemd's final-sigterm delivered SIGTERM to the orphaned subprocess, handleSignals caught it but immediately entered a 10-second sleep before cancelling the command's context. Even after the sleep, the context cancellation had no effect on bbolt.Open, so the subprocess remained stuck until systemd sent SIGKILL at 90s.

Fix: made db.New accept a context.Context and race the bbolt open against context cancellation in a goroutine:

func New(ctx context.Context, dbPath string, opts ...Option) (*PackagesDB, error) {
    ch := make(chan result, 1)
    go func() {
        db, err := bbolt.Open(dbPath, 0644, &bbolt.Options{Timeout: o.timeout, ...})
        ch <- result{db, err}
    }()
    select {
    case <-ctx.Done():
        return nil, ctx.Err()
    case r := <-ch:
        ...
    }
}

The context is now threaded from the installer command's signal-cancellable context (cmd.ctx) all the way down to db.New. When get-states receives SIGTERM, handleSignals sleeps 10s then calls stop(), which cancels cmd.ctx, which unblocks db.New immediately — the subprocess exits cleanly within ~10s instead of 90s.

Test plan

• Deploy to a host running the fleet setup script and confirm systemctl stop datadog-agent-installer completes in ≤15s
• Verify systemctl restart datadog-agent-installer no longer hangs 90s between stop and start
• Run dda inv test --targets=./pkg/fleet/installer/db/... and ./pkg/fleet/installer/...