Cloud & infrastructure

Argo CD self-heal race condition skips database migration hooks

A race condition in Argo CD causes sync hooks to be skipped when a deny window closes, leading to schema mismatches despite a healthy status.

Illustration of a race condition in synchronization mechanisms
Illustration generated for this article

A recent investigation by CNCF Ambassador Afzal Ansari reveals a critical race condition in Argo CD that can cause database migrations to fail silently. When a sync deny window closes, automated self-healing may trigger a partial sync that skips hooks, leaving applications running new code against old schemas. This issue affects setups using automated sync with prune and self-heal enabled alongside sync windows.

What happened

A customer reported that their Argo CD applications showed a successful sync status, yet the associated database migration had not executed. The team had pushed a release while a deny sync window was active. Upon the window's expiration, five regional applications attempted to roll out the same commit. Three applications successfully ran the migration, but two did not. For 36 to 44 minutes, the affected services operated with code expecting a new schema while the database remained on the old version.

The failure was difficult to detect because the Argo CD user interface reported the applications as Synced and Healthy. The operation log indicated that all tasks had run successfully. The only indicator of the problem was found in the controller logs, which showed a self-heal attempt count of one and listed only a subset of resources. This confirmed that Argo CD had performed a drift correction rather than a full deployment, and drift corrections do not execute hooks.

Ansari reproduced the issue in a controlled lab environment using a simple application with a ConfigMap, Deployment, and a Sync hook Job for migration. By pushing a release during an active deny window and timing the next periodic refresh, he triggered the failure in five out of six attempts. The application appeared to sync the new version, but the migration job retained the label of the previous release, proving the hook had been skipped.

How it works

The root cause lies in how Argo CD handles different types of application refreshes. A level-2 refresh is driven by a reconciliation timer, typically occurring every two to three minutes, and compares the live state against the target revision in Git. A level-1 refresh is triggered by changes to managed objects and reuses existing controller knowledge. When a sync window closes, there is no explicit event to announce it. The next refresh, whether level-1 or level-2, simply observes that syncing is now permitted.

During the deny window, timer-driven level-2 refreshes continue to compare the new commit and update the status.sync.revision field, even though no actual sync occurs. If a level-1 refresh arrives first after the window closes, it uses this recorded revision for comparison. The controller evaluates the revision changes, finds no difference between the recorded revision and itself, and concludes that no changes are pending. Consequently, the alreadyAttemptedSync check passes, and the controller initiates a self-heal operation.

Self-healing performs a partial sync, updating only the resources that are out of sync. Crucially, partial syncs skip all hooks, including PreSync, Sync, and PostSync hooks. In contrast, if a level-2 refresh had arrived first, it would have compared the last synced revision against the target revision, detected the mismatch, and triggered a full sync that includes hooks. The bug effectively mistakes the last compared revision for the last synced revision on the level-1 path.

Key details

  • The issue occurs when automated sync, prune, and self-heal are enabled alongside sync deny windows.
  • Partial syncs triggered by self-healing skip all resource hooks, regardless of their phase.
  • The RunHooksOnPartialSync option does not resolve the issue because it is ignored for automated syncs.
  • Changing hook types from Sync to PreSync does not prevent the skip, as partial syncs ignore all hook types.
  • The race condition depends on whether a level-1 or level-2 refresh occurs first after a window closes.
  • Applications appear healthy and synced in the UI even when the migration hook has been skipped.

Why it matters

For teams relying on Argo CD for continuous delivery, this behavior undermines the reliability of database migrations managed via hooks. Engineers often treat hooks as guaranteed execution steps for schema changes, but this race condition demonstrates that they are conditional on the type of sync operation. A silent failure where code and schema diverge can lead to runtime errors, data corruption, or service outages that are difficult to diagnose due to the misleading healthy status in the dashboard.

This issue highlights a broader architectural risk in treating migrations as side effects of the sync process rather than part of the desired state. When migrations are decoupled from the core resource definitions, they become vulnerable to operational nuances like partial syncs. Teams must recognize that self-healing, while useful for correcting manual drift, introduces a path that bypasses critical lifecycle events. Understanding this distinction is essential for designing resilient deployment pipelines that do not depend on timing luck.

What you can do

  • Replace sync hooks with managed resources by removing hook annotations and using sync waves to order execution.
  • Assign unique names to migration jobs per release, such as including the image tag, to ensure they are treated as new resources.
  • Enable prune to clean up old migration jobs and avoid setting ttlSecondsAfterFinished to prevent premature deletion.
  • Disable self-heal on applications that require mandatory migrations, accepting that manual drift will remain out of sync until the next commit.
  • Test deliberately failing migrations in staging environments to verify that the new wave-based approach handles errors correctly.
  • Avoid relying on RunHooksOnPartialSync or changing hook phases as fixes, as these do not address the underlying race condition.
  • #argo cd
  • #kubernetes
  • #devops
  • #database migrations
  • #gitops

Tools from the Bytechap store

Keep reading

All stories