pFad - Phone/Frame/Anonymizer/Declutterfier! Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

URL: http://github.com/apache/airflow/pull/64127

ef="https://github.githubassets.com/assets/actions-0e714a98ea09295a.css" /> Add provider class registration checks to provider.yaml validation by potiuk · Pull Request #64127 · apache/airflow · GitHub
Skip to content

Add provider class registration checks to provider.yaml validation#64127

Merged
potiuk merged 4 commits intoapache:mainfrom
potiuk:add-provider-class-registration-checks
Mar 31, 2026
Merged

Add provider class registration checks to provider.yaml validation#64127
potiuk merged 4 commits intoapache:mainfrom
potiuk:add-provider-class-registration-checks

Conversation

@potiuk
Copy link
Copy Markdown
Member

@potiuk potiuk commented Mar 24, 2026

Add two new checks to run_provider_yaml_files_check.py that verify provider classes
are properly registered in provider.yaml, closing the gap where existing checks only
validated yaml→code (classes listed in YAML exist) but not code→yaml (classes in code
are listed).

New checks

1. Hook conn_type registration check

  • Imports all hook modules, finds BaseHook subclasses that define conn_type directly
  • Verifies the connection type is registered in connection-types
  • Async hooks sharing conn_type with an already-registered sync hook are allowed
  • Skips abstract classes

2. Comprehensive class registration check

  • Walks all provider source files (not just standard directories)
  • Finds concrete subclasses of BaseHook, BaseOperator, BaseSensorOperator,
    BaseTrigger, BaseExecutor, BaseNotifier
  • Verifies modules (or class paths for executors/notifications) are registered
  • Handles class aliases (e.g. send_*_notification = FooNotifier) via id() dedup
  • Handles __init__.py re-exports for executors
  • Skips tests/, example_dags/, decorators/ directories
  • Prints every discovered class with OK/MISSING status for transparency

Verified: 1,571 classes across all providers (936 operators, 307 hooks,
155 triggers, 152 sensors, 13 notifiers, 8 executors).

Also registers the missing LocalKubernetesExecutor in the cncf.kubernetes
provider, found by the new check.

Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Claude Opus 4.6)

Generated-by: Claude Code (Claude Opus 4.6) following the guidelines

@potiuk potiuk force-pushed the add-provider-class-registration-checks branch from 842c744 to 8f514ae Compare March 24, 2026 06:29
@potiuk potiuk force-pushed the add-provider-class-registration-checks branch from 8f514ae to e0032a8 Compare March 27, 2026 20:02
jason810496
jason810496 approved these changes Mar 28, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@jason810496 jason810496 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice improvement to guard the missing provider registration, thanks!

Features like assets or the CLI are not validated here as well, so perhaps we can handle them in a follow-up?

Comment thread scripts/in_container/run_provider_yaml_files_check.py
@potiuk potiuk force-pushed the add-provider-class-registration-checks branch from e0032a8 to 768f67a Compare March 28, 2026 20:14
@potiuk potiuk requested a review from shahar1 as a code owner March 28, 2026 20:14
potiuk added 2 commits March 30, 2026 02:43
@potiuk
Add provider class registration checks to provider.yaml validation
383c8ce 
Add two new checks to run_provider_yaml_files_check.py:

1. Check that hook classes defining conn_type are registered in
   connection-types — catches hooks with unique conn_type values that
   are not listed in provider.yaml (async variants sharing conn_type
   with a registered sync hook are allowed).

2. Check that all provider Hook/Operator/Sensor/Trigger/Executor/Notifier
   classes are registered in provider.yaml — walks all provider source
   files, imports them, and verifies every concrete subclass of base
   types has its module (or class path for executors/notifications)
   registered in the appropriate section.

Also register the missing LocalKubernetesExecutor in the cncf.kubernetes
provider.yaml, which was found by the new check.
@potiuk
Fix mypy errors: use correct BaseHook import path
cd179f2
@potiuk potiuk force-pushed the add-provider-class-registration-checks branch from 768f67a to cd179f2 Compare March 30, 2026 00:43
potiuk added 2 commits March 30, 2026 03:00
@potiuk
Add validation for secrets-backends, auth-managers, logging, and db-m…
59e2035 
…anagers

Extends the provider class registration checks to also cover
BaseSecretsBackend, BaseAuthManager, FileTaskHandler (logging),
BaseDagBundle, and BaseDBManager subclasses, addressing the gap
noted in review.

Adds both directions of validation:
- yaml-to-code: classes listed in provider.yaml exist and are importable
- code-to-yaml: classes in provider code are registered in provider.yaml
@potiuk
Fix provider.yaml logging registration for Redis and HDFS
a93a6e6 
- Redis: Fix incorrect module path (missing .log. segment)
- HDFS: Add missing logging section for HdfsTaskHandler
@potiuk
Copy link
Copy Markdown
Member Author

potiuk commented Mar 30, 2026

Nice improvement to guard the missing provider registration, thanks!

Features like assets or the CLI are not validated here as well, so perhaps we can handle them in a follow-up?

Feel free :). Claude is your friend - all ther was entirely Claude generated.

@potiuk potiuk merged commit 3dc0d9e into apache:main Mar 31, 2026
144 of 146 checks passed
@potiuk potiuk deleted the add-provider-class-registration-checks branch March 31, 2026 00:15
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 31, 2026

Backport failed to create: v3-1-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
❌ v3-1-test Commit Link

You can attempt to backport this manually by running:

cherry_picker 3dc0d9e v3-1-test

This should apply the commit to the v3-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

Subham-KRLX pushed a commit to Subham-KRLX/airflow that referenced this pull request Apr 3, 2026
@potiuk @Subham-KRLX
Add provider class registration checks to provider.yaml validation (a…
043335c 
…pache#64127)

* Add provider class registration checks to provider.yaml validation

Add two new checks to run_provider_yaml_files_check.py:

1. Check that hook classes defining conn_type are registered in
   connection-types — catches hooks with unique conn_type values that
   are not listed in provider.yaml (async variants sharing conn_type
   with a registered sync hook are allowed).

2. Check that all provider Hook/Operator/Sensor/Trigger/Executor/Notifier
   classes are registered in provider.yaml — walks all provider source
   files, imports them, and verifies every concrete subclass of base
   types has its module (or class path for executors/notifications)
   registered in the appropriate section.

Also register the missing LocalKubernetesExecutor in the cncf.kubernetes
provider.yaml, which was found by the new check.

* Fix mypy errors: use correct BaseHook import path

* Add validation for secrets-backends, auth-managers, logging, and db-managers

Extends the provider class registration checks to also cover
BaseSecretsBackend, BaseAuthManager, FileTaskHandler (logging),
BaseDagBundle, and BaseDBManager subclasses, addressing the gap
noted in review.

Adds both directions of validation:
- yaml-to-code: classes listed in provider.yaml exist and are importable
- code-to-yaml: classes in provider code are registered in provider.yaml

* Fix provider.yaml logging registration for Redis and HDFS

- Redis: Fix incorrect module path (missing .log. segment)
- HDFS: Add missing logging section for HdfsTaskHandler
Suraj-kumar00 pushed a commit to Suraj-kumar00/airflow that referenced this pull request Apr 7, 2026
@potiuk @Suraj-kumar00
Add provider class registration checks to provider.yaml validation (a…
06d871d 
…pache#64127)

* Add provider class registration checks to provider.yaml validation

Add two new checks to run_provider_yaml_files_check.py:

1. Check that hook classes defining conn_type are registered in
   connection-types — catches hooks with unique conn_type values that
   are not listed in provider.yaml (async variants sharing conn_type
   with a registered sync hook are allowed).

2. Check that all provider Hook/Operator/Sensor/Trigger/Executor/Notifier
   classes are registered in provider.yaml — walks all provider source
   files, imports them, and verifies every concrete subclass of base
   types has its module (or class path for executors/notifications)
   registered in the appropriate section.

Also register the missing LocalKubernetesExecutor in the cncf.kubernetes
provider.yaml, which was found by the new check.

* Fix mypy errors: use correct BaseHook import path

* Add validation for secrets-backends, auth-managers, logging, and db-managers

Extends the provider class registration checks to also cover
BaseSecretsBackend, BaseAuthManager, FileTaskHandler (logging),
BaseDagBundle, and BaseDBManager subclasses, addressing the gap
noted in review.

Adds both directions of validation:
- yaml-to-code: classes listed in provider.yaml exist and are importable
- code-to-yaml: classes in provider code are registered in provider.yaml

* Fix provider.yaml logging registration for Redis and HDFS

- Redis: Fix incorrect module path (missing .log. segment)
- HDFS: Add missing logging section for HdfsTaskHandler
@potiuk potiuk mentioned this pull request Apr 8, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eladkal eladkal eladkal approved these changes

@jason810496 jason810496 jason810496 approved these changes

@jscheffl jscheffl jscheffl approved these changes

@vincbeck vincbeck vincbeck approved these changes

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@gopidesupavan gopidesupavan Awaiting requested review from gopidesupavan gopidesupavan is a code owner

@amoghrajesh amoghrajesh Awaiting requested review from amoghrajesh amoghrajesh is a code owner

@bugraoz93 bugraoz93 Awaiting requested review from bugraoz93 bugraoz93 is a code owner

@jedcunningham jedcunningham Awaiting requested review from jedcunningham jedcunningham is a code owner

@hussein-awala hussein-awala Awaiting requested review from hussein-awala hussein-awala is a code owner

@shahar1 shahar1 Awaiting requested review from shahar1 shahar1 is a code owner

Assignees

No one assigned

Labels

area:dev-tools area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@potiuk @eladkal @jason810496 @jscheffl @vincbeck
pFad - Phonifier reborn

Pfad - The Proxy pFad © 2024 Your Company Name. All rights reserved.




Check this box to remove all script contents from the fetched content.



Check this box to remove all images from the fetched content.


Check this box to remove all CSS styles from the fetched content.


Check this box to keep images inefficiently compressed and original size.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy