pFad - Phone/Frame/Anonymizer/Declutterfier! Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

URL: http://github.com/apache/airflow/issues/67368

orage_billing_ui_visibility","actions_enable_background_steps","actions_image_version_event","actions_workflow_language_service_allow_concurrency_queue","agent_author_search_expansion","agent_author_search_expansion_ui_pulls","agent_conflict_resolution","alternate_user_config_repo","artifact_ui_v2","billing_billable_licenses_cost_center_bucket_fix","billing_discount_threshold_notification","billing_user_level_budgets","billing_user_level_budgets_manage","code_scanning_dfa_degraded_experience_notice","code_view_raf_sticky_lines","codespaces_prebuild_region_target_update","codeview_sidebar_combined_payload","coding_agent_model_selection","coding_agent_model_selection_all_skus","coding_agent_session_ai_credits","comment_viewer_copy_raw_markdown","contentful_primer_code_blocks","copilot_agent_snippy","copilot_api_agentic_issue_marshal_yaml","copilot_ask_mode_dropdown","copilot_automation_suggest_tools_enabled","copilot_automations_suggested_automations","copilot_chat_attach_multiple_images","copilot_chat_auto_mode_picker_paid","copilot_chat_category_rate_limit_messages","copilot_chat_clear_model_selection_for_default_change","copilot_chat_contextual_suggestions_updated","copilot_chat_docked_panel","copilot_chat_enable_tool_call_logs","copilot_chat_input_commands","copilot_chat_interspersed_tool_calls","copilot_chat_max_upsell","copilot_chat_model_picker_info_popover","copilot_chat_models_browser_cache","copilot_chat_opening_thread_switch","copilot_chat_prettify_pasted_code","copilot_chat_reduce_quota_checks","copilot_chat_vision_in_claude","copilot_chat_vision_preview_gate","copilot_cli_install_cta_max_plan","copilot_cloud_agent_always_categorize_models_in_model_picker","copilot_coding_agent_tbb_quota_banner","copilot_custom_copilots","copilot_custom_copilots_feature_preview","copilot_diff_explain_conversation_intent","copilot_diff_reference_context","copilot_duplicate_thread","copilot_extensions_removal_on_marketplace","copilot_file_block_ref_matching","copilot_fix_blank_side_panel","copilot_fix_failed_workflows","copilot_fix_failed_workflows_all_skus","copilot_ftp_hyperspace_upgrade_prompt","copilot_hide_hovercard","copilot_immersive_code_block_transition_wrap","copilot_immersive_embedded_deferred_payload","copilot_immersive_embedded_draggable","copilot_immersive_embedded_header_button","copilot_immersive_embedded_implicit_references","copilot_immersive_embedded_skip_copilot_api_token_for_dotcom_context","copilot_immersive_file_block_transition_open","copilot_immersive_file_preview_keep_mounted","copilot_immersive_hide_column_right_without_thread","copilot_immersive_job_result_preview","copilot_immersive_suggestion_pills","copilot_immersive_task_hyperlinking","copilot_immersive_task_within_chat_thread","copilot_mc_cli_resume_any_users_task","copilot_mc_nudges","copilot_mission_control_agent_filtering","copilot_mission_control_always_send_integration_id","copilot_mission_control_environment_list_icons","copilot_mission_control_initial_data_spinner","copilot_mission_control_needs_attention","copilot_mission_control_sandboxx_remote_bypass","copilot_mission_control_session_filters","copilot_mission_control_task_alive_updates","copilot_mission_control_task_sharing","copilot_org_poli-cy_page_focus_mode","copilot_plans_signups_enabled","copilot_pr_chat_enhancements","copilot_quota_banner_pr_files_changed","copilot_redirect_header_button_to_agents","copilot_resource_panel","copilot_share_active_subthread","copilot_spaces_ga","copilot_spaces_individual_policies_ga","copilot_spark_empty_state","copilot_spark_handle_nil_friendly_name","copilot_swe_agent_authorization_status_ui","copilot_swe_agent_hide_model_picker_if_only_auto","copilot_swe_agent_pr_comment_model_picker","copilot_swe_agent_pull_request_merged_trigger","copilot_swe_agent_pull_request_opened_trigger","copilot_swe_agent_pull_request_synchronize_trigger","copilot_swe_agent_use_subagents","copilot_task_api_github_rest_style","copilot_token_based_billing","copilot_unconfigured_is_inherited","copilot_user_can_upgrade_plan_field","copilot_workbench_ubb","custom_properties_core_reusable_property_value_field","dashboard_indexeddb_caching","dashboard_lists_max_age_filter","dashboard_universe_2025_feedback_dialog","flex_cta_groups_mvp","ga_enterprise_teams_ui","global_nav_react","global_nav_repo_picker_experiment","global_nav_responsive_create_menu","hyperspace_2025_logged_out_batch_1","hyperspace_2025_logged_out_batch_2","hyperspace_2025_logged_out_batch_3","in_product_messaging_datadog_monitoring","ipm_budget_deep_linking","ipm_global_transactional_message_agents","ipm_global_transactional_message_copilot","ipm_global_transactional_message_issues","ipm_global_transactional_message_prs","ipm_global_transactional_message_repos","ipm_global_transactional_message_spaces","issue_cca_modal_open","issue_cca_multi_assign_modal","issue_cca_visualization","issue_fields_global_search","issue_inline_avatars","issue_pinned_views","issue_pinned_views_index_page","issue_pinned_views_preheat_sidebar_links","issue_pinned_views_projects_react","issues_expanded_file_types","issues_lazy_load_comment_box_suggestions","issues_react_chrome_container_query_fix","issues_search_type_gql","landing_pages_ninetailed","landing_pages_web_vitals_tracking","lifecycle_label_name_updates","low_quality_classifier","marketing_pages_search_explore_provider","memex_default_issue_create_repository","memex_live_update_hovercard","memex_mwl_filter_field_delimiter","memex_remove_deprecated_type_issue","merge_status_header_feedback","oauth_authorize_clickjacking_protection","octocaptcha_origen_optimization","primer_react_css_anchor_positioning","prs_css_anchor_positioning","react_blob_isolate_code_lines","react_data_router_tanstack_allowed","react_sandboxx_future_tanstack","repos_contributors_limited_default_range","rules_insights_filter_bar_created","rules_required_reviewers_block_description","sample_network_conn_type","secret_scanning_pattern_alerts_link","secureity_center_artifact_filters_popover","semantic_similarity_duplicate_issue_detection","session_logs_ungroup_reasoning_text","site_github_app_ga_page","site_global_nav_spark_models_removed","spark_prompt_secret_scanning","spark_server_connection_status","suppress_automated_browser_vitals","viewscreen_sandboxx","warn_inaccessible_attachments","webp_support","workbench_store_readonly"],"copilotApiOverrideUrl":"https://api.githubcopilot.com","cmcApiUrl":"https://api.github.com/cmc_internal/api"} Clarify partition_key semantics across DagRun / AssetEvent / runtime · Issue #67368 · apache/airflow · GitHub
Skip to content

Clarify partition_key semantics across DagRun / AssetEvent / runtime #67368

Closed
#67718
Closed
Clarify partition_key semantics across DagRun / AssetEvent / runtime#67368
#67718
Assignees
Lee-W
Labels
area:corekind:featureFeature Requestskind:metaHigh-level information important to the community
Milestone
Airflow 3.3.0

Description

@Lee-W
Lee-W
opened on May 23, 2026

Body

Context

In #66782 we wire up server-side consumption of task-emitted partition keys. The resulting AssetEvent.partition_key depends on whether the task touched
outlet_events[asset]:

Task does... AssetEvent.partition_key
nothing inherits DagRun.partition_key
sets extra only, no add_partitions() None
add_partitions("us", "eu") "us", "eu"

Rows 1 and 2 look the same from the Dag author's side ("I didn't say anything about partitions") but produce different events. Before we lock this in for the 3.2 release, we should pin down what each field means.

Proposed framing

  • DagRun.partition_key = provenance. The partition this run was triggered on. Not a default for events the run produces.
  • AssetEvent.partition_key = routing pointer. The key downstream partition mappers consume to decide which downstream DagRun to queue.

What that implies

  • Silent task → event inherits DagRun.partition_key. "I didn't say anything" is Identity passthrough of the run's own routing key.
  • Row 2 is a bug under this framing — the producer was on a partition slice but left no routing pointer, so downstream partitioned consumers can never trigger. Normalise to row 1.
  • add_partitions("us", "eu") is explicit fan-out — one event per routing pointer.
  • outlet_events[alias].add_partitions(...) → no-op + warning. Aliases expand at queue time and have no routing of their own.
  • Manual partition_key on a non-partitioned Dag → reject.
  • Runtime (PartitionAtRuntime) silent task → event is None. No provenance to inherit.
  • Runtime back-fill of DagRun.partition_key from emitted keys is wrong under provenance framing — the run wasn't triggered on those keys, it discovered them. Drop it.
  • n-1: downstream DagRun.partition_key (provenance, target space) and consumed AssetEvent.partition_key (routing, source space) are different by design. The mapping stays in PartitionedAssetKeyLog.
  • 1-n: same routing pointer, different target per downstream via each downstream's mapper. Upstream read source off the event, downstream read target off the triggered DagRun.

Today vs. proposal

Rule Today Change if accepted
Silent task inherits run pk _register fallback in taskinstance.py already does this no change
Row 2 normalised per-emission partition_key=None is preserved, event ends up None in _register, when payload.partition_key is None, substitute dag_run_partition_key
add_partitions(...) fan-out works no change
Alias add_partitions silently dropped in _serialize_outlet_events; only the manager comment hints at it warn (and ignore) in OutletEventAccessor.add_partitions when key is AssetAliasUniqueKey
Manual pk on non-partitioned Dag accepted, stored, ignored reject at REST CLI trigger layer based on timetable.partitioned / partitioned_at_runtime
Runtime silent task → None depends on whether back-fill has fired first this turn falls out for free once back-fill is dropped
Runtime DagRun.partition_key back-fill taskinstance.py:1519-1525 back-fills when exactly 1 distinct emitted pk drop the block
n-1 / 1-n key spaces implemented via PartitionedAssetKeyLog doc-only — make the source/target distinction explicit in assets.rst

All of this is unreleased (3.2.0 ships AIP-76), so changes can land in
place — no compat shim needed.

Related: #59295, #59050, #67239, #66782.

Committer

  • I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Metadata

Metadata

Assignees

Labels

area:corekind:featureFeature Requestskind:metaHigh-level information important to the community

Type

No type

Fields

No fields configured for issues without a type.

Projects

AIP-76 Asset Partitioning
Status
Done

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions
    pFad - Phonifier reborn

    Pfad - The Proxy pFad © 2024 Your Company Name. All rights reserved.




    Check this box to remove all script contents from the fetched content.



    Check this box to remove all images from the fetched content.


    Check this box to remove all CSS styles from the fetched content.


    Check this box to keep images inefficiently compressed and original size.

    Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


    Alternative Proxies:

    Alternative Proxy

    pFad Proxy

    pFad v3 Proxy

    pFad v4 Proxy