pFad - Phone/Frame/Anonymizer/Declutterfier! Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

URL: http://github.com/apache/arrow/pull/49549

ssets/global-a40b6ece39d70d4a.css" /> GH-49548: [C++][FlightRPC] Decouple Flight Serialize/Deserialize from gRPC transport by raulcd · Pull Request #49549 · apache/arrow · GitHub
Skip to content

GH-49548: [C++][FlightRPC] Decouple Flight Serialize/Deserialize from gRPC transport#49549

Merged
raulcd merged 3 commits intoapache:mainfrom
raulcd:GH-49548
Mar 31, 2026
Merged

GH-49548: [C++][FlightRPC] Decouple Flight Serialize/Deserialize from gRPC transport#49549
raulcd merged 3 commits intoapache:mainfrom
raulcd:GH-49548

Conversation

@raulcd
Copy link
Copy Markdown
Member

@raulcd raulcd commented Mar 18, 2026

Rationale for this change

Currently the Serialize/Deserialize APIs are gRPC dependent. This means that any code that needs to encode or decode Flight data must depend on gRPC C++ internals. After some discussions around trying to build a PoC using gRPC's generic API with gRPC's BidiReactor we discussed that these primitives should be made gRPC agnostic.

What changes are included in this PR?

  • Move the serialization/deserialization logic from cpp/src/arrow/flight/transport/grpc/serialization_internal.{h,cc} to cpp/src/arrow/flight/serialization_internal.cc
  • Create new arrow::Result<arrow::BufferVector> SerializePayloadToBuffers(const FlightPayload& msg) gRPC agnostic function.
  • Create new arrow::Result<arrow::flight::internal::FlightData> DeserializeFlightData(const std::shared_ptr<arrow::Buffer>& buffer) gRPC agnostic function.
  • Keep the existing serialize/deserialize functions for gRPC as simple wrappers on top of the new serialization functions to implement the grpc::ByteBuffer and grpc::Slice details.
  • Add utility arrow::Result<BufferVector> SerializeToBuffers() const; to FlightPayload struct.
  • Add roundtrip tests to serialize/deserialize

Are these changes tested?

Yes, both by existing tests and new tests.

Are there any user-facing changes?

No

@github-actions
Copy link
Copy Markdown

⚠️ GitHub issue #49548 has been automatically assigned in GitHub to PR creator.

@raulcd
Copy link
Copy Markdown
Member Author

raulcd commented Mar 18, 2026

@github-actions crossbow submit -g cpp

@github-actions
Copy link
Copy Markdown

Revision: e852d66

Submitted crossbow builds: ursacomputing/crossbow @ actions-c88270abef

Task Status
example-cpp-minimal-build-static GitHub Actions
example-cpp-minimal-build-static-system-dependency GitHub Actions
example-cpp-tutorial GitHub Actions
test-build-cpp-fuzz GitHub Actions
test-conda-cpp GitHub Actions
test-conda-cpp-valgrind GitHub Actions
test-debian-13-cpp-amd64 GitHub Actions
test-debian-13-cpp-i386 GitHub Actions
test-debian-experimental-cpp-gcc-15 GitHub Actions
test-fedora-42-cpp GitHub Actions
test-ubuntu-22.04-cpp GitHub Actions
test-ubuntu-22.04-cpp-20 GitHub Actions
test-ubuntu-22.04-cpp-bundled GitHub Actions
test-ubuntu-22.04-cpp-emscripten GitHub Actions
test-ubuntu-22.04-cpp-no-threading GitHub Actions
test-ubuntu-24.04-cpp GitHub Actions
test-ubuntu-24.04-cpp-bundled-offline GitHub Actions
test-ubuntu-24.04-cpp-gcc-13-bundled GitHub Actions
test-ubuntu-24.04-cpp-gcc-14 GitHub Actions
test-ubuntu-24.04-cpp-minimal-with-formats GitHub Actions
test-ubuntu-24.04-cpp-thread-sanitizer GitHub Actions

@raulcd raulcd added CI: Extra: C++ Run extra C++ CI CI: Extra: Package: Linux Run extra Linux Packages CI labels Mar 18, 2026
@raulcd
Copy link
Copy Markdown
Member Author

raulcd commented Mar 19, 2026

CI failures are unrelated.

I've run some comparison benchmarks between the old code path and the new code path, the full info can be seen here:
https://gist.github.com/raulcd/f139ccaaeb700a3ffe23a0c914e48c7c

Basically run the following to validate:
./arrow-flight-benchmark --num_streams=4 --records_per_stream=1000000

Summary:

DoGet benchmark, 4 streams, 1M records/stream (arrow-flight-benchmark)

Build Branch Run 1 Run 2 Run 3 Median
Debug main 1354.94 1394.40 1263.38 1354.94
Debug GH-49548 1431.46 1390.77 1433.68 1431.46
Release main 2515.47 2408.77 2461.71 2461.71
Release GH-49548 2447.93 2213.12 2546.33 2447.93

All values in MB/s.

@raulcd
Copy link
Copy Markdown
Member Author

raulcd commented Mar 23, 2026

@zanmato1984 I saw you were taking a look, would you mind a review?
ping @pitrou @lidavidm

@AlenkaF
Copy link
Copy Markdown
Member

AlenkaF commented Mar 27, 2026

Also cc @benibus in case the topic is of interest.

@raulcd raulcd requested review from kou, pitrou and zanmato1984 March 31, 2026 07:56
@github-actions github-actions bot removed CI: Extra: C++ Run extra C++ CI CI: Extra: Package: Linux Run extra Linux Packages CI labels Mar 31, 2026
Copy link
Copy Markdown
Member

@lidavidm lidavidm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the late response, this seems reasonable to me.

@github-actions github-actions bot added awaiting merge Awaiting merge awaiting review Awaiting review awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review awaiting review Awaiting review awaiting merge Awaiting merge labels Mar 31, 2026
@raulcd raulcd merged commit b3120c1 into apache:main Mar 31, 2026
81 of 83 checks passed
@raulcd raulcd removed the awaiting changes Awaiting changes label Mar 31, 2026
@raulcd raulcd deleted the GH-49548 branch March 31, 2026 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

pFad - Phonifier reborn

Pfad - The Proxy pFad © 2024 Your Company Name. All rights reserved.





Check this box to remove all script contents from the fetched content.



Check this box to remove all images from the fetched content.


Check this box to remove all CSS styles from the fetched content.


Check this box to keep images inefficiently compressed and original size.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy