Skip to content

[fix](arrow-flight) Support table type metadata and filters - #67899

Open
hutiefang76 wants to merge 4 commits into
apache:masterfrom
hutiefang76:frank/doris-flight-table-types
Open

hutiefang76 wants to merge 4 commits into
apache:masterfrom
hutiefang76:frank/doris-flight-table-types

Conversation

@hutiefang76

@hutiefang76 hutiefang76 commented Sep 12, 2026 •

Copy link
Copy Markdown

What problem does this PR solve?

Related to #67578 (partial A3).

Flight SQL clients can obtain FlightInfo for GetTableTypes, but reading its ticket fails with UNIMPLEMENTED. GetTables also forwards only the first table_types entry to a service that treats only VIEW as a filter: [VIEW, BASE TABLE] loses base tables, while [BASE TABLE] or an unknown type can return excluded rows.

Implement the type stream using the same Doris names as GetTables, sorted according to Flight SQL. Match the complete filter against the existing authorized listing before collecting optional schemas. Request only name/type metadata to avoid unused table statistics, and retain the session guard and Arrow buffer cleanup on failed sends.

Release note

Flight SQL clients can enumerate supported table types and filter table metadata by multiple types accurately.

Check List (For Author)

  • Test on the merged head:
    • FE Checkstyle passed with 0 violations.
    • The official selected FE unit-test command below was attempted with JDK 17 and Thrift 0.24.0. It stopped before test execution because the project's grpc-java 1.34.0 generator is an Intel macOS executable that this ARM host cannot run. No current-head unit-test success is claimed.
    • Earlier PR-revision evidence: 30 selected unit tests and the SQL/Flight JDBC regression suite passed; the 19 new cases produced 15 failures/errors on the earlier unmodified baseline. Those results have not been rerun successfully after this merge.
    • No native-BE/data-scan cluster or ADBC driver test was run in this follow-up.
  • Behavior changed: Yes. GetTableTypes now returns BASE TABLE, SYSTEM VIEW and VIEW. Empty type filters retain all authorized tables; unknown-only filters return no rows.
  • Does this need documentation: No new configuration or API schema.

Validation command (JDK 17, matching Thrift 0.24.0 compiler):

bash run-fe-ut.sh --run org.apache.doris.arrowflight.FlightSqlTableTypesTest,org.apache.doris.arrowflight.FlightSqlSchemaHelperTableTypesTest,org.apache.doris.arrowflight.DorisFlightSqlProducerTest,org.apache.doris.arrowflight.FlightSqlSchemaHelperArrowTypeTest

Current follow-up merges master and resolves the FlightSqlSchemaHelper import conflict, retaining both the new Arrow MapType/StructType/Type support and the table-type implementation. No local toolchain configuration or generated sources are included.

AI assistance: DeepSeek and Codex assisted review, conflict resolution, validation and this description.

### What problem does this PR solve?

Related to apache#67578 (partial A3).

Flight SQL clients can obtain FlightInfo for GetTableTypes, but reading its ticket fails with UNIMPLEMENTED. GetTables also forwards only the first table_types entry to a service that treats only VIEW as a filter: [VIEW, BASE TABLE] loses base tables, while [BASE TABLE] or an unknown type can return excluded rows.

Implement the type stream using the same Doris names as GetTables, sorted according to Flight SQL. Match the complete filter against the existing authorized listing before collecting optional schemas. Request only name/type metadata to avoid unused table statistics, and retain the session guard and Arrow buffer cleanup on failed sends.

### Release note

Flight SQL clients can enumerate supported table types and filter table metadata by multiple types accurately.

### Check List (For Author)

- Test:
  - [x] Unit Test: 30 passed via run-fe-ut.sh, including 19 new regression cases and 11 existing producer/schema cases.
  - Baseline: the 19 new cases produced 15 failures/errors on unmodified master, reproducing UNIMPLEMENTED and incorrect filtering.
  - Coverage includes both schema modes, filter order, unknown/duplicate types, external catalog propagation, serialized column alignment and repeated success/failure buffer cleanup.
  - FE Checkstyle: 0 violations.
  - No live-cluster or end-to-end ADBC driver test was run.
- Behavior changed: Yes. GetTableTypes now returns BASE TABLE, SYSTEM VIEW and VIEW. Empty type filters retain all authorized tables; unknown-only filters return no rows.
- Does this need documentation: No new configuration or API schema.

Validation command (JDK 17, matching Thrift 0.24.0 compiler):

```sh
bash run-fe-ut.sh --run org.apache.doris.arrowflight.FlightSqlTableTypesTest,org.apache.doris.arrowflight.FlightSqlSchemaHelperTableTypesTest,org.apache.doris.arrowflight.DorisFlightSqlProducerTest,org.apache.doris.arrowflight.FlightSqlSchemaHelperArrowTypeTest
```
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

### What problem does this PR solve?

Related PR: apache#67899

Problem Summary: Exercise SQL-created base tables and views through the Arrow
Flight JDBC metadata APIs, covering type enumeration and complete type filters.
The suite executes DDL over Flight and consumes GetTableTypes/GetTables streams.

### Release note

None

### Check List (For Author)

- Test: official regression runner, 1 suite passed against a test FE with real
  Flight/MySQL services and the built-in mocked BE; 30 related FE unit tests
  passed. No native BE or data-scan integration was exercised.
- Behavior changed: No, test-only addition
- Does this need documentation: No
@hutiefang76

Copy link
Copy Markdown
Author

I’m syncing this with current master. The conflict is in the schema-helper imports; I’ll keep the newer Arrow type support and rerun the Flight metadata tests before pushing.

### What problem does this PR solve?

Related PR: apache#67899

Problem Summary: Resolve the schema-helper import conflict while retaining master Arrow type support and the table-types metadata fix.

### Release note

None; this commit synchronizes the existing PR with master.

### Check List (For Author)

- Test: FE Checkstyle passed with zero violations. Official selected FE unit-test command attempted; blocked before execution by the Intel-only grpc-java 1.34.0 generator on this ARM macOS host.
- Behavior changed: No additional behavior beyond the existing PR and master.
- Does this need documentation: No.

Signed-off-by: hutiefang76 <137664623+hutiefang76@users.noreply.github.com>
@hutiefang76

Copy link
Copy Markdown
Author

Merged master in 58c16ac and resolved the import conflict, retaining the newer Arrow type support. FE Checkstyle reports zero violations.

I attempted the selected tests through run-fe-ut.sh, but the build stops before test execution at the Intel-only grpc-java 1.34.0 generator on this ARM Mac. I have updated the description so the earlier 30-test and SQL/Flight regression results are not mistaken for verification of this merged head. The local toolchain files are not part of the patch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants