Repository navigation
[fix](arrow-flight) Support table type metadata and filters - #67899
hutiefang76 wants to merge 4 commits into
Conversation
### What problem does this PR solve? Related to apache#67578 (partial A3). Flight SQL clients can obtain FlightInfo for GetTableTypes, but reading its ticket fails with UNIMPLEMENTED. GetTables also forwards only the first table_types entry to a service that treats only VIEW as a filter: [VIEW, BASE TABLE] loses base tables, while [BASE TABLE] or an unknown type can return excluded rows. Implement the type stream using the same Doris names as GetTables, sorted according to Flight SQL. Match the complete filter against the existing authorized listing before collecting optional schemas. Request only name/type metadata to avoid unused table statistics, and retain the session guard and Arrow buffer cleanup on failed sends. ### Release note Flight SQL clients can enumerate supported table types and filter table metadata by multiple types accurately. ### Check List (For Author) - Test: - [x] Unit Test: 30 passed via run-fe-ut.sh, including 19 new regression cases and 11 existing producer/schema cases. - Baseline: the 19 new cases produced 15 failures/errors on unmodified master, reproducing UNIMPLEMENTED and incorrect filtering. - Coverage includes both schema modes, filter order, unknown/duplicate types, external catalog propagation, serialized column alignment and repeated success/failure buffer cleanup. - FE Checkstyle: 0 violations. - No live-cluster or end-to-end ADBC driver test was run. - Behavior changed: Yes. GetTableTypes now returns BASE TABLE, SYSTEM VIEW and VIEW. Empty type filters retain all authorized tables; unknown-only filters return no rows. - Does this need documentation: No new configuration or API schema. Validation command (JDK 17, matching Thrift 0.24.0 compiler): ```sh bash run-fe-ut.sh --run org.apache.doris.arrowflight.FlightSqlTableTypesTest,org.apache.doris.arrowflight.FlightSqlSchemaHelperTableTypesTest,org.apache.doris.arrowflight.DorisFlightSqlProducerTest,org.apache.doris.arrowflight.FlightSqlSchemaHelperArrowTypeTest ```
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
### What problem does this PR solve? Related PR: apache#67899 Problem Summary: Exercise SQL-created base tables and views through the Arrow Flight JDBC metadata APIs, covering type enumeration and complete type filters. The suite executes DDL over Flight and consumes GetTableTypes/GetTables streams. ### Release note None ### Check List (For Author) - Test: official regression runner, 1 suite passed against a test FE with real Flight/MySQL services and the built-in mocked BE; 30 related FE unit tests passed. No native BE or data-scan integration was exercised. - Behavior changed: No, test-only addition - Does this need documentation: No
|
I’m syncing this with current master. The conflict is in the schema-helper imports; I’ll keep the newer Arrow type support and rerun the Flight metadata tests before pushing. |
### What problem does this PR solve? Related PR: apache#67899 Problem Summary: Resolve the schema-helper import conflict while retaining master Arrow type support and the table-types metadata fix. ### Release note None; this commit synchronizes the existing PR with master. ### Check List (For Author) - Test: FE Checkstyle passed with zero violations. Official selected FE unit-test command attempted; blocked before execution by the Intel-only grpc-java 1.34.0 generator on this ARM macOS host. - Behavior changed: No additional behavior beyond the existing PR and master. - Does this need documentation: No. Signed-off-by: hutiefang76 <137664623+hutiefang76@users.noreply.github.com>
|
Merged master in 58c16ac and resolved the import conflict, retaining the newer Arrow type support. FE Checkstyle reports zero violations. I attempted the selected tests through run-fe-ut.sh, but the build stops before test execution at the Intel-only grpc-java 1.34.0 generator on this ARM Mac. I have updated the description so the earlier 30-test and SQL/Flight regression results are not mistaken for verification of this merged head. The local toolchain files are not part of the patch. |
What problem does this PR solve?
Related to #67578 (partial A3).
Flight SQL clients can obtain FlightInfo for GetTableTypes, but reading its ticket fails with UNIMPLEMENTED. GetTables also forwards only the first table_types entry to a service that treats only VIEW as a filter: [VIEW, BASE TABLE] loses base tables, while [BASE TABLE] or an unknown type can return excluded rows.
Implement the type stream using the same Doris names as GetTables, sorted according to Flight SQL. Match the complete filter against the existing authorized listing before collecting optional schemas. Request only name/type metadata to avoid unused table statistics, and retain the session guard and Arrow buffer cleanup on failed sends.
Release note
Flight SQL clients can enumerate supported table types and filter table metadata by multiple types accurately.
Check List (For Author)
Validation command (JDK 17, matching Thrift 0.24.0 compiler):
Current follow-up merges master and resolves the FlightSqlSchemaHelper import conflict, retaining both the new Arrow MapType/StructType/Type support and the table-type implementation. No local toolchain configuration or generated sources are included.
AI assistance: DeepSeek and Codex assisted review, conflict resolution, validation and this description.