-
Notifications
You must be signed in to change notification settings - Fork 1.2k
docs(bigquery-jdbc): add developer and maintainer guide #13879
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Neenu1995
wants to merge
8
commits into
pr-1-user-guide-properties
Choose a base branch
from
pr-2-developer-guide
base: pr-1-user-guide-properties
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+179
−0
Open
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
f637e4b
docs(bigquery-jdbc): add developer and maintainer guide
Neenu1995 16b58e8
Merge branch 'pr-1-user-guide-properties' into pr-2-developer-guide
Neenu1995 c887cd1
Merge branch 'pr-1-user-guide-properties' into pr-2-developer-guide
Neenu1995 958dc37
docs(bigquery-jdbc): add Table of Contents to DEVELOPMENT.md
Neenu1995 4f30a68
Update java-bigquery-jdbc/DEVELOPMENT.md
Neenu1995 24a0d7e
Merge branch 'pr-1-user-guide-properties' into pr-2-developer-guide
Neenu1995 e9fd7c2
Merge branch 'pr-2-developer-guide' of github.com:googleapis/google-c…
Neenu1995 2a2a626
Merge branch 'pr-1-user-guide-properties' into pr-2-developer-guide
Neenu1995 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,164 @@ | ||
| # BigQuery JDBC Developer & Contributor Guide | ||
|
|
||
| This guide details the architectural design, core abstractions, coding principles, and testing workflows for developers contributing to the `google-cloud-bigquery-jdbc` module. | ||
|
|
||
| --- | ||
|
|
||
| ## 1. Core Architecture & Component Map | ||
|
|
||
| The driver is structured to provide high performance, zero-allocation MDC log tracing, strict JDBC compliance, and seamless execution over the Google Cloud BigQuery REST and Storage APIs. | ||
|
|
||
| ```mermaid | ||
| graph TD | ||
| Client[Client Application / BI Tool] -->|DriverManager.getConnection| Driver[BigQueryDriver] | ||
| Driver -->|Parses URI & Options| UrlUtil[BigQueryJdbcUrlUtility] | ||
| Driver -->|Configures Logging| RootLogger[BigQueryJdbcRootLogger] | ||
| Driver -->|Creates| Conn[BigQueryConnection] | ||
| Conn -->|Dynamic Context Proxy| Proxy[BigQueryJdbcContextProxy] | ||
| Proxy -->|MDC Tracing| Mdc[BigQueryJdbcMdc] | ||
| Proxy -->|Delegates Exec| DirectConn[Client Session] | ||
| Conn -->|Type Mapping & Coercion| Coercion[BigQueryJdbcTypeMappings & BigQueryCoercion] | ||
| Conn -->|REST / Storage API| BQSDK[google-cloud-bigquery] | ||
| ``` | ||
|
|
||
| ### Key Abstractions | ||
|
|
||
| - **`BigQueryDriver`**: JDBC entry point registered with `java.sql.DriverManager`. Intercepts `jdbc:bigquery://` URLs, initializes early logger state, and instantiates `BigQueryConnection`. | ||
| - **`BigQueryConnection`**: Represents an active BigQuery session, holding dataset defaults, connection configuration maps, and transaction/session state (`EnableSession=true`, `session_id`). | ||
| - **`BigQueryJdbcUrlUtility`**: Parses and validates connection string parameters using a bounded LRU parse cache (`PARSE_CACHE`) to avoid heavy allocations during frequent connection creation. | ||
| - **`BigQueryJdbcContextProxy`**: A dynamic proxy layer (`java.lang.reflect.Proxy`) wrapping JDBC statements, connections, and metadata. Intercepts calls to propagate ThreadLocal MDC parameters (`connectionId`) across execution threads and enforce state validation (`checkClosed()`). | ||
| - **`BigQueryJdbcTypeMappings` & `BigQueryCoercion`**: Centralized mapping logic handling standard JDBC-to-BigQuery SQL type mappings (`StandardSQLTypeName`) and object coercions (`Date`, `Timestamp`, `BigDecimal`, etc.). | ||
| - **`BigQueryArrowResultSet`**: Custom result set implementation accelerating large query result retrieval via the BigQuery Storage Read API gRPC stream. | ||
|
|
||
| --- | ||
|
|
||
| ## 2. Developer Guardrails & Rules of Engagement | ||
|
|
||
| > [!IMPORTANT] | ||
| > **Adhere strictly to the following guardrails when making code changes:** | ||
|
|
||
| 1. **Visibility Principle**: Always default to the most restrictive access level (`private`, package-private, or `@InternalApi`). Do **NOT** expose classes or methods as `public` unless strictly required by standard JDBC interfaces. | ||
| 2. **Explicit Class Imports**: Always write explicit `import` statements. Do **NOT** use wildcard star imports or inline fully qualified class names (e.g., use `import java.math.BigDecimal;` instead of `java.math.BigDecimal` inline). | ||
| 3. **Logger Preference**: Always prefer `BigQueryJdbcCustomLogger` over `java.util.logging.Logger`. Format strings using `String.format(...)` before logging, as `BigQueryJdbcRootLogger` evaluates `record.getMessage()` directly. | ||
| 4. **Exception Handling**: Always throw exceptions from the `com.google.cloud.bigquery.exception` package (`BigQueryJdbcException`, `BigQueryJdbcSqlSyntaxErrorException`, `BigQueryConversionException`). | ||
| 5. **No Mocking of Final JDK Classes**: Do **NOT** mock final JDK types (`BigDecimal`, `LocalDate`, `Instant`, `UUID`) with Mockito. Mocking final JDK classes is unstable and can cause JVM crashes under JDK 21+. Always construct real instances in unit tests. | ||
|
|
||
| --- | ||
|
|
||
| ## 3. Build & Test Playbook | ||
|
|
||
| Builds and test tasks are managed via the module [Makefile](file:///usr/local/google/home/neenushaji/IdeaProjects/google-cloud-java/java-bigquery-jdbc/Makefile). | ||
|
|
||
| ### Local Build Commands | ||
|
|
||
| ```bash | ||
| # Build & install module locally | ||
| make install | ||
|
|
||
| # Clean project target directory | ||
| make clean | ||
|
|
||
| # Format code and check linter compliance | ||
| make lint | ||
| ``` | ||
|
|
||
| ### Running Unit Tests | ||
|
|
||
| ```bash | ||
| # Run all unit tests | ||
| make unittest | ||
|
|
||
| # Run a specific unit test class | ||
| make unittest test=BigQueryPreparedStatementTest | ||
|
|
||
| # Run a specific unit test method | ||
| make unittest test=BigQueryPreparedStatementTest#testSetObjectWithTemporalTypes | ||
| ``` | ||
|
|
||
| ### Running Integration Tests | ||
|
|
||
| > [!WARNING] | ||
| > Integration tests connect to real GCP BigQuery resources and require valid GCP credentials. | ||
|
|
||
| ```bash | ||
| # Set GCP service account credentials | ||
| export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json | ||
|
|
||
| # Run a specific integration test | ||
| make integration-test test=ITBigQueryJDBCTest#testValidServiceAccountAuthenticationOAuthPvtKey | ||
| ``` | ||
|
|
||
| ### Dockerized Execution | ||
|
|
||
| If local Java/Maven environments are not available, use the dockerized environment: | ||
|
|
||
| ```bash | ||
| # Start an interactive shell session inside Docker container | ||
| make docker-session | ||
|
|
||
| # Run unit tests inside Docker | ||
| make docker-unittest | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## 4. Logging Architecture & Developer Conventions | ||
|
|
||
| The driver uses a custom logging subsystem built on top of `java.util.logging`: `BigQueryJdbcCustomLogger` and `BigQueryJdbcRootLogger`. | ||
|
|
||
| ### Instantiating Loggers | ||
|
|
||
| - **For Instance Components** (`BigQueryConnection`, `BigQueryStatement`, `BigQueryDatabaseMetaData`): | ||
| Use `this.toString()` to include instance identity in logger output: | ||
| ```java | ||
| private final BigQueryJdbcCustomLogger LOG = new BigQueryJdbcCustomLogger(this.toString()); | ||
| ``` | ||
| - **For Static / Utility Components** (`BigQueryJdbcUrlUtility`, `BigQueryJdbcTypeMappings`): | ||
| Use the class name: | ||
| ```java | ||
| private static final BigQueryJdbcCustomLogger LOG = | ||
| new BigQueryJdbcCustomLogger(BigQueryJdbcTypeMappings.class.getName()); | ||
| ``` | ||
|
|
||
| ### Developer Logging Rules & Conventions | ||
|
|
||
| 1. **Method Entry / Exit Tracing**: | ||
| Methods at `FINER` level must log entrance and exit points: | ||
| ```java | ||
| public ResultSet executeQuery(String sql) throws SQLException { | ||
| LOG.finer("++enter++"); | ||
| try { | ||
| // ... execution logic ... | ||
| return rs; | ||
| } finally { | ||
| LOG.finer("++exit++"); | ||
| } | ||
| } | ||
| ``` | ||
| 2. **Format Placeholders (Zero Allocation)**: | ||
| Avoid string concatenation in log calls. Use formatting placeholders or `Supplier<String>` lambdas to prevent unneeded string allocation when the log level is disabled: | ||
| ```java | ||
| // Recommended: Use printf-style formatting | ||
| LOG.fine("Executing query on dataset: %s, table: %s", datasetId, tableId); | ||
|
|
||
| // Recommended: Use supplier lambda for expensive calculations | ||
| LOG.fine(() -> "Parsed properties: " + complexObject.toDebugString()); | ||
| ``` | ||
| 3. **Caller Inference & MDC Propagation**: | ||
| - `BigQueryJdbcCustomLogger` automatically wraps log records in `BigQueryJdbcLogRecord`, which inspects the stack trace to accurately infer caller class and method names. | ||
| - `BigQueryJdbcMdc` maintains `connectionId` in a `ThreadLocal` context. When logging from proxy or worker threads, always ensure MDC context is preserved or propagated via `BigQueryJdbcContextProxy`. | ||
|
|
||
| --- | ||
|
|
||
| ## 5. Pre-PR Checklist | ||
|
|
||
| Before submitting a Pull Request: | ||
|
|
||
| - [ ] All new classes and methods use the narrowest possible visibility scope (`private` or package-private). | ||
| - [ ] No inline fully qualified names or wildcard star imports are present. | ||
| - [ ] All logger instances use `BigQueryJdbcCustomLogger`. | ||
| - [ ] Method entrance/exit logging (`++enter++` / `++exit++`) is included for complex internal routines. | ||
| - [ ] All method changes and feature additions are covered by corresponding JUnit 5 tests. | ||
| - [ ] Unit tests pass cleanly without Mockito `UnnecessaryStubbingException` warnings. | ||
| - [ ] All `Statement`, `ResultSet`, or `DatabaseMetaData` objects returned by public entry points are properly wrapped via `BigQueryJdbcContextProxy.wrap()`. | ||
| - [ ] Code formatting and linting pass via `make lint`. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.