[mattcasters]

Parquet Output mapped a Hop Date to a Parquet TIMESTAMP(MILLIS) column. BigQuery then imported that column as a timestamp, and the fields grid had no way to choose DATE or any other logical type.

Each output field can now set a Parquet type, plus precision and scale for Decimal. Leaving the type empty keeps the previous mapping, so existing pipelines still write Date as TIMESTAMP(MILLIS). Get Fields proposes Date for a Hop date, TimestampMicros for a timestamp, and Decimal when a BigNumber fits in a decimal. An explicit type replaces the column after the automatic schema conversion and is the type written to the file. Invalid precision, scale, Int32 range, and unknown type names fail before a file is written.

./mvnw -pl plugins/tech/parquet test -Dtest=ParquetLogicalTypeTest,ParquetFieldTest,ParquetOutputMetaTest,ParquetWriteSupportTest,ParquetOutputTest,HopToParquetTypeMatrixTest -Dsurefire.failIfNoSpecifiedTests=false passed (86 tests). ./mvnw -pl plugins/tech/parquet test -Dsurefire.failIfNoSpecifiedTests=false passed (511 tests). ./mvnw -pl plugins/tech/parquet spotless:apply made no further edits. The full mvn clean install apache-rat:check was not run, and Hop GUI was not started.

fixes #3864

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Run mvn clean install apache-rat:check to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
• If you have a group of commits related to the same change, please squash your commits into one and force push your branch using git rebase -i.
• Mention the appropriate issue in your description (for example: addresses #123), if applicable.

To make clear that you license your contribution under the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.

• I hereby declare this contribution to be licensed under the Apache License Version 2.0, January 2004
• In any other case, please file an Apache Individual Contributor License Agreement.

[bamaer]

Blockers

• Get Fields proposes Date for a Hop Date, which drops the time of day. ParquetFieldType.forValueMeta maps TYPE_DATE to Date, the Parquet DATE type that stores days only (ParquetFieldType.java:115). Until now the writer stored a Hop Date as TIMESTAMP(MILLIS) (ParquetOutput.avroType). Database timestamp columns usually arrive as Hop Date, because "supports timestamp data type" is off by default on connections. So Table Input → Parquet Output → Get Fields now writes 2024-03-05 14:22:10 as 2024-03-05, without a warning.

  Suggested fix: propose TimestampMillis for TYPE_DATE, so Get Fields matches what the writer did before. Users who want a DATE column (for example for BigQuery) can still select Date. Update getFieldsProposesDateForADateAndTheDefaultForTheOthers (ParquetLogicalTypeTest.java:221) and the sentence about Get Fields in parquet-file-output.adoc:189.