Skip to content

fix: DataFormatter trailing-comma scaling and literals between digits - #1232

Merged
bengbengbalabalabeng merged 2 commits into
mainfrom
fix/dataformatter-trailing-commas
Oct 17, 2026
Merged

bengbengbalabalabeng merged 2 commits into
mainfrom
fix/dataformatter-trailing-commas

Conversation

@pjfanning

Copy link
Copy Markdown
Member

Ports apache/poi#1396 to Fesod's DataFormatter. Fixes #1120.

Warning

This PR and #1231 (the "check DataFormatter against Excel" test) will conflict. Whichever is merged second needs a rebase.
#1231's test asserts that each known failure still differs from Excel. This PR fixes 167 of those cases, so after both are in, #1231's UNSUPPORTED_FORMATS / KNOWN_FAILURES lists need those entries removed:

  • the trailing-comma formats 0,, #,,, 0.#, and #,.#,
  • the fraction formats with escaped text, such as |#\:#=/=#| and |0\:0=/=?|

I checked by running #1231's test on this branch: the only failures were "now matches Excel" cases, and there were no regressions. Matches go from 151 to 318 of the 569 Excel cases.

Note

Overlap with #1121. #1121 (@nkuprins) fixes the same issue (#1120) in the same code, so only one of the two should be merged. All 14 test cases from #1121's scalesByTrailingCommas also pass on this branch. The two take different approaches:

  • fix: correct scaling of trailing-comma formats #1121 keeps DecimalFormat and adds an extra power of ten for digits after an escaped or quoted dot.
  • This PR follows POI. Any format with literal text between digit placeholders goes to POI's CellFormat, which also handles cases the extra-shift approach can't, e.g. 0".0"0, → 123.05, 0\.00,;-0\.0, (a different literal per section), and fractions with literal text.

Keeping the same logic as POI also makes a later switch to POI's DataFormatter easier.

Changes

1. Trailing commas no longer over-scale.
The issue #2319 workaround also divided by ten for each character between the . and the ,. It used indexOf, so even formats without a dot were affected. That has been removed.

Format Value Before After (= Excel)
0, 1234567 124 1235
#,, 1234567 0 1
0.#, 1234567 123.5 1234.6

The division is now exact (BigDecimal.divide(divider), no rounding), so the decimal places survive until formatting. #,##0.0, gives 1,234.6.

2. ! is an escape character, like \.
0!.0, → 0.9 and 0!! → 5!. A ! inside quotes, inside brackets, or after _ or * is left alone.

3. Number formats with literal text between digit placeholders go to CellFormat.
DecimalFormat can't express these. For #2319's format 0\.0,, 9234 → 0.9 and 123456 → 12.3. The format is cached as a CellFormatWrapper.

Tests

  • DataFormatterTest: new scalesByTrailingCommas, keepsLiteralTextBetweenDigits and treatsExclamationMarkAsEscape tables. These are POI's #1396 cases, plus the 0, / #,, / 0.#, regressions and #2319's values.
  • TrailingCommaFormatReadTest: reads the #2319 workbooks (temp/issue2319/test1.xlsx and test2.xlsx, still in the test resources) end to end and asserts the formatted values. The original Issue2319Test was removed in refactor(examples): Fesod Examples Module Refactoring #770, and it only printed results without asserting anything.
  • ./mvnw -pl fesod-sheet test -Dmaven.test.skip=false: 1112 tests, 0 failures. spotless:check is clean.

🤖 Generated with Claude Code

Port of apache/poi#1396.

- Trailing commas scaled by an extra power of ten for each character between
  the decimal point and the comma (the issue #2319 workaround), e.g. "0,"
  showed 1234567 as "124" instead of "1235". Remove that and divide exactly,
  so formats such as "#,##0.0," keep their decimal places.
- Excel treats '!' as an escape like '\'; rewrite it as '\' outside quotes,
  brackets and '_'/'*' operands.
- Number formats with quoted or escaped text between digit placeholders, e.g.
  "0\.0," (issue #2319: shows 9234 as "0.9"), cannot be expressed with
  DecimalFormat, so they are now formatted by CellFormat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Test Results

1 178 tests  +25   1 178 ✅ +25   2m 22s ⏱️ -6s
  129 suites  -  1       0 💤 ± 0 
  129 files    -  1       0 ❌ ± 0 

Results for commit a1b6175. ± Comparison against base commit 56e8327.

This pull request removes 8 and adds 33 tests. Note that renamed tests count towards both.
org.apache.fesod.sheet.benchmark.util.BenchmarkFileUtilTest ‑ createTestDataDirectoryIsIdempotent
org.apache.fesod.sheet.benchmark.util.BenchmarkFileUtilTest ‑ createTestFilePlacesFileInTestDataDirectory
org.apache.fesod.sheet.benchmark.util.DataGeneratorTest ‑ datesStayWithinAnchoredWindows
org.apache.fesod.sheet.benchmark.util.DataGeneratorTest ‑ defaultConstructorUsesFixedSeed
org.apache.fesod.sheet.benchmark.util.DataGeneratorTest ‑ differentSeedsProduceDifferentData
org.apache.fesod.sheet.benchmark.util.DataGeneratorTest ‑ generateTestDataMatchesDatasetSize
org.apache.fesod.sheet.benchmark.util.DataGeneratorTest ‑ generatedRowsAreComplete
org.apache.fesod.sheet.benchmark.util.DataGeneratorTest ‑ sameSeedProducesIdenticalData
org.apache.fesod.sheet.metadata.format.DataFormatterTest ‑ keepsLiteralTextAfterTrailingCommas(String, String, String)[1]
org.apache.fesod.sheet.metadata.format.DataFormatterTest ‑ keepsLiteralTextAfterTrailingCommas(String, String, String)[2]
org.apache.fesod.sheet.metadata.format.DataFormatterTest ‑ keepsLiteralTextAfterTrailingCommas(String, String, String)[3]
org.apache.fesod.sheet.metadata.format.DataFormatterTest ‑ keepsLiteralTextAfterTrailingCommas(String, String, String)[4]
org.apache.fesod.sheet.metadata.format.DataFormatterTest ‑ keepsLiteralTextAfterTrailingCommas(String, String, String)[5]
org.apache.fesod.sheet.metadata.format.DataFormatterTest ‑ keepsLiteralTextAfterTrailingCommas(String, String, String)[6]
org.apache.fesod.sheet.metadata.format.DataFormatterTest ‑ keepsLiteralTextBetweenDigits(String, String, String)[10]
org.apache.fesod.sheet.metadata.format.DataFormatterTest ‑ keepsLiteralTextBetweenDigits(String, String, String)[1]
org.apache.fesod.sheet.metadata.format.DataFormatterTest ‑ keepsLiteralTextBetweenDigits(String, String, String)[2]
org.apache.fesod.sheet.metadata.format.DataFormatterTest ‑ keepsLiteralTextBetweenDigits(String, String, String)[3]
…

♻️ This comment has been updated with latest results.

@nkuprins

Copy link
Copy Markdown
Contributor

Great PR! Thank you very much!

@skytin1004 skytin1004 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM @pjfanning. I verified the changes against the issue and ran the tests locally. All tests passed, and I did not find a blocker.

Formats with quoted or escaped text after trailing scaling commas, such as
0.0,"万" or #,##0,,"M", fell back to the raw value because the stripped
pattern is not valid for DecimalFormat, and a quoted comma in #,##0.0,","
was taken as an extra scaling comma. Route these to CellFormat too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pjfanning added a commit to apache/poi that referenced this pull request Oct 17, 2026
…mat (#1426)

* DataFormatter: format literal text after trailing commas with CellFormat

Formats with quoted or escaped text after trailing scaling commas, such as
0.0,"万" or #,##0,,"M", fell back to the raw value because the stripped
pattern is not valid for DecimalFormat, and a quoted comma in #,##0.0,","
was treated as another scaling comma. Route these sections to CellFormat,
like formats with literal text between digit placeholders.

Reported in apache/fesod#1232, whose DataFormatter was ported from POI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Note that the trailing-comma literal cases were checked with Excel

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

@bengbengbalabalabeng bengbengbalabalabeng left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@bengbengbalabalabeng
bengbengbalabalabeng merged commit 50dc2c9 into main Oct 17, 2026
11 checks passed
@bengbengbalabalabeng
bengbengbalabalabeng deleted the fix/dataformatter-trailing-commas branch October 17, 2026 11:57
pjfanning added a commit that referenced this pull request Oct 17, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bengbengbalabalabeng added a commit that referenced this pull request Oct 19, 2026
* test: check DataFormatter against text computed by Excel

Adds DataFormatterAgainstExcelTest, a port of POI's TestDataFormatterAgainstExcel
(apache/poi#1398). The TEXT(value, format) results cached in Excel-saved sample
workbooks from POI's test data are compared with what Fesod's DataFormatter
produces. Chinese date/time and [DBNum] cases from POI's TestDataFormatter
(apache/poi#1400) are also included.

Cases that do not match are listed, grouped by cause, and asserted to still
differ, so the lists document current behaviour and give regression coverage
for changes to the formatter, such as switching to POI's DataFormatter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: drop cases fixed by #1232 from the known failures

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Bengbengbalabalabeng <70380092+bengbengbalabalabeng@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] A number format ending in a comma scales the value by an extra power of ten

4 participants