Migrate Jira search from removed /rest/api/3/search to /rest/api/3/se…

[harshasannapaneni]

Migrate Jira search from removed /rest/api/3/search to /rest/api/3/search/jql

Closes #7104

Description

Atlassian permanently removed the /rest/api/3/search endpoint (CHANGE-2046, effective May 1, 2025). All Jira Cloud instances now return HTTP 410 GONE for this endpoint. This change migrates the Jira source plugin to the replacement /rest/api/3/search/jql endpoint.

Key changes:

• Updated search endpoint URL from rest/api/3/search to rest/api/3/search/jql
• Migrated from offset-based (startAt) to cursor-based (nextPageToken) pagination per the new API contract
• Added nextPageToken and isLast fields to the SearchResults model
• Updated pagination loop in JiraService to use cursor-based iteration

Issues Resolved

Closes #7104

Check List

• New functionality includes testing.
• New functionality has a documentation issue. Please link to it in this PR.
  • New functionality has javadoc added
• Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 849b105)

Here are some key observations to aid the review process:

🧪 PR contains tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ Recommended focus areas for review Missing First Page The pagination loop condition while (!isLast && nextPageToken != null) uses the token from the response to continue, but a proper cursor-based iteration should terminate only when isLast is true. If the first (or any) page returns isLast=false with a valid nextPageToken, this works; however, on the initial iteration nextPageToken starts as null and the loop always fires once (do-while). After that first request, if the response has isLast=false but nextPageToken=null (transient/malformed response), the loop exits silently with a warning and some results may be missed. Consider failing loudly or retrying instead of only logging a warning, since silent partial ingestion can lead to data loss. do { SearchResults searchIssues = jiraRestClient.getAllIssues(jql, nextPageToken); List<IssueBean> issueList = new ArrayList<>(searchIssues.getIssues()); totalFound += issueList.size(); nextPageToken = searchIssues.getNextPageToken(); isLast = Boolean.TRUE.equals(searchIssues.getIsLast()); addItemsToQueue(issueList, itemInfoQueue); } while (!isLast && nextPageToken != null); if (!isLast && nextPageToken == null) { log.warn("Jira search pagination ended unexpectedly: isLast={} but nextPageToken is null. Some results may be missing.", isLast); }

[github-actions]

PR Code Suggestions ✨

Latest suggestions up to 849b105

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Explicitly request required issue fields *The new /rest/api/3/search/jql endpoint has different semantics for the fields and expand parameters compared to the old endpoint, and by default returns only the id field unless fields is explicitly specified. Without a fields query parameter, downstream processing may not receive expected issue data (e.g., project, updated, created). Verify that required fields are being requested, or add a fields query parameter (e.g., fields=all or a specific list). data-prepper-plugins/saas-source-plugins/jira-source/src/main/java/org/opensearch/dataprepper/plugins/source/jira/rest/JiraRestClient.java [92-101] UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl(url) .queryParam(MAX_RESULT, FIFTY) .queryParam(JQL_FIELD, jql) + .queryParam("fields", "*all") .queryParam(EXPAND_FIELD, EXPAND_VALUE); if (nextPageToken != null) { builder.queryParam(NEXT_PAGE_TOKEN, nextPageToken); } URI uri = builder.buildAndExpand().toUri(); Suggestion importance[1-10]: 7 __ Why: This is a valid and important concern; the new /rest/api/3/search/jql endpoint by default returns only the id field, which could cause downstream data loss if fields like project, created, updated are not explicitly requested. However, it is presented as a verification suggestion.	Medium
	Fix pagination termination logic The loop condition !isLast && nextPageToken != null will terminate after the first page if the response omits isLast (null) and has no nextPageToken, but it also fails to fetch subsequent pages when only isLast=false is returned without a token on the very first iteration. More critically, on the very first call nextPageToken starts as null, so if the API returns isLast=null and no token, the loop exits after one iteration correctly — but the condition should preferably rely on isLast alone (per Jira's new API contract) to avoid premature termination. Consider looping while !isLast and breaking with a warning only when the token is unexpectedly null. data-prepper-plugins/saas-source-plugins/jira-source/src/main/java/org/opensearch/dataprepper/plugins/source/jira/JiraService.java [103-113] do { SearchResults searchIssues = jiraRestClient.getAllIssues(jql, nextPageToken); List<IssueBean> issueList = new ArrayList<>(searchIssues.getIssues()); totalFound += issueList.size(); nextPageToken = searchIssues.getNextPageToken(); isLast = Boolean.TRUE.equals(searchIssues.getIsLast()); addItemsToQueue(issueList, itemInfoQueue); -} while (!isLast && nextPageToken != null); + if (!isLast && nextPageToken == null) { + log.warn("Jira search pagination ended unexpectedly: isLast=false but nextPageToken is null. Some results may be missing."); + break; + } +} while (!isLast); Suggestion importance[1-10]: 4 __ Why: The suggestion refactors the loop condition to rely primarily on isLast, which is a reasonable style improvement, but the existing logic already handles termination correctly with the warning log after the loop. The impact is minor.	Low

---

Previous suggestions

Suggestions up to commit 681baad

Category	Suggestion	Impact
Possible issue	Explicitly request issue fields in new endpoint *The new /rest/api/3/search/jql endpoint does not support the expand=renderedFields behavior in the same way and requires an explicit fields parameter, otherwise it may return only id and key (no issue fields at all). Verify that fields are requested explicitly (e.g., fields=all or a specific list); otherwise downstream code relying on issue fields will break. data-prepper-plugins/saas-source-plugins/jira-source/src/main/java/org/opensearch/dataprepper/plugins/source/jira/rest/JiraRestClient.java [92-101] UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl(url) .queryParam(MAX_RESULT, FIFTY) .queryParam(JQL_FIELD, jql) + .queryParam("fields", "*all") .queryParam(EXPAND_FIELD, EXPAND_VALUE); if (nextPageToken != null) { builder.queryParam(NEXT_PAGE_TOKEN, nextPageToken); } URI uri = builder.buildAndExpand().toUri(); Suggestion importance[1-10]: 7 __ Why: This is a valid concern: the new /rest/api/3/search/jql endpoint has different default field behavior and may return only id/key without explicit fields param, which could break downstream processing.	Medium
	Prevent infinite pagination loop on repeated tokens The loop's termination condition is fragile: if the API returns isLast=false but no nextPageToken (or if isLast is null on the last page), the loop exits silently or infinitely re-fetches the same page when nextPageToken stays non-null but isLast is null. Also, when isLast is null but there are no more issues, this may loop forever. Consider terminating when isLast is true OR nextPageToken is null, and guard against repeated tokens or empty responses to avoid infinite loops. data-prepper-plugins/saas-source-plugins/jira-source/src/main/java/org/opensearch/dataprepper/plugins/source/jira/JiraService.java [104-110] SearchResults searchIssues = jiraRestClient.getAllIssues(jql, nextPageToken); List<IssueBean> issueList = new ArrayList<>(searchIssues.getIssues()); totalFound += issueList.size(); + String previousToken = nextPageToken; nextPageToken = searchIssues.getNextPageToken(); isLast = Boolean.TRUE.equals(searchIssues.getIsLast()); addItemsToQueue(issueList, itemInfoQueue); + if (nextPageToken != null && nextPageToken.equals(previousToken)) { + log.warn("Jira pagination returned the same nextPageToken twice; stopping to avoid infinite loop."); + break; + } } while (!isLast && nextPageToken != null); Suggestion importance[1-10]: 5 __ Why: Adding a guard against infinite pagination loops (same nextPageToken returned twice) is a reasonable defensive measure, though the underlying API contract likely handles this correctly. Moderate impact.	Low
General	Fix misleading pagination warning condition The warning condition can never be true as written: the loop only exits when isLast is true or nextPageToken is null; combined with !isLast, this branch is reachable, but isLast will always be false here, making the log message misleading. Additionally, a null isLast from the API is treated identically to false, which may spam warnings for normal completion. Consider only warning when the API response explicitly indicated more pages exist. data-prepper-plugins/saas-source-plugins/jira-source/src/main/java/org/opensearch/dataprepper/plugins/source/jira/JiraService.java [111-113] } while (!isLast && nextPageToken != null); - if (!isLast && nextPageToken == null) { - log.warn("Jira search pagination ended unexpectedly: isLast={} but nextPageToken is null. Some results may be missing.", isLast); + if (!isLast && nextPageToken == null && Boolean.FALSE.equals(isLast)) { + log.warn("Jira search pagination ended unexpectedly: isLast=false but nextPageToken is null. Some results may be missing."); } Suggestion importance[1-10]: 4 __ Why: The observation about the warning being potentially misleading when isLast is null is valid, but the improved code's added condition Boolean.FALSE.equals(isLast) is redundant with !isLast here in effect and provides only marginal improvement.	Low

Suggestions up to commit e9e6db4

Category	Suggestion	Impact
Possible issue	Explicitly request issue fields from new endpoint *The new /rest/api/3/search/jql endpoint requires an explicit fields parameter to return issue data; without it, the response may omit fields relied upon downstream. Additionally, the expand parameter behavior differs on this endpoint. Verify that required fields (e.g., all or specific field names) are requested via a fields query parameter. data-prepper-plugins/saas-source-plugins/jira-source/src/main/java/org/opensearch/dataprepper/plugins/source/jira/rest/JiraRestClient.java [92-95] UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl(url) .queryParam(MAX_RESULT, FIFTY) .queryParam(JQL_FIELD, jql) - .queryParam(EXPAND_FIELD, EXPAND_VALUE); + .queryParam(EXPAND_FIELD, EXPAND_VALUE) + .queryParam("fields", "*all"); Suggestion importance[1-10]: 6 __ Why: The new /rest/api/3/search/jql endpoint indeed behaves differently regarding fields, and this suggestion raises a valid concern about downstream data completeness, though it needs verification against Jira API docs.	Low
	Prevent potential infinite pagination loop The loop termination condition !isLast && nextPageToken != null can cause an infinite loop if the API returns isLast=false but without a nextPageToken, or if isLast is null and a token is present indefinitely. Also, if the first response has isLast=null and no token, the loop still executes at least once (do-while) which is fine, but subsequent iterations should stop as soon as either signal indicates completion. Consider using || semantics or explicitly breaking when nextPageToken is null regardless of isLast. data-prepper-plugins/saas-source-plugins/jira-source/src/main/java/org/opensearch/dataprepper/plugins/source/jira/JiraService.java [104-110] SearchResults searchIssues = jiraRestClient.getAllIssues(jql, nextPageToken); List<IssueBean> issueList = new ArrayList<>(searchIssues.getIssues()); totalFound += issueList.size(); nextPageToken = searchIssues.getNextPageToken(); isLast = Boolean.TRUE.equals(searchIssues.getIsLast()); addItemsToQueue(issueList, itemInfoQueue); - } while (!isLast && nextPageToken != null); + } while (!isLast && nextPageToken != null && !nextPageToken.isEmpty()); Suggestion importance[1-10]: 4 __ Why: The added !nextPageToken.isEmpty() check is a minor defensive improvement; the existing condition !isLast && nextPageToken != null already handles null tokens, so the impact is small.	Low

[dinujoh]

can we verify builder.queryParam(NEXT_PAGE_TOKEN, nextPageToken) call called ?

[harshasannapaneni]

Good catch, will add the verification

[dinujoh]

can we verify searchIssues.getNextPageToken() was called multiple times ?

[harshasannapaneni]

Good catch, will add the verification.

[github-actions]

Persistent review updated to latest commit 681baad

[github-actions]

Persistent review updated to latest commit 849b105