[Proposal] Move the OpenSearch 2.12.0 release to Feb 20 2024

[bbarani]

Is your feature request related to a problem? Please describe

OpenSearch 2.12.0 release is currently scheduled to be released on Jan 23 2024. We are proposing to move the release date of OpenSearch 2.12.0 version to Feb 20 2024 to accommodate the below action items,

• Upgrade Lucene version in OpenSearch 2.12.0 version to 9.9.1 instead of Lucene 9.9.0 version to resolve critical bugs
• Ensure readiness across all the repositories to support JDK21 upgrade
• Test the JDK21 + Lucene 9.9.1 changes thoroughly before releasing the 2.12.0 version
• Fix flaky, unreliable tests across OpenSearch repositories

Describe the solution you'd like

The code freeze date for OpenSearch 2.12.0 release is currently set to Jan 09 2024 with release date schedule for Jan 23 2024. We are proposing to move the code freeze date for OpenSearch 2.12.0 release to Feb 06 2024 with release date set to Feb 20 2024.

This proposal will help the team to upgrade, test OpenSearch Lucene version to 9.9.1 along with the JDK21 thoroughly in 2.12.0 version. Also, the additional time would allow the teams to close the flaky tests before the 2.12.0 release which would help in improving the quality of OpenSearch artifacts.

Describe alternatives you've considered

Keep the release date for 2.12.0 version as planned.

Additional context

No response

[anirudha]

The code freeze date for OpenSearch 2.12.0 release is currently set to Jan 09 2024 with release date schedule for Jan 23 2024. We are proposing to move the code freeze date for OpenSearch 2.12.0 release to Feb 06 2024 with release date set to Feb 20 2024.

what do we need to do to make this a reality? @bbarani

[macohen]

How long is this up for a vote? Can we say this should stay open until end of the day on January 2nd?

BTW, if you want to vote, thumbs up or thumbs down the description posted by @bbarani.

[dtaivpp]

@macohen I think that sounds reasonable.

[krisfreedain]

Agree @macohen - good suggestion.

[AmiStrn]

No objection to postponing this. 1 month is not significant, especially with the proposed critical bug fixes.
Does anyone have any downside to this?
I see that many items would anyway be moving to 2.13 at this rate.
@deshsidd regarding the postponing of phase 1 of your PR -> what is your opinion on the proposed release date?

[navneet1v]

Do we have a decision here? Is the release date moved?

[Gaganjuneja]

Any update on this?

[macohen]

I suggested that we leave this open until end of the day on Jan 2, but failed to give a time. Let's say 5PM PST. It does look like we will delay, but I think we should stick to timelines on votes as a community.

[CEHENKLE]

I think moving to out to pick up 9.9.1 (and thereby get fixes for those bugs) makes some sense. The risk/trade-off we're making is that the longer you go before you have a release, the more changes you have to integrate. We've had problems in the past, even on a 6 week cycle, with people batching changes at the last minute. Are there steps we could take to make this release go smoother/derisk 12 week merge?

[dblock]

We have merged code changes to re-add support for JDK8 to opensearch-java that has been asked for by many. Delaying 2.12 will delay that as the client is tightly coupled, so that would be a reason not to do it. We'll also have to work on reverting these changes and the re-merging them on a bigger diff. See opensearch-project/opensearch-java#777.

cc: @reta

[macohen]

@bbarani what do you think? @dblock could this be considered a patch release? Just looking for a way...

[dblock]

I am not sure what's best @macohen. We can live without the release of course, I am just throwing in a reason not to.

[Pallavi-AWS]

hi @dblock and others, are there other reasons besides opensearch-java to not delay the release? Getting to the right version of Lucene is important for us, hence we are inclined to delay.

[reta]

hi @dblock and others, are there other reasons besides opensearch-java to not delay the release? Getting to the right version of Lucene is important for us, hence we are inclined to delay.

@bbarani @Pallavi-AWS just in case, we have reverted the changes in opensearch-java 2.x branch (related to JDK-8), but note, this feature could only be merged after 2.12.0 release.

[sohami]

We also recently found a memory leak issue with concurrent segment search flow (Ref here) and fix for that is in lucene 9.9. We will need this change for GA release of concurrent segment search. Would be great if we can consider moving to lucene 9.9 in 2.12 as suggested in this proposal.

[macohen]

Thinking through this more, I do think we should try to stick to the release train model: every 6 weeks(ish) we start the window. If your changes are on the train, you're in the release. If not, then you wait for the next one. We do run the risk of having a ton of last minute updates as per @CEHENKLE, but I don't really know how to force anyone to merge changes more frequently. I also ack that there are 0 thumbs down on the vote, so we should move the release out this time.

These changes to JDK-21 and Lucene 9.9.1 are going to have a large impact across repositories. How soon after the 23rd (or actual release date) will 9.9.1 be integrated into 3.0 and backported to 2.x for testing/integration for other repos?

[bbarani]

We usually follow the ~6 week cadence for minor and patch releases as mentioned in this documentation. As mentioned by @macohen this proposal is to request an exception only for 2.12.0 release to accommodate Lucene and JDK version upgrade as the changes impact broad surface area. We would like to also use this additional time to close as many long pending flaky tests as possible before the 2.12.0 release.

[reta]

We would like to also use this additional time to close as many long pending flaky tests as possible before the 2.12.0 release.

👍 , personally, the benefits of delivering stable release (that incorporates quite large scale changes, as in case of 2.12.0) is far outweigh few weeks delay.

[macohen]

I'm in. @bbarani if you're satisfied, will you close this and update the roadmap accordingly? I'm just going to keep talking here otherwise and that won't do anyone any good.😉

[Pallavi-AWS]

Reading all comments on the benefits, I am inclined to make an an exception and delay 2.12.0 release. Thanks.

[macohen]

@krisfreedain will update the release calendar for the new date and then close this issue. Thanks all for the input and discussion.

[krisfreedain]

2.12.0 Release date has been updated on the calendar. Thank you all for the discussion & the engagement on voting.

[mashah]

Folks,

Thank you for warning us about the delayed release until Feb 20. In the future, I would object to all releases that are delayed without sufficient voice from non-AWS contributors.

I do agree that we want to have a stable release. That said, it seems that AWS needs and timelines seem to be prioritized over others.

[mashah]

We at Aryn have been waiting for the conversational memory and RAG features to move from experimental to GA. That is supposed to happen in 2.12 and our customers are waiting for that. The 1 month delay is critical for us.

[CEHENKLE]

Sorry to reopen, but I had a thought: We could hold the 2.12.0 of Jan 23rd, and then pull in the 2.13.0 by 2 weeks so it's a 4 week cadence rather than a 6 week.

Pros:

• It gets out features faster, and biases towards releasing rather than not releasing.
• We don't have a long(er) delay between releases, which reduces the risk of large merge conflicts
• It stays truer to the spirit of a release train model -- we go when we go.
• It puts some distance between two large changes (Lucene and JDK) if we split them up between 2.12 and 2.13

Cons:

• 4 weeks is tight for the team to release, but (IMHO as someone who's not doing the release, so take this with a boulder-sized grain of salt) is doable. We could move out the 1.3.x release to make some room and take some pressure off the team.

Thoughts?

[mashah]

Thanks @CEHENKLE !

We're in favor for keeping the 2.12.0 to Jan 23rd. We're also in favor for bringing in the 2.13.0 in by 2 weeks, to have a 4 week cadence.

Pros:

1. We're moving faster as a project.
2. We more firmly establish a "catch the train" model for releases. Community has more confidence in the release dates.
3. Each release is more manageable.

Cons:

1. Shorter timeline for AWS team and a revert from what was decided earlier.

[Pallavi-AWS]

@CEHENKLE @mashah we need to account time to upgrade to JDK21 for vector and core performance improvements and Lucene 9.9.1 for critical bug fixes in 2.12. We are currently working through both these upgrades. After the release calendar was updated with the new date, the community and repo teams have been planning with the new date. We can certainly improve the process for release date changes going forward (we can have a separate discussion on an improved process).

[CEHENKLE]

Thanks, @Pallavi-AWS for the follow up. Given you want more calendar time to work on 9.9.1, wouldn't it make sense to release what we have now at the planned time at the end of Jan, and then follow up in Feb with the critical fixes?

Thanks,
/C

[msfroh]

For the record, Lucene 9.9.1 was just merged into main: opensearch-project/OpenSearch#11421

Sure enough, the automatic backport to 2.x failed, but @mch2 said that resolving it should be smooth.

[Pallavi-AWS]

@CEHENKLE Lucene 9.9.1 fixes memory leak bugs, which we have been waiting for. The work needed to upgrade is in-flight. Additionally, JDK21 provides a range of significant performance improvements, including vector search where OpenSearch is seeing rapid adoption.

[bbarani]

@mashah @CEHENKLE The changes to support Lucene 9.9.1 and JDK21 seems to have already merged in to 2.x line across all repositories and reverting it might not be a straightforward process at this point in time. Also, as mentioned by @Pallavi-AWS, Lucene 9.9.1 fixes memory leak bugs, which we have been waiting for. The work needed to upgrade is in-flight. Additionally, JDK21 provides a range of significant performance improvements, including vector search. Having said that, are you ok with the proposed release date of Feb 20 2024 OR do we need to debate this further? We should take the decision by Jan 16 2024 to avoid any further delays. CC: @macohen @reta @elfisher

[CEHENKLE]

@bbarani I think the best thing we can take out of this is to revisit it in the retro how we make decisions and make changes going forward, but untangling things technically is probably more expensive that we can afford now. So I am okay with closing this and executing as planned.

[bbarani]

Closing this issue as there's no additional feedback. 2.12.0 Release date has been updated on the calendar and scheduled to be released on Feb 20 2024. We will schedule a retro meeting to discuss on release date changes.