OpenSearch 1.2.0 Release Retro for Engineering Effectiveness

[peternied]

The members of @opensearch-project/engineering-effectiveness team had an small retro about the 1.2.0 release. Here are the raw notes, I'll make sure to create issues/trackbacks here and then close this ticket out, but it will live on the the project history.

1.1.0 vs 1.2.0

• Reduced back and forth for requests, improved in 1.2.0
  • “Use of slack should be at a minimum”
• Docker testing process was a big improvement, reduced EE team demands
• Smoother, reusing processes rather than running for the first time. All teams were more familiar with the tooling
• Not clear why the dates were pushed, lost in the messages in the release channel.
  • a summary could be better?
• Not seeing improvements in BWC and Performance tests
• Running automated tests was much better, used automation for Dashboards
• Maven release was much easier
  • There was an issue with 1.3 snapshots
• Automated many things that were clunky and were smoothly, but didn’t ship new features
• Big release process changes
  • Socialization / Education could be improved
• Reduced meeting overhead, and created a flywheel for teams to improve
• Jenkinsfiles are hugely improvement
  • testability
  • sharing libraries
• traceability
• Team is moving faster
• Releasing older components is harder
  • Previously we did it manually
  • Now building a new process
• Need to keep legacy release process in mind

Preparation Phase

• Missed that Windows / RPM platform weren’t going to make it
• What is the way for a new component to be added?
  • Add documentation for how a new component should be added #1256
• “What is the scope of the release manager if we aren’t making any changes?”
  • Should look into how we do this and improve templates
  • Add guides for scenario 'component is not shipping' #1255
• "How to use this issue" section was useful
• Document the end-to-end OpenSearch release process #1258
  • We don’t completely follow the release template, need to update it
• Document the function of the release manager #1257
  • Alway cycle release manager
• Motivation team alignment
  • Not certain about phase meeting
  • Wasn’t clear what the days should have been aligned to
  • Wasn’t clear the priority
• Instructions assume staggered release
  • this introduces time lag
  • We should always be attempt to prepared for the next major/minor release

Development Phase

• OpenSearch / OpenSearch Dashboard arrived early for plugin dependencies

Release Testing Phase

• Features were pushed in quickly with low quality
• ‘Testing’ quality widely varies
  • Want visibility on the testing
  • No visibility into manual testing verses automation
• Accepted many bugs fixes
• Performance testing
  • testing system has 5% error
  • No comparison process, no way to close the gaps
  • Stop running performance tests to get back time?

Releasing Phase

• Released even when the original release manager was unavailable
• Missing automation to transfer artifacts to from staging to production buckets
• Still haven’t made jenkins public infra

[bbarani]

Closing this ticket as we have already recorded the retro action items in this ticket.