Testing and Release process

[davepagurek]

Topic

To improve the stability of releases, how would everyone feel about drafting a beta release before doing an official release?

Currently, when we do a release, we discover a number of new bugs as the features get stress tested, which we then work on for a bit to solve, and then either wait for the next major release, or come up with a minor release. Potentially, before doing the release, we could make a beta build for people to test with. If enough people would be interested in experimenting with new features and letting us know what bugs they encounter, it could help us figure out what needs to get fixed before a release, and also make it easier for people to test out new features.

Is beta testing p5 something people are interested in?

[Qianqianye]

Thanks @davepagurek! Would love to hear what @limzykenneth thinks on this, since he worked on most of the code for the release process.

[limzykenneth]

It depends a bit on what this would be, there are two possible types I'm thinking of.

1. A nightly build where a rolling version is built off of current main everyday that is replaced daily.
2. A rc (release candidate) build where we branch off of main to create the beta version for bugs to be worked out (that are backported to main) but won't be including any new commits to main.

1 should be relatively easy to do, but might not be that effective at catching bug before release, while 2 would probably complicate the contribution process, especially when a new release is being prepared. Unless you are thinking of some other mechanism @davepagurek?

[davepagurek]

I was thinking something like 2, but not so much about making a branch that only has bug fixes, since I agree that would be too complicated. I was thinking just about making a build for people a bit before release as a way of soliciting some new bug reports. We kinda do something similar already from contributors testing builds off of main, but then all bug reports for new features are coming from just a few of us testing them.

[limzykenneth]

@davepagurek That would be somewhere in between the two I feel, with some of the pros from both and some of the cons too.

Pros:

• Easy and simple setup, no need more release candidate branch
• Have a beta version for testing

Cons:

• Possible that commits/PR occuring after the beta release contains bugs too that make it into the final release
• Or alternatively (less desirable) prevent larger changes to be merged between beta relase and final release

[davepagurek]

Possible that commits/PR occuring after the beta release contains bugs too that make it into the final release

I think this is something that already occurs in our current process, so while this is still a con, do you think it would still be enough of an improvement to try out?

[limzykenneth]

Yeah I don't mind giving it a go, mainly want to point out potential limitation to this approach. We'll do this for the next release then.

[limzykenneth]

@davepagurek Just to note this is now implemented. I'll try to create a prerelease version later today.

[limzykenneth]

https://github.com/processing/p5.js/releases/tag/v1.9.1-1 if you haven't seen it yet