To host stats.jenkins.io GSoC 2024 project in jenkins-infra

[krisstern]

Service(s)

Helpdesk, stats.jenkins.io

Summary

We are in the process of redeveloping the frontend for the data presentation of the Jenkins Infra Statitics project, and will need help migrating the GSoC contributor's repo (https://github.com/shlomomdahan/stats.jenkins.io/) from his personal GitHub account to one in the jenkins-infra org.

@lemeurherve @gounthar @shlomomdahan

Reproduction steps

N/A

[lemeurherve]

@shlomomdahan could you initiate the transfer of your repository to https://githubcom/jenkins-infra please?

See https://docs.github.com/en/repositories/creating-and-managing-repositories/transferring-a-repository#transferring-a-repository-owned-by-your-personal-account

[lemeurherve]

Action plan

• CI
  • Add job to ci.jenkins.io
    • https://ci.jenkins.io/job/Websites/ folder
    • Same job configuration as https://ci.jenkins.io/job/Websites/job/contributors/
    • Add repository access to ci.jenkins.io GitHub app (App ID: 74951)
  • Add job to infra.ci.jenkins.io
    • feat: add stats.jenkins.io job to infra.ci.jenkins.io kubernetes-management#5310
  • Create pipeline
    • Build/lint/test/typos check
    • Netlify preview on pull request
      • Add site to jenkins-infra Netlify account (can be done by uploading dummy content or by using netlify
      • feat: add dedicated stats.jenkins.io PR previews job kubernetes-management#5345
      • feat: dedicated previews pipeline stats.jenkins.io#53
        CLI)
    • feat: add Jenkinsfile stats.jenkins.io#3
  • updatecli manifest to set nodejs version to the one defined in packer-images
    • Ref: Disable renovate for nodejs and use updatecli to retrieve last version from packer-images repository contributor-spotlight#124
    • chore: add updatecli manifest to retrieve nodejs version from packer-images stats.jenkins.io#5
  • Add status checks to repository branch protection
    
• Production
  • Host the service
    • Add a new.stats.origin CNAME record to jenkins.io pointing to publick8s
    • Add a new.stats CNAME record to jenkins.io pointing to publick8s
      • feat: add new.stats jenkins.io CNAME record azure-net#257
    • Nginx based helm chart
      • Reusing https://github.com/jenkins-infra/helm-charts/tree/main/charts/nginx-website
    • Storage account & file share
      • No backup as the content is generated from the infra-statistics repository
      • 5Gio size, should be more than enough
      • feat: add stats.jenkins.io storage account and file share azure#729
    • Deployment on publick8s cluster
      • Create secret in jenkins-infra/charts-secrets (private repo) with storage account credentials
        • https://github.com/jenkins-infra/charts-secrets/commit/33ad9f6fff020917a3d78afb6c19412dd4bef8ad
      • Create release
        • feat(publick8s): add stats.jenkins.io service kubernetes-management#5325
  • Setup Fastly cache
    • Not until we can measure if it's needed, see To host stats.jenkins.io GSoC 2024 project in jenkins-infra #4132 (comment)
  • Add Datadog monitor
    • feat: add stats.jenkins.io synthetics monitoring datadog#251
• CD
  • Add deployment step to the pipeline (publication from infra.ci.jenkins.io)
    • Use azcopy and fine grained File Share SAS token
      • Create a service principal to interact with the File Share from infra.ci.jenkins.io
        • feat: create a service principal to interact with stats-jenkins-io File Share azure#733
      • Add event in Jenkins Infra shared calendar as reminder for updating File Share expiration date in three months
        
      • Add an updatecli manifest to update service principal end date
        • chore(updatecli): track infraci_stats_jenkins_io_fileshare_serviceprincipal_writer end date azure#737
      • Add credentials to infra.ci.jenkins.io
        • https://github.com/jenkins-infra/charts-secrets/commit/67d0e595754f5d0acb991c70c20b4371f182b345
        • feat(privatek8s/infra.ci.jenkins.io): create azure service principal credentials for stats.jenkins.io kubernetes-management#5334
  • feat: add production deployment stage stats.jenkins.io#43
• Miscellaneous
  • Add CODEOWNERS
    • chore: add CODEOWNERS stats.jenkins.io#27
  • Add Renovate
    • Add repository access to Renovate GitHub app
    • chore: Configure Renovate stats.jenkins.io#19
    • Pin Dependencies dashboard Renovate issue
      • Dependency Dashboard stats.jenkins.io#22
  • Add a service and the corresponding label to jenkins-infra/helpdesk issue templates
    • Already included since bfff995
  • Setup Matomo site
    • Not a Gatsby website, need to be implemented
  • ???
  • Profit
• Relocate this new website from new.stats.jenkins.io to stats.jenkins.io

Similar to:

• Start a new repo under jenkins-infra for the Jenkins Contributor Spotlight feature on jenkins.io #3809 (comment)
• To host versioned jenkins.io docs on docs.jenkins.io #3885 (comment)

[lemeurherve]

@krisstern transfered the repository to jenkins-infra org (thanks!).

I've set @krisstern as "maintainer", and https://github.com/orgs/jenkins-infra/teams/core as "admin".

[krisstern]

Thanks @lemeurherve!

[dduportal]

Question about Fastly: what is the rationale behind using Fastly in this case (compared to serving the website ourselves)?

[lemeurherve]

Using it as CDN, same logic as for other websites like jenkins.io, stories.jenkins.io, contributors.jenkins.io, docs.jenkins.io.

The website would be served from a nginx on publick8s at (new.)stats.origin.jenkins.io like the others.

Am I missing something?

[dduportal]

Using it as CDN, same logic as for other websites like jenkins.io, stories.jenkins.io, contributors.jenkins.io, docs.jenkins.io.

The website would be served from a nginx on publick8s at (new.)stats.origin.jenkins.io like the others.

Am I missing something?

For jenkins.io, the rationale has always been because of the amount of requests costs too much in outbound bandwidth. But it was measured.

do we have current stats for stats.jenkins.io ?

the question is related to: do we need a CDN ? If no then what is the point.

the same question I have for stories and contributors but i never had the answer or at least i do not see anything around numbers and facts for it.

[lemeurherve]

As stats.jenkins.io is currently hosted on GitHub Pages, I'm not sure we can access traffic stats from it.
I found https://github.blog/2014-01-07-introducing-github-traffic-analytics/ but it doesn't seem active anymore.

Then I don't think there's a lot of traffic on these pages, we can start serving it without Fastly then see what are the actual numbers in term of visits.

Removing the Fastly step points and reworking jenkins-infra/azure-net#257

[lemeurherve]

FTR, I invited @shlomomdahan as writer on stats.jenkins.io.

Could be revised when a dedicated team is created.

[lemeurherve]

Opened jenkins-infra/stats.jenkins.io#3 to add a pipeline for building the website on ci.jenkins.io and infra.ci.jenkins.io (pending jenkins-infra/kubernetes-management#5310).

[krisstern]

Hi @lemeurherve, would it be possible to add @gounthar as a co-maintainer?

[lemeurherve]

https://github.com/orgs/jenkins-infra/teams/stats-jenkins-io team including you and @gounthar created and set as "maintainer" on https://github.com/jenkins-infra/stats.jenkins.io/.

[dduportal]

Can we also add @Vandit1604 and @shlomomdahan as the repo maintainers along with @gounthar and me?

For @dduportal @smerle33: they should be added to the @jenkins-infra/stats-jenkins-io team (which has the "maintainer" role on the repository) and their possible custom/personal roles removed.

Just wanted to check, has this task been completed yet?

Hi @krisstern, it has been done:

• Both @gounthar and you had been set up as "maintainers" of the team. It means you will be able to add/change/remove users in the team
• Invitations sent to @Vandit1604 and @shlomomdahan to they can join the team

[dduportal]

Update:

• The new website is online as per Replacing existing stats.jenkins.io code with https://github.com/jenkins-infra/stats.jenkins.io #4265 🥳
• However, switching to the new site did had the following consequences which require additional (and mandatory) work in the upcoming weeks:
  • Plugin site generation was broken as it retrieve data from stats.jenkins.io. This data is missing in the new site so it has to be migrated.
    • See fix(utils.mjs) use the old stats.jenkins.io website (temporarily?) plugin-site#1890 and use the old stats.jenkins.io website (temporarily?) plugin-site#1893
    • Temporarily patch has been applied to avoid rollbacking, giving time for the proper fixes to be applied: we've set up the old site to be available at https://old.stats.jenkins.io until data is available on the new site (or not used by plugin site).

[lemeurherveCB]

(copied from #4265)

As noted in badges/shields#10522 (comment), there are far more consumers than we (I) initially though: https://github.com/search?q=stats.jenkins.io&type=code 😱

Links to (old.)stats.jenkins.io were pointing to files (still) stored on GitHub (currently on the gh-pages, generated from the pipeline of https://github.com/jenkins-infra/infra-statistics), while the new frontend doesn't serve them.

To avoid breaking existing consumers and without serving these files from our service, I propose to put in place redirections from the three folders of https://github.com/jenkins-infra/infra-statistics/tree/gh-pages (then from the data branch) to https://raw.githubusercontent.com/jenkins-infra/infra-statistics/gh-pages/ with the help of jenkins-infra/helm-charts#1332, WDYT?

Ex:
https://stats.jenkins.io/plugin-installation-trend/view-job-filters.stats.json
would redirect to
https://raw.githubusercontent.com/jenkins-infra/infra-statistics/gh-pages/plugin-installation-trend/view-job-filters.stats.json

Note: as https://stats.jenkins.io/jenkins-stats, https://stats.jenkins.io/plugin-installation-trend & https://stats.jenkins.io/pluginversions are not actual pages of the new frontend, this shouldn't cause conflict.
The remaining issue that I see with this proposition is for consumers not following redirections that would still need to adapt their fetching process.

[halkeye]

why take over the old domain at all? its pretty bad practice to break old files, and stats has always been raw metrics, csvs, and stuff

why not call the new site data.jenkins.io or dataviz and keep them seperate?

[lemeurherveCB]

Good point.

[timja]

I think fine to take over the domain but old links need to be maintained

[dduportal]

As underlined by @timja in another thread, any reason not to rollback and re-architecture?
For instance the new frontend in the github page site so it merges both sites ?

[lemeurherveCB]

Yeah that should be doable, the content is static. I take the blame here I should have thought about that possibility. I'll try on a fork tomorrow then propose a plan to adapt the service and decommission the resources that won't be used anymore if that's OK for you.

[dduportal]

Yeah that should be doable, the content is static. I take the blame here I should have thought about that possibility. I'll try on a fork tomorrow then propose a plan to adapt the service and decommission the resources that won't be used anymore if that's OK for you.

That's OK in term of planning 👍 thanks!

@krisstern @lemeurherve @gounthar WDYT if we roll back the DNS to the old site to repair user use cases, and it will let you time to plan, act and retest?

[krisstern]

Sure @dduportal, the arrangement is okay with me

[dduportal]

Just realized something: jenkins-infra/helm-charts#1332 (comment)

Just a point about the GitHub Pages: it only accepts a single CNAME.
Looks like it is a problem here as we have to choose between old.stats.jenkins.io or stats.jenkins.io if we rollback.

1. If we rollback, we'll break users already migrated to use the old.stats.jenkins.io. Including plugin site (but we can revert the PRs quickly for this specific one).
2. GitHub Page is useful for not paying bandwidth, BUT it is not easy for it to manage multiple CNAME (which make sense: it's free). Have to think about it for the new architecture.

• Also: in term of permissions, it is a nightmare for ensuring production works as expected for the Jenkins Infra team. Any repository maintainer can accidentally break it.

[timja]

Also: in term of permissions, it is a nightmare for ensuring production works as expected for the Jenkins Infra team. Any repository maintainer can accidentally break it.

I would say not a real concern, you are highly unlikely to touch this area.

[lemeurherveCB]

I propose to revise the decision to not use Fastly: activating it would allow us to serve files from our service without worrying about the bandwidth, while letting old.stats.jenkins.io in place for consumers who already switched.

[timja]

Sounds good to me, any concerns about using fastly?

Enough credits?

[dduportal]

Sounds good to me, any concerns about using fastly?

Enough credits?

As discussed privately with @lemeurherve :

Cons of setting up Fastly:

• Additional work (to set it up, changing DNSes) as it is an additional component
• We're not sure if it is really needed here (e.g. how much data is really served by stats.jio?)
• Need to monitor if it's not adding too much usage in the sponsorship (I don't know, we can look now but we'll have to check it monthly)

Pros of using Fastly:

• Solve the bandwidth issue
• Fire and forget for the stats.jio website
• Same pattern as www.jenkins.io, plugins.jenkins.io, etc.

Any objection if we start without Fastly (less work and @lemeurherve can focus on solving the misssing files), wait for 1 month, check usage and enable Fastly eventually if needed?

[timja]

Sounds good to me

[lemeurherveCB]

Opened jenkins-infra/stats.jenkins.io#168 to restore broken links by serving all existing static files from stats.jenkins.io while letting old.stats.jenkins.io in place for now.

[lemeurherveCB]

jenkins-infra/stats.jenkins.io#168 has been deployed, links are restored 🙂

Ex: https://stats.jenkins.io/plugin-installation-trend/view-job-filters.stats.json

[lemeurherveCB]

Proposition for the next steps:

• Add old.stats.jenkins.io as host to the new service ingress on publick8s (and remove new.stats.jenkins.io host as this CNAME isn't defined in azure-net anymore)
• Change old.stats.jenkins.io CNAME to point to the new service on publick8s instead of the GitHub pages
• Add old.stats.jenkins.io host to the existing datadog monitor
• Disable GitHub pages publication in infra-statistics
• Change infra-statistics "receiving" branch from gh-pages to data and propagate the change to its pipeline and in stats.jenkins.io's retrieve-infra-statistics-data.sh script
• Revert fix(utils.mjs) use the old stats.jenkins.io website (temporarily?) plugin-site#1890 & use the old stats.jenkins.io website (temporarily?) plugin-site#1893

[dduportal]

The production deployment process is broken since jenkins-infra/stats.jenkins.io#168
as per @lemeurherveCB messag in https://matrix.to/#/!JLUOInpEYmxJIYXlzs:matrix.org/$lK7-7uRQPpVDLey3LeH0KpMj5EJlxIJx7rGDxTsVlLA?via=g4v.dev&via=gitter.im&via=matrix.org

The azcopy keeps failing on subsequent replay or builds.

Currently investigating.

[dduportal]

Currently investigating.

Failures from error logs are described in jenkins-infra/stats.jenkins.io#168 (comment)

[dduportal]

Additional issue with the new website (and the addon of old files) which need to reconsider/fine tune the "try file" method (due to frontend managed routing) - #4291