Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GITEA] add new gitea service (release/languages) #9781

Merged
merged 13 commits into from
Dec 18, 2023
Merged
2 changes: 2 additions & 0 deletions core/server/server.js
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@ const publicConfigSchema = Joi.object({
},
restApiVersion: Joi.date().raw().required(),
},
gitea: defaultService,
gitlab: defaultService,
jira: defaultService,
jenkins: Joi.object({
Expand Down Expand Up @@ -168,6 +169,7 @@ const privateConfigSchema = Joi.object({
gh_client_id: Joi.string(),
gh_client_secret: Joi.string(),
gh_token: Joi.string(),
gitea_token: Joi.string(),
gitlab_token: Joi.string(),
jenkins_user: Joi.string(),
jenkins_pass: Joi.string(),
Expand Down
9 changes: 9 additions & 0 deletions doc/server-secrets.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,15 @@ These settings are used by shields.io for GitHub OAuth app authorization
but will not be necessary for most self-hosted installations. See
[production-hosting.md](./production-hosting.md).

### Gitea

- `GITEA_ORIGINS` (yml: `public.services.gitea.authorizedOrigins`)
- `GITEA_TOKEN` (yml: `private.gitea_token`)

A Gitea [Personal Access Token][gitea-pat] is required for accessing private content. If you need a Gitea token for your self-hosted Shields server then we recommend limiting the scopes to the minimal set necessary for the badges you are using.

[gitea-pat]: https://docs.gitea.com/development/api-usage#generating-and-listing-api-tokens

### GitLab

- `GITLAB_ORIGINS` (yml: `public.services.gitlab.authorizedOrigins`)
Expand Down
90 changes: 90 additions & 0 deletions services/gitea/gitea-base.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
import { BaseJsonService } from '../index.js'

export default class GiteaBase extends BaseJsonService {
static auth = {
passKey: 'gitea_token',
serviceKey: 'gitea',
}

async fetch({ url, options, schema, httpErrors }) {
CanisHelix marked this conversation as resolved.
Show resolved Hide resolved
return this._requestJson(
this.authHelper.withBearerAuthHeader({
schema,
url,
options,
httpErrors,
}),
)
}

async fetchPage({ page, requestParams, schema }) {
CanisHelix marked this conversation as resolved.
Show resolved Hide resolved
const {
options: { searchParams: existingQuery, ...restOptions } = {},
...rest
} = requestParams

requestParams = {
options: {
searchParams: {
...existingQuery,
...{ page },
},
...restOptions,
},
...rest,
}

const { res, buffer } = await this._request(
this.authHelper.withBearerAuthHeader({
...requestParams,
}),
)

const json = this._parseJson(buffer)
const data = this.constructor._validate(json, schema)

return { res, data }
}

async fetchPaginatedArrayData({
url,
options,
schema,
httpErrors,
firstPageOnly = false,
}) {
const requestLimit = 100
const requestParams = {
url,
options: {
headers: { Accept: 'application/json' },
searchParams: { limit: requestLimit },
...options,
},
httpErrors,
}

const {
res: { headers },
data,
} = await this.fetchPage({ page: 1, requestParams, schema })
const numberOfItems = headers['x-total-count']

let numberOfPages = 1
if (numberOfItems > 0) {
numberOfPages = Math.ceil(numberOfItems / requestLimit)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. I have not had a really detailed look at this pagination code, but I'm not sure this is going to work.
If we look at
https://docs.gitea.com/next/development/api-usage
(which I think is relevant)

MAX_RESPONSE_ITEMS is set to 50

says to me that

a) The default max allowable value of requestLimit for a gitea server is 50
b) This is a number that is going to be different for different gitea servers, so we can't safely assume its value

so I think trying to assume we can work out the number of pages up-front based on our hard-coded value of requestLimit and the total number of items feels like it is logic based on a bad assumption.

Tbh, the way we do this for the GitHub badges is we just request the first page of releases ordered by date, and if you want the latest semver then the 'search space' for that is the first page of results, mostly for performance reasons. We've yet to have anyone pop up whose latest release by semver isn't in the first page of results.

Tbh, I'd be happy with that solution here too. When I flagged that fetchPage was unused code, I assumed we'd probably just delete it.

Given we have to deal with multiple different implementations, if we do want to try and implement pagination like this, we'd need to infer the page size rather than assuming it up-front, but as I say, I'd also be happy to just gloss over it if we can.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have MAX_RESPONSE_ITEMS for the absolute maximum, and DEFAULT_PAGING_NUM for the default pagination value which can be different per every instance. We do get a header for x-total-count for how many items are available, and if this is greater than the number of items in the array we could calculate the pages.

When I found the pagination was not working in Gitlab's code too I thought to myself... who even has more than 100 releases to even scan through to find the latest SemVer! Happy to put this back and just assume the first page too.

}

if (numberOfPages === 1 || firstPageOnly) {
return data
}

const pageData = await Promise.all(
[...Array(numberOfPages - 1).keys()].map((_, i) =>
this.fetchPage({ page: ++i + 1, requestParams, schema }),
),
)

return [...data].concat(...pageData.map(p => p.data))
}
}
48 changes: 48 additions & 0 deletions services/gitea/gitea-base.spec.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
import Joi from 'joi'
import { expect } from 'chai'
import nock from 'nock'
import { cleanUpNockAfterEach, defaultContext } from '../test-helpers.js'
import GiteaBase from './gitea-base.js'

class DummyGiteaService extends GiteaBase {
static route = { base: 'fake-base' }

async handle() {
const data = await this.fetch({
schema: Joi.any(),
url: 'https://codeberg.org/api/v1/repos/CanisHelix/shields-badge-test/releases',
})
return { message: data.message }
}
}

describe('GiteaBase', function () {
describe('auth', function () {
cleanUpNockAfterEach()

const config = {
public: {
services: {
gitea: {
authorizedOrigins: ['https://codeberg.org'],
},
},
},
private: {
gitea_token: 'fake-key',
},
}

it('sends the auth information as configured', async function () {
const scope = nock('https://codeberg.org')
.get('/api/v1/repos/CanisHelix/shields-badge-test/releases')
.matchHeader('Authorization', 'Bearer fake-key')
.reply(200, { message: 'fake message' })
expect(
await DummyGiteaService.invoke(defaultContext, config, {}),
).to.not.have.property('isError')

scope.done()
})
})
})
13 changes: 13 additions & 0 deletions services/gitea/gitea-helper.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
const documentation = `
Note that only internet-accessible Gitea compatible instances are supported, for example
[https://codeberg.org](https://codeberg.org).
`

function httpErrorsFor() {
return {
403: 'private repo',
404: 'user or repo not found',
}
}

export { documentation, httpErrorsFor }
80 changes: 80 additions & 0 deletions services/gitea/gitea-languages-count.service.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
import Joi from 'joi'
import { nonNegativeInteger, optionalUrl } from '../validators.js'
import { metric } from '../text-formatters.js'
import { pathParam, queryParam } from '../index.js'
import { documentation, httpErrorsFor } from './gitea-helper.js'
import GiteaBase from './gitea-base.js'

/*
We're expecting a response like { "Python": 39624, "Shell": 104 }
The keys could be anything and {} is a valid response (e.g: for an empty repo)
*/
const schema = Joi.object().pattern(/./, nonNegativeInteger)

const queryParamSchema = Joi.object({
gitea_url: optionalUrl,
CanisHelix marked this conversation as resolved.
Show resolved Hide resolved
}).required()

export default class GiteaLanguageCount extends GiteaBase {
static category = 'analysis'

static route = {
base: 'gitea/languages/count',
pattern: ':user/:repo',
queryParamSchema,
}

static openApi = {
'/gitea/languages/count/{user}/{repo}': {
get: {
summary: 'Gitea language count',
description: documentation,
parameters: [
pathParam({
name: 'user',
example: 'go-gitea',
}),
pathParam({
name: 'repo',
example: 'gitea',
}),
queryParam({
name: 'gitea_url',
example: 'https://codeberg.org',
required: true,
}),
],
},
},
}

static defaultBadgeData = { label: 'languages' }

static render({ languagesCount }) {
return {
message: metric(languagesCount),
color: 'blue',
}
}

async fetch({ user, repo, baseUrl }) {
// https://codeberg.org/api/swagger#/repository/repoGetLanguages
return super.fetch({
schema,
url: `${baseUrl}/api/v1/repos/${user}/${repo}/languages`,
httpErrors: httpErrorsFor('user or repo not found'),
})
}

async handle(
{ user, repo },
{ gitea_url: baseUrl = 'https://codeberg.org' },
CanisHelix marked this conversation as resolved.
Show resolved Hide resolved
CanisHelix marked this conversation as resolved.
Show resolved Hide resolved
) {
const data = await this.fetch({
user,
repo,
baseUrl,
})
return this.constructor.render({ languagesCount: Object.keys(data).length })
}
}
27 changes: 27 additions & 0 deletions services/gitea/gitea-languages-count.tester.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
import Joi from 'joi'
import { createServiceTester } from '../tester.js'

export const t = await createServiceTester()

t.create('language count (empty repo)')
.get(
'/CanisHelix/shields-badge-test-empty.json?gitea_url=https://codeberg.org',
)
.expectBadge({
label: 'languages',
message: '0',
})

t.create('language count')
.get('/CanisHelix/shields-badge-test.json?gitea_url=https://codeberg.org')
.expectBadge({
label: 'languages',
message: Joi.number().integer().positive(),
})

t.create('language count (user or repo not found)')
.get('/CanisHelix/does-not-exist.json?gitea_url=https://codeberg.org')
.expectBadge({
label: 'languages',
message: 'user or repo not found',
})
Loading
Loading