From baf91344b34be13f6ddd4956c281562a1540ee27 Mon Sep 17 00:00:00 2001 From: "David E. Wheeler" Date: Fri, 12 Jul 2024 14:28:35 -0400 Subject: [PATCH] RFC: Meta Spec v2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a new RFC for Meta Spec v2. Based on the [PGXN Meta Sketch] and additional research. Notable differences from v1: * Introduce the term "package" separate from "distribution". The "package" is the bundle of objects being distributed as a…package. * Use only one version for the entire distribution, rather than separate versions for each extension included in the distribution. This allows the deletion of the notes on comparing versions. * Make "Source Distribution" the formal term, but mostly refer to it as "Distribution". Will be distinguished from binary distributions that ship with their own metadata (see #2). * Use JSON data types as the base types instead of generic "list" and "map" types. * Add "Path", "purl", and "Platform" types * Use SPDX License Expressions instead of Perl Software::License-based structures. * Use the term "property" to describe object key/value pairs, to align with JSON Schema. * Use an array of objects to describe maintainers. * Replace the `provides` property with `contents`, with support for multiple kinds of PostgreSQL extensions, including TLEs, loadable modules, and background workers. * Move the `tags` property to the new `classifications` object, and add support for curated categories borrowed from [Trunk]. * Replace `no_index` with `ignore` and use the gitignore format instead of separate lists of files and directories. * Rename `prereqs` to `packages` and move it into the new `dependencies` property, which also has `postgres`, `pipeline`, `platforms`, and `variations` properties. Use [purls]. to specify dependencies, so that any supported packaging dependency can be specified, as well as PGXN packages and Postgres core extensions and tools. * Remove `release_status`; we'll instead depend on [SemVer] to indicate pre-releases. * Simplify the `resources` object and add `badges` to it. * Add the `artifacts` property, so the extension author can include links to other packages or sources for a release. Also configure `#` to hide a line in `json` code blocks and use it to encode proper JSON objects without showing the surrounding braces. Readers can hit the eye button that appears on hover to make the hidden lines appear. [PGXN Meta Sketch]: https://justatheory.com/2024/03/rfc-pgxn-metadata-sketch/ [Trunk]: https://pgt.dev [purls]: https://github.com/package-url/purl-spec/blob/master/PURL-SPECIFICATION.rs [SemVer] https://semver.org --- book.toml | 3 + text/0001-meta-spec-v1.md | 3 + text/0003-meta-spec-v2.md | 1208 +++++++++++++++++++++++++++++++++++++ 3 files changed, 1214 insertions(+) create mode 100644 text/0003-meta-spec-v2.md diff --git a/book.toml b/book.toml index 0eadf83..ab42a15 100644 --- a/book.toml +++ b/book.toml @@ -16,3 +16,6 @@ heading-split-level = 0 [output.html.playground] runnable = false + +[output.html.code.hidelines] +json = "#" \ No newline at end of file diff --git a/text/0001-meta-spec-v1.md b/text/0001-meta-spec-v1.md index b9d1ba0..1388908 100644 --- a/text/0001-meta-spec-v1.md +++ b/text/0001-meta-spec-v1.md @@ -7,6 +7,9 @@ * **Category:** Packaging * **Pull Request:** [pgxn/rfcs#1](https://github.com/pgxn/rfcs/pull/1) +> **Note:** This RFC has been superseded by [RFC--3](0003-meta-spec-v2.md). +> However, it remains fully supported by the PGXN toolchain. + Name ==== diff --git a/text/0003-meta-spec-v2.md b/text/0003-meta-spec-v2.md new file mode 100644 index 0000000..460f4bf --- /dev/null +++ b/text/0003-meta-spec-v2.md @@ -0,0 +1,1208 @@ +{{#title PGXN RFC–3 — Meta Spec v2}} +* **RFC:** 3 +* **Title:** PGXN Meta Spec v2 +* **Slug:** `meta-spec-v2` +* **Start Date:** 2024-07-11 +* **Status:** Proposed Standard +* **Category:** Packaging +* **Supersedes:** [RFC--1](0001-meta-spec-v1.md "Meta Spec v1") +* **Pull Request:** [pgxn/rfcs#3](https://github.com/pgxn/rfcs/pull/3) +* **Implementation Issue:** TBD + +# RFC--3 --- Meta Spec v2 + +> **Note:** This RFC Supersedes but does not replace [PGXN Meta Spec +> v1](0001-meta-spec-v1.md). + +## Abstract + +This document describes version 2.0.0 of the [PGXN] source distribution +metadata specification, also known as the "PGXN Meta Spec." PGXN metadata +ships with PGXN source distribution archives, and serves to describe the their +contents for the benefit of automated indexing, distribution, discovery, +full-text search, binary packaging, and more. + +## Introduction + +Distribution metadata describe important properties of source code [archive +files][archive file] distributed on the PostgreSQL Extension Network, or +[PGXN]. Tools that build PGXN source distribution archives must create a +metadata file in accordance with this specification and include it in the +distribution archive for use by automated tools that index, examine, package, +or install PGXN source distributions. + +## Guide-level explanation + +TODO. + + + +### Terminology ### + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", +"SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be +interpreted as described in [IETF RFC 2119]. + +This RFC makes use of the following additional terms: + +#### Package #### + +A collection of extensions that are released, versioned, and distributed +together. + +#### Source Distribution #### + +An [archive file] of the source code for the release of a [Package](#package), +together with metadata defined by this spec, distributed for other developers +to build, install, and use. The file name of a Source Distribution consists of +the package name, a dash, and the version, e.g., `pgtap-1.14.3.zip`. + +Usually referred to as a "Distribution", including in this document. The full +term "Source Distribution" is used where necessary to distinguish from binary +distributions of a [Package](#package) (not covered by this RFC). + +#### Extension #### + +A software component that extends the capabilities of a PostgreSQL database or +cluster. Extensions **MAY** be `CREATE EXTENSION` [extensions], [background +workers], command-line apps, [loadable modules], shared libraries, and more. + +#### Consumer #### + +Code that reads a metadata file, deserializes it into a data structure in +memory, or interprets its elements. + +#### Producer #### + +Code that constructs a metadata data structure, serializes it into a byte +stream, and/or writes it to disk. + +## Reference-level explanation + +### Data Types ### + +Properties in the [Structure](#structure) section describe data elements, each +of which has an associated data type as described herein. Each is based on the +primitive types defined by [JSON]: *object*, *array*, *string*, *number*, and +*boolean*. Other types are subtypes of these primitives and define compound +data structures or define constraints on the values of a data element. + +#### Boolean #### + +A *Boolean* is used to provide a true or false value. It **MUST** be +represented as a defined (not `null`) value. + +#### String #### + +A *String* is data element containing a non-zero length sequence of Unicode +characters. + +#### Array #### + +An *Array* is an ordered collection of one or more data elements. Items in an +Array **MAY** be of mixed types. + +#### Object #### + +An **Object** is an unordered set of key/value pairs, or "properties". +Property values are indexed by their corresponding [String](#string) keys. An +Object's values **MAY** be of mixed types. + +#### Term #### + +A *Term* is a [String](#string) that **MUST** be at least two characters long, +and contain no slash (`/`), backslash (`\`), control, or space characters. + +#### Tag #### + +A *Tag* is a [String](#string) that **MUST** be at least two and no more than +255 characters long, and contain no slash (`/`), backslash (`\`), or control +characters. + +#### URI #### + +*URI* is a [String](#string) containing a valid Uniform Resource Identifier or +Locator as defined by [IETF RFC 3986]. + +#### Path #### + +*Path* is a [String](#string) with a relative file path that identifies a file +in the [Distribution](#source-distribution). The path **MUST** be specified +with unix conventions. + +#### Version #### + +A *Version* is a [String](#string) containing a value that describes the +version number of extensions or distributions, and adhere to the format of the +[Semantic Versioning 2.0.0 Specification][semver] with the exception of [Build +metadata], which is reserved for use by downstream packaging systems. + +#### Version Range #### + +A *Version Range* is a [String](#string) that describes a range of +[Versions](#version) that **MAY** be present or installed to fulfill +dependencies. + +The simplest format for a Version Range is just the [Versions](#version) +itself, e.g. `2.4.2`. This means that **at least** version 2.4.2 must be +present. Versions may also be truncated to their major or minor parts, as +appropriate. For example, `2.4` means that **at least** version 2.4.0 must be +present. + +Alternatively, a version range **may** use the operators `<` (less than), `<=` +(less than or equal), `>` (greater than), `>=` (greater than or equal), `==` +(equal), and `!=` (not equal). For example, the specification `< 2` means that +any version less than version 2 is suitable. + +For more complicated situations, version specifications **may** be AND-ed +together using commas. The specification `>= 1.2, != 1.5.2, < 2.0` +indicates a version that must be **at least** 1.2, **less than** 2.0, and +**not equal to** 1.5.2. + +#### License String #### + +A *License String* is a [String](#string) identifying a licenses. Its values +are restricted to and defined by the [SPDX License List]. Examples: + +* `PostgreSQL` +* `MIT` +* `Apache-2.0` +* `BSD-2-Clause` +* `GPL-2.0-only` + +#### License Expression #### + +A *License Expression* is a [String](#string) that represents one or more +licenses by combining [License Strings](#license-string) into a single value. +The format is defined by [SPDX Standard License Expression]. Examples: + +* `LGPL-2.1-only OR MIT` +* `LGPL-2.1-only AND MIT AND BSD-2-Clause` +* `GPL-2.0-or-later WITH Bison-exception-2.2` +* `LGPL-2.1-only OR BSD-3-Clause AND MIT` + +#### purl #### + +A *purl* is a [String](#string) containing a valid package in the format +defined by the [purl spec]. All known [purl Types] **MAY** be used, as well as +`pgxn` for PGXN packages and `postgres` for PostgreSQL core [contrib] or +development packages. Some examples: + +* `pkg:pgxn/pgtap` +* `pkg:postgres:pg_regress` +* `pkg:generic/python3@3` +* `pkg:pypi/pyarrow@11.0.0` + +#### Platform #### + +A *Platform* is a [String](#string) that identifies a computing platform as a +one to three dash-delimited substrings: An OS name, the OS version, and the +architecture: `$os-$version-$architecture`. + +If the string contains no dash, it represents only the OS. If it contains a +single dash, the values represent the OS and the architecture. The complete +list of values will be determined by the [bulid farm animals]. Some likely +Examples: + +* `linux`: Any Linux +* `linux-amd64`: Any Linux on amd64/x86_64 +* `gnulinux-amd64`: [GNU] Linux on amd64/x86_64 +* `musllinux-1.2-arm64`: [musl] Linux v1.2 on arm64/aarch64 +* `darwin`: Any Darwin (macOS) +* `darwin-23.5.0-arm64`: Darwin (macOS) 23.5.0 on arm64/aarch64 + +### Structure + +The metadata structure is an [Object](#object). This section describes valid +properties of the [Object](#object). + +Any properties not described in this specification document (whether top-level +or within [Objects](#object) described herein) are considered *custom +properties* and **MUST** begin with an "x" or "X" and be followed by an +underscore; i.e., they **MUST** match the regular expression pattern +`^[xX]_.`. If a custom property refers to an [Object](#object), properties +within it do not need an "x_" or "X_" prefix. + +Metadata [Consumers](#consumer) **MAY** ignore any or all custom properties. +All other properties not described herein are invalid and **SHOULD** be +ignored by [Consumers](#consumer). [Producers](#producer) **MUST NOT** +generate or output invalid properties. + +For each property, one or more examples are provided followed by a +description. The description begins with the version of spec in which the +property was added or in which the definition was modified, whether the +property is **REQUIRED** or **OPTIONAL**, and the data type of the value. +These items are in parentheses, brackets, and braces, respectively. + +If a data type is an [Object](#object), valid sub-properties will be described +as well. + +All examples are represented as [JSON]. + + + +#### Required Properties + +##### abstract ##### + +``` json +#{ + "abstract": "Unit testing for PostgreSQL" +#} +``` + +(Spec 1) [required] {[String](#string)} + +This is a short description of the purpose of the [Package](#package) provided +by the [Distribution](#source-distribution). + +##### maintainers ##### + +```json +#{ + "maintainers": [ + { + "name": "David E. Wheeler", + "url": "https://pgxn.org/user/theory" + } + ] +#} +``` + +```json +#{ + "maintainers": [ + { + "name": "David E. Wheeler", + "email": "theory@pgxn.org", + "url": "https://pgxn.org/user/theory" + }, + { + "name": "Josh Berkus", + "email": "jberkus@pgxn.org" + } + ] +#} +``` + +(Spec 2) [required] {[Array](#array) of [Objects]{#object}} + +This property indicates the person(s) to contact concerning the +[Distribution](#source-distribution). Each [Object](#object) in the +[Array](#array) consists of the following properties: + +* **name**: The name of the maintainer. **REQUIRED**. +* **email**: The email address of the maintainer. +* **url**: The URL for the maintainer. + +Either `email` or `url` or both **MUST** be present. + +This property provides a general contact list independent of other structured +fields provided within the [resources](#resources) field, such as `issues`. +The addressee(s) can be contacted for any purpose including but not limited +to: (security) problems with the [Distribution](#source-distribution), +questions about the [Distribution](#source-distribution), or bugs in the +[Distribution](#source-distribution). + +A [Distribution](#source-distribution)'s original author is usually the +contact listed within this field. Co-maintainers, successor maintainers, or +mailing lists devoted to the distribution **MAY** also be listed in addition +to or instead of the original author. + +##### license ##### + +``` json +#{ + "license": "MIT" +#} +``` + +``` json +#{ + "license": "MIT AND BSD-2-Clause" +#} +``` + +(Spec 1) [required] {[License String](#license-string) or [License Expression](#license-expression)} + +One or more licenses that apply to some or all of the files in the +[Distribution](#source-distribution). For [License +Expressions](#license-expression), the [Distribution](#source-distribution) +documentation **SHOULD** be consulted to clarify the interpretation of +multiple licenses. + +##### contents ###### + +``` json +#{ + "contents": { + "extensions": { + "pair": { + "sql": "sql/pair.sql", + "doc": "doc/pair.md", + "abstract": "A key/value pair data type", + "preload": "session", + "tle": true, + "control": "pair.control" + } + } + } +#} +``` + +``` json +#{ + "contents": { + "workers": { + "pair_pruner": { + "bin": "bin/pair_pruner", + "doc": "doc/pair_pruner.md", + "abstract": "A worker to periodically prune pairs" + } + }, + "modules": { + "lib_pair": { + "lib": "lib/lib_pair", + "doc": "doc/lib_pair.md", + "abstract": "A library hooking function calls to convert pairs to named parameters", + "load": "shared_preload_libraries" + } + } + } +#} +``` + +``` json +#{ + "contents": { + "apps": { + "pair_rand": { + "bin": "bin/pair_rand", + "doc": "doc/pair_rand.md", + "abstract": "Command to generate random pairs of strings" + } + }, + "libraries": { + "ruby_pair": { + "dir": "lib/gems", + "abstract": "Ruby libraries required to run the extension" + } + } + } +#} +``` + +(Spec 1) [required] {[Object](#object) of [Objects](#object) of [Terms](#term)} + +A description of what's included in the [Package](#package) provided by the +[Distribution](#source-distribution). This information is used by [PGXN] to +build indexes identifying in which [Package](#package) various +[Extensions](#extension) can be found. + +The properties of `contents` identify the types of [Extensions](#extension) in +the [Distribution](#source-distribution). At least one property **MUST** be +present in the `contents` object. The properties are as follows: + +* **extensions**: [Object](#object) describing `CREATE EXTENSION` + [extensions]. Properties are extension name [Terms](#term) pointing to + [Objects](#object) with the following fields: + * **sql**: A [Path](#path) pointing to the SQL file used by `CREATE + EXTENSION`. **REQUIRED**. + * **control**: A [Path](#path) pointing to the [control file] used by + `CREATE EXTENSION`. **REQUIRED**. + * **doc**: A [Path](#path) pointing to the main documentation file for + the extension, which **SHOULD** be more than a README. + * **abstract**: A [String](#string) containing a short description of + the extension. + * **tle**: A [Boolean](#boolean) that, when `true`, indicates that the + extension can be used as a [trusted language extension]. +* **workers**: [Object](#object) describing [background workers]. Properties + are worker name [Terms](#term) pointing to [Objects](#object) with the + following properties: + * **src**: A [Path](#path) pointing to the main source file for the + background worker. **REQUIRED**. + * **doc**: A [Path](#path) pointing to the main documentation file for + the background worker, which **SHOULD** be more than a README. + * **abstract**: A [String](#string) containing a short description of + the background worker. +* **apps**: [Objects](#object) describing applications, command-line or + otherwise. Properties are are app name [Terms](#term) pointing to + [Objects](#object) with the following properties: + * **src**: A [Path](#path) pointing to the main source file for the app. + **REQUIRED**. + * **doc**: A [Path](#path) pointing to the main documentation file for + the app, which **SHOULD** be more than a README. + * **abstract**: A [String](#string) containing a short description of + the app. +* **modules**: [Objects](#object) describing [loadable modules] that can be + loaded into Postgres. Properties are module name [Terms](#term) pointing + to [Objects](#object) with the following properties: + * **src**: A [Path](#path) pointing to the main source file for the + module. **REQUIRED**. + * **doc**: A [Path](#path) pointing to the main documentation file for + the module, which **SHOULD** be more than a README. + * **abstract**: A [String](#string) containing a short description of + the module. + * **preload**: A [String](#string) that indicates that the extension's + libraries **MAY** be loaded in advance. Its three possible values are: + `shared`, `session`, and `local`. Extensions that require early or + late loading of their module **MAY** optionally append a space and + then either `early` or `late`. +* **libraries**: [Objects](#object) listing other libraries that **MAY** + ship in the package and need to be installed but are not [loadable + modules], such as a dynamic library used by an app. Properties are library + name [Terms](#term) pointing to [Objects](#object) with the following + properties: + * **src**: A [Path](#path) pointing to the main source file or directory + of files for the library. **REQUIRED**. + * **doc**: A [Path](#path) pointing to the main documentation file for + the library, which **SHOULD** be more than a README. + * **abstract**: A [String](#string) containing a short description of + the app. + +##### meta-spec ##### + +``` json +#{ + "meta-spec": { + "version": "2.0.0", + "url": "https://pgxn.org/meta/v2/spec.txt" + } +#} +``` + +(Spec 1) [required] {[Map](#Map)} + +This field indicates the [Version](#Version) of the PGXN Meta Spec that +**SHOULD** be used to interpret the metadata. Consumers **MUST** check this +key as soon as possible and abort further metadata processing if the meta-spec +[Version](#version) is not supported by the consumer. + +The following properties are valid, but only `version` is **REQUIRED**. + +* **version**: The [Version](#version) of the PGXN Meta Spec against which + the metadata object was generated. + +* **url**: The [URI](#uri) of the metadata specification corresponding to + the given version. This is strictly for human-consumption and **SHOULD + NOT** impact the interpretation of the metadata structure. + +##### name ##### + +Example: + +``` json +#{ + "name": "pgTAP" +#} +``` + +(Spec 1) [required] {[Term](#Term)} + +This property is the name of the [Package](#package) provided by the +[Distribution](#source-distribution). This is usually the same as the name of +the "main extension" in the [contents](#contents) of the [Package](#package), +but **MAY** be completely unrelated. This value will be used in the +[Distribution](#source-distribution) file name on [PGXN]. + +##### version ##### + +``` json +#{ + "version": "1.3.6" +#} +``` + +(Spec 1) [required] {[Version](#version)} + +This property gives the version of the [Distribution](#source-distribution) to +which the metadata structure refers. Its value **MUST** be a +[Version](#version). + +All of the items listed in [contents](#contents) will be considered to have +this version; any references they make to a version, such as the [control +file], **SHOULD** be compatible with this version. + +#### Optional Fields #### + +##### description ##### + +``` json +#{ + "description": "pgTAP is a suite of database functions that make it easy to write TAP-emitting unit tests in psql scripts or xUnit-style test functions." +#} +``` + +(Spec 1) [optional] {[String](#string)} + +A longer, more complete description of the purpose or intended use of the +[Package](#package) provided by the [Distribution](#source-distribution), +answering the question "what is this thing and what value is it?" + +##### generated_by ##### + +Example: + +``` json +#{ + "generated_by": "Module::Build::PGXN version 0.42" +#} +``` + +(Spec 1) [optional] {[String](#string)} + +This property indicates the tool that was used to create this metadata. There +are no defined semantics for this property, but it is traditional to use a +string in the form "Software package version 1.23", or the maintainer's name +if the metadata was generated by hand. + +##### classification ##### + +``` json +#{ + "classifications": { + "tags": [ + "testing", + "pair", + "parameter" + ], + "categories": [ + "Machine Learning" + ] + } +#} +``` + +(Spec 2) [optional] {[Object](#object) of [Arrays](#array) of [Tags](#tag)} + +Classification metadata associates additional information about the +[Package](#package) provided by the [Distribution](#source-distribution) to +improve discovery. This [Object](#object) **MUST** contain at least one of the +following properties: + +* **tags**: An [Array](#array) of one or more keyword [Tags](#tag)s that + describe the distribution. +* **categories**: An [Array](#array) of at least one and no more than three + of the following [Strings](#string) that categorize the distribution: + * Analytics + * Auditing and Logging + * Change Data Capture + * Connectors + * Data and Transformations + * Debugging + * Index and Table Optimizations + * Machine Learning + * Metrics + * Orchestration + * Procedural Languages + * Query Optimizations + * Search + * Security + * Tooling and Admin + +#### ignore #### + +``` json +#{ + "ignore": [ + "/src/private", + "/src/file.sql", + "*.html" + ] +#} +``` + +(Spec 2) [optional] {[Array](#Array) of [Strings](#string)} + +This [Array](#array) describes any files or directories that are private to +the [Distribution](#source-distribution) and **SHOULD** be ignored by indexing +or search tools. Values are [Paths](#path) or [Strings](#string) based on a +subset of the [gitignore format]. + +##### dependencies ##### + +``` json +#{ + "dependencies": { + "postgres": { + "version": "14" + }, + } +#} +``` + +``` json +#{ + "dependencies": { + "postgres": { + "version": ">= 12, < 17", + "with": [ "xml", "uuid", "perl" ] + }, + "pipeline": "pgxs", + "packages": { + "build": { + "requires": { + "pkg:generic/awk": 0, + "pkg:generic/perl": "5.20" + }, + "recommends": { + "pkg:generic/jq": 0, + "pkg:generic/perl": "5.40" + } + } + } + } +#} +``` + +``` json +#{ + "dependencies": { + "pipeline": "pgrx", + "platforms": [ + "linux-amd64", + "linux-amd64v3", + "gnulinux-arm64", + "musllinux-amd64", + "darwin-23.5.0-arm64" + ], + "packages": { + "configure": { + "requires": { "pkg:cargo/cargo-pgrx": "=0.11.4" } + }, + "test": { + "requires": { + "pkg:postgres/pg_regress": 0, + "pkg:postgres/plpgsql": 0, + "pkg:pgxn/pgtap": "1.1.0" + } + }, + "run": { + "requires": { + "pkg:postgres/plperl": 0, + "pkg:pgxn/hostname": 0 + } + } + } + } +#} +``` + +``` json +#{ + "dependencies": { + "postgres": { + "version": ">= 15, < 16" + }, + "pipeline": "pgxs", + "platforms": [ + "linux-amd64", "linux-arm64", + "darwin-amd64", "darwin-arm64" + ], + "packages": { + "configure": { + "requires": { + "pkg:cargo/cargo-pgrx": "=0.11.4", + "pkg:generic/bison": 0, + "pkg:generic/cmake": 0, + "pkg:generic/flex": 0, + "pkg:generic/readline": 0, + "pkg:generic/openssl": 0, + "pkg:generic/pkg-config": 0 + } + }, + "run": { + "requires": { + "pkg:generic/penblas": 0, + "pkg:generic/python3": 0, + "pkg:generic/readline": 0, + "pkg:generic/openssl": 0, + "pkg:generic/bison" + }, + "recommends": { + "pkg:pypi/pyarrow": "11.0.0", + "pkg:pypi/catboost": 0, + "pkg:pypi/lightgbm": 0, + "pkg:pypi/torch": 0, + "pkg:pypi/langchain": 0 + } + } + }, + "variations": [ + { + "where": { + "platforms": ["linux"] + }, + "dependencies": { + "packages": { + "run": { + "recommends": { + "pkg:pypi/auto-gptq": 0, + "pkg:pypi/xformers": 0 + } + } + } + } + } + ] + } +#} +``` + +(Spec 2) [optional] {[Object](#object)} + +This property identifies dependencies required to configure, build, test, +install, and run the [Package](#package) provided by the +[Distribution](#source-distribution), expressed as [purls](#purl). These +include not only PGXN packages, but also external libraries, system +dependencies, and versions of PostgreSQL --- as well as any OS and +architectures ([arm64], [amd64], etc.). + +[Consumers](#consumer) **SHOULD** use this data to determine what dependencies +to install. + +Properties: + +* **platforms**: An [Array](#array) of one or more [Platform](#platform) + strings that identify OSes and architectures supported by the + [Package](#package) provided by the [Distribution](#source-distribution). + If this property is not present, [Consumers](#consumer) **SHOULD** assume + that the [Package](#package) supports any platform that PostgreSQL + supports. This property is typically needed only when the + [Package](#package) depends on platform-specific features. + +* **postgres**: An [Object](#object) describing the versions of PostgreSQL + required by the [Package](#package) provided by the + [Distribution](#source-distribution). The object supports the following + properties: + * **version**: A [Version Range](#version-range) identifying the + supported versions of PostgreSQL. **REQUIRED**. + * **with**: An [Array](#array) of [Terms](#term) that correspond + features that are required to be compiled into PostgreSQL. Each + corresponds to the appropriate `--with` [configure flags]. + **OPTIONAL**. + +* **pipeline**: A [Term](#term) identifying the build pipeline required to + configure, build, test, and install the [Package](#package) provided by + the [Distribution](#source-distribution). Supported values + **MAY** include: + + * pgxs + * meson + * pgrx + * autoconf + * gem + * cpan + * pip + * go + * rust + + If this field is not present, [Consumers](#consumer) **MAY** use + heuristics to ascertain the pipeline to use, such as the presence or + absence of a `configure.sh`, `Makefile`, or `Cargo.toml` file. + +* **packages**: An [Object](#object) defining dependencies required for + different phases of the build process. The supported property names are + `configure`, `build`, `test`, `run`, and `develop`. Values are + [Objects](#object) with at least one of the properties `requires`, + `recommends`, `suggests`, and `conflicts` pointing to [Objects](#object) + with [purls](#purl) property keys pointing to [Version + Range](#version-range) values. + + See the [Package Spec](#packages-spec) for the full definition of this + property. + +* **variations**: An [Array](#array) of [Object](#object)s that define + dependency variations. Each object contains two properties: + * **where**: An [Object](#object) containing the subset of the + [dependencies](#dependencies) to identify a variation, such as + `{ "platforms": ["gnulinux-arm64", "gnulinux-amd64"] }` for Linux + configurations, or `{"postgres": { "version": ">= 16, < 17" }}` for + PostgreSQL versions. **MUST NOT** include a `variations` property. + * **dependencies**: An [Object](#object) containing the subset of + [dependencies](#dependencies) required for the `where` property's + configuration. **MUST NOT** include a `variations` property. + +##### resources ##### + +``` json +#{ + "resources": { + "homepage": "https://pair.example.com", + "issues": "https://github.com/example/pair/issues", + "docs": "https://pair.example.com/docs", + "support": "https://github.com/example/pair/discussions", + "repository": "https://github.com/example/pair", + "badges": [ + { + "alt": "Test Status", + "src": "https://test.packages.postgresql.org/github.com/example/pair.svg" + } + ] + } +#} +``` + +(Spec 2) [optional] {[Object](#object)} + +This property provides external information about the [Package](#package) +provided by the [Distribution](#source-distribution). [Consumers](#consumer) +**MAY** use this data for links and displaying useful information about the +package. + +The `resources` object **MUST** contain at least one of the following +properties: + +* **homepage**: [URI](#uri) for the official home of the project on the web. +* **issues**: [URI](#uri) for the package's issue tracking system. +* **repository**: [URI](#uri) for the package's source code repository. +* **docs**: [URI](#uri) for the package's documentation. +* **support**: [URI](#uri) for support resources and contacts for the + package. +* **badges**: An [Array](#array) of [Objects](#object) linking to badge + images that **SHOULD** follow the [Shields badge specification]. It + **MUST** have at least one entry, and all entries require two properties: + * **src**: The [URI](#uri) for the badge. + * **alt**: Alt text for the badge. + +##### artifacts ##### + +``` json +#{ + "artifacts": [ + { + "type": "source", + "url": "https://github.com/theory/pg-pair/releases/download/v1.1.0/pair-1.1.0.zip", + "sha256": "2b9d2416096d2930be51e5332b70bcd97846947777a93e4a3d65fe1b5fd7b004" + }, + { + "type": "binary", + "url": "https://github.com/theory/pg-pair/releases/download/v1.1.0/pair-1.1.0-linux-amd64.tar.gz", + "sha1": "12d9bc5cfb6bc3c453627eac69511f48be63cfc0" + }, + { + "type": "binary", + "url": "https://github.com/theory/pg-pair/releases/download/v1.1.0/pair-1.1.0-linux-arm64.tar.gz", + "sha1": "787dc39137f7d1510a33ab0a1b8905cd5f3f72d1" + } + ] +#} +``` + +(Spec 2) [optional] {[Array](#array)} + +An [Array](#array) of [Objects](#objects) describing links and checksums for +downloading the [Package](#package) provided by the +[Distribution](#source-distribution) in one or more formats, including source +code, binaries, system packages, and more. [Consumers](#consumer) my use this +information to determine the best option for installing an extension on a +particular system. Useful for projects that publish their own binaries, such +as in GitHub releases. + +The [Array](#array) **MUST** have at least one [Object](#object). The +properties of each [Object](#object) are: + +* **url**: A [URI](#uri) to download the artifact. **REQUIRED**. +* **sha256** or **sha512**: A [String](#string) containing a SHA-256 or + SHA-512 checksum in hex format. **REQUIRED**. +* **type**: The type of artifact. **MUST** be a single lowercase word + describing the artifact, such as none of `binary`, `source`, `rpm`, + `homebrew`, etc. **REQUIRED**. +* **platform**: A [Platform](#platform) string identifying the platform the + artifact was built for. **RECOMMENDED** for packages compiled for a + specific platform, such as a C [Extension](#extension) compiled for + `linux-arm64`. + +Each URL **MUST** properly resolve and the checksum **MUST** match. + +### Packages Spec ### + +The `packages` sub-property of the [dependencies](#dependencies) property +defines the relationship between a [Distribution](#source-distribution) and +external dependencies --- including other PGXN [Packages](#package), system +packages, and third-party packages --- expressed as [purls](#purl). The +structure is an [Object](#object) that specifies package dependencies into +*Phases* of activity in the installation process, and *Relationships* that +indicate how dependencies **SHOULD** be resolved. + +For example, to specify that the [PGXN] extension `pgtap` is required during +the `test` phase, this entry would appear in the +[Distribution](#source-distribution) metadata: + +``` json +#{ + "packages": { + "test": { + "requires": { "pkg:pgxn/pgtap": 0 } + } + } +#} +``` + +All known [purl Types] **SHOULD** be used to identify dependencies. +[Producers](#producer) **MAY** specify dependencies of two additional types as +appropriate: + +* **`pkg:pgxn`**: Packages distributed via [PGXN]. These **MUST** include + package name, e.g., `pkg:pgxn/pair`. +* **`pkg:postgres`**: Dependencies distributed as part of the PostgreSQL + core, including [contrib] or development packages such as [auto_explain], + [dblink], [pg_regress], and [pg_isolation_regress]. Example: + `pkg:postgres/dblink`. + +Versions **SHOULD** not be included in [purls](#purl), but in the [Version +Range](#version-range) values the [purls](#purl) properties point to. For +example, this specification requires the `pair` PGXN package version 1.2.0 or +greater but not 1.5.2: + + +``` json +#{ + "packages": { + "test": { + "requires": { "pkg:pgxn/pair": ">= 1.2, != 1.5.2" } + } + } +#} +``` + +[Producers](#producer) **SHOULD** avoid OS-specific [purls](#purl) such as +`pkg:rpm:/libreadline-dev` unless the [Distribution](#source-distribution) +supports only OSes that provide such packages. See the "variations" property +of the [dependencies](#dependencies) object for platform-specific dependency +specification. + +[Consumers](#consumer) **SHOULD** use [Repology] to resolve `pkg:generic` +[purls](#purl) to packages specific to the platform on which an extension is +being built. This is useful for specifying system dependencies that vary by +name and packaging system. Otherwise, they **MAY** use whatever techniques or +heuristics are appropriate to install dependencies. + +#### Phases #### + +Requirements for regular use **MUST** be listed in the `runtime` phase. Other +requirements **SHOULD** be listed in the earliest stage in which they are +required, and [Consumers](#consumer) **MUST** accumulate and satisfy +requirements across phases before executing the action. For example, `build` +requirements **MUST** also be available during the `test` phase. + + before action | requirements that **MUST** be met +----------------|----------------------------------- + ./configure | configure + make | configure, runtime, build + make test | configure, runtime, build, test + +Consumers that install the [Package](#package) provided by the +[Distribution](#source-distribution) **MUST** ensure that *runtime* +requirements are also installed and **MAY** install dependencies from other +phases. + + after action | requirements that **MUST** be met +----------------|----------------------------------- + make install | runtime + +* **configure**: The configure phase occurs before any dynamic configuration + has been attempted. Dependencies required by the configure phase **MUST** + be available for use before the build tool has been executed. + +* **build**: The build phase is when the + [Distribution](#source-distribution)'s source code is compiled (if + necessary) and otherwise made ready for installation. + +* **test**: The test phase is when the + [Distribution](#source-distribution)'s automated test suite is run. Any + dependency needed only for testing and not for subsequent use **SHOULD** + be listed here. + +* **runtime**: The runtime phase refers not only to when the contents of the + [Package](#package) provided by the [Distribution](#source-distribution) + are installed, but also to its continued use. Any package that is a + dependency for regular use of this [Package](#package) **SHOULD** be + indicated here. + +* **develop**: The develop phase's packages are needed to work on the + [Package](#package)'s source code as its maintainer does. These tools + might be needed to build a release archive, to run maintainer-only tests, + or to perform other tasks related to developing new versions of the + [Package](#package). + +#### Relationships #### + +* **requires**: These dependencies **MUST** be installed for proper + completion of the phase. + +* **recommends**: **RECOMMENDED** dependencies are *strongly* encouraged and + **SHOULD** be satisfied except in resource constrained environments. + +* **suggests**: These dependencies are **OPTIONAL**, are suggested for + enhanced operation of the described distribution, and **MAY** be + satisfied. + +* **conflicts**: These dependencies **MUST NOT** be installed when the phase + is in operation. This is a very rare situation, and the conflicts + relationship **SHOULD** be used with great caution, or not at all. + +### Serialization ### + +Distribution metadata **SHOULD** be serialized as JSON-encoded data and +packaged with distributions as the file `META.json`. + +## Drawbacks + +This design adds significant complexity over and above that provided by the +[v1 spec](0001-meta-spec-v1.md). The features and services planned for PGXN, +including automated binary packaging require that complexity, however. That +said, the [v1 spec](0001-meta-spec-v1.md) remains valid and can continue to be +used by projects unless they need additional metadata to support new features. + +We plan to develop an [SDK] and [CLI] to reduce or eliminate the toil of +maintaining the metadata, and to automate conversion from the [v1 +spec](0001-meta-spec-v1.md) when new features are needed. + +## Rationale and alternatives + +There are currently no developer-focused alternatives to the PGXN Meta Spec. +Other extension registries, such as [Trunk], [PGXMan], [StackBuilder], and +the community [Yum] and Apt repositories decouple maintenance of build +metadata and dependencies, in particular, from the developer. Instead they're +independently and manually managed by volunteers or employees of teh companies +who developed them. + +They also tend to have formats specific to the platforms they serve, rather +than to PostgreSQL specifically. + +Without this metadata, it would be difficult to develop a cross-platform, +community binary packaging registry of all PGXN distributions. With this +format, even if extension authors don't adopt the new format, PGXN maintainers +can develop PGXN Meta Spec v2 overlays to merge with developer [v1 +spec](0001-meta-spec-v1.md) files, both to assist with build automation and to +create pull requests for the original projects. + +## Prior art + +The PGXN Meta Spec v2 is based on the [v1 spec](0001-meta-spec-v1.md), which +is itself based on the [CPAN Meta Spec]. Dependency specification is inspired +by the [purl spec] and [Repology], as well as the [RPM Packaging Guidelines]. + +## Unresolved questions + +* Can we rely on [Repology] to resolve system dependencies? +* How, exactly, do we configure loadable modules, esp when used with an + extension? +* Is it better to bundle dependencies into a package as loadable modules, or + to create separate distributions that just contain the loadable modules + and depend on them? +* Should any of this be deferred? For example, maybe omit the `archives` + property. +* Do we need to continue to treat package names as globally unique, or can + we categorize under username? This has implications for integration into + existing PGXN services; perhaps it'd be better to keep uniqueness as it is + and look to change it at some future time? +* What controls do we need to encode into badges in the + [resources](#resources) object? Do we limit URL formats or domains to + prevent crapping things up? Does it matter? +* Can we get away with allowing only one version for the entire + distribution? This is a change from the [v1 spec](0001-meta-spec-v1.md), + which allowed independent versions for each extension included in a + distribution. Many people found the version variation confusing. Will + developers find it too painful to switch? Very few extensions on PGXN + currently include more than one extension, so hopefully not. + +## Future possibilities + +Successful adoption of this standard will mean most extensions use it to +distribute on PGXN. We'll separately need an initiative to work with extension +authors to onboard them into PGXN, perhaps by pull requests adding the +metadata and CI workflows. + +Or perhaps in the future we'll be able to recognize extensions on GitHub and +other repositories and automatically index them. Seems tricky without +[Go-style module paths]. + +## See Also ## + +* [CPAN Meta Spec] +* [PGXN] +* [JSON] +* [Semantic Versioning 2.0.0][semver] +* [Repology] +* [purl spec] + +## Contributors ## + +The PGXN Meta Spec was originally based on the [CPAN Meta Spec], which was +written by Ken Williams in 2003 and has since been updated by Randy Sims, +David Golden, Ricardo Signes, Adam Kennedy, and contributors. + + [PGXN]: https://pgxn.org "PostgreSQL Extension Network" + [source code repository]: https://github.org/pgxn/pgxn-meta-spec + [PostgreSQL License]: https://www.postgresql.org/about/licence/ + [CC BY-SA 4.0]: https://creativecommons.org/licenses/by-sa/4.0/ "Attribution-Sharealike 4.0 International" + [Github Flavored Markdown]: https://github.github.com/gfm/ + [Markdown]: https://daringfireball.net/projects/markdown/ + [master.pgxn.org/meta/spec.txt]: https://master.pgxn.org/meta/spec.txt + [pgxn.org/spec/]: https://pgxn.org/spec/ + [archive file]: https://en.wikipedia.org/wiki/Archive_file + "Wikipedia: “Archive file”" + [`semver`]: https://pgxn.org/dist/semver/ + [`vector`]: https://pgxn.org/dist/vector/ + [`citus`]: https://pgxn.org/dist/citus/ + [`CREATE EXTENSION` statement]: https://www.postgresql.org/docs/current/static/sql-createextension.html + [IETF RFC 2119]: https://www.ietf.org/rfc/rfc2119.txt + [JSON]: https://json.org/ + [IETF RFC 3986]: https://www.rfc-editor.org/info/rfc3986 + "RFC 3986: Uniform Resource Identifier (URI): Generic Syntax" + [purl spec]: https://github.com/package-url/purl-spec/blob/master/PURL-SPECIFICATION.rst + "package-url/purl-spec: A minimal specification a “mostly universal” package URL" + [percent-encoded]: https://github.com/package-url/purl-spec/blob/master/PURL-SPECIFICATION.rst#character-encoding + "Package URL specification: Character encoding" + [purl Types]: https://github.com/package-url/purl-spec/blob/master/PURL-TYPES.rst + "Package URL Type definitions" + [semver]: https://semver.org/ + [SPDX License List]: https://github.com/spdx/license-list-data/ + [SPDX Standard License Expression]: https://spdx.github.io/spdx-spec/v3.0/annexes/SPDX-license-expressions/ + [control file]: https://www.postgresql.org/docs/current/extend-extensions.html + [trusted language extension]: https://github.com/aws/pg_tle + "pg_tle: Framework for building trusted language extensions for PostgreSQL" + [background workers]: https://www.postgresql.org/docs/current/bgworker.html + "PostgreSQL Docs: Background Worker Processes" + [loadable modules]: https://www.postgresql.org/docs/16/gist-extensibility.html + [gitignore format]: https://git-scm.com/docs/gitignore + [bulid farm animals]: https://buildfarm.postgresql.org/cgi-bin/show_members.pl + [configure flags]: https://www.postgresql.org/docs/current/install-make.html#CONFIGURE-OPTIONS-FEATURES + [Repology API]: https://repology.org/api "Repology, the packaging hub: API" + [contrib]: https://www.postgresql.org/docs/current/contrib.html + [auto_explain]: https://www.postgresql.org/docs/current/auto-explain.html + [dblink]: https://www.postgresql.org/docs/current/dblink.html + [pg_regress]: https://github.com/postgres/postgres/tree/master/src/test/regress + [pg_isolation_regress]: https://github.com/postgres/postgres/tree/master/src/test/isolation + [Shields badge specification]: https://github.com/badges/shields/blob/master/spec/SPECIFICATION.md + [CPAN Meta Spec]: https://metacpan.org/pod/CPAN::Meta::Spec + [musl]: https://musl.libc.org/ + [GNU]: https://www.gnu.org/software/libc/ + [SDK]: https://en.wikipedia.org/wiki/Software_development_kit + "Wikipedia: Software development kit" + [CLI]: https://en.wikipedia.org/wiki/Command-line_interface + "Wikipedia: Command-line interface" + [Repology]: https://repology.org/ "Repology, the packaging hub" + [RPM Packaging Guidelines]: https://docs.fedoraproject.org/en-US/packaging-guidelines/ + [Go-style module paths]: https://go.dev/ref/mod#module-path