ModuleDraft 0.2, for comment

Access module (draft 0.2)

Status: draft for comment, with one implementation (Tangle's /omc/v1). Namespace: org.openmusiccollection.access. Needs: Core 1.0.

Access is the protocol apps use to read, add to and change a person's collection at its home, with that person's consent. It is what levels 1 (Read) and 2 (Write) of conformance are built on. It returns core records as JSON over HTTP, so everything in the core applies unchanged, above all the keep-unknown-data rule.

#Consent

OAuth 2.1: authorization code with PKCE (S256) for every client, rotating refresh tokens, RFC 8414 metadata at /.well-known/oauth-authorization-server, RFC 7009 revocation. Requests carry Authorization: Bearer <token>.

A host MAY offer more scopes for its own features. A home's consent screen says what the app asks for in these words, and, under a view, which lists and sources it is limited to ("Only: From Bandcamp email").

#Who the app is

An app is known by its client_id. It may be one the home gave it when the app registered, or, so that an app can connect to a home that has never heard of it, an https URL the app owns: an OAuth Client ID Metadata Document. The document at that URL is JSON with the app's client_name, logo_uri, client_uri and redirect_uris. A home SHOULD accept such a client_id: it fetches the document when the person is asked to consent, shows the name and the host it came from, and redirects only to the URIs listed there. /me says whether the home does ("client_id_metadata": true). Registered clients keep working.

#Apps that only add

library:add is for apps that bring music in and need nothing back: one that follows labels and adds their new releases, one that reads music links from the person's email. Such an app:

• adds items with POST {base}/items and puts them in a list (?list=, or POST {base}/lists/{id}/entries), for lists it created (POST {base}/lists, which needs no more than library:add) or that the person picked when connecting it;
• learns nothing it didn't send. Each addition answers with an id and one word: created, existing or held;
• can't read items, lists, sources, records, the collection file or the change feed (403 insufficient_scope), and gets no OMC-Cursor on its writes, which would count changes it can't see;
• can't probe for ids: adding an id the home doesn't know to its list answers as unchanged, like one already there, never 404.

The lists it fills are ordinary lists of the collection, which move home with it.

#Connections and views

Each app the person connects is a connection, kept by the home. The person decides what each connection can reach by giving it a view:

• What is in a view: the chosen lists, sources and record types, the items in those lists, and the items with a copy on those sources. A list the app creates joins its view.
• What an app sees: only its view. GET {base}/me says what it is: "view": { "lists": "all" | [...], "sources": "all" | [...], "records": "all" | [...], "write": true }. A collection file served under a view that isn't everything says so in its header, "ext": { "org.openmusiccollection.access": { "view": {...} } }, so a reader knows that records outside it were left out rather than deleted.
• Changes outside the view are refused with 403 out_of_view.
• An app that collects new releases might see only its own list, no sources, and the person's follows ("records": ["org.openmusiccollection.feeds/follow"]), so the labels it watches stay in the collection; a DJ app might see everything on the person's DJ library source, read only.

A connection's view, scopes and sync position are the home's state. None of it is in the collection.

#What an app can do with a list

Whether an app may add to, remove from, reorder, rename or delete a list depends on the platform the list lives on (a source may not allow reordering), what kind of list it is there, what the home itself can do, and the connection's scopes and view. So it is computed per app, when the app asks, and never stored. Every response that carries a list (GET /lists, GET /lists/{id}, POST /lists, PATCH /lists/{id}, the /entries requests) carries

"can": { "add": true, "remove": true, "reorder": false, "rename": true, "delete": false }

add and remove are adding and removing entries, reorder placing or moving an entry to a position rather than only adding at the end, rename and delete the list itself. A smart list answers add: false. can belongs to the response: it MUST NOT appear in collection files, and a host ignores it when an app sends a list back.

GET {base}/lists/can answers can for every list in the app's view in one request, { "lists": { "<list id>": { "add": true, … }, … } }, so an app can show which lists it may change as soon as it has read the collection file, which carries no can. When a list's can changes (a scope granted, a source disconnected), /changes reports the list as changed.

#Reading

{base} is the host's Access base URL, for example https://api.example.com/omc/v1.

Pages are { "data": [...], "next_cursor": "…" or null, "has_more": true | false }.

Lookups by content answer a page like ?q=, empty when nothing matches. They are how an app finds the home's record for something it knows from elsewhere, since ids are local.

/me modules lists the modules the home supports, by namespace and version, with Core as "core": "1.0". /me limits: data_bytes (the most the host keeps in one app's own namespace on one record), entries_per_request, ids_per_request, import_records, additions_per_hour (POST /items). Limits are the home's choice. Module data is outside data_bytes: each module says what a home that supports it must accept (the DJ module: at least 1 MB per copy).

#A list's version

Every change to a list's entries gives the list a new version, an opaque string. Access returns it with the list, as a member and in ETag, and it is never in collection files. An app names the version it read whenever it addresses entries by position, so the home can tell when the list has changed since.

#Changes

GET {base}/changes?cursor=… answers { "changes": [{ "type": "item", "id": "…", "op": "upsert" }], "cursor": "…", "has_more": false }. type is item, list, source or record (whose id is <type>/<id>); op is upsert or delete. Read the record again to see its new state. Without cursor, the answer is the current cursor and no changes.

An app keeps up by downloading /collection once, keeping its OMC-Cursor, and reading /changes from there. The cursor is the connection's, so it travels in the response, never in the collection file.

Who changed it. An entry carries by, the client_id of the app that made the change through Access; it is absent when the person or the home made it. An app uses it to skip its own changes. by describes the change, not the record: it is never written into records.

Only what the app can see. The feed lists only changes to records in the app's view.

A server that can't hold /events open can poll /changes. Webhooks, if a home offers them, are the home's own feature.

#Writing

#Entries by position

An ordered list may hold an item more than once, so its entries are addressed by position: the 0-based index in entries at the list's version. A request that names a position names the version it was read at. If the list has changed since, the home refuses with 409 version_conflict and the list's current version; the app reads the list again and retries. Every answer carries the new version. before equal to the number of entries means the end. Moving is one change, so it reaches a platform that has a move as a move.

Unordered lists hold an item at most once and are addressed by item id, without a version. An ordered list takes an item again only where its platform allows repeats; elsewhere the addition changes nothing and is reported in unchanged, like every idempotent write.

#Adding an item

POST {base}/items takes an item as in a collection file, without an id or with one of the app's own, and copies on sources by platform instead of by id when the app doesn't know the home's sources: { "platform": "bandcamp", "ref": "…", "url": "…", "ownership": "none" }. The home combines it with an item it has, or adds it:

{ "id": "itm_…", "result": "created" }

result is one of:

• created: the home added it;
• existing: the collection already had it, and id is that item;
• held: the item waits for the person to let it in (Held), and has no id until they do. The home keeps the item as the app sent it. new says whether it is newly held or was already waiting.

POST {base}/items?hold=true asks the home to hold the item even when it would add it straight away, for apps that bring in more than the person will want (new arrivals from a dozen shops). A home that holds nothing ignores it.

POST {base}/items?list=<id> also puts the item in that list (at the end), once it is in the collection; the answer's list says what happened: added, already (it was there), or not_added (a held item before the person lets it in, or a list that can't hold that kind). A home sets the item's via to the app's name when it creates it. An app that has library:read gets the item too, in item.

An item with only a link (a release a label just announced) is a copy with a url and ownership: none whenever the link is on a registered platform, or on a source that publishes a descriptor. If the person has no source for that platform, the home adds one, without account. Such a copy can be played, and matches exactly when the person later buys or saves the same release there. A link anywhere else (a record shop's page) has no id another collection shares, so it is not a copy: it goes in via ({ "name": "Hard Wax", "url": "…" }), and the item comes in with no copies. An item needs a link or identifiers to be added.

#Importing

POST {base}/import takes a piece of a collection file (application/x-ndjson, header optional) whose records refer to each other by the app's own ids. The home combines it into the collection by the core's rules and answers with what it did:

{ "matched":   [{ "type": "item", "id": "a7", "home_id": "itm_…", "by": "copy" }],
  "created":   [{ "type": "list", "id": "l1", "home_id": "lst_…" }],
  "suggested": [{ "type": "item", "id": "a9", "home_id": "itm_…", "candidates": ["itm_…"], "by": "title" }],
  "held":      [{ "type": "item", "id": "a8" }],
  "left":      [{ "type": "source", "id": "s3", "reason": "not connected to this home" }],
  "ids": { "a7": "itm_…", "l1": "lst_…", "a9": "itm_…" } }

by says how a record matched (id for the home's own records, account, copy, ref, musicbrainz, isrc, upc, title). A suggested match is created as a new record and reported with the records it may be the same as, so the person can merge them at the home. Held records wait for the person to accept them (as with POST /items), and left ones the home couldn't take, each with a reason (a source it has to connect itself, data over its limits): nothing is dropped without saying so. ?dry_run=true answers the same without changing anything, so an app can show the person what an import would do. ?conflicts=home (the default) keeps the home's values where both sides have one; ?conflicts=import takes the piece's. ids maps the app's ids to the home's, for the app to use from then on. Lists in the piece need lists:write.

#How PATCH works

• A PATCH body is an RFC 7396 merge patch: members replace those stored; null removes one; objects (including ext, per namespace) merge member by member; arrays are replaced whole.
• The host's own fields (the core fields it keeps itself, such as title) can't be changed this way: they MUST be left out or sent unchanged. A changed value is refused with 422 field_not_writable, naming the field in param. So an app can always send back a whole record it read.
• Dates the host manages (added_at, created_at) and response members (can) are ignored on write.
• Everything else is the app's data: ext namespaces, ids, provenance, via, fields the host doesn't know, and data on copies. The host keeps it and gives it back unchanged.
• The host's own namespace in ext is the host's: what an app sends there is ignored, so a record read earlier can be sent back whole even after the host's data in it has changed.
• Copies are matched by source and ref. Sending copies replaces the app's copy data; a host whose copies follow a platform MAY refuse to drop one (422 field_not_writable), and MAY refuse a copy on a source it can't hold (422 unknown_source).
• The person's own values. A host SHOULD let an app write rating as the person's own rating, and MAY let it write another core field the person corrected (a copy's ownership), recorded with provenance origin user. Values an app measured or looked up go on the copy they came from, in the copy's ext.
• List entries change only through /entries; a PATCH that changes entries is refused.
• Data over the host's data_bytes limit for the app's own namespace, or over a module's limit, is refused with 413 data_too_large, never cut.

#Who may write a namespace

Everyone may read every namespace. Who may write one is a rule about apps, and only through Access. The person's own edits (in the home, by file or by folder) are never restricted, and a file or folder has no write rules at all. Through Access, a home MAY stop one app writing another's namespace, by these rules:

A write refused this way answers 403 namespace_not_yours, naming the namespace in param.

Writes are idempotent: adding an entry that is already in an unordered list (or in an ordered list whose platform doesn't allow repeats), or removing one that isn't, changes nothing and is reported in unchanged.

Every successful write answers with an OMC-Cursor header: the change-feed position just after the write. An app that reads /changes from there sees its own write already applied.

#Errors

{ "error": { "code": "…", "message": "…", "request_id": "…", "param": "…" } }. Codes include missing_token, invalid_token, insufficient_scope, rate_limited, not_found variants (item_not_found, list_not_found, source_not_found, record_not_found), field_not_writable, unknown_source, data_too_large, not_supported, invalid_cursor, out_of_view, missing_url (an item needs a link or identifiers), too_many_records (an import over import_records), version_conflict (409: the list changed since the version the request names), namespace_not_yours (403).

#Listens

A home that keeps listens takes them at POST {base}/listens (scope listens:write), in the body ListenBrainz's submit-listens takes, so a player that scrobbles to ListenBrainz needs no new code:

{ "listen_type": "import",
  "payload": [{ "listened_at": 1793001600,
                "track_metadata": { "artist_name": "Mira Holt", "track_name": "Night Drive",
                                    "additional_info": { "isrc": "QZES72600001",
                                                         "spotify_id": "https://open.spotify.com/track/0exampleTrackId000001",
                                                         "submission_client": "Example Player" } } }] }

listen_type is single or import (a batch); playing_now is accepted and not stored. The home matches each listen to an item by its identifiers and links, as it combines, and updates that item's play_count and last_played_at, which /changes then reports. The app that sent a listen is known from its token. GET {base}/listens answers ListenBrainz's own shape ({ "payload": { "listens": [...] } }), and a home's export of the history is the same. Listens are never in the collection file.

#Checking a host

The reference validator checks a live host: OMC_TOKEN=… omc-validate access https://api.example.com/omc/v1. It validates /me and the collection file, checks that the file carries no cursor, can or version and the response does, reads /lists/can and one list's version, writes probe data onto one item, checks the next export gives it back unchanged, and removes it again.

#Open questions

1. Should /changes say how a list changed (entries added and removed), saving a read per change?
2. Views by role or kind (only saved lists, only releases) as well as by list, source and record type?
3. Should a sender learn what the person did with what it sent to be held? Today it can tell only by seeing the item appear, if it can read the collection.
4. Device sign-in (RFC 8628) for apps without a web address. Left until one needs it; personal access tokens cover a person's own scripts.

What was decided, and when, is in the changelog.

#Prior art

The Tangle API, which this module generalises; JSPF for playlists; the Spotify Web API and the OpenSubsonic API for shapes developers already know.