Content Index

Editor’s Draft,

This version:
https://wicg.github.io/content-index/spec/
Issue Tracking:
GitHub
Inline In Spec
Editors:
(Google)
(Google)

Abstract

An API for websites to register their offline enabled content with the browser.

Status of this document

This specification was published by the Web Platform Incubator Community Group. It is not a W3C Standard nor is it on the W3C Standards Track. Please note that under the W3C Community Contributor License Agreement (CLA) there is a limited opt-out and other conditions apply. Learn more about W3C Community and Business Groups.

1. Introduction

High quality offline-enabled web content is not easily discoverable by users right now. Users would have to know which websites work offline, or they would need to have an installed PWA, to be able to browse through content while offline. This is not a great user experience as there no entry points to discover available content. To address this, the spec covers a new API which allows developers to tell the browser about their specific content.

The content index allows websites to register their offline enabled content with the browser. The browser can then improve the website’s offline capabilities and offer content to users to browse through while offline. This data could also be used to improve on-device search and augment browsing history.

Using the API can help users more easily discover content for the following example use cases:

1.1. Example

Registering an offline news article in a service worker.
function deleteArticleResources(id) {
  return Promise.all([
    caches.open('offline-articles')
        .then(articlesCache => articlesCache.delete(`/article/${id}`)),
    // This is a no-op if the function was called as a result of the
    // `contentdelete` event.
    self.registration.index.delete(id),
  ]);
}

self.addEventListener('activate', event => {
  // When the service worker is activated, remove old content.
  event.waitUntil(async function() {
    const descriptions = await self.registration.index.getAll();
    const oldDescriptions =
        descriptions.filter(description => shouldRemoveOldArticle(description));
    await Promise.all(
        oldDescriptions.map(description => deleteArticleResources(description.id)));
  }());
});

self.addEventListener('push', event => {
  const payload = event.data.json();

  // Fetch & store the article, then register it.
  event.waitUntil(async function() {
    const articlesCache = await caches.open('offline-articles');
    await articlesCache.add(`/article/${payload.id}`);
    await self.registration.index.add({
      id: payload.id,
      title: payload.title,
      description: payload.description,
      category: 'article',
      icons: payload.icons,
      url: `/article/${payload.id}`,
    });

    // Show a notification if urgent.
  }());
});

self.addEventListener('contentdelete', event => {
  // Clear the underlying content after user-deletion.
  event.waitUntil(deleteArticleResources(event.id));
});    

In the above example, shouldRemoveOldArticle is a developer-defined function.

2. Privacy Considerations

Firing the contentdelete event may reveal the user’s IP address after the user has left the page. Exploiting this can be used for tracking location history. The user agent SHOULD limit tracking by capping the duration of the event.

When firing the contentdelete event, the user agent SHOULD prevent the website from adding new content. This prevents spammy websites from re-adding the same content, and the users from being shown content they just deleted.

Displaying all registered content can cause malicious websites to spam users with their content to maximize exposure. User agents are strongly encouraged to not surface all content, but rather choose the appropriate content to display based on a set of user agent defined signals, aimed at improving the user experience.

3. Infrastructure

3.1. Extensions to service worker registration

A service worker registration additionally has:

3.2. Content index entry

A content index entry consists of:

3.2.1. Display

The user agent MAY display a content index entry (entry) at any time, as long as entry exists in a service worker registration's content index entries.

Note: User agents should limit surfaced content to avoid showing too many entries to a user.

To display a content index entry (entry), the user agent MUST present a user interface that follows these rules:

3.2.2. Undisplay

To undisplay a content index entry (entry), the user agent MUST remove all UI associated with running display on entry.

4. Algorithms

4.1. Delete a content index entry

To delete a content index entry for entry (a content index entry), run these steps:
  1. Let id be entry’s description's id.

  2. Let contentIndexEntries be entry’s service worker registration's content index entries.

  3. Enqueue the following steps to entry’s service worker registration's entry edit queue:

    1. Undisplay entry.

    2. Remove contentIndexEntries[id].

    3. Fire a content delete event for entry.

4.2. Activate a content index entry

To activate a content index entry for entry (a content index entry), run these steps:
  1. Let activeWorker be entry’s service worker registration's active worker.

  2. If activeWorker is null, abort these steps.

  3. Let newContext be a new top-level browsing context.

  4. Queue a task to run the following steps on newContext’s Window object’s environment settings object's responsible event loop using the user interaction task source:

    1. HandleNavigate: Navigate newContext to entry’s launch url with exceptions enabled and replacement enabled.

    2. If the algorithm steps invoked in the step labeled HandleNavigate throws an exception, abort these steps.

    3. Let frameType be "`top-level`".

    4. Let visibilityState be newContext’s active document's visibilityState attribute value.

    5. Let focusState be the result of running the has focus steps with newContext’s active document as the argument.

    6. Let ancestorOriginsList be newContext’s active document's relevant global object's Location object’s ancestor origins list's associated list.

    7. Let serviceWorkerEventLoop be activeWorker’s global object's event loop.

    8. Queue a task to run the following steps on serviceWorkerEventLoop using the DOM manipulation task source:

      1. If newContext’s Window object’s environment settings object's creation URL's origin is not the same as the activeWorker’s origin, abort these steps.

      2. Run Create Window Client with newContext’s Window object’s environment settings object, frameType, visibilityState, focusState, and ancestorOriginsList as the arguments.

Is creating a new Browsing Context the right thing to do here? (issue)

4.3. Fire a content delete event

To fire a content delete event for entry (a content index entry), fire a functional event named "contentdelete" using ContentIndexEvent on entry’s service worker registration with the following properties:
id

entry’s description's id.

5. API

5.1. Extensions to ServiceWorkerGlobalScope

partial interface ServiceWorkerGlobalScope {
  attribute EventHandler oncontentdelete;
};

5.1.1. Events

The following is the event handler (and its corresponding event handler event type) that must be supported, as event handler IDL attributes, by all objects implementing ServiceWorker interface:

event handler event type event handler Interface
contentdelete oncontentdelete ContentIndexEvent

5.2. Extensions to ServiceWorkerRegistration

partial interface ServiceWorkerRegistration {
  [SameObject] readonly attribute ContentIndex index;
};

A ServiceWorkerRegistration has a content index (a ContentIndex), initially a new ContentIndex whose service worker registration is the context object's service worker registration.

The index attribute’s getter must return the context object's content index.

5.3. ContentIndex

ContentIndex

In only one current engine.

FirefoxNoneSafariNoneChromeNone
OperaNoneEdgeNone
Edge (Legacy)NoneIENone
Firefox for AndroidNoneiOS SafariNoneChrome for Android84+Android WebView84+Samsung InternetNoneOpera Mobile60+
enum ContentCategory {
  "",
  "homepage",
  "article",
  "video",
  "audio",
};

dictionary ContentDescription {
  required DOMString id;
  required DOMString title;
  required DOMString description;
  ContentCategory category = "";
  sequence<ImageResource> icons = [];
  required USVString url;
};

[Exposed=(Window,Worker)]
interface ContentIndex {
  Promise<undefined> add(ContentDescription description);
  Promise<undefined> delete(DOMString id);
  Promise<sequence<ContentDescription>> getAll();
};

ContentIndex/getAll

In only one current engine.

FirefoxNoneSafariNoneChromeNone
OperaNoneEdgeNone
Edge (Legacy)NoneIENone
Firefox for AndroidNoneiOS SafariNoneChrome for Android84+Android WebView84+Samsung InternetNoneOpera Mobile60+

A ContentIndex has a service worker registration (a service worker registration).

5.3.1. add()

ContentIndex/add

In only one current engine.

FirefoxNoneSafariNoneChromeNone
OperaNoneEdgeNone
Edge (Legacy)NoneIENone
Firefox for AndroidNoneiOS SafariNoneChrome for Android84+Android WebView84+Samsung InternetNoneOpera Mobile60+
The add(description) method, when invoked, must return a new promise promise and run these steps in parallel:
  1. Let registration be the context object's service worker registration.

  2. If registration’s active worker is null, reject promise with a TypeError and abort these steps.

  3. If any of description’s id, title, description, or url is the empty string, reject promise with a TypeError and abort these steps.

  4. Let launchURL be the result of parsing description’s url with context object's relevant settings object's API base URL.

    Note: A new service worker registration might be introduced later with a narrower scope.

  5. Let matchedRegistration be the result of running Match Service Worker Registration algorithm with launchURL as its argument.

  6. If matchedRegistration is not equal to registration, reject promise with a TypeError and abort these steps.

  7. If registration’s active worker's set of extended events does not contain a FetchEvent, reject promise with a TypeError and abort these steps.

  8. Let icons be an empty list.

  9. Optionally, the user agent MAY select icons to use from description’s icons. In which case run the following steps for each image resource (resource) of description’s icons' selected icons after successfully parsing it:

    1. Let response be the result of awaiting a fetch using a new request with the following properties:

      URL

      resource’s src.

      Client

      context object's relevant settings object.

      Keepalive flag

      Set.

      Destination

      "`image`".

      Mode

      "`no-cors`".

      Credentials mode

      "`include`".

    2. If response is a network error, reject promise with a TypeError and abort these steps.

    3. If response cannot be decoded as an image, reject promise with a TypeError and abort these steps.

    4. Append response to icons.

  10. Let entry be a new content index entry with:

    description

    description.

    launch url

    launchURL

    service worker registration

    registration.

    icons

    icons

  11. Let id be description’s id.

  12. Let contentIndexEntries be registration’s content index entries.

  13. Enqueue the following steps to registration’s entry edit queue:

    1. Set contentIndexEntries[id] to entry.

    2. Optionally, the user agent MAY display entry.

    3. Resolve promise with undefined.

Note: Adding a description with an existing ID would overwrite the previous value.

5.3.2. delete()

ContentIndex/delete

In only one current engine.

FirefoxNoneSafariNoneChromeNone
OperaNoneEdgeNone
Edge (Legacy)NoneIENone
Firefox for AndroidNoneiOS SafariNoneChrome for Android84+Android WebView84+Samsung InternetNoneOpera Mobile60+
The delete(id) method, when invoked, must return a new promise promise and run these steps in parallel:
  1. Let registration be the context object's service worker registration.

  2. Let contentIndexEntries be registration’s content index entries.

  3. Enqueue the following steps to registration’s entry edit queue:

    1. Undisplay contentIndexEntries[id].

    2. Remove contentIndexEntries[id].

    3. Resolve promise with undefined.

5.3.3. getAll()

The getAll() method, when invoked, must return a new promise promise and run these steps in parallel:
  1. Let registration be the context object's service worker registration.

  2. Let contentIndexEntries be registration’s content index entries.

  3. Let descriptions be an empty list.

  4. Enqueue the following steps to registration’s entry edit queue:

    1. For each id → entry of contentIndexEntries:

      1. Append entry’s description to descriptions.

    2. Resolve promise with descriptions.

5.4. ContentIndexEvent

ContentIndexEvent

In only one current engine.

FirefoxNoneSafariNoneChromeNone
OperaNoneEdgeNone
Edge (Legacy)NoneIENone
Firefox for AndroidNoneiOS SafariNoneChrome for Android84+Android WebView84+Samsung InternetNoneOpera Mobile60+
dictionary ContentIndexEventInit : ExtendableEventInit {
  required DOMString id;
};

[Exposed=ServiceWorker]
interface ContentIndexEvent : ExtendableEvent {
  constructor(DOMString type, ContentIndexEventInit init);
  readonly attribute DOMString id;
};

Conformance

Document conventions

Conformance requirements are expressed with a combination of descriptive assertions and RFC 2119 terminology. The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in the normative parts of this document are to be interpreted as described in RFC 2119. However, for readability, these words do not appear in all uppercase letters in this specification.

All of the text of this specification is normative except sections explicitly marked as non-normative, examples, and notes. [RFC2119]

Examples in this specification are introduced with the words “for example” or are set apart from the normative text with class="example", like this:

This is an example of an informative example.

Informative notes begin with the word “Note” and are set apart from the normative text with class="note", like this:

Note, this is an informative note.

Conformant Algorithms

Requirements phrased in the imperative as part of algorithms (such as "strip any leading space characters" or "return false and abort these steps") are to be interpreted with the meaning of the key word ("must", "should", "may", etc) used in introducing the algorithm.

Conformance requirements phrased as algorithms or specific steps can be implemented in any manner, so long as the end result is equivalent. In particular, the algorithms defined in this specification are intended to be easy to understand and are not intended to be performant. Implementers are encouraged to optimize.

Index

Terms defined by this specification

Terms defined by reference

References

Normative References

[DOM]
Anne van Kesteren. DOM Standard. Living Standard. URL: https://dom.spec.whatwg.org/
[FETCH]
Anne van Kesteren. Fetch Standard. Living Standard. URL: https://fetch.spec.whatwg.org/
[HTML]
Anne van Kesteren; et al. HTML Standard. Living Standard. URL: https://html.spec.whatwg.org/multipage/
[IMAGE-RESOURCE]
Aaron Gustafson; Rayan Kanso; Marcos Caceres. Image Resource. 29 March 2021. WD. URL: https://www.w3.org/TR/image-resource/
[INFRA]
Anne van Kesteren; Domenic Denicola. Infra Standard. Living Standard. URL: https://infra.spec.whatwg.org/
[PAGE-VISIBILITY]
Jatinder Mann; Arvind Jain. Page Visibility (Second Edition). 29 October 2013. REC. URL: https://www.w3.org/TR/page-visibility/
[RFC2119]
S. Bradner. Key words for use in RFCs to Indicate Requirement Levels. March 1997. Best Current Practice. URL: https://tools.ietf.org/html/rfc2119
[SERVICE-WORKERS-1]
Alex Russell; et al. Service Workers 1. 19 November 2019. CR. URL: https://www.w3.org/TR/service-workers-1/
[URL]
Anne van Kesteren. URL Standard. Living Standard. URL: https://url.spec.whatwg.org/
[WebIDL]
Boris Zbarsky. Web IDL. 15 December 2016. ED. URL: https://heycam.github.io/webidl/

IDL Index

partial interface ServiceWorkerGlobalScope {
  attribute EventHandler oncontentdelete;
};

partial interface ServiceWorkerRegistration {
  [SameObject] readonly attribute ContentIndex index;
};

enum ContentCategory {
  "",
  "homepage",
  "article",
  "video",
  "audio",
};

dictionary ContentDescription {
  required DOMString id;
  required DOMString title;
  required DOMString description;
  ContentCategory category = "";
  sequence<ImageResource> icons = [];
  required USVString url;
};

[Exposed=(Window,Worker)]
interface ContentIndex {
  Promise<undefined> add(ContentDescription description);
  Promise<undefined> delete(DOMString id);
  Promise<sequence<ContentDescription>> getAll();
};

dictionary ContentIndexEventInit : ExtendableEventInit {
  required DOMString id;
};

[Exposed=ServiceWorker]
interface ContentIndexEvent : ExtendableEvent {
  constructor(DOMString type, ContentIndexEventInit init);
  readonly attribute DOMString id;
};

Issues Index

Is creating a new Browsing Context the right thing to do here? (issue)