Subresource Loading with Web Bundles

Draft Community Group Report,

This version:
https://wicg.github.io/webpackage/subresource-loading.html
Issue Tracking:
GitHub
Inline In Spec
Editors:
(Google Inc.)
(Google)

Abstract

How UAs load subresources from Web Bundles.

Status of this document

This specification was published by the Web Platform Incubator Community Group. It is not a W3C Standard nor is it on the W3C Standards Track. Please note that under the W3C Community Contributor License Agreement (CLA) there is a limited opt-out and other conditions apply. Learn more about W3C Community and Business Groups.

1. Introduction

This section is non-normative.

The Subresource Loading with Web Bundles specification describes a way to load a large number of resources efficiently using a format that allows multiple resources to be bundled, Web Bundles. This specification describes how web browsers load those resources. It is expressed as several monkeypatches to the [HTML], [FETCH] and [CSP] specification which call algorithms defined here.

Note: This specification is under construction. See #708.

2. Structures

A fetched web bundle is a representation of a web bundle format defined in [draft-ietf-wpack-bundled-responses-latest].

A web bundle fetch entry is a struct with the following items:

A better name for web bundle fetch entry?

A web bundle fetch entry entry is used by a registration in Document document if document’s web bundle registration list contains a web bundle registration whose fetch entry is entry.

A bundle rule is a struct with the following items:

A web bundle registration is a struct with the following items:

A web bundle parse result is a struct with the following items:

Each environment settings object will get a web bundle registration list algorithm, which returns a list of web bundle registrations.

A Document has a web bundle registration list, which is a list of web bundle registrations. It is initially empty.

In set up a window environment settings object, settings object’s web bundle registration list returns the web bundle registration list of window’s associated Document.

In set up a worker environment settings object, settings object’s web bundle registration list returns an empty list.

A Document has a web bundle fetch entry list, which is a list of web bundle fetch entries. It is initially empty.

While list is used for web bundle fetch entry list, the order shouldn’t be important.

Not supported for workers.

3. HTML monkeypatches

To process web bundles in the prepare the script element algorithm consistently with existing script types (i.e. classic or module), we make the following changes:

Note: Because we don’t make web bundle result a new subclass of script, other script execution-related specs are left unaffected.

3.1. Prepare the script element

Inside the prepare the script element algorithm, we make the following changes:

NOTE: CSPs are applied to inline web bundles at Step 15 of prepare the script element, just like applied to classic/module scripts.

To prepare a web bundle, given an HTMLScriptElement element, a string sourceText and a URL baseURL:

  1. Let parse result be the result of parse a web bundle string given sourceText and baseURL.

  2. If this throws an exception:

    1. Set the script’s result to a new web bundle result with its registration is null and its error to rethrow is the exception thrown.

    2. Mark as ready.

    3. Return.

  3. Let document be element’s node document.

  4. Set fetch entry to null.

  5. For each r in document’s web bundle fetch entry list:

    1. If r’s source is parse result’s source and r’s credentials is parse result’s credentials, then:

      1. If r is not used by a registration in document, then set fetch entry to r.

        NOTE: This implies that another script element whose the script’s result's registration's fetch entry is r was removed. This is to ensure that web bundle fetch entries are not destructed and re-fetched when HTMLScriptElements with the same web bundle source and credentials are removed and added.

  6. If fetch entry is null:

    1. Set fetch entry to a new web bundle fetch entry with its source is parse result’s source, its credentials is parse result’s credentials, its state is "fetching", and its fetched bundle is null.

    2. Append fetch entry to document’s web bundle fetch entry list.

    3. In parallel, fetch a web bundle fetch entry.

  7. Let registration be a new web bundle registration with its fetch entry is fetch entry and its rule is parse result’s rule.

  8. Append registration to document’s web bundle registration list.

  9. Set the script’s result to a new web bundle result with its registration is registration and its error to rethrow is null.

  10. Mark as ready.

3.2. Firing events

In execute the script element, add the following case to Step 6:

To process events for a web bundle given an HTMLScriptElement element:

  1. Let result be the script’s result of element.

  2. Assert: element’s the script’s type is "webbundle".

  3. Assert: result is an web bundle result.

  4. Await asynchronously until either of the following conditions met:

    Note: Unlike other script types, we wait asynchronously here for fetch web bundle before firing load events at HTMLScriptElement. We don’t delay the load event here, because we mark as ready synchronously in prepare the script element. This is intentional because fetch a web bundle is similar to preloading.

  5. If the script’s result of element is null, then return.

    Note: This can happen when element was removed from the document during the previous step.

    Note: This is specified consistently with whatwg/html#2673. Currently we don’t fire error events in this case. If we change the decision at whatwg/html#2673 to fire error events, then change this step accordingly.

  6. Assert: element’s node document is equal to element’s preparation-time document.

  7. If result’s error to rethrow is not null, then:

    1. Report the exception given result’s error to rethrow.

      There are no relevant script, because web bundle result isn’t a script. This needs to wait for whatwg/html#958 before it is fixable.

    2. Return.

  8. Assert: result’s registration is not null.

  9. If result’s registration's fetch entry's state is "failed":

    1. Fire an event named error at element.

    2. Return.

  10. Assert: result’s registration's fetch entry's state is "fetched".

  11. Fire an event named load at element.

3.3. Removing

If script element is removed from the document, user agents must run the following algorithm:

  1. If the script’s type is not "webbundle", then return.

  2. If the script’s result is null, then return.

  3. Assert: the script’s result is an web bundle result.

  4. Let registration be the script’s result's registration.

  5. Set the script’s result to null.

  6. If registration is null, then return.

  7. Let document be the node document.

  8. Assert: document’s web bundle registration list contains registration.

  9. Remove registration from document’s web bundle registration list.

  10. Queue a microtask to perform the following steps:

    1. Let fetch entry be registration’s fetch entry.

    2. If fetch entry is used by a registration in document, then return.

    3. Remove fetch entry from document’s web bundle fetch entry list.

      Note: It is possible that document’s web bundle fetch entry list doesn’t contain fetch entry even before this step, if fetch entry is used by web bundle registrations of multiple script elements and the script elements are removed.

      Note: At this point, fetch entry can no longer used by subsequent subresource fetches nor subsequent prepare a web bundle calls, but its fetched bundle can be still in use by ongoing fetches.

4. Fetch monkeypatches

4.1. Monkeypatch fetch

In fetch, before

  1. Let taskDestination be null.

add the following step:

  1. If the result of find a matching web bundle registration given request is null, set request’s service-workers mode to "none".

Note: This means that no service workers will get events for a subresource loading from a webbundle.

4.2. Monkeypatch fetch scheme

Add "uuid-in-package" to the schemes listed in fetch scheme.

Note: This ensures that the navigate algorithm uses the process a navigate fetch algorithm for uuid-in-package: URLs.

Note: The origin of a URL whose scheme is "uuid-in-package" is an opaque origin.

4.3. Monkeypatch HTTP-network-or-cache fetch

In HTTP-network-or-cache fetch, before

8.22. Set httpCache to the result of determining the HTTP cache partition, given httpRequest.

add the following steps:

  1. Set the response to the result of fetch a subresource from web bundle, given httpRequest.

    1. If response is network error, return network error.

    2. If response is non-null, skip the steps 8.22-8.24 and goto the step 9.

      Note: That means a subresource from a webbundle never interacts with HttpCache. We plan to support HttpCache as a feature enhancement in the future.

5. CSP monkeypatches

5.1. Monkeypatch Does request match source list?

Rewrite Does request match source list? to run these steps:

  1. Let url be request’s current url.

  2. If url’s scheme is "uuid-in-package", then:

    1. Let registration be the result of running find a matching web bundle registration given request.

    2. If registration is not null, then set url to registration’s fetch entry's source.

  3. Returns the result of executing Does url match source list in origin with redirect count? on url, source list, policy’s self-origin, and request’s redirect count.

Note: This means that CSP restrictions are evaluated against the bundle’s URL instead of to the uuid-in-package: URL. See #651 for the detailed motivation.

5.2. Monkeypatch Does response to request match source list?

Rewrite Does response to request match source list? to run these steps:

  1. Let url be response’s url.

  2. If url’s scheme is "uuid-in-package", then:

    1. Let registration be the result of running find a matching web bundle registration given request.

    2. If registration is not null, then set url to registration’s fetch entry's source.

  3. Returns the result of executing Does url match source list in origin with redirect count? on url, source list, policy’s self-origin, and request’s redirect count.

Note: This means that CSP restrictions are evaluated against the bundle’s URL instead of to the uuid-in-package: URL. See #651 for the detailed motivation.

6. Algorithms

6.1. Parsing

To parse a web bundle string, given a string sourceText and a URL baseURL:

  1. Let parsed be the result of parsing JSON into Infra values given sourceText.

  2. If parsed is not a map, then throw a TypeError indicating that the top-level value needs to be a JSON object.

  3. If parsed["source"] does not exist, then throw a TypeError.

  4. If parsed["source"] is not a string, then throw a TypeError.

  5. Let source be the result of parsing parsed["source"] with baseURL as the base URL.

  6. If source is null, then throw a TypeError.

  7. Let credentials be "same-origin".

  8. If parsed["credentials"] exists, then:

    1. If parsed["credentials"] is "omit", then set credentials to "omit".

    2. Otherwise, if parsed["credentials"] is "include", then set credentials to "include".

  9. Let resources be an empty list.

  10. If parsed["resources"] exists, then:

    1. If parsed["resources"] is not a list, then throw a TypeError.

    2. Set resources to the result of parsing a url list given parsed["resources"] and source.

  11. Let scopes be an empty list.

  12. If parsed["scopes"] exists, then:

    1. If parsed["scopes"] is not a list, then throw a TypeError.

    2. Set scopes to the result of parsing a url list given parsed["scopes"] and source.

  13. If parsed’s keys contains any items besides "source", "credentials", "resources" or "scopes", report a warning to the console that an invalid top-level key was present in the web bundle string.

    Note: This can help detect typos. It is not an error, because that would prevent any future extensions from being added backward-compatibly.

  14. Let rule be bundle rule whose resources are resources and whose scopes are scopes.

  15. Return the web bundle parse result whose source is source, whose credentials are credentials and whose rule is rule .

To parse a URL list, given a list originalList and a URL baseURL:

  1. Let parsed URL list be an empty list.

  2. For each item of originalList,

    1. If item is a string, then

      1. Let URL be the result of parsing item with baseURL as the base URL.

      2. If URL is not null, append URL to parsed URL list.

  3. Return parsed URL list.

6.2. Fetching a web bundle

To fetch a web bundle given web bundle fetch entry fetch entry and fetch params fetch params:

  1. Assert: fetch entry’s state is "fetching".

  2. Let request be fetch params’s request.

  3. Set request’s url to fetch entry’s source.

    Note: Source URL is resolved on document’s base URL.

  4. Set request’s destination to "webbundle",

  5. Set request’s mode to "cors",

  6. Set request’s credentials mode to fetch entry’s credentials.

  7. Set request’s service-workers mode to "none".

  8. Append a header, a tuple of ("Accept", "application/webbundle;v=b2"), to request’s header list.

    Note: The final [draft-ietf-wpack-bundled-responses-latest] will use a version of 1, but this specification tracks what’s actually implemented in browsers, which still uses draft versions.

  9. Fetch request with processResponse algorithm set to process web bundle response which is partially applied with fetch entry.

    Note: Chromium’s current implementation doesn’t allow a nested bundle. A Web bundle is never fetched from other web bundles.

6.3. Process web bundle response

To process web bundle response given web bundle fetch entry fetch entry and response response:

  1. If response’s status is an ok status,

    1. Parse response’s body as a Web Bundle ([draft-ietf-wpack-bundled-responses-latest]).

      Note: response’s body might not be fully available at this moment. UA might parse the bundle by incrementally reading a body asynchronously in order to serve a subresource as early as possible.

      Note: In parsing, Chromium’s experimental implementation only accepts "b2" as a web bundle format version number ([draft-ietf-wpack-bundled-responses-latest]).

    2. When the parse algorithm asynchronously completes, set fetch entry’s fetched bundle to the result of parsing and fetch entry’s state be "fetched". If parsing fails, or any other conformance is violated, set fetched bundle to null and state to "failed".

  2. Otherwise, set fetch entry’s state to "failed".

6.4. Fetching subresources from a web bundle

To fetch a subresource from web bundle given request httpRequest:

  1. Let registration be the result of running find a matching web bundle registration given httpRequest.

  2. If registration is not null:

    1. Let response be the result of get response from web bundle fetch entry given httpRequest’s url and registration’s fetch entry.

    2. If response is null, return a network error.

      Note: This means a browser does not fallback to fetch a subresource from network.

    3. Otherwise, return response.

  3. Return null.

Note: Returning null here can fallback to HTTP cache and ordinal network fetch, unlike returning a network error above.

To get response from web bundle fetch entry given url url and web bundle fetch entry fetch entry:

  1. If fetch entry’s state is "fetching", await until state becomes "fetched" or "failed" asynchronously.

  2. If fetch entry’s state is "failed", return null.

  3. Assert: fetch entry’s fetched bundle is non-null.

  4. Returns response from fetch entry’s fetched bundle given url ([draft-ietf-wpack-bundled-responses-latest]). If a representation of url is not found in fetched bundle, return null.

6.5. Finding a matching registration

To find a matching web bundle registration given request httpRequest:

  1. Let url be httpRequest’s url.

  2. For each registration of httpRequest’s client's web bundle registration list:

    1. Let rule be registration’s rule.

    2. If url’s scheme is not "uuid-in-package", then

      1. If url’s origin and registration’s fetch entry's source's origin are not same origin, then continue.

      2. Let allowed path be the result of shortening registration’s fetch entry's source's path.

      3. If url’s path doesn’t start with allowed path, then continue.

    3. If rule’s resources contains url, then return registration.

    4. If url starts with any of rule’s scopes, then return registration.

  3. Return null.

Conformance

Document conventions

Conformance requirements are expressed with a combination of descriptive assertions and RFC 2119 terminology. The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in the normative parts of this document are to be interpreted as described in RFC 2119. However, for readability, these words do not appear in all uppercase letters in this specification.

All of the text of this specification is normative except sections explicitly marked as non-normative, examples, and notes. [RFC2119]

Examples in this specification are introduced with the words “for example” or are set apart from the normative text with class="example", like this:

This is an example of an informative example.

Informative notes begin with the word “Note” and are set apart from the normative text with class="note", like this:

Note, this is an informative note.

Conformant Algorithms

Requirements phrased in the imperative as part of algorithms (such as "strip any leading space characters" or "return false and abort these steps") are to be interpreted with the meaning of the key word ("must", "should", "may", etc) used in introducing the algorithm.

Conformance requirements phrased as algorithms or specific steps can be implemented in any manner, so long as the end result is equivalent. In particular, the algorithms defined in this specification are intended to be easy to understand and are not intended to be performant. Implementers are encouraged to optimize.

Index

Terms defined by this specification

Terms defined by reference

References

Normative References

[CONSOLE]
Dominic Farolino; Robert Kowalski; Terin Stock. Console Standard. Living Standard. URL: https://console.spec.whatwg.org/
[CSP]
Mike West; Antonio Sartori. Content Security Policy Level 3. URL: https://w3c.github.io/webappsec-csp/
[DOM]
Anne van Kesteren. DOM Standard. Living Standard. URL: https://dom.spec.whatwg.org/
[FETCH]
Anne van Kesteren. Fetch Standard. Living Standard. URL: https://fetch.spec.whatwg.org/
[HTML]
Anne van Kesteren; et al. HTML Standard. Living Standard. URL: https://html.spec.whatwg.org/multipage/
[INFRA]
Anne van Kesteren; Domenic Denicola. Infra Standard. Living Standard. URL: https://infra.spec.whatwg.org/
[RFC2119]
S. Bradner. Key words for use in RFCs to Indicate Requirement Levels. March 1997. Best Current Practice. URL: https://datatracker.ietf.org/doc/html/rfc2119
[URL]
Anne van Kesteren. URL Standard. Living Standard. URL: https://url.spec.whatwg.org/
[WEBIDL]
Edgar Chen; Timothy Gu. Web IDL Standard. Living Standard. URL: https://webidl.spec.whatwg.org/

Informative References

[DRAFT-IETF-WPACK-BUNDLED-RESPONSES-LATEST]
Web Bundles. URL: https://wpack-wg.github.io/bundled-responses/draft-ietf-wpack-bundled-responses.html

Issues Index

A better name for web bundle fetch entry?
While list is used for web bundle fetch entry list, the order shouldn’t be important.
Not supported for workers.
There are no relevant script, because web bundle result isn’t a script. This needs to wait for whatwg/html#958 before it is fixable.