KV Storage

Draft Community Group Report,

This version:
https://wicg.github.io/kv-storage/
Editor:
Domenic Denicola (Google)
Participate:
GitHub WICG/kv-storage (new issue, open issues)
Commits:
GitHub spec.bs commits

Abstract

This specification details a high level asynchronous key/value storage API, layered on top of IndexedDB, and as a spiritual successor to the original localStorage.

Status of this document

This specification was published by the Web Platform Incubator Community Group. It is not a W3C Standard nor is it on the W3C Standards Track. Please note that under the W3C Community Contributor License Agreement (CLA) there is a limited opt-out and other conditions apply. Learn more about W3C Community and Business Groups.

1. Introduction

This section is non-normative.

The localStorage API is widely used, and loved for its simplicity. However, its synchronous nature leads to terrible performance and cross-window synchronization issues.

This specification proposes a new API, called KV storage, which is intended to provide an analogously simple interface, while being asynchronous. Along the way, it embraces some additional goals:

A conversion of the HTML Standard’s localStorage example to use KV storage might look like the following:
<p>
  You have viewed this page
  <span id="count">an untold number of</span>
  time(s).
</p>
<script type="module">
  import { storage } from "std:kv-storage";

  (async () => {
    let pageLoadCount = await storage.get("pageLoadCount") || 0;
    ++pageLoadCount;

    document.querySelector("#count").textContent = pageLoadCount;

    await storage.set("pageLoadCount", pageLoadCount);
  })();
</script>

As a side note, observe how, in contrast to the original example which performs up to five storage operations, our example only performs two. Also, it updates the UI as soon as possible, instead of delaying the UI update until we’ve set the new page load count.

The KV storage API design can take some credit for this, as by forcing us to explicitly state our await points, it makes it more obvious that we’re performing a potentially-expensive storage operation.

2. The std:kv-storage built-in module

The below is not yet valid Web IDL. It depends on the following outstanding Web IDL pull requests:

Note that until at least the first two of these are merged and the associated tooling is updated, syntax highlighting and cross-linking will not work for the below IDL block. Sorry about that.

[SecureContext,Exposed=(Window,Worker)]
module kv-storage {
  [Constructor(DOMString name), SameRealmBrandCheck]
  interface StorageArea {
    Promise<void> set(any key, any value);
    Promise<any> get(any key);
    Promise<void> delete(any key);
    Promise<void> clear();

    async_iterable<any, any>;

    [SameObject] readonly attribute object backingStore;
  };

  readonly attribute kv-storage.StorageArea storage;
};
import { storage } from "std:kv-storage"

Returns the default storage area. It is a pre-constructed instance of the StorageArea class, meant to be a convenience similar to localStorage.

import { StorageArea } from "std:kv-storage"

Returns the constructor for the StorageArea class, to allow the creation of isolated storage areas.

If the module is not imported in a secure context, the import statements will fail, as persistent storage is a powerful feature.

This specification defines a new built-in module, "std:kv-storage". Its exports are:

storage

An instance of the StorageArea class, backed by a database named "kv-storage:default".

StorageArea

The StorageArea class

The evaluation steps for the std:kv-storage module in realm are:

  1. Let defaultStorageArea be a new StorageArea in realm.

  2. Set defaultStorageArea.[[DatabaseName]] to "kv-storage:default".

  3. Set defaultStorageArea.[[DatabasePromise]] to null.

  4. Set defaultStorageArea.[[BackingStoreObject]] to null.

  5. Set the module attribute storage of this module to defaultStorageArea.

3. The StorageArea class

Each StorageArea instance must also contain the [[DatabaseName]], [[DatabasePromise]], and [[BackingStoreObject]] internal slots. The following is a non-normative summary of their meaning:

[[DatabaseName]]
A string containing the name of the backing IndexedDB database.
[[DatabasePromise]]
A promise for an IDBDatabase object, lazily initialized when performing any database operation.
[[BackingStoreObject]]
The object returned by the backingStore getter, cached to ensure identity across gets.

The [SameRealmBrandCheck] extended attribute used in the IDL gives us semantics identical to using JavaScript’s WeakMap or the proposed private class fields. This ensures StorageArea does not use any magic, like the platform’s usual cross-realm brand checks, which go beyond what can be implemented in JavaScript. [ECMA-262] [CLASS-FIELDS]

3.1. constructor(name)

storage = new StorageArea(name)

Creates a new StorageArea that provides an async key/value store view onto an IndexedDB database named `kv-storage:${name}`.

This does not actually open or create the database yet; that is done lazily when other methods are called. This means that all other methods can reject with database-related exceptions in failure cases.

The StorageArea(name) constructor, when invoked, must run these steps:
  1. Set this.[[DatabaseName]] to the concatenation of "kv-storage:" and name.

  2. Set this.[[DatabasePromise]] to null.

  3. Set this.[[BackingStoreObject]] to null.

3.2. set(key, value)

await storage.set(key, value)

Asynchronously stores the given value so that it can later be retrieved by the given key.

Keys have to follow the same restrictions as IndexedDB keys: roughly, a key can be a number, string, array, Date, ArrayBuffer, DataView, typed array, or an array of these. Invalid keys will cause the returned promise to reject with a "DataError" DOMException.

Values can be any value that can be structured-serialized for storage. Un-serializable values will cause a "DataCloneError" DOMException. The value undefined will cause the corresponding entry to be deleted.

The returned promise will fulfill with undefined on success.

  1. If key is not allowed as a key, return a promise rejected with a "DataError" DOMException.

  2. Return the result of performing a database operation given this object, "readwrite", and the following steps operating on transaction and store:

    1. If value is undefined, then

      1. Perform the steps listed in the description of IDBObjectStore's delete() method on store, given the argument key.

    2. Otherwise,

      1. Perform the steps listed in the description of IDBObjectStore's put() method on store, given the arguments value and key.

    3. Let promise be a new promise.

    4. Add a simple event listener to transaction for "complete" that resolves promise with undefined.

    5. Add a simple event listener to transaction for "error" that rejects promise with transaction’s error.

    6. Add a simple event listener to transaction for "abort" that rejects promise with transaction’s error.

    7. Return promise.

3.3. get(key)

value = await storage.get(key)

Asynchronously retrieves the value stored at the given key, or undefined if there is no value stored at key.

Values retrieved will be structured-deserialized from their original form.

  1. If key is not allowed as a key, return a promise rejected with a "DataError" DOMException.

  2. Return the result of performing a database operation given this object, "readonly", and the following steps operating on transaction and store:

    1. Let request be the result of performing the steps listed in the description of IDBObjectStore's get() method on store, given the argument key.

    2. Let promise be a new promise.

    3. Add a simple event listener to request for "success" that resolves promise with request’s result.

    4. Add a simple event listener to request for "error" that rejects promise with request’s error.

    5. Return promise.

3.4. delete(key)

await storage.delete(key)

Asynchronously deletes the entry at the given key. This is equivalent to storage.set(key, undefined).

The returned promise will fulfill with undefined on success.

  1. If key is not allowed as a key, return a promise rejected with a "DataError" DOMException.

  2. Return the result of performing a database operation given this object, "readwrite", and the following steps operating on transaction and store:

    1. Perform the steps listed in the description of IDBObjectStore's delete() method on store, given the argument key.

    2. Let promise be a new promise.

    3. Add a simple event listener to transaction for "complete" that resolves promise with undefined.

    4. Add a simple event listener to transaction for "error" that rejects promise with transaction’s error.

    5. Add a simple event listener to transaction for "abort" that rejects promise with transaction’s error.

    6. Return promise.

3.5. clear()

await storage.clear()

Asynchronously deletes all entries in this storage area.

This is done by actually deleting the underlying IndexedDB database. As such, it always can be used as a fail-safe to get a clean slate, as shown below.

The returned promise will fulfill with undefined on success.

  1. If this.[[DatabasePromise]] is not null, return the result of transforming this.[[DatabasePromise]] by fulfillment and rejection handlers that both perform the following steps:

    1. Set this.[[DatabasePromise]] to null.

    2. Return the result of deleting the database given by this.[[DatabaseName]].

  2. Otherwise, return the result of deleting the database given by this.[[DatabaseName]].

To delete the database given a string name:

  1. Let promise be a new promise.

  2. Let request be the result of performing the steps listed in the description of IDBFactory's deleteDatabase() method on the current IDBFactory, given the argument name.

  3. If those steps threw an exception, catch the exception and reject promise with it.

  4. Otherwise:

    1. Add a simple event listener to request for "success" that resolves promise with undefined.

    2. Add a simple event listener to request for "error" that rejects promise with request’s error.

  5. Return promise.

This method can be used to recover from unexpected modifications to the backing store. For example,
// This upgrade to version 100 breaks the "cats" storage area: since StorageAreas
// assume a version of 1, "cats" can no longer be used with KV storage.
const openRequest = indexedDB.open("kv-storage:cats", 100);
openRequest.onsuccess = () => {
  openRequest.onsuccess.close();
};

(async () => {
  const area = new StorageArea("cats");

  // Due to the above upgrade, all other methods will reject:
  try {
    await area.set("fluffy", new Cat());
  } catch (e) {
    // This will be reached and output a "VersionError" DOMException
    console.error(e);
  }

  // But clear() will delete the database entirely:
  await area.clear();

  // Now we can use it again!
  await area.set("fluffy", new Cat());
  await area.set("tigger", new Cat());

  // Also, the version is back down to 1:
  console.assert(area.backingStore.version === 1);
})();

3.6. Iteration

The StorageArea interface supports asynchronous iteration.

for await (const key of storage.keys()) { ... }

Retrieves an async iterator containing the keys of all entries in this storage area.

Keys will be yielded in ascending order; roughly, segregated by type, and then sorted within each type. They will be key round-tripped from their original form.

for await (const value of storage.values()) { ... }

Asynchronously retrieves an array containing the values of all entries in this storage area.

Values will be ordered as corresponding to their keys. They will be structured-deserialized from their original form.

for await (const [key, value] of storage.entries()) { ... }
for await (const [key, value] of storage) { ... }

Asynchronously retrieves an array of two-element [key, value] arrays, each of which corresponds to an entry in this storage area.

Entries will be ordered as corresponding to their keys. Each key and value will be key round-tripped and structured-deserialized from its original form, respectively.

All of these iterators provide live views onto the storage area: modifications made to entries sorted after the last-returned one will be reflected in the iteration.

To get the next iteration result:

  1. If this’s relevant realm is not equal to the current realm, then return a promise rejected with a TypeError exception.

    This should be handled by WebIDL once the various features involved land.

  2. Return the result of performing a database operation given this, "readonly", and the following steps operating on transaction and store:

    1. Let lastKey be current state.

    2. Let range be the result of getting the range for lastKey.

    3. Let keyRequest be the result of performing the steps listed in the description of IDBObjectStore's getKey() method on store, given the argument range.

    4. Let valueRequest be the result of performing the steps listed in the description of IDBObjectStore's get() method on store, given the argument range.

      Note: The iterator returned from keys() discards the value. Implementations could avoid constructing valueRequest in that case.

    5. Let promise be a new promise.

    6. Add a simple event listener to valueRequest for "success" that performs the following steps:

      1. Let key be keyRequest’s result.

      2. If key is undefined, then:

        1. Resolve promise with undefined.

      3. Otherwise:

        1. Let value be valueRequest’s result.

        2. Resolve promise with (key, value, key).

    7. Add a simple event listener to keyRequest for "error" that rejects promise with keyRequest’s error.

    8. Add a simple event listener to valueRequest for "error" that rejects promise with valueRequest’s error.

    9. Return promise.

To illustrate the live nature of the async iterators, consider the following:
await storage.set(10, "value 10");
await storage.set(20, "value 20");
await storage.set(30, "value 30");

const keysSeen = [];
for await (const key of storage.keys()) {
  if (key === 20) {
    await storage.set(15, "value 15");
    await storage.delete(20);
    await storage.set(25, "value 25");
  }
  keysSeen.push(key);
}

console.log(keysSeen);   // logs 10, 20, 25, 30

That is, calling keys() does not create a snapshot as of the time it was called; it returns a live asynchronous iterator, that lazily retrieves the next key after the last-seen one.

Assuming you knew that that you only stored JSON-compatible types in the StorageArea storage, you could use the following code to send all locally-stored entries to a server:
const entries = [];
for await (const entry of storage.entries()) {
  entries.push(entry);
}

fetch("/storage-receiver", {
  method: "POST",
  body: entries,
  headers: {
    "Content-Type": "application/json"
  }
});

3.7. backingStore

{ database, store, version } = storage.backingStore

Returns an object containing all of the information necessary to manually interface with the IndexedDB backing store that underlies this storage area:

  • database will be a string equal to "kv-storage:" concatenated with the database name passed to the constructor. (For the default storage area, it will be "kv-storage:default".)

  • store will be the string "store".

  • version will be the number 1.

It is good practice to use the backingStore property to retrieve this information, instead of memorizing the above factoids.

  1. If this.[[BackingStoreObject]] is null, then:

    1. Let backingStoreObject be ObjectCreate(%ObjectPrototype%).

    2. Perform CreateDataProperty(backingStoreObject, "database", this.[[DatabaseName]]).

    3. Perform CreateDataProperty(backingStoreObject, "store", "store").

    4. Perform CreateDataProperty(backingStoreObject, "version", 1).

    5. Perform SetIntegrityLevel(backingStoreObject, "frozen").

    6. Set this.[[BackingStoreObject]] to backingStoreObject.

  2. Return this.[[BackingStoreObject]].

Consider a checklist application, which tracks the Pokémon a user has collected. It might use a StorageArea storage like so:
bulbasaur.onchange = () => storage.set("bulbasaur", bulbasaur.checked);
ivysaur.onchange = () => storage.set("ivysaur", ivysaur.checked);
venusaur.onchange = () => storage.set("venusaur", venusaur.checked);
// ...

(Hopefully the developer quickly realizes that the above will be hard to maintain, and refactors the code into a loop. But in the meantime, their repetitive code makes for a good example, so let’s take advantage of that.)

The developer now realizes they want to add an evolution feature, e.g. for when the user transforms their Bulbasaur into an Ivysaur. They might first implement this like so:

bulbasaurEvolve.onclick = async () => {
  await storage.set("bulbasaur", false);
  await storage.set("ivysaur", true);
};

However, our developer starts getting bug reports from their users: if the users happen to open up the checklist app in a second tab while they’re evolving in the first tab, the second tab will sometimes see that their Bulbasaur has disappeared, without ever turning into an Ivysaur! A Pokémon has gone missing!

The solution here is to step beyond the comfort zone of KV storage, and start using the full power of IndexedDB: in particular, its transactions feature. The backingStore getter is the gateway to this world:

const { database, store, version } = storage.backingStore;
const request = indexedDB.open(database, version);
request.onsuccess = () => {
  const db = request.result;

  bulbasaurEvolve.onclick = () => {
    const transaction = db.transaction(store, "readwrite");
    const store = transaction.objectStore(store);

    store.put("bulbasaur", false);
    store.put("ivysaur", true);

    db.close();
  };
};

Satisfied with their web app’s Pokémon integrity, our developer is now happy and fulfilled. (At least, until they realize that none of their code has error handling.)

4. Supporting operations and concepts

To add a simple event listener, given an EventTarget target, an event type string type, and a set of steps steps:

  1. Let jsCallback be a new JavaScript function object, created in the current realm, that performs the steps given by steps. Other properties of the function (such as its name and length properties, or [[Prototype]]) are unobservable, and can be chosen arbitrarily.

  2. Let idlCallback be the result of converting jsCallback to an EventListener.

  3. Perform the steps listed in the description of EventTarget's addEventListener() method on target given the arguments type and idlCallback.

The current IDBFactory is the IDBFactory instance returned by the following steps:

  1. Assert: the current global object includes WindowOrWorkerGlobalScope.

  2. Return the result of performing the steps listed in the description of the getter for WindowOrWorkerGlobalScope's indexedDB attribute on the current global object.

To perform a database operation given a StorageArea area, a mode string mode, and a set of steps steps that operate on an IDBTransaction transaction and an IDBObjectStore store:

  1. Assert: area.[[DatabaseName]] is a string (and in particular is not null).

  2. If area.[[DatabasePromise]] is null, initialize the database promise for area.

  3. Return the result of transforming area.[[DatabasePromise]] by a fulfillment handler that performs the following steps, given database:

    1. Let transaction be the result of performing the steps listed in the description of IDBDatabase's transaction() method on database, given the arguments "store" and mode.

    2. Let store be the result of performing the steps listed in the description of IDBTransaction's objectStore() method on transaction, given the argument "store".

    3. Return the result of performing steps, passing along transaction and store.

To initialize the database promise for a StorageArea area:

  1. Set area.[[DatabasePromise]] to a new promise.

  2. If the current global object does not include WindowOrWorkerGlobalScope, reject area.[[DatabasePromise]] with a TypeError, and return.

  3. Let request be the result of performing the steps listed in the description of IDBFactory's open() method on the current IDBFactory, given the arguments area.[[DatabaseName]] and 1.

  4. If those steps threw an exception, catch the exception, reject area.[[DatabasePromise]] with it, and return.

  5. Add a simple event listener to request for "success" that performs the following steps:

    1. Let database be request’s result.

    2. Check the database schema for database. If the result is false, reject area.[[DatabasePromise]] with an "InvalidStateError" DOMException and abort these steps.

    3. Add a simple event listener to database for "close" that sets area.[[DatabasePromise]] to null.

      This means that if the database is closed abnormally, future invocations of perform a database operation will attempt to reopen it.

    4. Add a simple event listener to database for "versionchange" that performs the steps listed in the description of IDBDatabase's close() method on database, and then sets area.[[DatabasePromise]] to null.

      This allows attempts to upgrade the underlying database, or to delete it (e.g. via the clear() method), to succeed. Without this, if two StorageArea instances were both open referencing the same underlying database, clear() would hang, as it only closes the connection maintained by the StorageArea it is invoked on.

    5. Resolve promise with database.

  6. Add a simple event listener to request for "error" that rejects promise with request’s error.

  7. Add a simple event listener to request for "upgradeneeded" that performs the following steps:

    1. Let database be request’s result.

    2. Perform the steps listed in the description of IDBDatabase's createObjectStore() method on database, given the arguments "store".

    3. If these steps throw an exception, catch the exception and reject area.[[DatabasePromise]] with it.

To check the database schema for an IDBDatabase database:

  1. Let objectStores be database’s connection's object store set.

  2. If objectStores’s size is not 1, return false.

  3. Let store be objectStores[0].

  4. If store’s name is not "store", return false.

  5. If store has a key generator, return false.

  6. If store has a key path, return false.

  7. If any indexes reference store, return false.

  8. Return true.

Check the database schema only needs to be called in the initial setup algorithm, initialize the database promise, since once the database connection has been opened, the schema cannot change.

A value value is allowed as a key if the following steps return true:

  1. If Type(value) is Number or String, return true.

  2. If IsArray(value) is true, return true.

  3. If value has a [[DateValue]] internal slot, return true.

  4. If value has a [[ViewedArrayBuffer]] internal slot, return true.

  5. If value has an [[ArrayBufferByteLength]] internal slot, return true.

  6. Return false.

A value being allowed as a key means that it can at least plausibly be used as a key in the IndexedDB APIs. In particular, the values which are allowed as a key are a subset of those for which IndexedDB’s convert a value to a key algorithm will succeed.

Most notably, using the allowed as a key predicate ensures that IDBKeyRange objects, or any other special object that is accepted as a query in future IndexedDB specification revisions, will be disallowed. Only straightforward key values are accepted by the KV storage API.

Key round-tripping refers to the way in which JavaScript values are processed by first being passed through IndexedDB’s convert a value to a key operation, then converted back through its convert a key to a value operation. Keys returned by the keys() or entries() methods will have gone through this process.

Notably, any typed arrays or DataViews will have been "unwrapped", and returned back as just ArrayBuffers containing the same bytes. Also, similar to the structured-serialization/deserialization process, any "expando" properties or other modifications will not be preserved by key round-tripping.

For primitive string or number values, there’s no need to worry about key round-tripping; the values are indistinguishable.

To get the range for a key key:

  1. If key is not yet started, then return the result of performing the steps listed in the description of the IDBKeyRange.lowerBound() static method, given the argument −Infinity.

    The intent here is to get an unbounded key range, but this is the closest thing we can get that is representable as an IDBKeyRange object. It works equivalently for our purposes, but will behave incorrectly if Indexed DB ever adds keys that sort below −Infinity. See some discussion on potential future improvements.

  2. Otherwise, return the result of performing the steps listed listed in the description of the IDBKeyRange.lowerBound() static method, given the arguments lastKey and true.

The special value not yet started can be taken to be any JavaScript value that is not equal to any other program-accessible JavaScript value (but is equal to itself). It is used exclusively as an argument to the get the range for a key algorithm.

A newly created object or symbol, e.g. const nys = {} or const nys = Symbol(), would satisfy this definition.

5. Appendix: is this API perfectly layered?

The APIs in this specification, being layered on top of Indexed DB as they are, are almost entirely well-layered, in the sense of building on low-level features in the way promoted by the Extensible Web Manifesto. However, it fails in two ways, both around ensuring the encapsulation of the implementation: [EXTENSIBLE]

Eventually we hope to introduce the ability for web authors to write code that gets these same benefits, instead of locking them up so that only web platform APIs like KV storage can achieve this level of encapsulation. That’s a separate effort, however, which is best followed in the above-linked issue threads.

Acknowledgments

The editor would like to thank Andrew Sutherland, Kenneth Rohde Christiansen, Jake Archibald, Jan Varga, Joshua Bell, Ms2ger, and Victor Costan for their contributions to this specification.

Conformance

This specification depends on the Infra Standard. [INFRA]

Index

Terms defined by this specification

Terms defined by reference

References

Normative References

[DOM]
Anne van Kesteren. DOM Standard. Living Standard. URL: https://dom.spec.whatwg.org/
[ECMA-262]
ECMAScript Language Specification. URL: https://tc39.github.io/ecma262/
[HTML]
Anne van Kesteren; et al. HTML Standard. Living Standard. URL: https://html.spec.whatwg.org/multipage/
[INDEXEDDB-2]
Ali Alabbas; Joshua Bell. Indexed Database API 2.0. URL: https://w3c.github.io/IndexedDB/
[INFRA]
Anne van Kesteren; Domenic Denicola. Infra Standard. Living Standard. URL: https://infra.spec.whatwg.org/
[PROMISES-GUIDE]
Domenic Denicola. Writing Promise-Using Specifications. 9 November 2018. TAG Finding. URL: https://www.w3.org/2001/tag/doc/promises-guide
[WebIDL]
Boris Zbarsky. Web IDL. URL: https://heycam.github.io/webidl/

Informative References

[CLASS-FIELDS]
Daniel Ehrenberg; Jeff Morrison. Public and private instance fields proposal. URL: https://tc39.github.io/proposal-class-fields/
[EXTENSIBLE]
The Extensible Web Manifesto. 10 June 2013. URL: https://extensiblewebmanifesto.org/
[FETCH]
Anne van Kesteren. Fetch Standard. Living Standard. URL: https://fetch.spec.whatwg.org/
[JSSTDLIB]
Standard Library Proposal. URL: https://github.com/tc39/proposal-javascript-standard-library/
[SECURE-CONTEXTS]
Mike West. Secure Contexts. URL: https://w3c.github.io/webappsec-secure-contexts/
[SERVICE-WORKERS-1]
Alex Russell; et al. Service Workers 1. URL: https://w3c.github.io/ServiceWorker/

Issues Index

This should be handled by WebIDL once the various features involved land.