This layer consists of a client-server system with capabilities of encrypting data in transit and at rest.

Validate Request (Server)

When a vault client makes a request to store, query, modify, or delete data in the vault, the server validates the request. Since the actual data and metadata in any given request is encrypted, such validation is necessarily limited and largely depends on the protocol and the semantics of the request.

Persist data (Server)

The mechanism a server uses to persist data, such as storage on a local, networked, or distributed file system, is determined by the implementation. The persistence mechanism is expected to adhere to the common expectations of a data storage provider, such as reliable storage and retrieval of data.

Persist global configuration (Server)

The configuration allows the the client to perform capability discovery regarding things like authorization, protocol, and replication mechanisms that are used by the server.

Enforcement of authorization policies (Server)

When a client makes a request to store, query, modify, or delete data in the vault, the server enforces any authorization policy that is associated with the request.

Encrypted data chunking (Client)

It is necessary that large data is chunked into sizes that are easily managed by a server. It is the responsibility of the client to set the chunk size of each resource and chunk large data into manageable chunks for the server. It is the responsibility of the server to deny requests to store chunks larger that it can handle. Each chunk is encrypted individually using authenticated encryption.

Resource structure (Client)

The process of storing encrypted data starts with the creation of a Resource by the client. If the data is less than the chunk size, it is embedded directly into the content. Otherwise, the data is sharded into chunks by the client (see next section), and each chunk is encrypted and sent to the server.

Encrypted resource structure (Client)

The process of creating the Encrypted Resource. If the data was sharded into chunks, this is done after the individual chunks are written to the server.

The following sections elaborate the architecture from a systematic and functional perspective. Also, components and dependencies are described.

There are different ways to get started in using the Storage Kit.

Ways of getting the Storage Kit:

• Docker: Quick way to launch and try out the latest builds. No build environment required.

• Local Build: Build and run the Storage Kit locally. Requires a JDK 16 build environment including, Gradle.

• Dependency (JVM): The Storage Kit can be used directly as JVM-dependency via Maven or Gradle.

Where to use the Storage Kit:

• CLI Tool: The Storage Kit comes with a command-line interface (CLI) tool, which offers a rich set of commands to run the entire functionality the Storage Kit provides. The CLI tool can be used by running the Docker container or the executable by the local build.

• REST API: In case you want to run the Storage Kit as a service, your application can access all functionalities via the REST API.

• Dependency (JVM): The Storage Kit can be used directly as JVM-dependency via Maven or Gradle.

This layer consists of a system that is capable of sharing data among multiple entities of versioning and replication and of performing privacy-preserving search in an efficient manner.

Encrypted search indexes (Client)

To enable privacy-preserving querying (where the search index is opaque to the server), the client must prepare a list of encrypted index tags (which are stored in the Encrypted Resource, alongside the encrypted data contents).

Versioning and replication (Client)

At least one versioning mechanism has to be supported. Replication is done by the client, not by the server (since the client controls the keys, knows about which other servers to replicate to, etc.). The versioning strategy may be implicit ("last write wins").

Sharing with other entities (Client)

An individual vault's choice of authorization mechanism determines how a client shares resources with other entities (authorization capability link or similar mechanism).

We embraced the controller-service model for all REST components: The web server maps a specific route to a specific controller, which itself does nothing other than handling the request and response data, it does not contain any business logic. All of that is separated into service managers.

Also, recognizable for all methods suffixed with -Docs in the controllers, is that our OpenAPI documentation, is configured and fed in-code, directly beneath the actual code. Having the code and it’s documentation nearly makes it very easy to maintain this documentation, e.g. to keep it up-to-date when the code changes, which can be forgotten easily if it was in external configuration files with special syntax.

The client REST API is still in beta testing.

Currently it is recommended to include the JVM client library as a dependency using Maven or Gradle.

Please let us know if you are interested in using the client in non-JVM languages, so we can prioritize this feature accordingly.

This layer consists of high-level server-side functions that work on top of the encrypted credentials.

Notifications (Server)

It is helpful if data storage providers are able to notify clients when changes to persisted data occurs. A server may optionally implement a mechanism by which clients can subscribe to changes in the vault.

Vault-wide integrity protection (Client)

Integrity protection is needed to prevent modification attacks. Some form of integrity protection must be integrated (e.g. HASHLINKS) to prevent the successful execution of such attacks.

The Storage Kit provides diverse functionality that can be segmented into three layers (based on the Confidential Storage specs and requirements by the Decentralized Identity Foundation):

• Layer 1 consists of a client-server system with capabilities of encrypting data in transit and at rest.

• Layer 2 consists of a system that is capable of sharing data among multiple entities of versioning and replication and of performing privacy-preserving search in an efficient manner.

• Layer 3 consists of high-level server-side functions that work on top of the encrypted credentials.

The following graphic illustrates these three layers and offers a functional perspective:

In a nutshell, the Storage Kit is a zero-trust solution for all things data storage and sharing.

The main purpose of the Storage Kit is to provide a highly secure and private infrastructure layer for data storage and data sharing for any application.

The importance of the Storage Kit becomes particularly evident in the context of Decentralized Identity and will be a core component for developers who are building solutions and use cases that involve keys, personal data or other secrets.

The following sections elaborate the basics of the Storage Kit to help you get started.

Key Traits

Here are the most important things you need to know about the Storage Kit:

• It is written in Kotlin/Java. It can be directly integrated (Maven/Gradle dependency) or run as RESTful web-service. A CLI tool allows you to run all functions manually.

• It is open source (Apache 2). You can use the code for free and without strings attached.

• It abstracts complexity and low-level functionality via different interfaces (CLI, APIs).

• It is a holistic solution that allows you to build use cases “end-to-end”. There is no need to research, combine or tweak different libraries to build pilots or production systems.

• It is modular, composable and built on open standards allowing you to customize and extend functionality with your own or third party implementations and to preventing lock-in.

• It is flexible in a sense that you can deploy and run it on-premise, in your (multi) cloud environment or as a library in your application.

Feature List

Find a full overview of all features here

Confidential Storage as a general system involves various actors that have to interoperate:

• EDVs - Encrypted Data Vaults allowing to store data, managing data access and performing queries on the stored data.

• Users - Human or nonhuman actor that wishes to access their EDVs.

• Clients - Programs or libraries supporting access to conform EDVs.

• Providers - Servers that provide an environment to create and manage EDVs.

• Services - Instances besides the user that wish to access data on a specific users EDV (needing the users consent to do so).

A user uses one or more clients to access to their EDVs (multiple clients devices may be used to access data using multiple devices, e.g. stationary and mobile, or for redundancy, to avoid loosing data access if one device fails).

Clients may be connected to multiple EDVs (for redundancy or performance reasons).

Services are a special kind of client that access EDVs through Service Data Requests (which is described later on). EDVs are hosted by providers, and are created through specific requests sent by a client.

Cryptography

We are using Nimbus JOSE and JWT for all JWE (JSON Web Encryption) related purposes, as it is the most popular JOSE Java library4 and the project team has already had experience with this library.

Bouncy Castle is used as cryptography-provider, providing Java Cryptography Extension (JCE) and Java Cryptography Architecture (JCA) implementations. It was chosen as they are one of the most commonly used - and thus tightly integrated - layer for cryptography in Java (and C), providing audited APIs, parts of it even being certified for FIPS-140-25 (Level 1).

Storage

To store the data at Layer A on the system, TPM-backed full disk encryption can be optionally utilized to disallow metadata access even in case of a physical breach. Clients are also secured with passphrase-based symmetric encryption, using PBES2 (published by RSA Laboratories).

Services come with their own configuration files.

The default mapping file is "service-matrix.properties", and looks like this:

e.g., to change the keystore service, simply replace the line

id.walt.services.keystore.KeyStoreService=id.walt.services.keystore.SqlKeyStoreService

with your own implementation mapping, e.g. for the Azure HSM keystore:

id.walt.services.keystore.KeyStoreService=id.walt.services.keystore.azurehsm.AzureHSMKeystoreService

To add a service configuration:

id.walt.services.keystore.KeyStoreService=id.walt.services.keystore.SqlKeyStoreService:sql.conf Service configuration is by default in HOCON format. Refer to the specific service on how their configuration is laid out.

Installation & Running the Project

Make sure you have Docker or a JDK 16 build environment including Gradle installed on your machine

Examples

Gradle

Maven

Required Maven repos: