This layer consists of a client-server system with capabilities of encrypting data in transit and at rest.

Validate Request (Server)

When a vault client makes a request to store, query, modify, or delete data in the vault, the server validates the request. Since the actual data and metadata in any given request is encrypted, such validation is necessarily limited and largely depends on the protocol and the semantics of the request.

Persist data (Server)

The mechanism a server uses to persist data, such as storage on a local, networked, or distributed file system, is determined by the implementation. The persistence mechanism is expected to adhere to the common expectations of a data storage provider, such as reliable storage and retrieval of data.

Persist global configuration (Server)

The configuration allows the the client to perform capability discovery regarding things like authorization, protocol, and replication mechanisms that are used by the server.

Enforcement of authorization policies (Server)

When a client makes a request to store, query, modify, or delete data in the vault, the server enforces any authorization policy that is associated with the request.

Encrypted data chunking (Client)

It is necessary that large data is chunked into sizes that are easily managed by a server. It is the responsibility of the client to set the chunk size of each resource and chunk large data into manageable chunks for the server. It is the responsibility of the server to deny requests to store chunks larger that it can handle. Each chunk is encrypted individually using authenticated encryption.

Resource structure (Client)

The process of storing encrypted data starts with the creation of a Resource by the client. If the data is less than the chunk size, it is embedded directly into the content. Otherwise, the data is sharded into chunks by the client (see next section), and each chunk is encrypted and sent to the server.

Encrypted resource structure (Client)

The process of creating the Encrypted Resource. If the data was sharded into chunks, this is done after the individual chunks are written to the server.

This layer consists of a system that is capable of sharing data among multiple entities of versioning and replication and of performing privacy-preserving search in an efficient manner.

Encrypted search indexes (Client)

To enable privacy-preserving querying (where the search index is opaque to the server), the client must prepare a list of encrypted index tags (which are stored in the Encrypted Resource, alongside the encrypted data contents).

Versioning and replication (Client)

At least one versioning mechanism has to be supported. Replication is done by the client, not by the server (since the client controls the keys, knows about which other servers to replicate to, etc.). The versioning strategy may be implicit ("last write wins").

Sharing with other entities (Client)

An individual vault's choice of authorization mechanism determines how a client shares resources with other entities (authorization capability link or similar mechanism).

This layer consists of high-level server-side functions that work on top of the encrypted credentials.

Notifications (Server)

It is helpful if data storage providers are able to notify clients when changes to persisted data occurs. A server may optionally implement a mechanism by which clients can subscribe to changes in the vault.

Vault-wide integrity protection (Client)

Integrity protection is needed to prevent modification attacks. Some form of integrity protection must be integrated (e.g. HASHLINKS) to prevent the successful execution of such attacks.

In a nutshell, the Storage Kit is a zero-trust solution for all things data storage and sharing.

The main purpose of the Storage Kit is to provide a highly secure and private infrastructure layer for data storage and data sharing for any application.

The importance of the Storage Kit becomes particularly evident in the context of Decentralized Identity and will be a core component for developers who are building solutions and use cases that involve keys, personal data or other secrets.

The following sections elaborate the basics of the Storage Kit to help you get started.

Key Traits

Here are the most important things you need to know about the Storage Kit:

• It is written in Kotlin/Java. It can be directly integrated (Maven/Gradle dependency) or run as RESTful web-service. A CLI tool allows you to run all functions manually.

• It is open source (Apache 2). You can use the code for free and without strings attached.

• It abstracts complexity and low-level functionality via different interfaces (CLI, APIs).

• It is a holistic solution that allows you to build use cases “end-to-end”. There is no need to research, combine or tweak different libraries to build pilots or production systems.

• It is modular, composable and built on open standards allowing you to customize and extend functionality with your own or third party implementations and to preventing lock-in.

• It is flexible in a sense that you can deploy and run it on-premise, in your (multi) cloud environment or as a library in your application.

Feature List

Find a full overview of all features here

The Storage Kit provides diverse functionality that can be segmented into three layers (based on the Confidential Storage specs and requirements by the Decentralized Identity Foundation):

• Layer 1 consists of a client-server system with capabilities of encrypting data in transit and at rest.

• Layer 2 consists of a system that is capable of sharing data among multiple entities of versioning and replication and of performing privacy-preserving search in an efficient manner.

• Layer 3 consists of high-level server-side functions that work on top of the encrypted credentials.

The following graphic illustrates these three layers and offers a functional perspective:

The client REST API is still in beta testing.

Currently it is recommended to include the JVM client library as a dependency using Maven or Gradle.

Please let us know if you are interested in using the client in non-JVM languages, so we can prioritize this feature accordingly.

The following sections elaborate the architecture from a systematic and functional perspective. Also, components and dependencies are described.

We embraced the controller-service model for all REST components: The web server maps a specific route to a specific controller, which itself does nothing other than handling the request and response data, it does not contain any business logic. All of that is separated into service managers.

Also, recognizable for all methods suffixed with -Docs in the controllers, is that our OpenAPI documentation, is configured and fed in-code, directly beneath the actual code. Having the code and it’s documentation nearly makes it very easy to maintain this documentation, e.g. to keep it up-to-date when the code changes, which can be forgotten easily if it was in external configuration files with special syntax.

There are different ways to get started in using the Storage Kit.

Ways of getting the Storage Kit:

• Docker: Quick way to launch and try out the latest builds. No build environment required.

• Local Build: Build and run the Storage Kit locally. Requires a JDK 16 build environment including, Gradle.

• Dependency (JVM): The Storage Kit can be used directly as JVM-dependency via Maven or Gradle.

Where to use the Storage Kit:

• CLI Tool: The Storage Kit comes with a command-line interface (CLI) tool, which offers a rich set of commands to run the entire functionality the Storage Kit provides. The CLI tool can be used by running the Docker container or the executable by the local build.

• REST API: In case you want to run the Storage Kit as a service, your application can access all functionalities via the REST API.

• Dependency (JVM): The Storage Kit can be used directly as JVM-dependency via Maven or Gradle.

Data models for data requests (serialized to and from JSON):

@Serializable
data class DataRequest(
    val context: String,
    val preferredDataType: String,
    val did: String,
    val responseUrl: String
)

@Serializable
data class DataResponse(
    val filesIndex: Map<String, String>,
    val capabilities: List<String>,
    val baseUrl: String,
    val edvId: String
)

The rest API is largely defined by the service provider.

It simply has to adhere to:

• having at least one POST endpoint

• with the DataResponse as request body

Docker Container

Building the Docker Container

./storagekit.sh build-docker

or with Podman

./storagekit.sh build-podman

or without script (manually) with docker:

docker build -t storagekit .

or with Podman

podman build -t storagekit .

Docker Compose

Run as RESTful service via Docker Compose:

docker-compose build
docker-compose up

For building the project JDK 16+ is required.

Wrapper

The walt.id wrapper script storagekit.sh is a convenient way for building and using the library on Linux.

./storagekit.sh {build|build-docker|build-podman|extract|execute (default)}

The script takes one of the the following arguments:

build|build-docker|build-podman|extract|execute.

For example, for building the project, simply supply the "build" argument:

./storagekit.sh build

Gradle

Manually with Gradle:

gradle clean build

After the Gradle build you can run the executable.

In build/distributions/ you have two archives, a .tar, and a .zip.

Extract either one of them, and run waltid-storagekit-1.0-SNAPSHOT/bin/waltid-storagekit.

cd build/distributions
tar xf waltid-stroagekit-1.0-SNAPSHOT.tar    # or unzip for the .zip
cd ../..  # go back to the root-directory

./build/distributions/waltid-storagekit-1.0-SNAPSHOT/bin/waltid-storagekit

Gradle

implementation("id.walt:waltid-storage-kit:VERSION")

Maven

  <dependency>
        <groupId>id.walt</groupId>
        <artifactId>waltid-storagekit</artifactId>
        <version>VERSION</version>
   </dependency>

Required Maven repos:

https://maven.walt.id/repository/waltid/

For building the project Gradle 7 as well as JDK 16 (or above) is required.

Building the application

First clone the Git repo and switch into the project folder:

git clone https://github.com/walt-id/waltid-storage-kit.git
cd waltid-storage-kit/

Services come with their own configuration files.

For the configuration of service -> implementation mappings, ServiceMatrix is used.

The default mapping file is "service-matrix.properties", and looks like this:

id.walt.services.essif.didebsi.DidEbsiService=id.walt.services.essif.didebsi.WaltIdDidEbsiService
id.walt.services.vc.JsonLdCredentialService=id.walt.services.vc.WaltIdJsonLdCredentialService
id.walt.services.vc.JwtCredentialService=id.walt.services.vc.WaltIdJwtCredentialService
id.walt.services.crypto.CryptoService=id.walt.services.crypto.SunCryptoService
id.walt.services.keystore.KeyStoreService=confidentialstorage.ssikitext.EncryptedKeyStore:crypto.conf
id.walt.services.key.KeyService=id.walt.services.key.WaltIdKeyService
id.walt.services.jwt.JwtService=id.walt.services.jwt.WaltIdJwtService
id.walt.services.vcstore.VcStoreService=id.walt.services.vcstore.FileSystemVcStoreService
id.walt.services.hkvstore.HKVStoreService=id.walt.services.hkvstore.FileSystemHKVStore:fsStore.conf
id.walt.services.context.ContextManager=id.walt.services.context.WaltIdContextManager
id.walt.signatory.Signatory=id.walt.signatory.WaltIdSignatory:signatory.conf
id.walt.custodian.Custodian=id.walt.custodian.WaltIdCustodian
id.walt.auditor.Auditor=id.walt.auditor.WaltIdAuditor
id.walt.services.essif.jsonrpc.JsonRpcService=id.walt.services.essif.jsonrpc.WaltIdJsonRpcService

e.g., to change the keystore service, simply replace the line

id.walt.services.keystore.KeyStoreService=id.walt.services.keystore.SqlKeyStoreService

with your own implementation mapping, e.g. for the Azure HSM keystore:

id.walt.services.keystore.KeyStoreService=id.walt.services.keystore.azurehsm.AzureHSMKeystoreService

To add a service configuration:

id.walt.services.keystore.KeyStoreService=id.walt.services.keystore.SqlKeyStoreService:sql.conf Service configuration is by default in HOCON format. Refer to the specific service on how their configuration is laid out.

The following sections elaborate more advanced conepts like Searchable Symmetric Encryoption (SEE), Authorization Capabilities, zCaps and Linked Data (LD) frameworks.

Installation & Running the Project

Make sure you have Docker or a JDK 16 build environment including Gradle installed on your machine

git clone https://github.com/walt-id/waltid-storage-kit.git

cd waltid-storage-kit/

docker build -t storagekit .

docker run storagekit

git clone https://github.com/walt-id/waltid-storage-kit.git

cd waltid-storage-kit/

./storagekit.sh build

alias storagekit="./build/distributions/waltid-storagekit-1.0-SNAPSHOT/bin/waltid-storagekit"

storagekit client

Examples

See also: Client CLI examples

Document options (document, doc, d):
  - document load   - Load a document from this EDV, cache it locally (gets updated automatically on notification)
  - document create - Create (sequence 0) a new document in this EDV and publish it to to EDV peers (e.g. backups)
  - document update - Recreate, resequence, and publish update to EDV peers (e.g. backups)
  - document delete - Unlink document from EDV, optionally remove all backups at replica nodes (notify peers)
  - document purge  - Purge document in all versions from EDV, optionally remove all backups at replica nodes (notify peers)
  - document cache  - Show cached documents
  - document search - Start remote-run encrypted-search operation

Index options (index, i):
  - index tree      - Displays the whole index tree (all EDVs, all documents)
  - index show      - Displays the index of the selected EDV
  - index documents - Filters the index of the selected EDV to base documents

EDV options (edv, e):
  - edv delegate - Delegate certain permissions to another keypair
  - edv add      - Create a new EDV to add to the session
  
  - Notification options:
    - edv notifications connect    - Connect to the notification channel for this EDV
    - edv notifications disconnect - Disconnect from the notification channel for this EDV

Data request options (datarequest, dr):
  - datarequest - Accept a data request

Session options (session, s):
  - session info   - Display stored information about the current session
  - session export - Exports the current session to use on another device, or as a backup
  - session switch - Switch to another session

Client options:
  - help/?    - Show this help
  - history   - Show command history
  - exit/quit - Quit the Confidential Storage Interactive Console Interface
                
> sess01 > 

Cryptography

We are using Nimbus JOSE and JWT for all JWE (JSON Web Encryption) related purposes, as it is the most popular JOSE Java library4 and the project team has already had experience with this library.

Bouncy Castle is used as cryptography-provider, providing Java Cryptography Extension (JCE) and Java Cryptography Architecture (JCA) implementations. It was chosen as they are one of the most commonly used - and thus tightly integrated - layer for cryptography in Java (and C), providing audited APIs, parts of it even being certified for FIPS-140-25 (Level 1).

Our is used for signing ZCap-LD capability delegations and invocations.

Storage

To store the data at Layer A on the system, TPM-backed full disk encryption can be optionally utilized to disallow metadata access even in case of a physical breach. Clients are also secured with passphrase-based symmetric encryption, using PBES2 (published by RSA Laboratories).

Confidential Storage as a general system involves various actors that have to interoperate:

• EDVs - Encrypted Data Vaults allowing to store data, managing data access and performing queries on the stored data.

• Users - Human or nonhuman actor that wishes to access their EDVs.

• Clients - Programs or libraries supporting access to conform EDVs.

• Providers - Servers that provide an environment to create and manage EDVs.

• Services - Instances besides the user that wish to access data on a specific users EDV (needing the users consent to do so).

A user uses one or more clients to access to their EDVs (multiple clients devices may be used to access data using multiple devices, e.g. stationary and mobile, or for redundancy, to avoid loosing data access if one device fails).

Clients may be connected to multiple EDVs (for redundancy or performance reasons).

Services are a special kind of client that access EDVs through Service Data Requests (which is described later on). EDVs are hosted by providers, and are created through specific requests sent by a client.

A client supports multiple EDV-groupings by utilizing a session-based storage container. It will be referred to simply as "session" hereafter.

Such a session consists of a unique identifier, key identifier, Decentralized Identifier (DID), list of EDV metadata as required to interact with the respective EDVs. EDV metadata consists of the EDV identifier, server URL, root delegation (from the EDV DID to the client’s DID) and an index key:

This container is not explicitly defined in the specifications, and was chosen to allow users to easy import and export of all relevant data to multiple of their devices.

Note that this container as a whole can be exported from a client and imported into other clients, allowing users to easily access their connected EDVs on multiple devices. To do so, the session will be encoded as JSON and encrypted to a JWT, using the clients master passphrase as the symmetric key.

When a client instance is started for the very first time, a number of things have to setup first to allow creating a EDV at a provider:

1. A master key has to be setup. For human-facing clients, this key is derived from a master passphrase. This symmetric master key will be used to encrypt all data-at-rest of the client instance.

2. A session is created. This session is initialized with a new Ed255191 based EdDSA public-private key-pair for requests to services and EDVs, and authorization with ZCaps.

3. This key is used to create the session DID - also known as "controller DID".

4. The controller DID is used to request a new EDV at a chosen provider. The request contains data about the client, most importantly the did:key.

5. The key receives the initial capability delegation from the root of trust. Several attributes are generated (e.g. IDs, sets up a did:key for the EDV) for the EDV.

When uploading a document, the users client will store a file key to the encrypted file index, chunk the file (explained below) and encrypt the chunks with the file key.

Encrypted search

1. The underlying encrypted search implementation parses the document structure (depending on file format, e.g. JSON, XML, etc.)

2. and creates a list of keywords that were found in the file. This is the search index for this file.

3. It gets encrypted with the encrypted search key. This is the encrypted index.

Chunking

1. The document is split into chunks. Chunks may not have a size exceeding the maximum of 1 MiB each. This is restricted per the Confidential Storage specification document.

2. Each chunk is individually encrypted with authenticated encryption using a file key.

3. An index (Resource Structure) is created, which is used to be able to recreate the file from the individual chunks later on. It gets encrypted, then being the encrypted chunk index (Encrypted Resource Structure).

Chunk transmission to the EDV

1. The encrypted chunks are sent to the EDV using individual request authorizations using ZCaps.

2. The encrypted chunk index and encrypted search index get stored in the EDV (each also being authorized using ZCap capability invocations).

All commands can be displayed by entering help at the session command prompt.

First startup

Master passphrase setup

[Master passphrase setup]

You have not setup a master passphrase.
A new one will be setup.

Please enter your desired master passphrase: 123456

Session wizard

[Add session]

Welcome to the session setup wizard!
Please follow the instructions below:

Enter new session identifier or enter "import" to import an exiting session.
Enter session identifier: session1

Create first EDV

Creating first EDV for session:
Enter provider url [http://localhost:7000]:

(you can just press enter here to go with the default,
otherwise, enter the remote URL)

EDV created: kGG5MGziJsak
Session "session1" was stored to local session saves.

The main CLI

[Session session1]

> session1 > 

Enter help for a list of all possible commands.

history shows the command history.

exit or quit will leave the interactive CLI.

Session management

Displaying information about the currently active session

> session1 > session info

[Session info]

ID:     session1
DID:    did:key:z6MkeVJPhuvkmKrXLdopdxbeX5XXQJSLaD4V21on9G9LY2Fa
Key ID: 187f31477eaa45978597c3520e1d5b8e

Linked EDVs:
1: kGG5MGziJsak @ http://localhost:7000

Exporting the currently active session

> session1 > session export

[Session export]

Exporting session "session1"...

Session token for "session1": eyJwMnM<snip>...
This session token is encrypted with your master passphrase.
The same master passphrase has to be used to decrypt the session token
  in the other client.

Switching the current active session

> session1 > session switch

[Select session]

Cached sessions:
- 1. session1
Type 'add' to add additional sessions using the session wizard.

Enter session identifier or index: 1

Importing a session from the session selection:

...
Enter session identifier or index: add

[Add session]

Welcome to the session setup wizard!
Please follow the instructions below:

Enter new session identifier or enter "import" to import an exiting session.
Enter session identifier: import

[Import session]

Enter session token: eyJwMnM...

Document management

Creating a document

> session1 > document create
Document identifier: doc1
Path/to/content.file: doc.txt
Generating file key...
Mapping chunks...
(Content size 30, chunks needed: 1)
Uploading EncryptedResourceStructure...
Done. Creating encrypted search index...
Uploading encrypted search index...
All done. File "doc1" is now stored to EDV "kGG5MGziJsak"!
Successfully created document!

Loading a document

> session1 > document load
Document identifier: doc1
Retrieved document "doc1" (below):
this is some document content

like document update.

Searching for a document

> session1 > document search
Enter keyword: content
Searchable Symmetric Encryption done.
Keyword "content" was found 1 time(s):
- 1. doc1

Deleting a document

> session1 > document delete
Document identifier: doc1
Successfully deleted document: doc1

Data requests