Storing Data

The Finsemble Storage Client interfaces with the Storage Service to provide access to a central location for persistent storage and retrieval of application data.

The Storage Client delivers one form of cross-origin resource sharing (CORS), specifically cross-origin storage. However, by using our Storage Client, different data stores can be transparently used:

  • Multiple users can be supported
  • Future storage enhancements can be inherited (e.g., storage backup, migration, permissions)
  • You can directly access HTML5's localStorage, IndexedDB, and other APIs from your component window

The Storage Client API supports a key-value schema. Each call to get and save must include a key.

All data is keyed off of a username, which is specified by the Authentication Service invoking the StorageClient.setUser method during startup. Because the username is always set automatically, you never need to do it. If authentication is disabled, the underlying Authentication Service sets the username to defaultUser.

The high-level storage architecture looks something like this:

Topics

Key-value is a simple concept. However, sometimes you need a little more flexibility than key-value provides. To allow further partitioning of data, we allow you to pass in a topic. Topics act like a namespace and help prevent key conflicts. All of Finsemble's core storage uses one of the three topics described below. You can use any topic you choose (e.g., an application or company name).

Topics can be mapped to specific storage adapters. For example, you may have a component that has a MySQL backend and another with a Redis backend. It's up to you.

By default, Finsemble saves to three topics which you can extend via config.

  • finsemble.workspace is storage for the workspace and is only accessed when workspaces are initialized or explicitly saved by the user.
  • finsemble.workspace.cache is temporary storage for high-frequency updates (e.g., window moves, resizes, opens, closes).
  • Finsemble is a general topic that contains everything outside of the workspace. User preferences fall under this topic.

Configuration

In your Finsemble config, you can specify storage adapters in the "servicesConfig.storage" setting. You can (and should) define your own topics for storage. The topics you define can be mapped to any adapter by inserting a corresponding entry under the topicToDataStoreAdapters property inside config.json. The property name (e.g., finsemble.workspace) is the topic, and the value is the adapter (e.g., IndexedDB). You can specify the default adapter using the defaultStorage setting:

"servicesConfig": {
	"storage": {
		"defaultStorage": "IndexedDBAdapter",
		"topicToDataStoreAdapters" : {
			"Finsemble" : "IndexedDBAdapter",
			"finsemble.workspace" : "IndexedDBAdapter",
			"finsemble.workspace.cache" : "IndexedDBAdapter"
		},
		"dataStoreAdapters": {
			"LocalStorageAdapter": "$applicationRoot/adapters/localStorageAdapter.js",
			"IndexedDBAdapter": "$applicationRoot/adapters/indexedDBAdapter.js"
		}
	}
	.....
}

Storage config is processed during the initialization of the Storage Service, so all related config settings will take place before any Storage Clients are initialized. If a topic doesn't have a storage adapter in the config, it defaults to the "defaultStorage" value. If no "defaultStorage" value exists, it then defaults to localStorage.


Custom storage adapters

The Storage Service receives requests from the Storage Client to save data under a particular topic and key. The Storage Service receives the request, determines which adapter should handle the request, and passes on the information. When the adapter has completed the operation, the Storage Service returns a message that everything is done.

If you want to use an adapter that uses a remote database (e.g., MongoDB, Oracle's relational database, etc.), you simply need to build an adapter that converts key-value requests into something that your backend can handle.

In general:

  • To build your adapter in the seed project, add its code to the folder /src/adapters. You can copy the localStorageAdapter from /src-built-in/adapters/localStorageAdapters.js or use the sample code below.
  • Once implemented, you must configure Finsemble to use your custom adapter in /configs/application/config.json under the finsemble.servicesConfig.storage section.
  • You must create an entry for the build process in the file /build/webpack/adapters.entries.json.

Note: The localStorageAdapter and IndexedDB adapter use the BaseStorage adapter to generate keys to store data. The BaseStorage adapter combines several variables available to it and can inherit the set methods for a couple of those variables, namely: setBaseName() and setUser(). The compound key generation function is simply:

	// return full underlying key (based baseName + userName + topic + key)	this.getCombinedKey = function (self, params) {		return self.baseName + ":" + self.userName + ":" + params.topic + ":" + params.key;	};

Creating a sample storage adapter

To illustrate how easy it is to create your own adapter, we're going to create an InMemoryStorageAdapter. Basically, the adapter will save data to memory. When you quit the app, nothing will be saved. This has limited practical purpose, but it neatly illustrates the ideas and code required to build a functional (and useful) storage adapter.

We don't want to break our example project, so we're going to write data to a custom topic. All data saved on that topic will go to our InMemoryStorageAdapter.

Create the file

Create a file called InMemoryStorageAdapter.js in the folder src/adapters.

Here's some sample code to get started:

/*
* The baseStorage model provides several utility functions, such as `getCombinedKey`, which will produce a compound key string (for use with a simple key:value store) incorporating the username, topic, and key. For example: For the default user, Finsemble topic and activeWorkspace key: `Finsemble:defaultUser:finsemble:activeWorkspace.`
.
*/
var BaseStorage = require("@chartiq/finsemble").models.baseStorage;
var { Clients: { Logger } } = require("@chartiq/finsemble");
//Because calls to this storage adapter will likely come from many different windows, we will log successes and failures in the central logger.
Logger.start();

const InMemoryStorageAdapter = function () {
	// #region Initializes a new instance of the InMemoryStorageAdapter.
	BaseStorage.call(this, arguments);

	this.myStorage = {};

	/**
	 * Save method.
	 * @param {object} params
	 * @param {string} params.topic A topic under which the data should be stored.
	 * @param {string} params.key The key whose value is being set.
	 * @param {any} params.value The value being saved.
	 * @param {function} cb callback to be invoked upon save completion
	 */
	this.save = (params, cb) => {
		//need to implement
		return cb(null, { status: "success" });
	};

	/**
	 * Get method.
	 * @param {object} params
	 * @param {string} params.topic A topic under which the data should be stored.
	 * @param {string} params.key The key whose value is being set.
	 * @param {function} cb callback to be invoked upon completion
	 */
	this.get = (params, cb) => {
		//need to implement
		return cb(null);
	};

	/**
	 * Returns all keys that we're saving data for.
	 * @param {*} params
	 * @param {*} cb
	 */
	this.keys = (params, cb) => {
		//need to implement
		return cb(null);
	};

	/**
	 * Delete method.
	 * @param {object} params
	 * @param {string} params.topic A topic under which the data should be stored.
	 * @param {string} params.key The key whose value is being deleted.
	 * @param {function} cb callback to be invoked upon completion
	 */
	this.delete = (params, cb) => {
		//need to implement
		cb(err, null);
	};

	/**
	 * This method should be used very, very judiciously. It's essentially a method designed to wipe the database for a particular user.
	 */
	this.clearCache = (params, cb) => {
		//need to implement
		return cb();
	};

	/**
	 * Wipes the storage container.
	 * @param {function} cb
	 */
	this.empty = (cb) => {
		//todo need to implement
	};
}

new InMemoryStorageAdapter("InMemoryStorageAdapter");

Note: You only need to implement the save, get, keys, and delete functions. The functions clearCache and empty are optional, but can be useful for clearing out data during testing; they should probably be disabled on the server side in production.

Config

Right now this file is doing nothing. To make it useful, you need to modify some code. First, all requests for the topic IntraSession should go to the new adapter. Second, you need to modify where the actual adapter sits (under dataStoreAdapters). Lastly, you need to put the file in the build.

To do this, go to config.json and change servicesConfig.storage as below.

	"storage": {
		"topicToDataStoreAdapters" : {
			"Finsemble" : "InMemoryStorageAdapter",
			"finsemble.workspace" : "InMemoryStorageAdapter",
			"finsemble.workspace.cache" : "IndexedDBAdapter",
			"IntraSession": "InMemoryStorageAdapter"
		},
		"dataStoreAdapters": {
			"LocalStorageAdapter": "$applicationRoot/adapters/localStorageAdapter.js",
			"IndexedDBAdapter": "$applicationRoot/adapters/indexedDBAdapter.js",
			"InMemoryStorageAdapter": "$applicationRoot/adapters/InMemoryStorageAdapter.js"
		}
}

Note: In practice, you should configure your new adapter for use for the finsemble and finsemble.workspace topic. When the user does a formal save operation, the data is copied from the finsemble.workspace.cache topic to the finsemble.workspace topic, copying it to remote storage. For the purposes of this example, the InMemoryStorageAdapter will not persist data.

Build

Next, if you are using the built-in Webpack/gulp build system provided by Finsemble, add the following definition to your build/webpack/webpack.adapters.entries.json file:

	"InMemoryStorageAdapter": {
		"output": "adapters/InMemoryStorageAdapter",
		"entry":"./src/adapters/InMemoryStorageAdapter.js"
	}

Implement adapter functions

To start, you have a member variable on our class called myStorage. It's a simple object that does nothing interesting. To make the adapter have simple functionality, give it the ability to save and retrieve data. Add in the following code in the appropriate places. Take a second to read through what's going on here before implementing these two methods in your src/adapters/InMemoryStorageAdapter.js.

	this.save = (params, cb) => {
		//Retrieves a key that looks like this:
		//applicationUUID:userName:topic:key
		const combinedKey = this.getCombinedKey(this, params);

		//Assign the value to the key on our storage object.
		this.myStorage[combinedKey] = params.value;

		return cb(null, { status: "success" });
	};

	this.get = (params, cb) => {
		const combinedKey = this.getCombinedKey(this, params);
		const data = this.myStorage[combinedKey];
		let err = null;
		if (!data) {
			err = `No data found for key ${params.key}`
		}
		return cb(err, data);
	};

Using authentication credentials with a storage adapter

In order to use credentials from Finsemble's Authentication service in your storage adapter, you must wait for authentication to complete and then use the Authentication Client to retrieve the credentials. The following pattern may be used:

const authStateHandler = function(err, notify) {
    if (err) {
        Logger.error("Received an error from AuthorizationState PubSub, error:", err);
    } else {
        let authStateData = notify.data;
        if (authStateData.state == "done") {
            AuthenticationClient.getCurrentCredentials(function(err, credentials) {
                if (err) {
                    Logger.error("Received an error when retrieving current credentials: ", err);
                } else {
                    //Do something with the credentials object here
                }
            });
        } else {
            Logger.log("Current auth state: " + authStateData.state);
        }
    }
};
RouterClient.subscribe("AuthorizationState", authStateHandler);

Testing the adapter

Make sure this thing is working. Generate a sample component using the Finsemble CLI. At your command prompt, use >finsemble-cli add component adapterTest. Put the following code into the JavaScript file created in src/components/adapterTest:

  function setData(key, value) {
	FSBL.Clients.StorageClient.save({ key: key, value: value, topic: "IntraSession" }, (err, data) => {
		console.log("StorageClient.set callback invoked");
		if (err) {
			console.error("StorageClient.save error for key", key, err);
		} else {
			console.log("Data saved successfully!");
		}
	});
}

function getData(key) {
	FSBL.Clients.StorageClient.get({ key: key, topic: "IntraSession" }, (err, data) => {
		console.log("StorageClient.get callback invoked");
		if (err) {
			console.error("StorageClient.get error for key", key, err);
		} else {
			console.log("Data found for", key, "data:", data);
		}
	});
}
//The component gets Webpacked. If you don't explictily make the function global, you won't be able to interact with it in the console later on.
window.getData = getData;
window.setData = setData;

Then, start Finsemble and spawn the sample adapterTest component. When adapterTest is loaded, click it and type CTRL+SHIFT+I. This will open the Chrome developer console so that you can test your adapter. Let's see what it does with a couple types of data:

setData("Chicken_Names", ["Edna", "Peepy", "Taylor", "Violet", "McKenna", "Harold"])
setData("Sporks", { seemUseful: true, areUnderUtilized: true })

When you see the callback logged for both sets, try to retrieve the data. First, reload the component by pressing CTRL+R (this clears the window object). This demonstrates that you didn't save this data locally. Once your component reloads, see what comes back when you try to retrieve the list of chicken names that we saved earlier. Enter the code below into your console.

getData("Chicken_Names");

Edna and her friends have flown all the way from the Storage Service back to a component. Without our storage adapter, this wouldn't be possible. While this is a silly example, it tackles the basics.


check   The Storage Client API supports a key-value schema, but you are encouraged to attach your own storage adapter that bridges to another database model.
 

Further reading

Advanced information about storing data can be found in the Storage Client documentation.

Read the Router tutorial to better understand how Finsemble sends and receives messages.