The purpose of this post is to give insights about the workings under the hood of OpenFire and assumes you already know about it’s existence.
Never heard of OpenFire? See: http://codebuffet.co/2014/09/08/announcing-openfire-revolutionary-database-backend-server/ for a introduction.
How does it work?
I like to learn by example, and that’s what we are going to do here. The way this paragraph is organized is by showing an example code in OpenFire, then explain how it works under the hood.
This set of examples + explanation is divided in 3 tasks OpenFire does: Writing, Reading, and Syncing data.
For the next paragraphs, Suppose we start off with a HTML file like this:
<!DOCTYPE html>
<html>
<script src="http://openfi.re/openfire.js"></script>
<script>
// Intialize our DB
// tech-demo is our namespace for this project
var db = new OpenFire("http://localhost:5454/tech-demo");
</script>
</html>
This code will load the JavaScript SDK, set up a real-time connection with the server, and registers the client with the server, ready to send or receive any new data.
Writing data
Writing data in OpenFire happens with the ‘set’ command. Let’s imagine we want to register a user.
db.child("users").push().set({
username: "peter",
email: "[email protected]",
settings: { newsletter: true, rights: 'admin' }
})
Child brings us a path deeper, in this case, in the users path. See paths kind of like folders on your computer.
Push creates a new child and assigns a random unique ID, to make users kind of behave like a list.
Set writes down the object, overwriting anything within it’s path.
As you can tell from the example above, you can see that we are writing an object with 2 primitive types (strings, numbers etc) and 1 object (the settings object), which contains primitive types again.
After writing, our database looks like this (note the randomly generated id):
{
"users": {
"o1vwFb2qtotSV_Yukq+i8rKsiMtxn9OZUimZ": {
"settings": {
"newsletter": true,
"rights": "admin"
},
"username": "peter",
"email": "[email protected]"
}
}
}
Users is now a rather small object, but you can imagine that with more than 100.000 users, the users object could become quite big. Fortunately, OpenFire flattens objects before being stored! The way this works is that objects and primitive types are split up in segments. Each object will be stored in it’s own path (think of the folders again like we talked about) I found this the best balance between scalability and storage efficiency.
Now let’s see how OpenFire stores our little Users objects:
OpenFire [Server] -> Memory is now:
{
"/tech-demo/users/o1vwFb2qtotSV_Yukq+i8rKsiMtxn9OZUimZ": {
"username": "peter",
"email": "[email protected]"
},
"/tech-demo/users/o1vwFb2qtotSV_Yukq+i8rKsiMtxn9OZUimZ/settings": {
"newsletter": true,
"rights": "admin"
}
}
The technology is called a path-value store. It’s a mix between a filesystem on your computer with folders and files, and a regular dictionary (key-value storage).
It allows ultra fast writes, because the server doesnt have to check if a parent object exists if you are writing a nested object.
Reading Data
When reading data in OpenFire, it will search for a object under the path what you are looking for, and, if it finds something, it will rebuild the object back to it’s original state. Let’s say you want to get the contents of ‘users/o1vwFb2qtotSV_Yukq+i8rKsiMtxn9OZUimZ’:
db.child("users").once("value", function(snapshot){
console.log(snapshot.val);
})
The best way to explain this is by using this gif:
It’s such a simple concept that brings tons of possibilities! Imagine it like folders and files on your computer again. Say you have everything about your users in 1 file, and you need to know the email of a user called ‘joe’. You would have to open the entire file in the memory of your computer, then start searching for this user.
Now imagine the same scenario, but then you got your users stored in folders. One folder for each user. Now you only have to open the folder belonging to joe, and open his file. This setup not only allows you to search faster, but it also takes less memory on your computer to do so. This is exactly what happens on OpenFire.
Syncing data
OpenFire sends data back and forth to it’s clients using real-time, serial data pushes. No HTTP API, no REST, no overhead.
Apart from using this to send and read data, you can also use it to sync in real-time. The way this works is by setting up a WebSocket connection, with a long-polling fallback for older browsers. This set-up happens as soon as you connect with your database. To sync with the database and other clients you subscribe to events.
db.child("users").on("child_added", function(snapshot){
console.log(snapshot.val);
})
This code code will call the function each time a new child is added to the users object.
OpenFire keeps track on what objects users are subscribed to and how many. Using this knoweldge, OpenFire only checks objects for changes that are subscribed on. This makes the publish-subscribe system extremely fast, because no unnecessary loops and checks are done in the background. Objects that are not subscribed on are written directly, objects that are subscribed on, will only be checked on the places that matter. These subscriptions are stored in the so-called Meta Table. This is a special object, that exists in every OpenFire instance, that resides in /_meta in. the root of your database. It is the warehouse for server-related data like configuration and in our case the subscription to this event.
No one can write to this object directly, only the server can.
Wrapping up
I hope I have given enough insights to show how OpenFire works. In case you would like to know more, feel free to comment. I’ll be happy to explain more of it’s inner workings!
