This document describes the network level architecture of MckoiDDB and gives practical examples of installing MckoiDDB over a private network.
Throughout this document the following terms will mean;
Before starting we will briefly cover the software and hardware capabilities needed to support a Mckoi Machine Node. Each machine will need a local file system with available space, an operating system that supports Java 1.6, and the machine must be connected to a private network. If the machine is connected to the Internet and has a public IP address, it should also have a second network interface to a private high bandwidth/low latency network with a private IP address.
If you do not own a network, the hardware that meets the MckoiDDB specifications can be leased from several dedicated and virtual Linux service providers such as Amazon's EC2, Linode, and Rackspace's Cloud Servers.
Note that while it is possible to create a MckoiDDB network where the machine nodes communicate with each other over the Internet, this kind of setup is very much discouraged. MckoiDDB expects a high quality low latency network for inter-machine communication. In addition, messages passed between MckoiDDB nodes are not encrypted.
All machines nodes have a configuration file called network.conf. The network.conf file contains the list of all the IP addresses of machines that are allowed to communicate with the node, and a list of all the other nodes in the network. This configuration file can be stored on a shared file system or published on an internal HTTP server so it only needs to be updated once whenever a machine is added or removed. The network.conf file defines three properties;
The following is an example network.conf for a network of 5 Machine Nodes. The Machine Nodes are located on IP addresses 192.168.30.0 through 192.168.30.4 on port 3900. The machines allowed to communicate with the network have IP addresses 192.168.30.0 through 192.168.30.11.
# The list of IP addresses allowed to connect to the network connect_whitelist = 192.168.30.0, 192.168.30.1, \ 192.168.30.2, 192.168.30.3, \ 192.168.30.4, 192.168.30.5, \ 192.168.30.6, 192.168.30.7, \ 192.168.30.8, 192.168.30.9, \ 192.168.30.10, 192.168.30.11 # The list of Machine Nodes on the network network_nodelist = 192.168.30.0:3900, 192.168.30.1:3900, \ 192.168.30.2:3900, 192.168.30.3:3900, \ 192.168.30.4:3900 # The time, in seconds, between checks of this # configuration file for changes. Defaults to 2 minutes. configcheck_timeout=120
A MckoiDDB installation must have a unique password defined called the 'network password'. The network password is used internally by the system whenever one machine node communicates with another, or when a database client connects to the network. The purpose of this network password is to help ensure there can be no misconfiguration where one MckoiDDB installation accidentally communicates with a separate one running on the same network. The network password must be set on each machine node and given to each database client that wishes to connect to the MckoiDDB network. Details of where the network password is set is given below.
All machine nodes must also have a node.conf configuration file. This file is used to define system specific settings such as the location of the data and log directories in the local file system, and what the MckoiDDB network password is. The node.conf file is deployed with each Mckoi Machine Node installation. If this configuration file is changed, the Machine Node must be restarted before the change takes effect. Fortunately the type of details in this configuration are static in nature and it is not expected this file will change after being deployed.
The following is an example node.conf that stores data and logs at the /mckoiddb/ subdirectory in the local file system. The network_password property is left for you to define;
# Choose a network password, network_password = # The location in the local filesystem of the directory # that stores stateful data of the roles running on the # machine. node_directory = /mckoiddb/data # The directory location in the local filesystem to store # logging information (uses the internal Java logging # format). log_directory = /mckoiddb/log # The minimum log level to report (inclusive), log_level = info
After the network.conf and node.conf files have been set, you are ready to install the Machine Node on a server. To start a Machine Node, upload the MckoiDDB distribution to the server and run the following Java command from the server's operating system shell (the following example would be run on the machine at 192.168.30.0);
Note: the command is entered as a single line
java -cp lib/mckoiddb.jar com.mckoi.runtime.MckoiMachineNode -nodeconfig node.conf -netconfig network.conf -host 192.168.30.0 -port 3900
Found in the distribution is an init.d style Daemon script for Linux. Follow the instructions in the distribution for using the script to start up the Machine Node at system boot.
Once all the Machine Nodes have been installed and are running, the nodes need to be given one or more roles to perform as part of the operation of the network. Before we do this, a quick overview of the architecture follows.
All data that is stored in a MckoiDDB network is saved in block files. Block files are fairly large files (usually multiple megabytes) located in the block/ sub-directory of the Machine Node's data directory. When data is written into a block file, MckoiDDB will try and make sure the block is replicated on at least three machines in the network. The reason for replicating the data is to ensure data continues to be available even when two machines fail or need to be taken down for maintenance. Additionally, replication gives multiple locations for clients to go to fetch data thereby providing load distribution and increasing performance.
To illustrate this, the diagram below shows a set of four blocks distributed over a network of five devices where each block is replicated three times. Any two devices in the network can fail yet the entire block file set will still remain available.
|Block #||Where block is found|
|1||Machine 1, Machine 3, Machine 5|
|2||Machine 2, Machine 3, Machine 5|
|3||Machine 1, Machine 2, Machine 4|
|4||Machine 1, Machine 4, Machine 5|
In MckoiDDB, both structural meta-data and user data are stored in these distributed block files. The block files are stored in a repository that database clients can connect to and read and write. In MckoiDDB terminology, the system role that maintains a block file repository on a machine is called the Block Server Role. The Block Server Role is one of three roles that is assignable to a Machine Node. The other two roles are the Manager Server Role and the Root Server Role.
The Manager Server Role is a service in the network that maintains a small database about the location of blocks in the network. The manager server resolves queries such as 'which servers should I go to find the block files for block n'. Additionally, a manager server is notified of any machine failures and serves as a central location for discovering real-time information about the current status of machines in the network. A new client on the network must query the manager server to discover information about network topology, such as the address of other nodes and the location of named partitions.
Finally, the Root Server Role manages a group of Path Instances. A Path Instance is basically a log of historical snapshots of a structured data model in the system. A Root Server is needed to manage transactional operations and updates (when a commit happens) on user data. Every update to the state of a Path Instance must be moderated through a root server to ensure consistency. For example, a Root Server may decide to veto a commit because an integrity rule would be violated by allowing it. This commit proposal and veto process is handled by a function that we call the Consensus Function. The Consensus Function is an important part of the MckoiDDB data model architecture, however a detailed explanation of it falls outside the scope of this document.
A minimal redundant MckoiDDB network must have three Block Servers, one Manager Server and one or more Root Server.
The MckoiDDB Console is used to assign roles to machines in your network. The following command will start the console application (assuming network.conf is in the current directory);
java -cp lib\mckoiddb.jar com.mckoi.runtime.AdminConsole -netconfig network.conf -netpassword [the network password]
You can find more specific information about the console application at the MckoiDDB Console Reference page. In this section we'll just be covering the commands used to assign machine roles.
Assuming you have installed MckoiDBB on a network of 5 machines on IP addresses 192.168.30.0 through 192.168.30.4 and you have setup the network as described previously in this document, the 'show network' console command should display the following;
MckoiDDB> show network MRB RC BC Server ------------------------------------ ... 192.168.30.0:3900 ... 192.168.30.1:3900 ... 192.168.30.2:3900 ... 192.168.30.3:3900 ... 192.168.30.4:3900 5 machines in the network. 0 manager 0 root 0 block machines.
If you receive any errors, it will likely be either because your network.conf file does not include all the machines in the network (including the machine you are running the console application on), or the Machine Node application has not been started on all the machines. See the sections above for the details on how to configure MckoiDDB for your network.
In this demonstration we will assign the Manager Server Role to the Machine Node at 192.168.30.0, a Root Server Role on 192.168.30.1, and the remaining Machine Nodes will be assigned as Block Server Roles. For this setup you would enter the following commands;
start role manager on 192.168.30.0:3900 start role root on 192.168.30.1:3900 start role block on 192.168.30.2:3900 start role block on 192.168.30.3:3900 start role block on 192.168.30.4:3900
Note that you can assign multiple roles to a Machine Node if you wish. You may find in the above setup that the servers running the manager and root server are underutilized, in which case you may decide to assign those machines as Block Server Roles also.
To check the role assignments that were made, the 'show network' console command should now display the following;
MckoiDDB> show network MRB RC BC Server ------------------------------------ M.. 192.168.30.0:3900 .R. 192.168.30.1:3900 ..B 192.168.30.2:3900 ..B 192.168.30.3:3900 ..B 192.168.30.4:3900 5 machines in the network. 1 manager 1 root 3 block machines.
Note the MRB column on the left of the table shows the role(s) assigned to the machine (Manager, Root, Block).
MckoiDDB is now installed and ready to go. At this point you will need to create at least one path instance before a client is able to connect and use the system. The Developing a Simple Database Guide is a good place to start to learn how to create a database application using the client API.