Cassandra Data Distribution using Partitioners

Before wading through the partitioners lets have a look how generally data is being distributed in a Cassandra
cluster. It includes a ring like a topology between the nodes. Here the data are being broken into tokens and are circulated
among the nodes through the Cassandra ring. Data is distributed across the cluster by the value of the token by using the
hash technique. Based on the value of the tokens the data will be evenly distributed among the cluster. The main advantage
of this method is the data retrieval process can be quick based on the token range. Now we can have can see what does a
partitioner role in a cluster in detail.

Partitioners

Partitioners allow how row keys should be sorted and how data will be distributed across your nodes in the cluster. The
read/write request to the cluster are evenly distributed when each part of the hash range receives the same number of
tokens on average. Based on the difference in the hash methods partitioners are classified into three types as below:

• Murmur3Partitioner
• RandomPartitioner
• Byte Ordered Partitioner

Murmur3 Partitioner

This is the default partitioning strategy for Cassandra. It provides fast hashing and good performance. It
uses MurmurHash hash values to distribute the data across the clusters.

Random Partitioner

It involves MD5 hash applied to place the keys on the node ring. An MD5 hash provides a natural way of
load balancing keys to nodes. Each data item is mapped to a token by calculating the MD5 hash of its key
The disadvantage of this method is that it causes inefficient range queries when keys specified in the range
might be in another ring.

Byte Ordered Partitioner

This method involves the distribution of the data lexically by key bytes in an ordered manner. It treats
the data as raw bytes, instead of converting them to strings. This is most likely to use when you want a
partitioner that doesn't want to validate the keys as being strings.

How to change a Partitioner?

So based on the need for the application you can choose any one of the above partitioners. In order
to apply it make the below change in the cassandra.yaml file. This file will be generally located in conf
directory.

So this is all about how a data is distributed among the Cassandra cluster and how partitioner helps more
for the sorting of data & retrieving them. Got any queries about partitioners to comment on them.

Read More

MongoDB Storage Engines

The storage engine is a vital component for managing and storing data in the memory as well as disk. MongoDB supports multiple storage engines with unique features for better performance. In this blog, we are going to discuss the various storage engines and their features.

Types of Storage Engine:

Our production workload will be different for each application, some will be writing intensive, some will be read and some required encryption etc. MongoDB provides flexibility to handle such workloads by providing multiple storage engines. Mentioned the storage engines below.

• Wired Tiger
• MMAPv1
• Encrypted
• In-memory

Let's see the key features of each storage engines.

Wired Tiger:

• Wired Tiger (WT) is the default storage engine from mongo 3.0

• WT storage engine uses document-level concurrency control for write operations so multiple clients can modify different documents of a collection at the same time.
• It uses only intent locks at the global, database and collection levels when the storage engine detects conflicts between two operations, one will incur a write conflict causing MongoDB to transparently retry that operation
• MongoDB utilizes both the WiredTiger internal cache and the filesystem cache. By default the wired tiger cache will use 50% of RAM minus 1 GB or 256 MB.
• Efficient use of CPU cores and RAM
• Allows for more tuning of storage engine than MMAP
• 7 to 10X better write performance
• 80% less storage with compression
• Compression minimizes storage use at the expense of additional CPU.

• Collection level data in the WiredTiger internal cache is uncompressed and uses a different representation from the on-disk format.

MMAP:

• The MMAP Storage engine uses memory mapped files to store its data
• A segment of virtual memory which has been assigned a direct byte-for-byte correlation with some portion of a file
• It is a traditional storage engine that allow great deal of performance for heavy read applications
• Data and indexes are mapped into virtual space
• Data access is placed into RAM
• When the OS runs out of RAM and an application requests for memory,then it will swap out memory to disk to make space for the newly requested data
• The operating system’s virtual memory subsystem manages MongoDB’s memory
• Deployments with enough memory to fit the application’s working data set in RAM will achieve the best performance.

Encrypted:

• Available in mongodb enterprise only.
• The default encryption mode that MongoDB Enterprise uses is the AES256-CBC
• All data files are fully encrypted from a file system
• Unencrypted state in memory and during transmission
• Master keys and database keys are used for encryption
• Data is encrypted with the database keys,master key encrypts the database keys
• Encryption is not a part in replication keys are not replicated
• In replication data is not natively encrypted over the wire
• Application Level Encryption provides encryption on a per-field or per-document basis within the application layer

In-memory:

• It is available in the enterprise editions starting from version 3.2.6.
• Handles ultra high throughput with low latency and high availability

• In-memory storage engine is part of general availability

• More predictable and low latency on less in-memory infrastructure
• Supports high level infrastructure based on zonal sharding
• MongodB rich query capability and indexing support

Third-party pluggable storage engines:

• MongoDB is providing support for 3rd party storage engines as “modules” that can be independently updated.
• When building MongoDB, any storage engine modules will be automatically detected, configured and integrated in the final binaries.
• The RocksDB storage engine is the first one to use this new module system for their MongoDB storage integration layer

• RocksDB for MongoDB is based on the key-value store optimized for fast storage.
• It is developed by facebook and designed to handle write-intensive workloads.

Storage Engine application API:

As mentioned, each application load will be different from other. Choosing the right storage engine will definetly boost the performance. Differentiated storage engine with respect to the workload which helps in choosing the right storage engine.

Comparison chart:

The overall feature comparison for all the storage engines are listed below:

Read More

MongoDB Free monitoring on Community version 4.0 explained

I was just playing around with MongoDB 4.0 community version & noticed a feature called free monitoring. In this article let have a detailed look of free monitoring, how to enable, check status and what are the monitoring metrics that are provided by MongoDB.

Free Monitoring:

Free monitoring is a service that is available in the MongoDB 4.0 Community versions. It provides us with some graphical/statistical data related to currently deployed instances which expire after 24 hours. So let's have a look how can we enable free monitoring & look at what is in it.

Enabling Monitoring:

This command enabled the free monitoring in the instance, one-time enabling is enough it is not necessary to enable each time to check the status. 

db.enableFreeMonitoring()

Check Monitoring status:

In order to check the current free monitoring status the above command which let us know the status. When once enabled, the monitored data is uploaded periodically to the cloud & it can be accessed via browsers through the provided URL. 

db.getFreeMonitoringStatus() 

Graph options:

Let's have a look over the various graph parameters that are included in the free monitoring.

Operation Execution:

This lists out the how many times the operations are being executed in the server. Operations include reads, writes & commands.

Disk Utilization:

Every disk has its own read & write speed. Disk utilization involves the sum of the total speed of data that is being written to and read from the disk. This graph plots out the maximum & average disk used by the drives.

Documents:

This returns the stats of the documents that are returned, inserted, updated & deleted

Memory:

MongoDB uses virtual as well as the resident memory. Resident memory is the memory that is used by RAM. In some cases, the operating system returns the imaginary address when MongoDB requests for memory address which is not real. If the journal is enabled then it returns another address for journal data, both together is the Virtual memory. Journal reference memory alone is known as the mapped memory.

Network IO:

This refers to the total network traffic that is being received and sent by MongoDB in bytes.

Opcounters:

Opcounters are nothing but the total list of the operations count that is being performed by the server. It included the operations such as insert, query, update, delete,getmore, commands.

Replicated Opcounters:

This refers the opcounters that are being replicated to the other secondaries in the replica cluster. The value of replicated opcounters will be present only if the instance is a replica set.

Query Targeting:

This chart depicts the query & objects that are scanned by the process.

Queue:

This parameter lists out the total number of reads,writes that are currently waiting under the queue or waiting for any other lock.

System CPU usage:

It explains about the complete system CPU usage with respect to the some of the specialized factors.

Read More

Consistency levels in Apache Cassandra explained

Cassandra is scalable column-oriented open source NoSQL database. It is the right choice for managing large amounts of structured, semi-structured, and unstructured data across multiple data centers when you need scalability and high availability without compromising performance. In this article, we are going to discuss how the read/write operations are maintained in a cluster and various consistency levels in Cassandra & how can they be applied to our business applications.

According to CAP theorem, it is impossible for a distributed system to simultaneously provide all three guarantees:

• Consistency -Every node contains same data at the same time
• Availability- At least one node must be available to serve data every time
• Partition tolerance -Failure of the system is very rare

"Cassandra is typically classified as an AP system, meaning that availability and partition tolerance are generally considered to be more important than consistency in Cassandra. But Cassandra can be tuned with replication factor and consistency level to also meet C.So Cassandra is eventually consistent."

Replication factor:

Before deep diving into the consistency levels its necessary to understand the term replication factor. It describes how many copies of your data exist. Based on the RF & the consistency levels it is easy to design a very good stable architecture in Cassandra.

The below terms explains how the write/read transactions serve its purpose:

Commit log − The commit log is a crash-recovery mechanism in Cassandra. Every write operation is written to the commit log.

Mem-table − A mem-table is a memory-resident data structure. After commit log, the data will be written to the mem-table. Sometimes, for a single-column family, there will be multiple mem-tables.

SSTable − It is a disk file to which the data is flushed from the mem-table when its contents reach a threshold value

Write path in Cassandra:

When a write is initiated its first captured by the commit logs. Later the data will be captured and stored in the mem-table. Whenever the mem-table is full, data will be written into the SStable data file. All writes are automatically partitioned and replicated throughout the cluster. Cassandra periodically consolidates the SSTables, discarding unnecessary data.

Read path in Cassandra:

For any read operations first, the values are fetched from the mem table and then Cassandra checks the bloom filter(cache) to find the appropriate SSTable that holds the required data.

Consistency Levels :

Consistency levels are used to manage the data consistency versus data availability. Below are the various levels of consistency that can be set to achieve the data consistency in the DB:

ALL- Writes/Reads must be written to the commit log and memtable on all in the cluster.

EACH_QUORUM- Writes/Reads must be written to the commit log and memtable on each quorum of nodes. Quorum is 51% of the nodes in a cluster.

QUORUM- Writes/Reads must be written to the commit log and memtable on a quorum of nodes across all data centers.

LOCAL_QUORUM- Writes/Reads must be written to the commit log and memtable on a quorum of nodes in the same datacenter as the coordinator.

ONE- Writes must/Reads be written to the commit log and memtable of at least one node.

TWO- Writes/Reads must be written to the commit log and memtable of at least two nodes.

THREE- Writes/Reads must be written to the commit log and memtable of at least three nodes.

LOCAL_ONE- Writes/Reads must be sent to and successfully acknowledged by, at least one node in the local datacenter.

ANY- Writes/Reads must be written to at least one node.

How to calculate the DB impact based on these parameters?

Its very easy to calculate the DB impacts for any given RF & read, write Consistency levels. For example, say let us set up a 5 node cluster with 3 RF, Read & Write Consistency level as quorum then the impact would be as below:

1.     Your reads are consistent

2.     You can survive the loss of 1 node without impacting the application.

3.     You can survive the loss of 1 node without data loss.

4.     You are really reading from 2 nodes every time.

5.     You are really writing to 2 nodes every time.

6.     Each node holds 60% of your data.

The same cluster scenario with Read & Write Consistency level as ONE will have the below impact.

1.     Your reads are eventually consistent

2.     You can survive the loss of 2 nodes without impacting the application.

3.     You can survive the loss of no nodes without data loss.

4.     You are really reading from 1 node every time.

5.     You are really writing to 1 node every time.

6.     Each node holds 60% of your data.

Thus the Cassandra cluster architecture can be defined according to our own business need with the optimal use of the resources to yield high performance.

Credits: You can use this Cassandra Parameters for Dummies to find out the impact: https://www.ecyrd.com/cassandracalculator/

Read More

Install MongoDB using yum on Linux(CentOS)

                MongoDB falls into NoSQL category of database also it is a document database that provides high performance, high availability, and easy scalability.Few of its advantages are high availability and scalability.In this post we are going to see how to install the mongodb using yum on linux.

Step 1:Create a repo file that is necessary to download the mongodb.

vim /etc/yum.repos.d/mongodb.repo

Step 2:Mongodb has two types of packages for 32-bit and 64-bit systems.

For 32-bit systems type the below information in repo file and save.

[mongodb-org-3.0]

name=MongoDB 3.0 Repository

baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/i686/

gpgcheck=0

enabled=1

         In the first line of the file specify you own necessary version of the mongodb to be downloaded.Here i have used version 3.0.

For 64-bit systems type the below information in repo file and save.

[mongodb-org-3.0]
name=MongoDB 3.0 Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64/
gpgcheck=0
enabled=1

         In the first line of the file specify you own necessary version of the mongodb to be downloaded.Here i have used version 3.0.

Note:It’s always better to download 64 bit for production rather than 32 bit because 32-bit MongoDB processes are limited to about 2 gb of data.But the 64 bit doesn't have that limitation. For development both versions can be used.

Step 3:Mongodb has a set of packages as below:

mongodb-org-server-3.0.4           //Mongodb server

mongodb-org-shell-3.0.4            //Mongo Shell

mongodb-org-mongos-3.0.4           //Mongo Shard

mongodb-org-tools-3.0.4            //Mongo Tools

And it contains all the above package together as a single package as below:

mongodb-org-3.0.4

To install all the packages issue the below command:

sudo yum install mongodb-org

To install mongodb server alone issue the below command:

sudo yum install mongodb-org-server-3.0.4

To install mongo shell alone issue the below command:

sudo yum install mongodb-org-shell-3.0.4

To install mongo shard alone issue the below command:

sudo yum install mongodb-org-mongos-3.0.4

To install mongo tools alone issue the below command:

sudo yum install mongodb-org-tools-3.0.4

Step 3:Start the installed mongodb server using the below command:

sudo service mongod start

The server is now started and to login to the mongo shell issue the below command:

mongo

To check the current status of mongod issue the below command:

sudo service mongod status

To stop the running mongod server use the below command:

sudo service mongod stop

It’s finished.We have installed the mongodb and now you can start exploring :)

Read More

View current user,host,port and database in mongo shell

                       Working would be great when there is some good interface. But when your working in the mongo shell we feel little uneasy to know the user, host, port, and database we are using at present. Even though we have certain commands to find out the current user, host, port, and database it looks like a time consuming one. Its difficult to find out them the shell. How can we make all these criteria to appear in the current shell we are working on?Okay, let's have a look at this post.
                   The normal mongo shell looks likes this with no any criteria specified:

                   The below script allows displaying the current user, host, port, and database in mongo shell. All you need to do is to paste the below-shown code into your mongo shell. 

 prompt = function() {
    user = db.runCommand({connectionStatus:1}).authInfo.authenticatedUsers[0]
    host = db.getMongo().toString().split(" ")[2]
    curDB = db.getName()
    if (user) {
       uname = user.user
    }
    else {
       uname = "local"
    }
    return uname + "@" + host + ":" + curDB + "> "
}  

                         And you can see some changes over there. So now the shell turns out like this:

 

                         The shell looks so cool now right. A simple post yet useful one.Hope this helps.

Read More