Apache solr replication step by step

Need for Replication :

• When there is a large search volume that cannot be handled by a single machine, so you need to distribute searches across multiple read-only copies of the index.
• If there is a high volume of indexing which consumes machine resources and reduces search performance on the indexing machine, so you need to separate indexing and searching.
• When we want to make a backup of the index

MASTER-SLAVE

• Distributes complete copies of a master index to one or more slave servers. 
• The master server continues to manage updates to the index. 
• All querying is handled by the slaves. 
• This enables Solr to scale to provide adequate responsiveness to queries against large search volumes.

Replication Terminology:

Index

A Lucene index is a directory of files. These files make up the searchable and returnable data of a Solr Core.

Distribution

The copying of an index from the master server to all slaves. 

Inserts and Deletes

• As inserts and deletes occur in the index, the directory remains unchanged. Documents are always inserted into newly created files. 
• Documents that are deleted are not removed from the files.
• They are flagged in the file, deletable and are not removed from the files until the index is optimized.

Master and Slave

• A Solr replication master is a single node which receives all updates initially and keeps everything organized.
• Solr replication slave nodes receive no updates directly, instead all changes (such as inserts, updates, deletes, etc.) are made against the single master node. 
• Changes made on the master are distributed to all the slave nodes which service all query requests from the clients.

Repeater

A node that acts as both a master and a slave.

Optimization

• A process that compacts the index and merges segments in order to improve query performance.
• Optimization should only be run on the master nodes. An optimized index may give query performance gains compared to an index that has become fragmented over a period of time with many updates. 
• Distributing an optimized index requires a much longer time than the distribution of new segments to an un-optimized index

Snapshot

A directory containing hard links to the data files of an index. Snapshots are distributed from the master nodes when the slaves pull them, "smart copying" any segments the slave node does not have in snapshot directory that contains the hard links to the most recent index data files.

Configuring the Replication RequestHandler on a Master Server:

commit-Triggers replication whenever a commit is performed on the master index.

optimize-Triggers replication whenever the master index is optimized.

startup-Triggers replication whenever the master index starts up.

MASTER SLAVE CONFIGURATION

MASTER

The configuration of the master are configured in the below file.

vim /var/solr/data/fortis/conf/solrconfig.xml

vim /var/solr/data/fortis/core.properties

Make the below changes in the core.properties file 

enable.master=true

enable.slave=false

SLAVE

To create a new collection use the below command

/usr/local/solr-6.6.2/bin/solr create -c prod

The configuration of the master are configured in the below file.

vim /var/solr/data/fortis/conf/solrconfig.xml

Make the below changes in the core.properties file 

vim /var/solr/data/fortis/core.properties

enable.master=false

enable.slave=true

Now we have setup the master-slave replication in the apache solr.

Read More

Apache SOLR introduction & installation

In this post, we are going to discuss about Apache Solr and how to install them step by step.

Why Solr?



Lucene:

• Search storage engine
• Solr uses concepts derived from Lucene
• Lucene is widely used in many projects among one of them is solr
• Used to index & search with high performance.
• Solr uses lucene as its backend

Solr:

• Search server
• Document oriented
• Stores data & indexing
• Searches including full-text search, stemming,hit-highlighting, faceted-search etc that cannot done by native databases
• Vertically and horizontally scalable
• Replication for high availability
• Sharding for distributed search
• Performs in memory,grouping,counting,similar products in single shot
• Exposed over HTTP,REST like api.
• The DataImportHandler provides a configuration driven way to import data from relational databases or XML files, into Solr in both “full import” and “incremental delta import” mode.

INVERTED INDEX:

• Searches document by unique word
• Similar to the index back of back

D1- I like Apache services.
D2- They include all kind of database & services support.
D3- I would recommend their services to my clients too.

Working:

• Define a schema. 
• Deploy Solr.
• Feed Solr documents for which your users will search.
• Expose search functionality in your application.

Solr schema:

• No schema
• Has index that contain docs
• Fields are used to index,search & store

Define a schema

The schema tells Solr about the contents of documents it will be indexing. In the online store example, the schema would define fields for the product name, description, price, manufacturer, and so on. 

Defining fields:

Indexing:

Indexing is a technique of adding Document’s content to Solr Index so that we can search them easily. Apache Solr uses Apache Lucene Inverted Index technique to Index it’s documents. That’s why Solr provides very fast searching feature.

Field analyzers: (Analyzer=tokenizer+filters)

Used both during ingestion, when a document is indexed, and at query time.Analyzers may be a single class or series of tokenizer and filter classes.

Alternative words-finish,complete

misspelled-google,gogle

Tokenizers:

Break field data into lexical units, or tokens.

Pre- Stripping html tags

Post-Stemming(replace) tables as collection

         Stop word filter (the,is,and)

Filters:(used for indexing)

• Examine a stream of tokens and keep them, transform or discard them, or create new ones. 
• Tokenizers and filters may be combined to form pipelines, or chains, where the output of one is input to the next. 
• Such a sequence of tokenizers and filters is called an analyzer and the resulting output of an analyzer is used to match query results or build indices.

eg)ram,RAM,Ram

Deploying Solr

Pre requisites:

yum update

yum install java-1.8.0-openjdk.x86_64

java -version

Installation:

wget http://apache.org/dist/lucene/solr/6.6.1/solr-6.6.1.tgz

tar zxvf solr-6.6.1.tgz
cp /opt/solr-6.6.1/bin/install_solr_service.sh .
rm -rf solr-6.6.1
./install_solr_service.sh solr-6.6.1.tgz
ps -ef | grep solr

Feed Solr documents for which your users will search Creating a project(core): A Core is an Index of texts and fields available in all documents. One Solr Instance can contain one or more Solr Cores. /opt/solr-6.6.1/bin/solr create -c jerwin Create new document : We can create a new document to the core with 3 fields and value using the below command on the terminal using curl: curl http://localhost:8983/solr/jerwin/update -d ' [ {"id" : "db1", "company_name" : "Mafiree", "location" : "Nagercoil" } ]'

View:

The inserted data can be viewed with the id specified.

curl http://localhost:8983/solr/jerwin/get?id=db1

Output can be viewed in the browser:

Hope this gives you simple introduction about SOLR,let me know if you have any concerns via comments.

Read More