Using ElasticSearch in Elixir

ElasticSearch is a document database, not in the typical sense, topped up with a Lucene for doing free form text search on your document. This is kind of the storage you want to use if you expect a lot of search on the entities you are going to have. As the name suggests it is also elastic which means it will scale out as your product also scales.

In this tutorial we are going to see how to use ElasticSearch in Elixir. ElasticSearch exposes its APIs as REST calls. So if you have to create a document, delete it, search it or do anything with ElasticSearch there is a REST call for it. This makes it usable from virtually any language easily. But who wants to call REST directly and had form the URLs (not me anyway). There are multiple client libraries for Erlang and as Elixir can use Erlang libraries as well so that should be fine. There is also one Elixir native client library as well. You can see the list of all client libraries for ElasticSearch here. We are going to explore two libraries which are following

1. erlastic_search
2. Tirexs

Setting up the environment

First thing first, lets set up the environment to play around. We need ElasticSearch set up so we can communicate with it at port 9200 and then we need Elixir set up so we can create a project using mix and write some code. Lets use docker to set up both things. Lets grab the docker files for ElasticSearch & Elixir. Fire up the glorious terminal and navigate to the path where you want to make a mess

$ git clone https://github.com/zabirauf/dockerfile.git
$ git clone https://github.com/dockerfile/elasticsearch.git
    
# Lets build the docker images
$ cd dockerfile/elixir
$ docker build -t img-elixir .
$ cd ../../elasticsearch
$ docker build -t img-elastic .

We are now going to create two docker containers, the first will be running the ElasticSearch server, lets call it doc-elastic, and the second one will be running Elixir and will be linked with the doc-elastic container so it can connect with the ElasticSearch server, we will call it doc-elixir. We will also share a folder between host and the doc-elixir so we can easily edit code in the host machine

$ docker run -d --name doc-elastic img-elastic
$ docker run -i -t -v /path/to/folder/on/host:/Project --name doc-elixir --link doc-elastic:doc-elastic img-elixir /bin/bash 

Lets see what we did there. In the first docker command we started the ElasticSearch container as a daemon and named it doc-elastic. In the second step we started a container containing Elixir, mapped our host folder at /path/to/folder/on/host to /Project in the container and also told docker to link this container with the previously created doc-elastic container, so we can access ElasticSearch server from doc-elixir. And then at the end we have access to the shell of doc-elixir so we can play around.

That wasn't hard, was it? Now lets go the next step

Using erlastic_search

Lets create a mix project and then add erlastic_search as a dependency. All the shell commands here will be executed in doc-elixir

$ cd /Project && mix new avenger avenger

This will create a folder called avenger and place the basic files for an Elixir project in it. Now we have to add the dependency on erlastic_search. Edit the mix.exs. You might have to change the permission if it does not allow it, what i did was sudo chmod -R 777 . on my host machine. The mixs.exs should be edited as follows

...
def application do
	[applications: [:logger, :erlastic_search]]
end
...
defp deps do
	[{:erlastic_search, github: "tsloughter/erlastic_search"}]
end
...

Then run mix do deps.get, compile, this should get all the dependencies of the project and then compile it. If you did everything right then it should compile properly. Lets create a file ErlasticSearch.ex inside lib and put the following in it

defmodule ErlasticSearch do
	require Record
    Record.defrecord :erls_params, Record.extract(:erls_params, from: "deps/erlastic_search/include/erlastic_search.hrl")
end

erlastic_search has a record type in Erlang which defines the settings for the ElasticSearch server. We need to import that to Elixir and that is the way to do it. Now lets create marvel.exs in lib folder. Its an exs which means it will be a script, i have done that so that we can easily play around with it. Put the following inside the marvel.exs

require ErlasticSearch

# When we linked doc-elastic with doc-elixir it added some environment variables,
# we can get the IP address of doc-elastic from that environment variable

# ss is short for server settings
ss = ErlasticSearch.erls_params(host: System.get_env("DOC-ELASTIC_PORT_9200_TCP_ADDR"))

#IO.puts "Creating an index"
#:erlastic_search.create_index(ss, "person")

# Creating some entries
IO.puts "Adding some entries"
:erlastic_search.index_doc(ss, "marvel", "characters", [{"name", "Iron man"}])
:erlastic_search.index_doc(ss, "marvel", "characters", [{"name", "Spider man"}])
:erlastic_search.index_doc(ss, "marvel", "characters", [{"name", "Thor"}])
:erlastic_search.index_doc(ss, "marvel", "characters", [{"name", "Hulk"}])

IO.puts "Searching for hulk..."
result = :erlastic_search.search(ss, "marvel", "name:Hulk")
IO.puts inspect result

Run iex -S mix and then compile c("lib/marvel.exs") that should print out a couple of things. So lets see what we did there. We created setting to connect to ElasticSearch server by getting the IP of doc-elastic from environment variables. Then we added some documents to ElasticSearch. At the end we search for the marvellous hulk and we found him. So you can now go ahead and write something in Elixir that does search.

Using Tirexs

Tirexs is another client for ElasticSearch that is created natively in Elixir and it also leverages macros to create a more readable code. The issue with the Tirexs was that it didn't work with Elixir v1.x.x. I have fixed the Tirexs and sent a pull request, until it gets to the repo you can get the dependency from my fork of Tirexs. You can start another container as described in Setting up the environment. Then use the bash of the container to create a project using mix and then edit the mix.exs

$ cd /Project && mix new avenger avenger

...
defp deps do
	[{:tirexs, github: "zabirauf/tirexs"}]
end
...

Then run mix do deps.get, compile, this should get all the dependencies of the project and then compile it. If you did everything right then it should compile properly. Now lets create marvel.exs in lib folder. Its an exs which means it will be a script, i have done that so that we can easily play around with it. Put the following inside the marvel.exs

import Tirexs.Bulk
import Tirexs.Search
require Tirexs.ElasticSearch

settings = Tirexs.ElasticSearch.config(uri: System.get_env("DOC-ELASTIC_PORT_9200_TCP_ADDR"))

Tirexs.Bulk.store [index: "marvel", refresh: true], settings do
  
    create id: 1, name: "Iron man", type: "characters"
    create id: 2, name: "Spider man", type: "characters"
    create id: 3, name: "Thor", type: "characters"
    create id: 4, name: "Hulk", type: "characters"
end

find_hulk = search [index: "marvel"] do
    query do
      string "name:Hulk"
    end
end

result = Tirexs.Query.create_resource(find_hulk, settings)
IO.puts inspect result

Run iex -S mix and then compile c("lib/marvel.exs") that should print out a couple of things. So lets see what we did there. We created setting to connect to ElasticSearch server by getting the IP of doc-elastic from environment variables. Then we added some documents to ElasticSearch. At the end we search for the marvellous hulk and we found him.
We can see that this code is much more readable than erlastic_search because of the Elixir macros.

Follow @zabirauf