Elasticsearch

From a startup to a listed company. 6 years of fun!

When I joined Elastic (formerly Elasticsearch) it was a startup with 10 employees + the founders. As one of those first employees I was invited (with #elkie and my wife) to the NYSE event where Elastic went listed as ESTC symbol. Some of us there (Rashid, Karel, Myself, Igor, Costin, Luca, Clinton). Yeah. You are not probably used to see us wearing a suit! :) If you want to read again my story, itā€™s there:
2019-01-10
7 min read
Featured Image

Enriching your postal addresses with Elastic stack - part 3

This blog post is part of a series of 3: Importing Bano dataset with Logstash Using Logstash to lookup for addresses in Bano index Using Logstash to enrich an existing dataset with Bano In the previous post, we described how we can transform a postal address to a normalized one with also the geo location point or transform a geo location point to a postal address. Letā€™s say we have an existing dataset we want to enrich.
2018-03-24
4 min read
Featured Image

Enriching your postal addresses with Elastic stack - part 2

This blog post is part of a series of 3: Importing Bano dataset with Logstash Using Logstash to lookup for addresses in Bano index Using Logstash to enrich an existing dataset with Bano In the previous post, we described how we indexed data coming from the BANO project so we now have indices containing all the french postal addresses. Letā€™s see what we can do now with this dataset. Searching for addresses Good. Can we use a search engine to search?
2018-03-23
12 min read
Featured Image

Enriching your postal addresses with Elastic stack - part 1

This blog post is part of a series of 3: Importing Bano dataset with Logstash Using Logstash to lookup for addresses in Bano index Using Logstash to enrich an existing dataset with Bano Iā€™m not really sure why, but I love the postal address use case. Often in my career I had to deal with that information. Very often the information is not well formatted so itā€™s hard to find the information you need when you have as an input a not so nice dataset.
2018-03-22
10 min read
Featured Image

5 years. What a milestone!

What a milestone! Can you imagine how changed the company in the last 5 years? From 10 employees when I joined to more than 700 now! If you want to read again my story, itā€™s there: 2013: Once upon a timeā€¦ 2014: Once upon a time: a year laterā€¦ 2015: Once upon a time: Make your dreams come true 2016: 3 years! Time flies! 2017: 4 years at elastic! Before speaking about what happened the last 5 years for me, letā€™s modify a bit the script I wrote last year.
2018-01-10
9 min read
Featured Image

4 years at elastic!

This post is starting to become a long series šŸ˜Š Yeah! Thatā€™s amazing! I just spent 4 years working at elastic and Iā€™m starting my happy 5th year! If you want to read again my story, itā€™s there: 2013: Once upon a timeā€¦ 2014: Once upon a time: a year laterā€¦ 2015: Once upon a time: Make your dreams come true 2016: 3 years! Time flies! This year, I will celebrate this by writing a new tutorialā€¦
2017-01-09
9 min read
Featured Image

Elasticsearch real integration tests with security enabled

In a recent post we have seen how to create real integration tests. Those tests launch a real elasticsearch cluster, then run some tests you write with JUnit or your favorite test framework then stop the cluster. But sometimes, you may want to add existing plugins in your integration test cluster. For example, you might want to use X-Pack to bring fantastic features such as: Security Alerting Monitoring Graph Reporting Letā€™s see how you can do that with Maven and Ant againā€¦
2016-08-03
2016-10-21
8 min read

Creating Elasticsearch Transport Action

This blog post is part of a series which will teach you: How to write a plugin for elasticsearch 5.0 using Maven. How to add a new REST endpoint plugin to elasticsearch 5.0. How to use Transport Action classes (what you are reading now). How I wrote the ingest-bano plugin which will be hopefully released soonish. In this plugin, new REST endpoints have been added. In the previous article, we discovered how to add a REST plugin.
2016-08-01
2016-10-20
11 min read

Adding a new REST endpoint to elasticsearch

This blog post is part of a series which will teach you: How to write a plugin for elasticsearch 5.0 using Maven. How to add a new REST endpoint plugin to elasticsearch 5.0 (what you are reading now). How I wrote the ingest-bano plugin which will be hopefully released soonish. In this plugin, new REST endpoints have been added. Imagine that you wish to add a new REST endpoint so you can send requests like:
2016-07-30
2016-10-19
7 min read

Elasticsearch real integration tests

Integration testsā€¦ How do you run them? Often, you are tempted to run services you want to test from JUnit for example. In elasticsearch, you can extend ESIntegTestCase class which will start a cluster of a given number of nodes. public class BanoPluginIntegrationTest extends ESIntegTestCase { public void testPluginIsLoaded() throws Exception { // Your code here } } But to be honest, the test you are running does not guarantee that you will have the same result in production.
2016-07-29
2016-10-18
13 min read

Creating an Ingest plugin for elasticsearch

This blog post is part of a series which will teach you: How to write a plugin for elasticsearch 5.0 using Maven. How to write an ingest plugin for elasticsearch 5.0 (what you are reading now). How I wrote the ingest-bano plugin which will be hopefully released soonish. Today, we will focus on writing an Ingest plugin for elasticsearch. Hey! Wait! You wrote Ingest? What is that? Ingest is a new feature coming in elasticsearch 5.
2016-07-28
2016-10-17
9 min read

Creating a plugin for elasticsearch 5.0 using Maven

Elasticsearch 5.0 switched to Gradle in October 2015. You can obviously write a plugin using Gradle if you wish and you could benefit from all the goodies elasticsearch team wrote when it comes to integration tests and so on. My colleague, Alexander Reelsen aka Spinscale on Twitter, wrote a super nice template if you wish to create an Ingest plugin for 5.0. Hey! Wait! You wrote Ingest? What is that? Ingest is a new feature coming in elasticsearch 5.
2016-07-27
2016-10-16
5 min read

And the beats go on!

Sounds like a cool music, right? At least this is one of my favorite tracks. May be some of you already know that, I enjoy doing some DeeJaying for my friends. But today, I want to speak about another kind of beats. Elastic beats! Elastic Beats Actually my favorite funky music track is a one from Georges Duke: Reach out! But this is another storyā€¦ Beats So what are beats? Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch
2016-03-17
11 min read
Featured Image

Understanding Zipf's law

I just discovered a nice video which explains the Zipfā€™s law. Iā€™m wondering if I can index the french lexique from UniversitĆ© de Savoie and find some funny things based on thatā€¦ Download french words wget http://www.lexique.org/listes/liste_mots.txt head -20 liste_mots.txt What do we have? Itā€™s a CSV file (tabulation as separator): 1_graph 8_frantfreqparm 0 279.84 1 612.10 2 1043.90 3 839.32 4 832.23 5 913.87 6 603.42 7 600.61 8 908.03 9 1427.45 a 4294.90 aa 0.
2016-01-05
4 min read
Featured Image

Building a directory map with ELK

I gave a BBL talk recently and while chatting with attendees, one of them told me a simple use case he covered with elasticsearch: indexing metadata files on a NAS with a simple ls -lR like command. His need is to be able to search on a NAS for files when a user wants to restore a deleted file. As you can imagine a search engine is super helpful when you have hundreds of millions files!
2015-12-10
9 min read
Featured Image

Index Twitter on found

Some months ago, I published a recipe on how to index Twitter with Logstash and Elasticsearch. I have the same need today as I want to monitor Twitter when we run the elastic FR meetup (join us by the way if you are in France!). Well, this recipe can be really simplified and actually I donā€™t want to waste my time anymore on building and managing elasticsearch and Kibana clusters anymore. Letā€™s use a Found by elastic cluster instead.
2015-11-17
4 min read
Featured Image

Next movie to watch based on recommendation

This article is based on Recommender System with Mahout and Elasticsearch tutorial created by MapR. It now uses the 20M MovieLens dataset which contains: 20 million ratings and 465 000 tag applications applied to 27 000 movies by 138 000 users and was released in 4/2015. The format with this recent version has changed a bit so I needed to adapt the existing scripts to the new format. Prerequisites Download the 20M MovieLens dataset. Unzip it.
2015-09-17
7 min read

Importing from a database without a database

Recently, I got a database MySQL dump and I was thinking of importing it into elasticsearch. The first idea which pops up was: install MySQL import the database read the database with Logstash and import into elasticsearch drop the database uninstall MySQL Well. I found that some of the steps are really not needed. I can actually use ELK stack and create a simple recipe which can be used to import SQL dump scripts without needing to actually load the data to a database and then read it again from the database.
2015-09-14
6 min read

Indexing Twitter with Logstash and Elasticsearch

Iā€™m often running some demos during conferences where we have a booth. As many others, Iā€™m using Twitter feed as my datasource. I have been using Twitter river plugin for many years but, you know, rivers have been deprecated. Logstash 1.5.0 provides a safer and more flexible way to deal with tweets with its twitter input. Letā€™s do it! Letā€™s assume that you have already elasticsearch 1.5.2, Logstash 1.5.0 and Kibana 4.0.2 running on your laptop or on a cloud instance.
2015-06-01
2015-06-02
4 min read
Featured Image

Reindex elasticsearch with Logstash

Sometimes, you would like to reindex your data to change your mapping or to change your index settings or to move from one server to another or to one cluster to another (think about multiple data centers for example). For the later you can use Snapshot and Restore feature but if you need to change any index settings, you need something else. With Logstash 1.5.0, you can now do it super easily using elasticsearch input and elasticsearch output.
2015-05-20
5 min read
Featured Image