-Xmx128gb -Xms128gb

adding more memory to my brain!

Elasticsearch Real Integration Tests

| Comments

NOTE: This article is now outdated. Please read Elasticsearch real integration tests (Updated for GA) instead!

Integration tests… How do you run them?

Often, you are tempted to run services you want to test from JUnit for example. In elasticsearch, you can extend ESIntegTestCase class which will start a cluster of a given number of nodes.

1
2
3
4
5
public class BanoPluginIntegrationTest extends ESIntegTestCase {
    public void testPluginIsLoaded() throws Exception {
      // Your code here
    }
}

But to be honest, the test you are running does not guarantee that you will have the same result in production.

Why this? Because some services might be disabled, some others might be mocked.

Instead, running tests against a real cluster as the one you would install locally on your machine is the way to go.

Let’s see how you can do that with Maven…

Creating an Ingest Plugin for Elasticsearch

| Comments

NOTE: This article is now outdated. Please read Creating an Ingest plugin for elasticsearch (Updated for GA) instead!

This blog post is part of a series which will teach you:

Today, we will focus on writing an Ingest plugin for elasticsearch.

Hey! Wait! You wrote Ingest? What is that?

Ingest is a new feature coming in elasticsearch 5.0. It helps you to transform your data on the fly while injecting it into elasticsearch. Read more in elastic blog post.

If you know me and my work before I joined elastic, I have always been in love with data crawling and transformation as I wrote myself some plugins called rivers.

Creating a Plugin for Elasticsearch 5.0 Using Maven

| Comments

NOTE: This article is now outdated. Please read Creating a plugin for elasticsearch 5.0 using Maven (Updated for GA) instead!

Elasticsearch 5.0 switched to Gradle in October 2015.

You can obviously write a plugin using Gradle if you wish and you could benefit from all the goodies elasticsearch team wrote when it comes to integration tests and so on.

My colleague, Alexander Reelsen aka Spinscale on Twitter, wrote a super nice template if you wish to create an Ingest plugin for 5.0.

Hey! Wait! You wrote Ingest? What is that?

Ingest is a new feature coming in elasticsearch 5.0. It helps you to transform your data on the fly while injecting it into elasticsearch. Read more in elastic blog post.

If you know me and my work before I joined elastic, I have always been in love with data crawling and transformation as I wrote myself some plugins called rivers.

This blog post is part of a series which will teach you:

And the Beats Go On!

| Comments

Sounds like a cool music, right? At least this is one of my favorite tracks.

May be some of you already know that, I enjoy doing some DeeJaying for my friends.

But today, I want to speak about another kind of beats. Elastic beats!

3 Years! Time Flies!

| Comments

3 years! Can you imagine that? Already 3 years spent working at elastic? Time flies!

2015 has been an uncommon year for me.

Not because Marty Mc Fly and Doc Emmett Brown finally arrived…

Not because, Han Solo, Leia and friends were finally back again…

But for technical and also personal reasons.

Building a Directory Map With ELK

| Comments

I gave a BBL talk recently and while chatting with attendees, one of them told me a simple use case he covered with elasticsearch: indexing metadata files on a NAS with a simple ls -lR like command. His need is to be able to search on a NAS for files when a user wants to restore a deleted file.

As you can imagine a search engine is super helpful when you have hundreds of millions files!

I found this idea great and this is by the way why I love speaking at conferences or in companies: you always get great ideas when you listen to others!

I decided then to adapt this idea using the ELK stack.

Index Twitter on Found

| Comments

Some months ago, I published a recipe on how to index Twitter with Logstash and Elasticsearch.

I have the same need today as I want to monitor Twitter when we run the elastic FR meetup (join us by the way if you are in France!).

Well, this recipe can be really simplified and actually I don’t want to waste my time anymore on building and managing elasticsearch and Kibana clusters anymore.

Let’s use a Found by elastic cluster instead.

Importing From a Database Without a Database

| Comments

Recently, I got a database MySQL dump and I was thinking of importing it into elasticsearch.

The first idea which pops up was:

  • install MySQL
  • import the database
  • read the database with Logstash and import into elasticsearch
  • drop the database
  • uninstall MySQL

Well. I found that some of the steps are really not needed.

I can actually use ELK stack and create a simple recipe which can be used to import SQL dump scripts without needing to actually load the data to a database and then read it again from the database.

Let’s do it!