-Xmx128gb -Xms128gb

adding more memory to my brain!

From a Startup to a Listed Company. 6 Years of Fun!

| Comments

When I joined Elastic (formerly Elasticsearch) it was a startup with 10 employees + the founders. As one of those first employees I was invited (with #elkie and my wife) to the NYSE event where Elastic went listed as ESTC symbol.

Some of us there (Rashid, Karel, Myself, Igor, Costin, Luca, Clinton). Yeah. You are not probably used to see us wearing a suit! :)

If you want to read again my story, it’s there:

Before speaking about what happened last year, I’d like to modify the way I’m ingesting my personal data in elasticsearch by using the real city name when I’m speaking in France. Why this? Because anytime I’m speaking in Paris or suburbs, I’m actually writing Paris in my report which is inaccurate.

Instead, I’d really like to have the real city name… How can I do this? Well, I do have the exact geo location points of each event (which is something I’m writing manually for now)… If you are a frequent visitor of this blog, that should ring a bell, right?

In 2018, I wrote some blog posts about transforming geo points to exact address:

Let’s use that now!

Enriching Your Postal Addresses With Elastic Stack - Part 3

| Comments

This blog post is part of a series of 3:

In the previous post, we described how we can transform a postal address to a normalized one with also the geo location point or transform a geo location point to a postal address.

Let’s say we have an existing dataset we want to enrich.

We will consider 3 scenarios:

  • We have a CSV file
  • We have a Relational Database, MySQL
  • We have data in elasticsearch

Let’s see how to enrich those datasets.

Enriching Your Postal Addresses With Elastic Stack - Part 1

| Comments

This blog post is part of a series of 3:

I’m not really sure why, but I love the postal address use case. Often in my career I had to deal with that information. Very often the information is not well formatted so it’s hard to find the information you need when you have as an input a not so nice dataset.

Let’s take a simple use case. I have a user in my database like:

1
2
3
4
5
6
7
8
9

{
  "name": "Joe Smith",
  "address": {
    "number": "23",
    "street_name": "r verdiere",
    "city": "rochelle",
    "country": "France"
  }
}

If you live in France you might notice that this address is fairly incomplete. If we want to send a letter to Joe, it’s going to be hard.

Also, I’d really like to display on a map, where my customers are located in France.

On the other way around, let’s say I’m collecting data from a mobile application where I’m meeting friends and I’d like to remember automatically what was the address of the location last time we met. Basically I have data like:

1
2
3
4
5
6
7

{
  "name": "Joe Smith",
  "location": {
    "lat": 46.15735,
    "lon": -1.1551
  }
}

And I would like to get back the postal address matching this location.

What can we do for that?

We need something to enrich the existing data by fixing the address and providing actual coordinates or the other way back.

5 Years. What a Milestone!

| Comments

What a milestone! Can you imagine how changed the company in the last 5 years? From 10 employees when I joined to more than 700 now!

If you want to read again my story, it’s there:

Before speaking about what happened the last 5 years for me, let’s modify a bit the script I wrote last year.

Elasticsearch Real Integration Tests With Security Enabled (Updated for GA)

| Comments

NOTE: This article is an updated version of Elasticsearch real integration tests with security enabled

In a recent post we have seen how to create real integration tests. Those tests launch a real elasticsearch cluster, then run some tests you write with JUnit or your favorite test framework then stop the cluster.

But sometimes, you may want to add existing plugins in your integration test cluster.

For example, you might want to use X-Pack to bring fantastic features such as:

  • Security
  • Alerting
  • Monitoring
  • Graph
  • Reporting

Let’s see how you can do that with Maven and Ant again…

Creating Elasticsearch Transport Action (Updated for GA)

| Comments

NOTE: This article is an updated version of Creating Elasticsearch Transport Action

This blog post is part of a series which will teach you:

In the previous article, we discovered how to add a REST plugin. It was a simple implementation as in HelloRestAction class we wrote something like:

1
2
3
4
5
6
7
8
9
10
11
12

@Override
protected RestChannelConsumer prepareRequest(RestRequest restRequest, NodeClient client) throws IOException {
    String name = restRequest.param("name");
    return channel -> {
        Message message = new Message(name);
        XContentBuilder builder = channel.newBuilder();
        builder.startObject();
        message.toXContent(builder, restRequest);
        builder.endObject();
        channel.sendResponse(new BytesRestResponse(RestStatus.OK, builder));
    };
}

But actually, you will probably like to execute some actions against a Node, call some internal services… So the implementation we wrote needs to be modified a bit.

Adding a New REST Endpoint to Elasticsearch (Updated for GA)

| Comments

NOTE: This article is an updated version of Adding a new REST endpoint to elasticsearch

This blog post is part of a series which will teach you:

Imagine that you wish to add a new REST endpoint so you can send requests like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

curl -XGET "http://localhost:9200/_hello?name=David&pretty"

# Or

curl -XGET "http://localhost:9200/_hello/David&pretty"

# Or

curl -XGET "http://localhost:9200/_hello?pretty" -d '{
    "name": "David"
}'

# Or

curl -XPOST "http://localhost:9200/_hello?pretty" -d '{
    "name": "David"
}'

And you want to get back something like:

1
2
3

{
  "message": "Hello David!"
}

Without any parameter:

1

curl -XGET "http://localhost:9200/_hello?pretty"

It should return:

1
2
3

{
    "message": "Hello World!"
}

Or get back a list of existing indices and the number of documents for a given type of indices with:

1

curl -XGET http://localhost:9200/_bano

Let’s see how to implement that!

Elasticsearch Real Integration Tests (Updated for GA)

| Comments

NOTE: This article is an updated version of Elasticsearch real integration tests

Integration tests… How do you run them?

Often, you are tempted to run services you want to test from JUnit for example. In elasticsearch, you can extend ESIntegTestCase class which will start a cluster of a given number of nodes.

1
2
3
4
5

public class BanoPluginIntegrationTest extends ESIntegTestCase {
    public void testPluginIsLoaded() throws Exception {
      // Your code here
    }
}

But to be honest, the test you are running does not guarantee that you will have the same result in production.

Why this? Because some services might be disabled, some others might be mocked.

Instead, running tests against a real cluster as the one you would install locally on your machine is the way to go.

Let’s see how you can do that with Maven…