David Pilato
Developer | Evangelist
Elastic
20+ years of experience, mostly in Java. Living in Cergy, France.
Self Introduction
Developer | Evangelist at elastic and creator of the Elastic French User Group. Frequent speaker about all things Elastic, in conferences, for User Groups and in companies with BBL talks. In my free time, I enjoy coding and DeeJaying, just for fun. Living with my family in Cergy, France.
Details
I discovered Elasticsearch project in 2011. After contributed to the project and created open source plugins for it, David joined elastic the company in 2013 where he is Developer and Evangelist. He also created and still actively managing the French spoken language User Group. At elastic, he mainly worked on Elasticsearch source code, specifically on open-source plugins. In his free time, he likes talking about elasticsearch in conferences or in companies (Brown Bag Lunches AKA BBLs). He is also author of FSCrawler project which helps to index your pdf, open office, whatever documents in elasticsearch using Apache Tika behind the scene.
Visited countries
You can see here the countries I have visited so far. Most of them are for business purpose but who said you can not do both: business and leisure?
Create your own visited countries map or check out the JavaScript Charts.
Free Lunches for Opensource Engineers
When I started at elastic some years ago, we were a few people in the company and I was feeling pretty much alone in France. I found a way to share my passion, meet new people and help them to get started with Elasticsearch. That's called BBLs.
The new decade - đ celebrating 11 years at Elastic
Here we go again⊠2023 is yet over and it helped me to realize what I love to do. 2023 Iâm so happy to be part of the Elastic family and to be able to help people around the world to be successful with the Elastic Stack. I can definitely say that interacting with the community is what I love to do. Itâs taking several forms. Of course, Iâm still enjoying going to conferences. You donât need to be a rock star to be a speaker.
Automatically update documentation with Github Actions
In some of my projects, I like to provide an accurate documentation. Which means that I want to have examples and documentation up to date. So, when I'm updating a library or a service I'm using in my code, I need to manually find and replace all the text to reflect the changes. This post describes how you can do this automatically using Github Actions, Maven and Dependabot.
The biggest milestone ever - đ 10 years at Elastic
Would I have been able to imagine such a thing when 10+ years ago I met Shay at Devoxx Belgium after his talk âSearch, the Final Frontierâ? 10 years! I have been working 10 years at Elastic. I have never working so long for a company. Thatâs my biggest work milestone ever! Remember the time Have you noticed that Iâm frequently using a music in my blog posts. Well, it makes sense as Iâm often DeeJaying for friends so music is everywhere in my life.
La potion magique pour faire avancer ta carriĂšre
Voici la transcription dâune prĂ©sentation que jâai eu le plaisir Ă donner lors du Camping des speakers 2022, dont il sâagissait de la premiĂšre Ă©dition. La potion magique pour faire progresser ta carriĂšre 10h15 - 15 minutes - Autour du Feu La recette de la potion magique ne se transmet quâaux seuls druides, normalement. Mais exceptionnellement, le conseil des druides de la forĂȘt des Carnutes mâa autorisĂ© Ă vous rĂ©vĂ©ler quelques uns des ingrĂ©dients qui constituent ce breuvage.
9 years! A whole new world.
I have been missing you! Indeed, last year, I have not been able to publish my anniversary blog post as Iâm used to do every year since I joined Elastic 9 years ago. That was for a technical reason actually. I was using a old and not updated blogging platform and it took me a looooong time before I was able to invest time to switch everything to Hugo. So here we go! This year celebrates my 9 years anniversary at elastic but also a new blogging system.
7 years! Finding the right balance
What a ride! 10 employees to around 2000 now. As I imagined 8 years ago, I still think that Elasticsearch (the product) and elastic (the company) are successful. Becoming a public company did not change a lot my daily activities. Iâm still on the road meeting/building the community, specifically in France and making sure people are sharing the same love that we have internally for the products we are building. Iâd like this year to focus this anniversary blog post on some items:
From a startup to a listed company. 6 years of fun!
When I joined Elastic (formerly Elasticsearch) it was a startup with 10 employees + the founders. As one of those first employees I was invited (with #elkie and my wife) to the NYSE event where Elastic went listed as ESTC symbol. Some of us there (Rashid, Karel, Myself, Igor, Costin, Luca, Clinton). Yeah. You are not probably used to see us wearing a suit! :) If you want to read again my story, itâs there:
Enriching your postal addresses with Elastic stack - part 3
This blog post is part of a series of 3: Importing Bano dataset with Logstash Using Logstash to lookup for addresses in Bano index Using Logstash to enrich an existing dataset with Bano In the previous post, we described how we can transform a postal address to a normalized one with also the geo location point or transform a geo location point to a postal address. Letâs say we have an existing dataset we want to enrich.
Enriching your postal addresses with Elastic stack - part 2
This blog post is part of a series of 3: Importing Bano dataset with Logstash Using Logstash to lookup for addresses in Bano index Using Logstash to enrich an existing dataset with Bano In the previous post, we described how we indexed data coming from the BANO project so we now have indices containing all the french postal addresses. Letâs see what we can do now with this dataset. Searching for addresses Good. Can we use a search engine to search?
Enriching your postal addresses with Elastic stack - part 1
This blog post is part of a series of 3: Importing Bano dataset with Logstash Using Logstash to lookup for addresses in Bano index Using Logstash to enrich an existing dataset with Bano Iâm not really sure why, but I love the postal address use case. Often in my career I had to deal with that information. Very often the information is not well formatted so itâs hard to find the information you need when you have as an input a not so nice dataset.
5 years. What a milestone!
What a milestone! Can you imagine how changed the company in the last 5 years? From 10 employees when I joined to more than 700 now! If you want to read again my story, itâs there: 2013: Once upon a time⊠2014: Once upon a time: a year later⊠2015: Once upon a time: Make your dreams come true 2016: 3 years! Time flies! 2017: 4 years at elastic! Before speaking about what happened the last 5 years for me, letâs modify a bit the script I wrote last year.
4 years at elastic!
This post is starting to become a long series đ Yeah! Thatâs amazing! I just spent 4 years working at elastic and Iâm starting my happy 5th year! If you want to read again my story, itâs there: 2013: Once upon a time⊠2014: Once upon a time: a year later⊠2015: Once upon a time: Make your dreams come true 2016: 3 years! Time flies! This year, I will celebrate this by writing a new tutorialâŠ
Elasticsearch real integration tests with security enabled
In a recent post we have seen how to create real integration tests. Those tests launch a real elasticsearch cluster, then run some tests you write with JUnit or your favorite test framework then stop the cluster. But sometimes, you may want to add existing plugins in your integration test cluster. For example, you might want to use X-Pack to bring fantastic features such as: Security Alerting Monitoring Graph Reporting Letâs see how you can do that with Maven and Ant againâŠCreating Elasticsearch Transport Action
This blog post is part of a series which will teach you: How to write a plugin for elasticsearch 5.0 using Maven. How to add a new REST endpoint plugin to elasticsearch 5.0. How to use Transport Action classes (what you are reading now). How I wrote the ingest-bano plugin which will be hopefully released soonish. In this plugin, new REST endpoints have been added. In the previous article, we discovered how to add a REST plugin.Adding a new REST endpoint to elasticsearch
This blog post is part of a series which will teach you: How to write a plugin for elasticsearch 5.0 using Maven. How to add a new REST endpoint plugin to elasticsearch 5.0 (what you are reading now). How I wrote the ingest-bano plugin which will be hopefully released soonish. In this plugin, new REST endpoints have been added. Imagine that you wish to add a new REST endpoint so you can send requests like:Elasticsearch real integration tests
Integration tests⊠How do you run them? Often, you are tempted to run services you want to test from JUnit for example. In elasticsearch, you can extend ESIntegTestCase class which will start a cluster of a given number of nodes. public class BanoPluginIntegrationTest extends ESIntegTestCase { public void testPluginIsLoaded() throws Exception { // Your code here } } But to be honest, the test you are running does not guarantee that you will have the same result in production.Creating an Ingest plugin for elasticsearch
This blog post is part of a series which will teach you: How to write a plugin for elasticsearch 5.0 using Maven. How to write an ingest plugin for elasticsearch 5.0 (what you are reading now). How I wrote the ingest-bano plugin which will be hopefully released soonish. Today, we will focus on writing an Ingest plugin for elasticsearch. Hey! Wait! You wrote Ingest? What is that? Ingest is a new feature coming in elasticsearch 5.Creating a plugin for elasticsearch 5.0 using Maven
Elasticsearch 5.0 switched to Gradle in October 2015. You can obviously write a plugin using Gradle if you wish and you could benefit from all the goodies elasticsearch team wrote when it comes to integration tests and so on. My colleague, Alexander Reelsen aka Spinscale on Twitter, wrote a super nice template if you wish to create an Ingest plugin for 5.0. Hey! Wait! You wrote Ingest? What is that? Ingest is a new feature coming in elasticsearch 5.And the beats go on!
Sounds like a cool music, right? At least this is one of my favorite tracks. May be some of you already know that, I enjoy doing some DeeJaying for my friends. But today, I want to speak about another kind of beats. Elastic beats! Elastic Beats Actually my favorite funky music track is a one from Georges Duke: Reach out! But this is another story⊠Beats So what are beats? Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch
3 years! Time flies!
3 years! Can you imagine that? Already 3 years spent working at elastic? Time flies! 2015 has been an uncommon year for me. Not because Marty Mc Fly and Doc Emmett Brown finally arrived⊠Not because, Han Solo, Leia and friends were finally back again⊠But for technical and also personal reasons. On a personal side, I had to deal with two major issues which cause some slow down in my professional activities. I had to cancel some conferences for instance.
Understanding Zipf's law
I just discovered a nice video which explains the Zipfâs law. Iâm wondering if I can index the french lexique from UniversitĂ© de Savoie and find some funny things based on that⊠Download french words wget http://www.lexique.org/listes/liste_mots.txt head -20 liste_mots.txt What do we have? Itâs a CSV file (tabulation as separator): 1_graph 8_frantfreqparm 0 279.84 1 612.10 2 1043.90 3 839.32 4 832.23 5 913.87 6 603.42 7 600.61 8 908.03 9 1427.45 a 4294.90 aa 0.
Building a directory map with ELK
I gave a BBL talk recently and while chatting with attendees, one of them told me a simple use case he covered with elasticsearch: indexing metadata files on a NAS with a simple ls -lR like command. His need is to be able to search on a NAS for files when a user wants to restore a deleted file. As you can imagine a search engine is super helpful when you have hundreds of millions files!
Index Twitter on found
Some months ago, I published a recipe on how to index Twitter with Logstash and Elasticsearch. I have the same need today as I want to monitor Twitter when we run the elastic FR meetup (join us by the way if you are in France!). Well, this recipe can be really simplified and actually I donât want to waste my time anymore on building and managing elasticsearch and Kibana clusters anymore. Letâs use a Found by elastic cluster instead.
Next movie to watch based on recommendation
This article is based on Recommender System with Mahout and Elasticsearch tutorial created by MapR. It now uses the 20M MovieLens dataset which contains: 20 million ratings and 465 000 tag applications applied to 27 000 movies by 138 000 users and was released in 4/2015. The format with this recent version has changed a bit so I needed to adapt the existing scripts to the new format. Prerequisites Download the 20M MovieLens dataset. Unzip it.Importing from a database without a database
Recently, I got a database MySQL dump and I was thinking of importing it into elasticsearch. The first idea which pops up was: install MySQL import the database read the database with Logstash and import into elasticsearch drop the database uninstall MySQL Well. I found that some of the steps are really not needed. I can actually use ELK stack and create a simple recipe which can be used to import SQL dump scripts without needing to actually load the data to a database and then read it again from the database.Indexing Twitter with Logstash and Elasticsearch
Iâm often running some demos during conferences where we have a booth. As many others, Iâm using Twitter feed as my datasource. I have been using Twitter river plugin for many years but, you know, rivers have been deprecated. Logstash 1.5.0 provides a safer and more flexible way to deal with tweets with its twitter input. Letâs do it! Letâs assume that you have already elasticsearch 1.5.2, Logstash 1.5.0 and Kibana 4.0.2 running on your laptop or on a cloud instance.
Reindex elasticsearch with Logstash
Sometimes, you would like to reindex your data to change your mapping or to change your index settings or to move from one server to another or to one cluster to another (think about multiple data centers for example). For the later you can use Snapshot and Restore feature but if you need to change any index settings, you need something else. With Logstash 1.5.0, you can now do it super easily using elasticsearch input and elasticsearch output.
Using JS Auth with found cluster
Using Found by elastic cluster helps a lot to have a ready to use and managed elasticsearch cluster. I started my own cluster yesterday to power brownbaglunch.fr website (work in progress) and it was ready to use after some clicks! Itâs a kind of magic! But I ran into an issue when you secure it and use the elasticsearch javascript client. Creating your cluster Found Console Adding ACL By default, your cluster is opened but you can fix that by opening âAccess Controlâ menu.
Advanced search for your Legacy application
I gave recently a talk at Voxxed Istanbul 2015 and Iâd like to share here the story of this talk. The talk was about adding a real search engine for your legacy application. Here âlegacyâ means an application which is still using SQL statements to execute search requests. Our current CRM application can visualize our customers. Each person is represented as a Person bean and have some properties like name, dateOfBirth, children, country, city and some metrics related to the number of clicks each person did on the car or food buttons on our mobile application (center of interests that is).
Devoxx France 2015
I gave recently a talk at Devoxx France 2015 with Colin Surprenant and Iâd like to share here some of the examples we used for the talk. The talk was about âwhat my data look like?â. We said that our manager was asking us to answer some questions: who are our customers? how do they use our services? what do they think about us on Twitter? Our CRM database So we have a PostgreSQL database containing our data.
Exploring Capitaine Train dataset
Recently I saw a tweet where Capitaine Train team started to open data they have collected and enriched or corrected. Ouvrez, ouvrez, les donnĂ©es structurĂ©es. Capitaine Train libĂšre les gares : https://t.co/y6DjWsbALF #opendata â Trainline France (@trainline_fr) April 23, 2015 I decided to play a bit with ELK stack and create a simple recipe which can be used with any other CSV like data. Prerequisites You will need: Logstash: Iâm using 1.5.0-rc3. Elasticsearch: Iâm using 1.
Using Log4J2 with Hibernate 4
I was trying to use Hibernate 4.3.8.Final with Log4J2 and I spent some hours to find why Hibernate was not using Log4J2 though it was declared in my pom.xml file. Actually, I hit issue JBLOGGING-107. The workaround is simply to add a more recent jboss-logging dependency than the one shipped by default with Hibernate 4.3.8.Final. <dependency> <groupId>org.jboss.logging</groupId> <artifactId>jboss-logging</artifactId> <version>3.2.1.Final</version> </dependency>Once upon a time: Make your dreams come true
Oh wait! Already 2 years spent working for Elasticsearch? Time flies! After the first year, I wrote that I did 58 talks in 4 countries, 37 towns for about 18 000 kilometers traveled. I was pretty sure that things would continue to grow. This year, I spoke 78 times! Around 2 talks per week! I did around 48 000 kilometers. 8 000 km more than the earthâs circumference! I still canât believe it⊠12 countries. And no need to say that I love giving talks and sharing my enthusiasm about Elasticsearch!
Once upon a time: a year later...
I joined Elasticsearch Inc one year ago. Those were pretty exciting days! But now⊠Itâs more than that! Really! You could think that after one year, my motivation would start to decrease. I have the total opposite feeling. Still excited by my job, by the company and by the project, but most of all by the amazing team Iâm lucky to work with! Everyone is different and each of us adds different value to Elasticsearch. Personally, I learn a lot from my co-workers.
Once upon a time...
Once upon a time⊠In fact 2 years ago, I was looking for a way to make Hibernate search distributed on multiple nodes. My first idea was to store indexes in a single database shared by my nodes. Yes, itâs a stupid idea in term of performances but I would like to try to build it. Digging for source code, I came to the JdbcDirectory class from the compass project. And I saw on the compass front page something talking about the future of Compass and Elasticsearch.
Il était une fois : un conte de fées élastique !
Il Ă©tait une fois⊠En fait, il y a 2 ans, je cherchais un moyen pour distribuer Hibernate search sur plusieurs noeuds. Ma premiĂšre idĂ©e Ă©tait de stocker les index dans une base de donnĂ©es partagĂ©e par les diffĂ©rents noeuds. Oui ! Il sâagit dâune idĂ©e stupide en terme de performances, mais jâavais envie dâessayer et de construire ce modĂšle. AprĂšs avoir cherchĂ© du code source, je suis finalement tombĂ© sur la classe JdbcDirectory du projet Compass.
ScrutMyDocs : un moteur de recherche pour documents
Avec Malloum, nous venons de publier notre premier projet open-source commun: Scrut My Docs ! Technical overview Nos objectifs Fournir une application web clĂ© en main permettant dâindexer des documents de vos disques locaux. Fournir Ă la communautĂ© Elasticsearch un modĂšle de base pour dĂ©velopper votre propre webapp pour une utilisation simple de recherche (« Ă la google »). Aider les dĂ©butants Elasticsearch Java avec des exemples concrets en Java Les technologies employĂ©es Elasticsearch ! et son Ă©cosystĂšme (rivers, plugins) Spring JSF Primefaces Comment dĂ©marrer ?
La factory Spring pour Elasticsearch est sortie !
Et voilĂ , la premiĂšre release de la factory spring vient dâĂȘtre faite. Vous pouvez donc maintenant lâutiliser dans vos projets Maven : <dependency> <groupId>fr.pilato.spring</groupId> <artifactId>spring-elasticsearch</artifactId> <version>0.0.1</version> </dependency> Le code source est disponible sur github.ProtĂ©ger son cluster Elasticsearch avec Jetty
Nativement, Elasticsearch expose lâensemble de ses services sans aucune authentification et donc une commande du type curl -XDELETE http://localhost:9200/myindex peut faire de nombreux dĂ©gĂąts non dĂ©sirĂ©s. De plus, si vous dĂ©veloppez une application JQuery avec un accĂšs direct depuis le poste client Ă votre cluster Elasticsearch, le risque quâun utilisateur joue un peu avec votre cluster est grand ! Alors, pas de panique⊠La sociĂ©tĂ© Sonian Inc. a open sourcĂ© son plugin Jetty pour Elasticsearch pour notre plus grand bonheur đUne factory Spring pour Elasticsearch
Le besoin Il existe dans Hibernate une fonctionnalitĂ© que jâaime beaucoup : la mise Ă jour automatique du schĂ©ma de la base en fonction des entitĂ©s manipulĂ©es. Mon besoin est de faire quasiment la mĂȘme chose avec Elasticsearch. Câest Ă dire que je souhaite pouvoir appliquer un mapping pour un type donnĂ© Ă chaque fois que je dĂ©marre mon projet (en lâoccurrence une webapp). En me basant sur le projet dĂ©veloppĂ© par Erez Mazor, jâai donc dĂ©veloppĂ© unefactory Spring visant Ă dĂ©marrer des clients (voire des noeuds) Elasticsearch.Mon talk sur Elasticsearch sĂ©lectionnĂ© pour Devoxx France
Bonjour, Câest avec une certaine Ă©motion et fiertĂ© que jâai appris samedi dernier la sĂ©lection de mon talk sur Elasticsearch Ă Devoxx France. Devoxx France est une confĂ©rence organisĂ©e du 18 au 20 avril 2012 Ă Paris, pour les DĂ©veloppeurs. Y faire parti au milieu de talents incroyables est vraiment un honneur. Je suis dâautant plus comblĂ© que je vais pouvoir parler du sujet qui me passionne depuis maintenant 1 an : Elasticsearch. A lâorigine, Shay Banon devait venir lui-mĂȘme nous parler de lâanalyse des donnĂ©es avec les facettes Elasticsearch, mais il ne pourra malheureusement pas ĂȘtre prĂ©sent.
Quel client Java pour elasticsearch ?
Il existe deux modes dâaccĂšs Ă elasticsearch en Java : Inscrire un noeud client dans le cluster elasticsearch Utiliser un client âsimpleâ Noeud client dans un cluster elasticsearch LâidĂ©e de cette mĂ©thode est de fabriquer un noeud elasticsearch (node) qui dĂ©marre avec les mĂȘmes caractĂ©ristiques quâun noeud dâindexation et de recherche sauf quâon lui prĂ©cise quâil nâhĂ©bergera pas de donnĂ©es. Pour cela, on utilise la propriĂ©tĂ© suivante : node.data=false Elle indique que le noeud que nous dĂ©marrons nâhĂ©bergera pas de donnĂ©es.Mon premier plugin elasticsearch : RSS River
Il existe dans elasticsearch la notion de river (riviĂšre) qui comme son nom le laisse supposer permet de voir sâĂ©couler des donnĂ©es depuis une source jusquâĂ elasticsearch. Au fur et Ă mesure que les donnĂ©es arrivent, la riviĂšre les transporte et les envoie Ă lâindexation dans elasticsearch. En standard, il existe 4 riviĂšres : CouchDB qui permet dâindexer toutes les nouveautĂ©s dâune base CouchDB. Voir aussi cet article Ă ce propos. RabbitMQ qui permet de rĂ©cupĂ©rer des documents dans une queue de traitement asynchrone (genre JMS) Twitter qui permet dâindexer votre flux de messages twitter par exemple Wikipedia qui permet dâindexer toutes les nouveautĂ©s de lâencyclopĂ©die au fur et Ă mesure de leurs publications Premiers pas Jâai commencĂ© par bidouiller un peu la riviĂšre CouchDB pour y apporter quelques fonctionnalitĂ©s dont mes collĂšgues avaient besoin :
Elasticsearch et les "facets"
Les aventures avec Elasticsearch se poursuivent. Combien de fois ai-je dit rĂ©cemment que ce projet est absolument gĂ©nial et quâil va constituer Ă mon sens un des projets majeurs des prochaines annĂ©es⊠Qui nâa pas besoin de moteur de recherche ? Qui sâest dĂ©jĂ âemmerdĂ©â Ă fabriquer ça lui-mĂȘme ou Ă utiliser des briques pouvant aider au prix dâune complexitĂ© plus ou moins grande de mise en oeuvre ? Je crois que nous sommes tous passĂ©s par lĂ !CouchDB
AprĂšs avoir testĂ© Elasticsearch, me voici parti pour regarder ce monde Ă©trange quâon appelle le NoSQL⊠A dire vrai, jâai entendu ce mot il y a quelques annĂ©es, sans jamais vraiment mây interesser⊠AprĂšs tout, une base de donnĂ©es non SQL, ça nâest tout simplement pas possible !!! Puis, Ă force de cotoyer le monde dâElasticsearch et les technos JSon et REST, je me lance. Pour des raisons trĂšs pratiques, je choisis CouchDB de Apache. Dâune part, il est directement intĂ©grable avec Elasticsearch, et Ă la lecture rapide de sa documentation, il semble rĂ©pondre Ă un des besoins auquel une Ă©quipe de mon pĂŽle de dĂ©veloppement est confrontĂ©e.
La recherche Ă©lastique...
Elasticsearch, un projet mature en quelques mois⊠A suivre de trĂšs prĂšs ! En cherchant un bout de code pour rendre la couche Hibernate Search facilement distribuable sur un cluster de machines JBoss, je suis tombĂ© sur le projet Elasticsearch. Au dĂ©but, un peu interloqué⊠Puis, je me lance⊠Je tĂ©lĂ©charge le projet. Je dĂ©zippe. Je lance⊠Miracle. En quelques secondes, je dispose dâun outil dans un Cloud, simple, me permettant dâindexer nâimporte quel type de document, de le rĂ©cupĂ©rer et de faire une recherche (au sens google du terme) sur nâimporte quel champ⊠Et cela, quelque soit la technologie employĂ©e (Java, C#, .Installation FusionForge 5.0 sur Redhat 5
Voici la suite de lâarticle sur lâinstallation dâune forge. Finalement, le temps dâobtenir une machine sous Redhat 5 a laissĂ© le temps Ă la team FusionForge de sortir une release finale de la version 5.0. Nous voilĂ donc lancĂ©s dans cette installation que je me propose de dĂ©crire ici. A noter que pour le moment la forge nâest pas totalement opĂ©rationnelle. Des Ă©volutions dans la configuration devront ĂȘtre menĂ©es et jâespĂšre pouvoir tenir Ă jour cet article pour les dĂ©crire.