I’m often running some demos during conferences where we have a booth. As many others, I’m using Twitter feed as my datasource.
Let’s do it!
Let’s assume that you have already elasticsearch 1.5.2, Logstash 1.5.0 and Kibana 4.0.2 running on your laptop or on a cloud instance.
Create first your Twitter application and open the “Keys and Access Tokens” tab.
consumer_secret (generate them if needed).
Note also your
access_token_secret (generate them if needed).
First define your twitter input to track whatever term you want. Let’s say here that I will collect data for dotScale conference (Elastic sponsors it so if you are around come to say hello at our booth!):
1 2 3 4 5 6 7 8 9 10
We won’t do any filtering as tweets come as JSON documents already well formed. We could of course omit some fields but let’s keep that simple:
1 2 3 4 5 6 7 8 9 10 11
We have set that we will use a
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
We are basically using something similar to logstash default template but we also disable
raw subfield for
message field and we define that
coordinates.coordinates is actually a geo_point.
Then, we can start logstash with this configuration and let it run forever…
If you send some tweets, you should be able to see them indexed in elasticsearch:
testing dotscale twitter wall. #ignore— David Pilato (@dadoonetest) 1 Juin 2015
This should give you some tweets back.
And now you can play with Kibana!
Open your data (but secure them first)!
If you want to share your results, you should secure your elasticsearch instance before opening it to the world!
I tried at first to add a Ngnix layer but I had hard time configuring it. I decided then to use Shield which is a free add-on for elasticsearch customers (yeah we have a fantastic support team who can definitely help you to build the best cluster ever).
Shield has a 30 days evaluation period so here I can use it as I will most likely track data only from few days before the conference and to some days after.
Then you can a new user who has the default
Give whatever password you want…
Modify Logstash configuration as now your elasticsearch output needs to provide credentials:
1 2 3 4 5 6 7 8 9 10 11 12
Restart Logstash and you’re done!
You probably want to also create another user who can access to Kibana4:
Set your password. And now you should be able to connect to Kibana4 using your username and password.
After dotScale update
I finally got this result after one day at dotScale.