Kibana Overview - Visualize and Manage Your Data
hello and welcome to the next lesson and the series Master the elastic stack all you need to get started with the elastic stack we're are covering how to start using kibana to understand the data that you sent to the cluster and walk through the interface and explore the features and how you can start using them we'll explore things like navigating the interface creating visualizations building dashboards and dive into analytics and look at some management tasks one of the learning objectives of this course is to be able to understand the data ingested into elastic search with visualizations and dashboards so we'll jump into the demo and we'll log into kibana this is the login page for kibana and as we mentioned before kibana is the graphical user interface you use to manage the cluster and manage the data on the cluster and create dashboards to be able to understand the data so we're going to log in with the elastic super user this is the homepage of kibana when you first log in you'll find the four major areas that elastic uh stack focuses on the search the observability security analytics they are all links to access the most commonly used features such as like dashboards and visualizations and other helpful links that will take you to some management tasks at the bottom of the screen here or you can upload a file to start exploring data or try some sample data and all of these links are available from the side menu and at the top we have a search bar where you can search um any app or feature or an index or a setting and while we're here I'm just going to the advanced settings I'm going to look for one setting to use as it's the dark mode I'm going to enable that and at the bottom I'm just going to click save and reload the page and this is how I like to use my Kean interface and if we go back quickly to advanced settings just under management stack management and under kibana here on the left we have advanced settings we have a search bar here that we can search the settings or you can start navigating them and look for the settings but I'm going to mention uh two or three important ones the default route is where you set the landing page when you log into kibana for example here we are landing at the homepage where we landed before you can change that to a custom dashboard that you want to see the first thing when you log into kibana you have these time filter quick ranges here this is where you click on the time range when you are looking at the dashboard or at the Discover app where you are looking at the logs and you can set custom ones if you want for example have one for yesterday or for this month or last month you can create those custom ones instead of other ones that you don't usually use now if you go back to the home screen and access analytics here this is where we can start building dashboards and visualizations we can access uh the Discover app where you can look uh at the logs or the documents and you can do more custom visualizations and create maps and and build some machine learning models and uh detection rules or you can access analytics from the side menu you have the same thing discover and dashboard and canvas and you have the visualize Library where you can go to all the visualizations created so we'll access the visualized library and I'm just going to turn off my camera here and there are some visualizations created already and loaded from file beat when we connected file beat to kibana they don't necessarily have any data because they are waiting for specific data sources so I'm going to create a visualization and I'm going to pick this lens tool here it's a drag and drop tool and the most important things here is to select the index that has the data you need to visualize and the time stamp or the time filter right now we're looking at today or the last 2 2 hours so from 12: a.m. all the way to the current time and we can change it to the last 30 minutes or hour or whatever you need um to see and here on the horizontal axis I'm going to pick date histogram and I'm going to pick the time stamp and it is going to select the time interval as Auto it will adjust automatically and then on the vertical axis I'm just going to keep it simple uh we have many functions we need to if you need to look at the average of something or the count or the minimum or maximum I'm just going to keep it simple and select count and by default it's looking at the count of Records or the count of logs that are coming from the firewall or we can drag and drop the fields here it's already selected by default and we can break them down for example I know of a field called Source name it's the source name of devices in my home network so now it's looking at the top three values and the rest are grouped under other if you don't want to see the other you can go to Advanced here select or turn off this option group remaining values as others so now it's looking at only three values or three Source devices and it's breaking the count of Records by these three devices I'm going to increase this number to see the top eight devices these are all on my home network if we want to save it give it a title and we can add it to a dashboard we can add it to an existing one or a new one or just no dashboard we can add a description and we can add tags if you want to group them based on tags just going to save to the library and now we have this visualization that we can see here I'm going to create another visualization and this time I'm going to choose one of these aggregation based visualizations and from here we have more options to create uh areas and heat maps and data tables and I'm I'm going to choose this tag cloud and it will ask you to choose a source I'm going to choose the file beat index aggregation is I'm going to choose count or keep it as count under buckets here I'm going to be breaking them down based on a term and I'm going to say destination autonomous system organization name click on update because it's by descending order the top five destination organization uh autonomous system organizations and these are the top 15 autonomous system organizations my home network is connecting to and we have some options here for the visualization itself for example the color palet the orientation so I'm going to save that and give it a title I'll just add it to a library without a dashboard we're going to be adding a dashboard next so from the left menu here I'm going to access dashboards under analytics and we have all these built-in dashboards that are loaded uh from filebeat they don't have any data they're waiting for specific data sources but we're going to create a custom dashboard this is a new dashboard we can create visualizations from here andess access that menu to create a visualization or we can add from the library we're going to add our custom created ones first one I called it count of records and the second was destination autonomous system names I'll close this menu and now I have these two visualization I can change the size I can place them where I need them and click on Save I'll give it a name firewall logs and I'll select this option to store the time with the dashboard now from here I'm going to create a visualization it'll open the same lens tool from here I'm going to select a table so I'll start with with metrics I'm going to select count I need to look at the count right now in the last 3 hours we have 49,000 uh Records or logs going to select Source IP for the rows and we are looking at the top five Source IPS and if I access here Advanced and get rid of the other I'll have the top five Source IPS with the count of Records they are generating and from here I'm just going to save and return and it will add that table to my dashboard and I can also adjust the size of this table and click on Save now I can click on switch to view mode and the tools to add visualizations from library and create new visualizations will disappear and now we have this view for our dashboard all right let's jump into one of the most powerful Tools in kibana that Discover app this is where you can explore your logs search through your massive indexes and start your investigations whether you're troubleshooting issues analyzing patterns or filtering data the Discover app is your go-to place for diving into the details to get started let's head over to the Discover tab under analytics you can access it from here on the main screen or main homepage or from the side menu under analytics and discover so I'm just going to click on Discover right here and the first things first first let's talk about the index pattern kibana needs an index pattern or an or a data view to know which elastic search data set to show here we'll choose here file beat this is my data from my firewall and now we have a context or a data set that kibana is able to fetch and display on here next let's set the time range kibana filters data based on time which is super helpful when you're looking for specific events you can click on the time picker here in the top right corner and we have some predefined ranges like the last 15 minutes last an hour last 7 days or you can set a custom range if you know exactly when the issue occurred you can click here and you can choose the date the hour I'll choose for example December 8 uh 12: a.m. to December 8th 500 a.m. now I have a custom time range I can look for data that happened during that time and another another cool feature here is you can stream the data in real time almost real time and you can refresh every few seconds or maybe every minute or you can turn it off and refresh whenever you need now let's talk about searching with the search bar at the top here we have a search bar to look for specific words phrases or fields we can search a word for example like warning and kibana instantly filters the results for us or if you want to get more precise you can search specific fields and values for example we can say log [Music] level and it will autop populate with whatever values are available for that field you can choose that you can add operations and or or logic operations or if you know if you're looking for a specific Source IP or a destination autonomous system organization name we can hit enter and now it filters everything that is going to that autonomous system organization on the left side here you'll see the fields list this shows all the available fields in your LW logs for example the event action The Source IPS Network application and all of these fields and they have their own search bar so you can find the fields that you're looking for and if you click on one of the fields you can see the top results or top values that are in that field and from here you can click the plus sign to create a filter up here and now it filters the data. r with this value for event action accept or you can click the minus it will negate that value so everything other than accept and with these fields you can click this plus sign here and add them to the view here and start building columns or a table every field you select is added as a column if I want to look at denied traffic I can click on the plus sign and create a filter for deny and let's say I want to filter for SMB traffic and now I have this View and if you want you can save this search by clicking here on Save and giving it a title and now if you come back to the Discover app and you want to open that search again you can click here on open and search for that saved search and it will populate those columns again and those filters added here again and now when you see a log entry that interests you you can click on it to expand it this opens a detailed View with all the fields and their values for that specific log you can even copy or export this data if needed or you can view it in Json format and you can copy this view as well to recap kiban Discover app is perfect for viewing and searching logs in real time filtering data using time fields and values exploring detailed information for troubleshooting saving and sharing searches to streamline your investigations and this is where you usually start exploring your data before you create visualizations and building dashboards now let's jump into another Main Service that the elastic stack offers the search capabilities this is where elastic search shines as a search engine whether you're powering a website an internet or an application it's fast powerful and customizable exactly what elastic search was built for at the elastic stack core elastic search is a full text search engine that can process huge amounts of data and return relevant results in milliseconds you can use elastic search to search documents websites product cataloges or logs anything that involves text or structured data think of it like Google but built specifically for your data you can start building a search experience from this link here at the home screen or from the side menu here under search first you need to index your data in elastic search this could be articles blog posts products or any documents you want searchable then you can test and find tune your search queries directly in kibana you can use all all of these methods to ingest your data into elastic search then you can test and fine-tune your search queries directly in Cabana kibana allows you to analyze how searches are performing tune relevance scoring and improve the user experience and some of the use cases is a website search for example like for an e-commerce store elastic search can deliver fast and relevant product product search results for an inonet site elas search makes internal documents and knowledge bases easy to search and navigate and for apps and services you can add realtime search capabilities to provide a seamless user experience to recap kibana makes it simple to work with elastic search as a search engine you can index your data test your queries and optimize the search experience to make your it or app fast efficient and userfriendly now we'll look at observability in kibana this is one of the most powerful features in kibana this is where you can monitor and troubleshoot your systems in real time by consolidating your logs metrics application traces and system availability if you're running applications websites or infrastructure this is your go-to place for operational insights observability is all about having visibility into your systems with giana's observability Section you can bring together data from multiple sources like servers applications uh databases and network devices all in one place instead of switching between different tools you get this single unified view for your operation here's what you can do in the observability section you can view and search logs from all of your systems in one place currently I don't have any data in the logs I'm not monitoring any systems my file beat is running to ingest 40 net CIS logs but it's not actually monitoring the logs of any system but here you can filter by time severity or keywords to investigate issues quickly here under infrastructure you can monitor the health and performance of your infrastructure like the CPU usage memory disk input output or network traffic we have APM or application performance monitoring you can trace the flow of your application requests to identify bottle next errors or latency issues again I don't have any data being monitored right now it's just a very clean install and we have the uptime monitoring under this section here called synthetics this is where you track the availability of your systems or websites with instant alerts if something goes down so why is observability important it helps you detect issues faster whether it's a slow query a high CPU Spike or a system outage and you get endtoend visibility from backend servers to front-end applications and it lets your team work collaboratively to resolve incidents quickly you can jump from logs to metrics or from an error Trace directly to the related logs all in just a few clicks and with observability you can move from reactive troubleshooting to to proactive monitoring it's all about having the right data at your fingertips to keep your systems running smoothly now we're going to talk about another important section in kibana Security in today's world the Cyber threats are everywhere and protecting the infrastructure is not optional kibana security section is designed to help you prevent detect and respond to threats across your environment all in one unified interface so you can access security here here is just a welcome screen to this section my current plan is basic so it's the free license I don't have many of the features uh but we can dive into dashboards this is where you can see or get a high level overview of your security posture including threat Trends system activity detected anomalies and you have have the rules section here kibana comes with hundreds of pre-built rules for identifying suspicious activity like fail logins or unexpected file changes and you can create your custom rules as well you can also get alerted in real time for any incidents there's also the cases section here this is where you document and track incidents uh from Discovery to resolution uh you can assign tasks to team members and keep everything organized in one place in timelines here you can visualize and correlate events investigate suspicious activity and um piece together the story of an incident quickly and effectively in the intelligence section here this is a licensed feature this section helps you leverage threat intelligence feeds so if you have a license you can get indicators of compromise ioc's like malicious IPS and domains and file hashes and all that and in the explore area this area provides a flexible way to dig deeper into your data so with kibana security you can prevent attacks proactively detect threats in real time and respond quickly and effectively to incidents all right let's move into the management section of kibana on the home page page here I'm going to scroll to the bottom and there are a few links here this is where you find T elastic search manage users uh monitor your stack and deploy Integrations and manage the index life cycles and more we have a link here for Dev tools and the stack management and we can also access them by the side menu here under management and first let's go to Dev tools this is where we can run queries uh test out index configurations and troubleshoot directly using the console it's a powerful space for developers and admins to interact with elastic search in real time for example we can run cat queries or the Compact and aligned text queries these are human readable API commands used to get cluster Health node status index stats and other essential information for example we can run the command get CAD Health we can click the play button here to send the request or press control enter on the keyboard and we can have some information at the top here but we can add some parameters here for example V for verbos and if I zoom in for example this is the time time stamp the cluster name a status of the cluster is green how many nodes we have how many shards how many primary shards next is Integrations Integrations is where you add monitoring for services like engine X and AWS or Azure it is plug-in play making data collection seamless we have Fleet this is where you manage all your elastic agents across different environments agents collect data from your systems and push it to elastic search and we will be covering Fleet in an upcoming video we have osquery this is a powerful tool for endpoint visibility it lets you query operating systems as if they were databases and you can ask questions like what processes are running are there any open ports it's perfect for investigations and auditing directly within kibana we have the monitoring section you get a complete view of elastic search log stash and Kiana's Health metrics like cluster status disk usage memory all in here and in a future video we'll be covering covering monitoring vastic search nodes with file beat and Metric beat and at the bottom here we have stack management the core for configuring elastic search itself from here you manage index patterns in just pipelines uh cross cluster applications uh index life cycle policies manage users and configure kibana data views this management section is packed with tools that help you fine tune and maintain elastic search and kibana whether you're building new queries managing users or monitoring cluster Health this section is essential for any admin or engineer working with the elastic stack in this video we took a look at how to start using kibana to understand the data ingested in your cluster we created some example visualizations and a dashboard and we looked at the main features of the interface up next setting up cluster monitoring stay tuned and I will see you in the next one
2025-01-08 00:11
