Logs ( machine data ) is an classical example of unstructured data which often comes in gigabyte of space. These data presents interesting “analytical” opportunities which can increase the insight of a user into their applications. While working on these opportunities, We want to share some of the insights of the IBM's latest offering IBM Log Analytics Work Group Edition ( currently available as open beta ).
IBM Log Analytics Work Group Edition ( LAWG ) provides an state of art framework for analysis of unstructured and semi-structured data.
A Hello World Log Analysis !
We will cover details of installation and getting your IBM LAWG running with some of the IBM WebSphere application logs to see the capability of big data analytics provided by latest IBM offering.
Installation: Installation of IBM LAWG is pretty straightforward :
- You need to get a RHEL 5/6 64 bit machine ( or virtual machine ).
- Create a new user account and add it to the root group.
- In case your machine is not having python json support, download this rpm ( or from this link ) and update your python package.
- Download the open beta build <ADD_LINK_HERE>, extract it and invoke ./install.sh ( in case you love your putty or terminal window, you need to add “-C” argument to install script ).
- Accept default setting and you are good to GO ! ( In case, you are interested in customizing the installation, please check this quick start guide)
Once you have successfully completed the installation you can verify if you are able to access IBM LAWG home page ( if you have accepted the default installation setting, you should see the following UI at http://your_rhel_host_name:9988/Unity/ ).
Log Sources are mapping to corresponding log files, so if you have IBM WebSphere SystemOut log file for which you want to perform analysis, you may want to create an log source as WAS-SystemOut. Similarly for an IBM Maximo application log file, you may create an log source named as Maximo-Application.
We will be demonstrating log source creation for a dummy log file which we can be created by running some scripts which are provided with IBM LAWG.
Please note: You require access to admin UI of IBM LAWG to create a log source.
- Navigate to Administrative Settings link, open admin UI ( you require administrative privilege to view admin UI ).
- Navigate to Data Source section.
- Create a new log source
- Enter log source name ( lets put “WAS-TestLogSource” )
- Description for your log source ( lets put “My first log source” in this field )
- Select SourceType as “WASSystemOut”.
- Select Collection as “WASSystemOut-Collection1”.
- Enter short host name of your RHEL machine as host name here
- In the log path text box, you can specify $LAWG_HOME/logsources/was/SystemOut.log ( eg. /home/unity/IBM/LogAnalyticsWorkgroup/logsources/was/SystemOut.log ).
Once you have above log source ready, we are good to create an artificial dummy log file and we will try to index that file and perform some cool search operations on top of it.
- Navigate to $LAWG_HOME/utilities/
- Execute unity_populate_was_log.sh
unity_populate_was_log.sh will create a SystemOut.log at $LAWG_HOME/logsources/was which contains ( as mentioned above ) superficial dummy data.
You are ready to Search !
After completing above steps, you are good to go and perform search over the log data.
- Open/refresh your IBM LAWG home page.
- Select time filter as “Last one year” ( Dummy data have some really old records ).
- Enter “*” as the search query.
- And Click on Search !
Bingo ! You have successfully completed the search.
This is the Hello World for IBM Log Analytics Work Group edition, we will start a deep dive in further blog post ...till then Happy Analytics !!!
do let us know your feedback on Installation and getting started.
Any questions, feel free to comment or visit this forum.