Pattern Finding In Log Data Using Hive on Hadoop

Authors: Swapna Sahu

DOI: https://doi.org/10.17605/OSF.IO/F7WAE

Short DOI: https://doi.org/ggkv9m

Country: India

Full-text Research PDF File:   View   |   Download

Abstract: Web log file ,in the computing context, is the log file which get routinely generated and maintained by a web server. Analysing web server access logs will give information regarding user’s behavior. Log files generate data which contain valuable information from the user which get stored in the web server. Server logs act as a guest sign-in sheet. Log files give information about the pages which had a heavy traffic and least. What sites refer visitors to your site? What pages that your visitors view? Because of the tremendous usage of web, the web log files are growing at faster rate and the size is becoming huge. Processing this explosive growth of log files using relational database technology has been facing a bottle neck. To analyse such large datasets we need parallel processing system and reliable data storage mechanism, Big data uses the Hadoop where massive quantity of information is processed using cluster of commodity hardware. In this paper we present the Hadoop framework for storing and processing large log files and also analysing through hive,
Hive is used in pre-processing of voluminous of log files and help us to find out the statics present in website and which help in our learning too.We can also perform optimization on hive query and we also compare the performance of both the analytical tools on analysing log files.

Keywords: Hadoop, data mining, logfile analysis, behaviour mining, web mining, hive, pig. (keywords)

Paper Id: 211

Published On: 2016-11-24

Published In: Volume 4, Issue 6, November-December 2016

Cite This: Pattern Finding In Log Data Using Hive on Hadoop - Swapna Sahu - IJIRMPS Volume 4, Issue 6, November-December 2016. DOI 10.17605/OSF.IO/F7WAE

Share this