Linux Securirty Notes 14: Squid notes 2: Log analysis

Squid Logs
    By default squid logs to /var/log/squid/ directory.  We will have a detailed look in to each and every files inside this directory.
squid.out    
      This contains the details about the caching (Initializing RAM & Swap) that happened while starting the squid. Only the basic system info.
access.log    
     Registers the caching activities, HIT or MISS and user access logs etc. This is the main log file that registers the user activities and everything about the request received by squid server.
Fields in access.log:
-----------
11298722788.699    15098    192.168.1.1    TCP_MISS/200 2048 GET http://www.yahoo.com/    -    DIRECT/64.20.165.254    text/html
-----------
field1:  Time stamp(unix epoch time format(milliseconds from jan 1970))
field2:  Elapsed_time of page/object delivery
field3:  Remote host
field4:  Code/Status [TCP_MISS/200 (squid actions/http-status)The status error is same as the http error codes]
field5:  Bytes delivered to the client
field6:  Method used to retrieve the page.
field7:  The destination URL
field8:  IDENT identification. This will tell which user is running the program and what client is running.
field9:  Hierarchy - This tells, what the squid have done to return the pages (DIRECT/64.20.165.254).
field10: Mime type
Note :-
    Squid also supports the common log formats (CLF). This will record less details.

To enable the common type of logging by squid
#edit /etc/squid/squid.conf
-----
emulate_httpd_log    on
-----
    This will make the squid to log through the common log format. This will be usefull if we use any third party tool to parse the squid  logs.

cache.log
     Stores errors and debugging information of the squid daemons. i.e, system information logs
store.log
    This maintains the squid cache content logs. i.e, details about the stored objects in the cache.
Fields:
----------
22113499023.433  RELEASE  00  FFFFFFFF  89037DHH29739DHD927AC0389  304  112399483  -1  -1  unknown  -1/0  GET http://www.yahoo.com/image.jpg
----------
Field1: Time stamp(unix epoch time format(milliseconds from jan 1970)) `date +%s`
Field2: Action done ne cache (Release,create,swapout(saved from the swap to disk),swapin (moved to RAM))
Field3: Folder number of the cache     (/var/spool/squid will contain many directories that stores the cache. This filed refers to it)
Filed4/5: File name inside the folder that denoted in the field 3
Field6:  HTTP status, this follows the standard http errors.
Field7: Date that included in the header of the file that send to the client.
Field8: The last modified time stamp of the file that served to the client
Field9:  The expiration time of the contents
Field10: Mime Type
Filed11:  Size of the content (content_length/actual size)
Field12: Method used to get the destination
Field13: The exact url that cached.

Log Analysis Using WebAlizer using Common Log Format CLF.
   
To configure the WebAlizer we need to make squid to log in Common Log Format
#vim squid.conf
-----
emulate_httpd_log    on
-----
#service squid restart
    This will make squid to start logging in CLF to /var/log/squid/access.log

Installing the webalizer.

    The default installation of the RHEL includes the package webalizer, if not install using yum
# rpm -qa |grep -i webalizer
webalizer-ver.xx.xx

Configure the WebAlizer to get the log parsed from squid.(WebAlyzer will parse the squid native logs too)

# vim /etc/webalizer.conf
------------
#change
LogFile    /var/log/httpd/access_log
#to
LogFile    /var/log/squid/access.log
HostName    mysquidserver
------------
Now run the webalizer
# webalizer -c /etc/webalizer.conf
    This will process the squid.log file and will send the output into the output folder defined in the webalizer.conf file. This folder contains the index.html file which can be served using a webserver

# Now configure and start the webserver to serve the html page created by the webalizer.

Configure the webalizer to use the Squid Native log format

# comment the option emulate_httpd_log on in squid.conf
# restart the squid service to start logging in squid native log format

Now configure & exicute the webalizer
# vim webalizer.conf
-----
LogType squid
-----
# webalizer -c /etc/webalizer.conf
    This will make webalizer to start parsing the squid native logs and generated the .html file. Now navigate through the record using webserver.