Squid Logs
By default squid logs to /var/log/squid/ directory. We will have a detailed look in to each and every files inside this directory.
squid.out
This contains the details about the caching (Initializing RAM & Swap) that happened while starting the squid. Only the basic system info.
access.log
Registers the caching activities, HIT or MISS and user access logs etc. This is the main log file that registers the user activities and everything about the request received by squid server.
Fields in access.log:
-----------
11298722788.699 15098 192.168.1.1 TCP_MISS/200 2048 GET http://www.yahoo.com/ - DIRECT/64.20.165.254 text/html
-----------
field1: Time stamp(unix epoch time format(milliseconds from jan 1970))
field2: Elapsed_time of page/object delivery
field3: Remote host
field4: Code/Status [TCP_MISS/200 (squid actions/http-status)The status error is same as the http error codes]
field5: Bytes delivered to the client
field6: Method used to retrieve the page.
field7: The destination URL
field8: IDENT identification. This will tell which user is running the program and what client is running.
field9: Hierarchy - This tells, what the squid have done to return the pages (DIRECT/64.20.165.254).
field10: Mime type
Note :-
Squid also supports the common log formats (CLF). This will record less details.
To enable the common type of logging by squid
#edit /etc/squid/squid.conf
-----
emulate_httpd_log on
-----
This will make the squid to log through the common log format. This will be usefull if we use any third party tool to parse the squid logs.
cache.log
Stores errors and debugging information of the squid daemons. i.e, system information logs
store.log
This maintains the squid cache content logs. i.e, details about the stored objects in the cache.
Fields:
----------
22113499023.433 RELEASE 00 FFFFFFFF 89037DHH29739DHD927AC0389 304 112399483 -1 -1 unknown -1/0 GET http://www.yahoo.com/image.jpg
----------
Field1: Time stamp(unix epoch time format(milliseconds from jan 1970)) `date +%s`
Field2: Action done ne cache (Release,create,swapout(saved from the swap to disk),swapin (moved to RAM))
Field3: Folder number of the cache (/var/spool/squid will contain many directories that stores the cache. This filed refers to it)
Filed4/5: File name inside the folder that denoted in the field 3
Field6: HTTP status, this follows the standard http errors.
Field7: Date that included in the header of the file that send to the client.
Field8: The last modified time stamp of the file that served to the client
Field9: The expiration time of the contents
Field10: Mime Type
Filed11: Size of the content (content_length/actual size)
Field12: Method used to get the destination
Field13: The exact url that cached.
Log Analysis Using WebAlizer using Common Log Format CLF.
To configure the WebAlizer we need to make squid to log in Common Log Format
#vim squid.conf
-----
emulate_httpd_log on
-----
#service squid restart
This will make squid to start logging in CLF to /var/log/squid/access.log
Installing the webalizer.
The default installation of the RHEL includes the package webalizer, if not install using yum
# rpm -qa |grep -i webalizer
webalizer-ver.xx.xx
Configure the WebAlizer to get the log parsed from squid.(WebAlyzer will parse the squid native logs too)
# vim /etc/webalizer.conf
------------
#change
LogFile /var/log/httpd/access_log
#to
LogFile /var/log/squid/access.log
HostName mysquidserver
------------
Now run the webalizer
# webalizer -c /etc/webalizer.conf
This will process the squid.log file and will send the output into the output folder defined in the webalizer.conf file. This folder contains the index.html file which can be served using a webserver
# Now configure and start the webserver to serve the html page created by the webalizer.
Configure the webalizer to use the Squid Native log format
# comment the option emulate_httpd_log on in squid.conf
# restart the squid service to start logging in squid native log format
Now configure & exicute the webalizer
# vim webalizer.conf
-----
LogType squid
-----
# webalizer -c /etc/webalizer.conf
This will make webalizer to start parsing the squid native logs and generated the .html file. Now navigate through the record using webserver.
Tuesday, October 20, 2009
Linux Securirty Notes 14: Squid notes 1: Introduction
SQUID
Intro
Squid is a caching proxy for the Web supporting HTTP, HTTPS, FTP, and more. It reduces bandwidth and improves response times by caching and reusing frequently-requested web pages. Squid has extensive access controls and makes a great server accelerator. It runs on most available operating systems,
including Windows and is licensed under the GNU GPL.
Thousands of web-sites around the Internet use Squid to drastically increase their content delivery. Squid can reduce your server load and improve delivery speeds to clients. Squid can also be used to deliver content from around the world - copying only the content being used, rather than inefficiently copying everything. Finally, Squid's advanced content routing configuration allows you to build content clusters to route and load balance requests via a variety of web servers.
" [The Squid systems] are currently running at a hit-rate (the web page that served from cache) of approximately 75%, effectively quadrupling the capacity of the Apache servers behind
them. This is particularly noticeable when a large surge of traffic arrives directed to a particular page via a web link from another site, as the caching efficiency for that page will be nearly 100%. "
The normal setup is caching the contents of an unlimited number of webservers for a limited number of clients. Another setup is “reverse proxy” or “webserver acceleration” (using http_port 80 accel vhost). In this mode, the cache serves an unlimited number of clients for a limited number of—or just one—web servers.
Initialising SQUID
1. install the squid
Squid can be installed by rpm. Squid uses lot of system resources to server the request. So best configuration on servers will defnitly improve the squid service.It uses the cache so more the RAM better the performance.
For eg:- for a squid server of 100 users
need a /var space of 150G
RAM - the more the better
Even if you install the squid in lower configuration, to server a large communit squid will still server by optimising the current Hardware config. But for better result it is recommanded with higher configuration.
/usr/sbin/squid is the daemon installed by the package
/usr/sbin/squidclient is the binary installed, which is used to make query to the squid server to check whether the cache is available in the local or remote squid server
Note:-
Squid has a master process that will spawn the childs to handle the process. it binds to the default port 3128. The child proces is binding to the port 3128 not the master process. The same squid process is also bind to the udp port 3130 which used distribute the load for other squid servers, which is a peer to peer communication to balance the load.
2. configure the squid
The client makes the request to the proxy server and the proxy server will contact the detination, caches the page and serves content. This cache will be used for later serve .The Access control should be configured to get it working. modify /etc/squid/squid.conf file to configure the squid
3. start the squid service
The rpm installation can be started using service and can be configured using chkconfig command
CONFIGURATION OF SQUID
Initial/simple configuration:
Here the acl is configured for the internal and the permission is given by the operator http_access. The squid operates from the top to bottom of the configuration file. so once if the search pattern is found squid will terminate the search in the config file and will start processing.
Testing the squid by squidclient:
Applying the proxy settings for textbased http client/shell based tools.
Steps to enabling the proxy in wget/lftp/lynx
Step 1:
Step 2:
now start the clients
Intro
Squid is a caching proxy for the Web supporting HTTP, HTTPS, FTP, and more. It reduces bandwidth and improves response times by caching and reusing frequently-requested web pages. Squid has extensive access controls and makes a great server accelerator. It runs on most available operating systems,
including Windows and is licensed under the GNU GPL.
Thousands of web-sites around the Internet use Squid to drastically increase their content delivery. Squid can reduce your server load and improve delivery speeds to clients. Squid can also be used to deliver content from around the world - copying only the content being used, rather than inefficiently copying everything. Finally, Squid's advanced content routing configuration allows you to build content clusters to route and load balance requests via a variety of web servers.
" [The Squid systems] are currently running at a hit-rate (the web page that served from cache) of approximately 75%, effectively quadrupling the capacity of the Apache servers behind
them. This is particularly noticeable when a large surge of traffic arrives directed to a particular page via a web link from another site, as the caching efficiency for that page will be nearly 100%. "
The normal setup is caching the contents of an unlimited number of webservers for a limited number of clients. Another setup is “reverse proxy” or “webserver acceleration” (using http_port 80 accel vhost). In this mode, the cache serves an unlimited number of clients for a limited number of—or just one—web servers.
Initialising SQUID
- install the squid
- configure the squid
- start the squid service
1. install the squid
Squid can be installed by rpm. Squid uses lot of system resources to server the request. So best configuration on servers will defnitly improve the squid service.It uses the cache so more the RAM better the performance.
For eg:- for a squid server of 100 users
need a /var space of 150G
RAM - the more the better
Even if you install the squid in lower configuration, to server a large communit squid will still server by optimising the current Hardware config. But for better result it is recommanded with higher configuration.
# rpm -qpl squid.ver.rpm
This will tell us that what are the changes that will be done to the system after the installation of the squid package.# rpm -ivh squid-ver.rpm
This will install the squid package./usr/sbin/squid is the daemon installed by the package
/usr/sbin/squidclient is the binary installed, which is used to make query to the squid server to check whether the cache is available in the local or remote squid server
Note:-
Squid has a master process that will spawn the childs to handle the process. it binds to the default port 3128. The child proces is binding to the port 3128 not the master process. The same squid process is also bind to the udp port 3130 which used distribute the load for other squid servers, which is a peer to peer communication to balance the load.
2. configure the squid
The client makes the request to the proxy server and the proxy server will contact the detination, caches the page and serves content. This cache will be used for later serve .The Access control should be configured to get it working. modify /etc/squid/squid.conf file to configure the squid
3. start the squid service
The rpm installation can be started using service and can be configured using chkconfig command
CONFIGURATION OF SQUID
Initial/simple configuration:
#vim squid.conf
#Go to Access controls sessionacl int src 192.168.1.0/255.255.255.0
http_access allow int
Here the acl is configured for the internal and the permission is given by the operator http_access. The squid operates from the top to bottom of the configuration file. so once if the search pattern is found squid will terminate the search in the config file and will start processing.
Testing the squid by squidclient:
#which squidclient
/usr/sbin/squidclient
This is a small utility that retrieves the objects in the squid server cahce.#squidclient --help
Shows the options in squidclient# squidclient -h localhost -p 3128 http://www.google.com
or#squidclient http://www.google.com
This will return the cache from the squid server located at localhost:3128. if the squid is running on localhost with the default port no need to specify the -h (host) of -p (port) options.# squidclient -g 3 http://www.google.com
This will tell that how speed the squid preformance to get the page http://www.google.com downloaded. and does for the 3 times (0-for infinite). we can verify the timing and can check whether the pages are cached or not. If there is caching happens then only the initial query will take time and others will take considerable less time. But if the webpage is not serverd from cache then all the request will have the same time. The caching permissions in the website (html code) is honoured by squid.# squidclient -v http://www.google.com
This will dump all the contents of the webpage to the STDOUT.#squidclient -h squidserver.domain.com -g 3 http://www.google.com
This will make query about the webcache to the given URL with the remote squid server.Applying the proxy settings for textbased http client/shell based tools.
Steps to enabling the proxy in wget/lftp/lynx
Step 1:
export http_proxy=http://proxy.domain.com:3128
This variable has been used by almost all the text based http clients.Step 2:
now start the clients
# wget http://remotewebiste.com
Now check the squid access log to find out the access request done by the client.# lftp http://remotewebiste.com
This will allow to download the http pages. This tool will honour the 'http_proxy' variable.#lynx http://remotewebiste.com
This will serve the webpage from the proxy
Subscribe to:
Posts (Atom)