AWStats logfile analyzer 7.0 Documentation

  


AWStats Installation, Configuration and Reporting

There are 3 steps to begin using AWStats:


I. Setup: Installation and configuration using awstats_configure.pl


A) Setup for an Apache or compatible web server (on Unix/Linux, Windows, MacOS...)

* Step 1:

(if you use a package provided with a Linux distribution or Windows installer, step 1 might have already been done; if you don't know, you can run this step again)

After downloading and extracting the AWStats package, you should run the awstats_configure.pl script to do several setup actions. You will find it in the AWStats tools directory (If using the Windows installer, the script is automatically launched):
perl awstats_configure.pl

* Step 2:

Once a configuration file has been created (by awstats_configure.pl, by your package installer or just by a manual copy of awstats.model.conf), it's important to verify that the "MAIN PARAMETERS" match your needs.  Open awstats.mysite.conf in your favorite text editor (i.e. notepad.exe, vi, gedit, etc) - don´t use a word processor - and make changes as required.

Particular attention should be given to these parameters:
- Verify the LogFile value.  It should be the full path of your server log file (You can also use a relative path from your awstats.pl directory, but a full path avoids errors).
- Verify the LogType value.  It should be "W" for analyzing web log files.
- Check if LogFormat is set to "1" (for "NCSA apache combined/ELF/XLF log format") or use a custom log format if you don't use the combined log format.
- Set the SiteDomain parameter to the main domain name or the intranet web server name used to reach the web site to analyze (Example: www.mysite.com). If you have several possible names for same site, use the main domain name and add the others to the list in the HostAlias parameter.
- You can also change other parameters if you want. The full list is described in Configurations/Directives options page.

Installation and configuration is finished. You can jump to the Process logs: Building/updating statistics database section.



B) Setup for Microsoft's IIS server

* Step 1:

Configure IIS to create logs in the "Extended W3C log format" (You can still use your own custom log format but setup is easier if you use the standard extended format). To do so, start the IIS management console snap-in, select the appropriate web site and open its Properties. Choose "W3C Extended Log Format", then Properties, then the Tab "Extended Properties" and uncheck everything under Extended Properties. Once they are all cleared, check just the following fields:
date
time
c-ip
cs-username
cs-method
cs-uri-stem
cs-uri-query
sc-status
sc-bytes
cs-version
cs(User-Agent)
cs(Referer)

To be sure the log format change is effective, you must stop IIS, backup it up (if you desire) and remove all of the old log files, restart IIS and go to your homepage. This is an example of the type of records you should find in the new log file:
2000-07-19 14:14:14 62.161.78.73 - GET / 200 1234 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+5.01;+Windows+NT+5.0) http://www.from.com/from.htm

* Step 2:

Copy the contents of the AWStats provided cgi-bin folder, from where the AWStats package put it on your local hard drive, to your server's cgi-bin directory (this includes awstats.pl, awstats.model.conf, and the lang, lib and plugins sub-directories).

* Step 3:

Move AWStats icon sub-directories and its content into a directory readable by your web server, for example C:\yourwwwroot\icon.

* Step 4:

Create a configuration file by copying awstats.model.conf to a new file named awstats.mysite.conf where "mysite" is a value of your choice but usually is the domain or virtual host name. This new file must be saved in the same directory as awstats.pl (i.e. cgi-bin).

* Step 5:

Edit your new awstats.mysite.conf file to match your specific environment:
- Change the LogFile value to the full path of your web server log file (You can also use a relative path from your awstats.pl (cgi-bin) directory).
- Change the LogType value to "W" for analyzing web log files.
- Change the LogFormat to 2 if you are using the "Extended W3C log format" described in step 1; in the case of a custom format, list the IIS fields being logged, for example:
LogFormat="date time c-ip cs-username cs-method cs-uri-stem cs-uri-query sc-status sc-bytes cs-version cs(User-Agent) cs(Referer)"
- Change the DirIcons parameter to reflect relative path of icon directory.
- Set the SiteDomain parameter to the main domain name or the intranet web server name used to reach the web site being analyzed (Example: www.mydomain.com).
- Set the AllowToUpdateStatsFromBrowser parameter to 1 if you don't have command line access and have only cgi access.
- Review and change other parameters if appropriate.

Installation and configuration is finished. You can jump to the Process logs: Building/Updating statistics database section.

C) Setup for other web servers

The setup process is similar to the setup for Apache or IIS.
Use LogFormat to value "3" if you have WebStar native log format. Use a personalized LogFormat if your log format is other.

D) Setup for other Internet servers, i.e. FTP, Mail, Streaming media

The setup process for other file formats is described in the relevant FAQ topics:  FAQ-COM090: FTP  FAQ-COM100: Mail and FAQ-COM110: Streaming media.


II. Process logs: Building/updating statistics database


* Update from command line (recommended):

The first log analysis should be done manually from the command line since the process may be long and it's easier to solve problems when you can see the command output (if you don't have Command Line access, skip to Step 2). The AWStats create (and update) statistics database command is:

perl awstats.pl -config=mysite -update

where mysite must be substituted with the domain/virtual host name you selected earlier during AWStats configuration.

AWStats will read the configuration file awstats.mysite.conf (or if not found, awstats.conf) and create/update its database with all summary information issued from analyzed log file.

AWStats statistics database files are saved in directory defined by the DirData parameter in configuration file.
When the create/update is finished, you should see a similar result on your screen:

Update for config "/etc/awstats/awstats.mysite.conf"
With data in log file "/pathtoyourlog/yourlog.log"...
Phase 1 : First bypass old records, searching new record...
Searching new records from beginning of log file...
Phase 2 : Now process new records (Flush history on disk after 20000 hosts)...
Jumped lines in file: 0
Parsed lines in file: 225730
 Found 122 dropped records,
 Found 87 corrupted records,
 Found 0 old records,
 Found 225521 new qualified records.

Dropped records are records discarded because they were not "user HTTP requests" or were requests matching AWStats filters (See the SkipHosts, SkipUserAgents, SkipFiles, OnlyHosts, OnlyUserAgents and OnlyFiles parameters). If you want to see which lines were dropped, you can add the -showdropped option on the command line.

Corrupted records are records that do not match the log format defined by the "LogFormat" parameter in the AWStats configuration file. All web servers will typically have a few corrupted records (<5%) even when everything works correctly. This can result for several reasons: 1) Web server internal bugs, 2) bad requests made by buggy browsers, 3) a dirty web server shutdown, such as unplugging the server... 

If all of your lines are corrupted and the LogFormat parameter in AWStats configuration file is correct, then there may be a setup problem with your web server log format. Don't forget that your LogFormat parameter in the AWStats configuration file MUST match the log file format you analyze.  If you want to see which lines are corrupted, you can add the -showcorrupted option on the command line.

Old records are simply records that were already processed by a previous update session. Although it is not necessary to purge your log file after each update process, it is highly recommended that you do so as often as possible.

New records are records in your log file that were successfully used to build/update the statistics database.

Note: A log analysis process might be slow (one second for each 4500 lines of your logfile with an Athlon 1Ghz, plus DNS resolution time for each different IP address in your logfile if DNSLookup is set to 1 and not already done in your log file).  See the Benchmarks page for more detailed information.


* Update from a browser:

AWStats statistics can also be updated from a browser, providing real-time statistics, by clicking the "Update now" link that appears when AWStats is used as a CGI (The URL is described in the next section 'Run reports: Building and reading reports').

Warning!!
To enable this link, the configuration file parameter AllowToUpdateStatsFromBrowser must be set to 1 (The link is not enabled by default).
Using the on-line update does not prevent you from running the update process automatically on a scheduled basis (the command is same as that of the first update process above).
For this, you have two choices:
- Include the update command in your logrotate process. See FAQ-COM120 for details.
- Or add instructions in your crontab (Unix/Linux) or your task scheduler (Windows), to regularly launch the Awstats update process. See FAQ-COM130 for details.

See the AWStats Benchmarks page for the recommended update/logrotate frequency.


III. Run reports: Building and reading reports


To see the analysis results, you have several options depending on your security policy.

Note: you must have created a statistics data base for the analysis period by processing your log files before you try to create reports.  See the previous section.

1. The first option is to build the main reports, in a static HTML page, from the command line, using the following syntax (skip to the second option if you only have CGI access):

perl awstats.pl -config=mysite -output -staticlinks > awstats.mysite.html

where mysite must be substituted with the domain/virtual host name you selected earlier during AWStats configuration.

To create specific individual reports, specify the report name on the command line as follows¹:
perl awstats.pl -config=mysite -output=alldomains -staticlinks > awstats.mysite.alldomains.html
perl awstats.pl -config=mysite -output=allhosts -staticlinks > awstats.mysite.allhosts.html
perl awstats.pl -config=mysite -output=lasthosts -staticlinks > awstats.mysite.lasthosts.html
perl awstats.pl -config=mysite -output=unknownip -staticlinks > awstats.mysite.unknownip.html
perl awstats.pl -config=mysite -output=alllogins -staticlinks > awstats.mysite.alllogins.html
perl awstats.pl -config=mysite -output=lastlogins -staticlinks > awstats.mysite.lastlogins.html
perl awstats.pl -config=mysite -output=allrobots -staticlinks > awstats.mysite.allrobots.html
perl awstats.pl -config=mysite -output=lastrobots -staticlinks > awstats.mysite.lastrobots.html
perl awstats.pl -config=mysite -output=urldetail -staticlinks > awstats.mysite.urldetail.html
perl awstats.pl -config=mysite -output=urlentry -staticlinks > awstats.mysite.urlentry.html
perl awstats.pl -config=mysite -output=urlexit -staticlinks > awstats.mysite.urlexit.html
perl awstats.pl -config=mysite -output=browserdetail -staticlinks > awstats.mysite.browserdetail.html
perl awstats.pl -config=mysite -output=osdetail -staticlinks > awstats.mysite.osdetail.html
perl awstats.pl -config=mysite -output=unknownbrowser -staticlinks > awstats.mysite.unknownbrowser.html
perl awstats.pl -config=mysite -output=unknownos -staticlinks > awstats.mysite.unknownos.html
perl awstats.pl -config=mysite -output=refererse -staticlinks > awstats.mysite.refererse.html
perl awstats.pl -config=mysite -output=refererpages -staticlinks > awstats.mysite.refererpages.html
perl awstats.pl -config=mysite -output=keyphrases -staticlinks > awstats.mysite.keyphrases.html
perl awstats.pl -config=mysite -output=keywords -staticlinks > awstats.mysite.keywords.html
perl awstats.pl -config=mysite -output=errors404 -staticlinks > awstats.mysite.errors404.html

¹If you prefer, you can use the awstats_buildstaticpages tool to build all these pages in one command, or to generate PDF files.

Notes:

a) You can also add a filter on the following reports: urldetail, urlentry, urlexit, allhosts, refererpages.  The filter can be a regexp (regular expression) on the full key you want AWStats to report on and is appended to the output parameter separated by a ":".

For example, to output the urldetail report, including only pages which contain /news in their URL, you can use the following command line:
perl awstats.pl -config=mysite -output=urldetail:/news -staticlinks > awstats.mysite.urldetailwithfilter.html

b) If you want to build a report for a particular month, add the options -month=MM -year=YYYY where MM is the month expressed as two digits, i.e. 03, and year is the four digit year.  To build a report for a full year, add the options -month=all -year=YYYY (warning: this is often resource intensive and might use a lot of memory and CPU.  Unix/Linux like operating systems might benefit from use of the "nice" command.)


2) The second option is to dynamically view your statistics from a browser.  To do this, use the URL:
http://www.myserver.mydomain/awstats/awstats.pl?config=mysite

where mysite specifies the configuration file to use (AWStats will use the file awstats.mysite.conf).

All output command line options (except -staticlinks and -logfile) are also available when using AWStats with a browser. Just use them as URL parameters: change "-option" to "&option", i.e.  http://www.myserver.mydomain/awstats/awstats.pl?month=MM&year=YYYY&output=unknownos

Reports are generated in real time from the statistics data base.  If this is slow, or putting too much load on your server, consider generating static reports instead.

If the AllowToUpdateStatsFromBrowser parameter is set to 1 in AWStats configuration file, you will also be able to run the update process from your browser. Just click on the link "Update now".