%% LyX 1.3 created this file.  For more info, see http://www.lyx.org/.
 %% Do not edit unless you really know what you are doing.
 \documentclass[english]{article}
 \usepackage{times}
 \usepackage[T1]{fontenc}
 \usepackage[latin1]{inputenc}
 \setcounter{secnumdepth}{4}
 \setcounter{tocdepth}{4}
 \usepackage{setspace}
 \onehalfspacing
 
 \makeatletter
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% User specified LaTeX commands.
 \usepackage[official]{eurosym}
 \usepackage{listings}
 \usepackage{color}
 \lstset{ 
  commentstyle=\color{blue}, 
  keywordstyle=\color{red}, 
  frame=single, 
  breaklines, 
  showstringspaces=false,
  tabsize=2, 
  numbers=left, 
  numberstyle=\tiny, 
  stepnumber=5, 
  numbersep=5pt,
  basicstyle=\normalfont\footnotesize\tt,
  language={[]Perl} 
  } 
 
 \usepackage{babel}
 \makeatother
 \begin{document}
 
 \title{Mirroring the Virus Database}
 
 
 \author{Luca Gibelli}
 
 \maketitle
 Some guidelines for people interested in contributing to the distribution
 of ClamAV virus database.
 
 
 \section{Introduction}
 
 
 \subsection{This doc}
 
 The latest version of this document is always available at http://www.clamav.net/doc/mirrors/.
 Before going any further, please check that you are reading the latest
 version.
 
 Japanese sysadmins can find a translated version of this doc at http://www.orange.co.jp/\textasciitilde{}masaki/clamav/mirror-howto-jp.html
 (not necessarily up to date).
 
 
 \subsection{Who is responsible for the virus database}
 
 The virusdb team take care of reviewing virus signatures, checking
 for new viruses in the wild and committing changes to the virus database
 file.
 
 The updates are released quite often (usually no less than three times
 a week). If you want to be notified whenever the virus database is
 updated subscribe to clamav-virusdb \emph{at} lists.clamav.net . 
 
 Every time the virusdb team updates the database, the ChangeLog will
 be posted to the mailing-list. 
 
 Visit for the list description and archives. 
 
 If you need to contact the virusdb team please write to: virus-team
 \emph{at} clamav.net
 
 
 \subsection{Virus submission}
 
 Whenever you find a new virus which is not detected by ClamAV you
 should send it to the virusdb team by filling the form at http://www.clamav.net/sendvirus.html.
 They will review your submission and update the database so that the
 whole ClamAV user community can take benefit from it.
 
 \textbf{Never} send virus samples to ClamAV mailing-lists or developers
 addresses.
 
 
 \subsection{Getting a copy of the latest virus database}
 
 The most important factor for an antivirus's efficiency is to be up
 to date. ClamAV comes with a tool to update the virus database automatically:
 its name is \emph{freshclam}.
 
 freshclam looks up the TXT record associated with \textit{current.cvd.clamav.net}
 and extracts the latest database version available from the string
 returned. If the local database is outdated, freshclam tries to connect
 to the hostnames listed in freshclam.conf (DatabaseMirror directive).
 If the first server in the list fails or the latest database is not
 available on that mirror (e.g. in case there has been a problem sync'ing
 the mirror), freshclam will sleep for 10 secs and then try again with
 the next one, and so on. 
 
 After freshclam downloads the new database, it sends a notify to clamd
 (if active) to reload the database.
 
 It is important for the machine running ClamAV to be able to make
 DNS lookups and to connect to port 80 of external hosts on Internet
 either directly or through a proxy. There are known problems with
 some transparent proxies caching what they shouldn't cache. If you
 should run into this kind of problem, please check your proxy configuration
 before reporting a bug.
 
 
 \section{Mirroring the database}
 
 
 \subsection{The need for mirrors}
 
 To prevent the spread of worms it is essential to check for updates
 frequently. ClamAV users often configure freshclam with a check interval
 of 30 minutes. 
 
 With an exponentially growing number of ClamAV users, the servers
 hosting the virus database files get easily overloaded. 
 
 Without mirrors, the traffic on our main site was 100GB/month (May
 2003). 
 
 On Feb 2004 the traffic on each mirror (11 in total) reached 120GB/month.
 \\
 Thanks to some improvements in freshclam and the increasing number
 of mirrors (currently 60), the traffic on each mirror was lowered
 to 40GB/month (Aug 2004). That makes about 2.5TByte/month of global
 traffic.
 
 Our users are encouraged to add the following directives to their
 freshclam.conf :
 
 DatabaseMirror db.XY.clamav.net
 
 DatabaseMirror db.local.clamav.net
 
 where XY stands for the country the server lives in %
 \footnote{a full list is available at http://www.iana.org/cctld/cctld-whois.htm%
 }
 
 Each db.XY.clamav.net record points to the mirrors available in that
 country%
 \footnote{For a complete list of the mirrors available in each country visit
 http://www.clamav.net/mirrors.html%
 } or, in case there are none, the continent.
 
 If freshclam can't connect to db.XY.clamav.net, it will fail back
 on db.local.clamav.net, which \textbf{attempts} to redirect the user
 to the closest pool of mirrors by looking up its ip source address
 in GeoIP database (http://www.maxmind.com/app/geoip\_country).%
 \footnote{See:
 
 http://www.iana.org/assignments/ipv4-address-space
 
 http://ip-to-country.webhosting.info/
 
 http://ftp.apnic.net/stats/apnic/
 
 http://www.ripe.net/db/erx/erx-ip/
 
 Some of the information were contributed by Walter Hop (from transip.nl).%
 } We are aware that looking up the ip source address is not an accurate
 method to find the user location from a network topology point of
 view. We accept the risk.
 
 
 \subsection{Requirements to become a mirror}
 
 We need fast reliable mirrors. Servers eligible for becoming mirrors
 have to provide:
 
 \begin{itemize}
 \item At least a 10Mbit/s link to the Internet%
 \footnote{Traffic is bursty, that's why we request such a large pipe%
 }
 \item Unlimited traffic
 \item At least 50MB of web space
 \item Support for our \emph{push-mirroring} system
 \item The mirror has to be available to all ClamAV users. We DO NOT support
 private mirrors.
 \item ssh 2 (read on)
 \end{itemize}
 We also appreciate (but do not require) having shell access to the
 server hosting the mirror. FTP access is no longer accepted. \\
 The virusdb team will use the account \emph{only} to update the virus
 database.
 
 
 \subsection{How to become a mirror}
 
 Before setting up a mirror contact \emph{luca -at- clamav.net}! 
 
 You have to follow these steps:
 
 \begin{enumerate}
 \item Set up a virtual host for \\
 http://database.clamav.net, http://db.{*}.clamav.net and http://clamav.your-domain.tld\\
 Note there is an asterisk in the second hostname. A literal asterisk.\\
 Do not replace it with your country code. \\
 If you are using name based virtual hosts%
 \footnote{You can check whether the mirror setup is correct or not, simply by
 adding a line like this:
 
 your-server-ip database.clamav.net
 
 to the /etc/hosts on your client machine. Then visit http://database.clamav.net
 and see if you can download files from your mirror's directory.%
 } see \\
 http://httpd.apache.org/docs/mod/core.html\#serveralias for more information.
 \\
 Here is an example for a typical setup:\\
 \\
 \emph{<VirtualHost 10.1.2.3> }\\
 \emph{ServerAdmin john@clamav.foo.com }\\
 \emph{DocumentRoot /home/users/clamavdb/public\_html }\\
 \emph{ServerName database.clamav.net}\\
 \emph{ServerAlias db.{*}.clamav.net}\\
 \emph{ServerAlias clamav.foo.com}\\
 \emph{</VirtualHost>}\\
 If you are not using Apache and you cannot create wildcard vhosts,
 you must use IP based virtual hosts!\\
 Please note that an http redirect (e.g. RedirectPermanent) is not
 enough! freshclam can't handle redirects yet. \emph{}\\
 If you are running Apache 2.x, please use the following directive
 for proper logging:\\
 \emph{LogFormat \char`\"{}\%h \%l \%u \%t \textbackslash{}\char`\"{}\%r\textbackslash{}\char`\"{}
 \%>s \%O \textbackslash{}\char`\"{}\%\{Referer\}i\textbackslash{}\char`\"{}
 \textbackslash{}\char`\"{}\%\{User-Agent\}i\textbackslash{}\char`\"{}\char`\"{}
 combinedrealsize }\\
 \emph{CustomLog /path/to/clamav-access.log combinedrealsize }\\
 See the {}``Statistics'' paragraph for more info.
 \item Create an account with login {}``clamavdb'' and give it write access
 to the virtual host's DocumentRoot. \\
 You may want to disable password authentication for this account and
 change the password to something obscure.\\
 The {}``clamavdb'' user's shell must be /bin/sh or /bin/bash . Otherwise
 the user won't be able to run the command associated with the ssh
 public key%
 \footnote{Take a look at the content of \emph{{}``authorized\_keys\_noshell''}:
 the only command which can be executed by the owner of the corresponding
 ssh private key is \textasciitilde{}/bin/clam-clientsync. We will
 only be able to trigger the execution of that script and nothing else!
 
 However, shell access is really appreciated. If you are willingly
 to give us shell access, use \emph{authorized\_key}s\emph{\_shell}
 instead which contains Luca Gibelli and Tomasz Papszun ssh public
 keys too.%
 }.
 \item Download the following files:\\
 clam-clientsync.conf\\
 clam-clientsync\\
 authorized\_keys\_shell\\
 authorized\_keys\_noshell\\
 authorized\_keys\_shell.sig\\
 authorized\_keys\_noshell.sig\\
 from http://www.clamav.net/doc/mirrors/
 \item Verify the signature using:\\
 \$ gpg --verify authorized\_keys\_noshell.sig authorized\_keys\_noshell\\
 \$ gpg --verify authorized\_keys\_shell.sig authorized\_keys\_shell\\
 My PGP public key is available on most keyservers and on ClamAV web
 site. It can eventually be verified by telephone. Contact me by email
 first.
 \item If you don't want to give us shell access, copy \emph{authorized\_keys\_noshell}
 to \emph{\textasciitilde{}clamavdb/.ssh/authorized\_keys}:\\
 \$ cp authorized\_keys\_noshell \textasciitilde{}/.ssh/authorized\_keys\\
 If you want to give us shell access, use \emph{authorized\_keys\_shell}
 instead:\\
 \$ cp authorized\_keys\_shell \textasciitilde{}clamavdb/.ssh/authorized\_keys\\
 \$ chmod go-w \textasciitilde{}clamavdb\\
 \$ chmod 700 \textasciitilde{}clamavdb/.ssh\\
 \$ chmod 600 \textasciitilde{}clamavdb/.ssh/authorized\_keys
 \item Copy clam-clientsync to \textasciitilde{}clamavdb/bin/\\
 Copy clam-clientsync.conf to \textasciitilde{}clamavdb/etc/\\
 chmod 600 \textasciitilde{}clamavdb/etc/clam-clientsync.conf\\
 chmod 755 \textasciitilde{}clamavdb/bin/clam-clientsync\\
 Everything must be owned by user clamavdb.\\
 The clam-clientsync requires the {}``lockfile'' program, which is
 part of the \emph{procmail} package. Before going any further, please
 check that {}``lockfile'' is available. 
 \item Send the server's details (ip address, country, virtual host aliases,
 available bandwidth and sysadmin's full name and email address) to
 \emph{luca} \emph{at} \emph{clamav.net} .
 \item Edit \textasciitilde{}clamavdb/etc/clam-clientsync.conf . If your
 DocumentRoot (see paragraph 1) is \emph{/home/users/clamavdb/public\_html}
 , your login is \emph{foo} and your password \emph{guessme}, then
 your clam-clientsync.conf will look like this: \\
 TO=/home/users/clamavdb/public\_html\\
 RSYNC\_USER=foo\\
 RSYNC\_PASSWORD=guessme\\
 EXCLUDE=\char`\"{}--exclude local\_{*}\char`\"{} 
 \item Reconfigure your packet filter to allow incoming connections on port
 22/tcp and outgoing connections to ports 873/tcp and 873/udp.\\
 You can furtherly restrict access to these ports by only allowing
 connections from/to the following IP addresses: \\
 194.109.142.194, 64.18.103.6, 194.242.226.43 .\\
 rsync.clamav.net is a round robin record which points to our master
 mirror servers. Any changes to this record will be announced on the
 clamav-mirrors mailing-list.
 \item You are welcome to put your company logo on the mirror home page.
 Just copy it to the DocumentRoot and rename it to {}``local\_logo.png''.
 The index.html is unique for every mirror. Please note that any file
 in the DocumentRoot whose name doesn't match {}``local\_{*}'' will
 be deleted at every mirror sync.
 \item Subscribe to clamav-mirrors \emph{at} lists.clamav.net: see \\
 http://lists.clamav.net/mailman/listinfo/clamav-mirrors for more info.
 \\
 Subscribe requests have to be approved. We will approve your subscription
 request only \emph{after} reviewing your server's info.
 \end{enumerate}
 When everything is done, your server's IP address will be added either
 to your country's dns record (db.XY.clamav.net) or one of the round
 robin record (db.<continent>.clamav.net) and your company will be
 listed on our mirrors list page.
 
 
 \subsection{Statistics}
 
 Although it's not required, we really appreciate if you can make access
 statistics of your mirror available to us. They should be available
 at http://your-mirror-host-name/local\_stats/ and they \textbf{must}
 be protected with login and password. You should use the same login
 and password you are using in your \textasciitilde{}clamavdb/etc/clam-clientsync.conf
 file.
 
 If possible, please tell your statistics generator to ignore requests
 made by the {}``ClamAV-MirrorCheck'' agent. 
 
 If you are using Webalizer, you can add the following directive to
 your conf. file:
 
 HideAgent ClamAV-MirrorCheck
 
 If you are using AWStats, you can add this one instead:
 
 SkipUserAgents=\char`\"{}ClamAV-MirrorCheck''
 
 Refer to your stats generator's manual for more info.
 
 \textbf{Important note} for Apache2 users: 
 
 As stated in the Apache documentation from http://httpd.apache.org/docs/2.0/mod/mod\_log\_config.html:
 
 \textit{Note that in httpd 2.0, unlike 1.3, the \%b and \%B format
 strings do not represent the number of bytes sent to the client, but
 simply the size in bytes of the HTTP response (which will differ,
 for instance, if the connection is aborted, or if SSL is used). The
 \%O format provided by mod\_logio will log the actual number of bytes
 sent over the network.}
 
 
 \subsection{Admin's duty}
 
 \begin{itemize}
 \item Scheduled downtimes should be announced on the clamav-mirrors mailing-list
 in advance.
 \item IP address changes should be notified in advance too. 
 \item Changes in the ssh host public key of the mirror host should be announced
 on the clamav-mirrors mailing-list.
 \item It is essential to be able to contact the sysadmin responsible for
 the mirror server and get a quick response. Whenever a problem with
 a mirror occurs we need to immediately find out its cause and act
 consequently.
 \end{itemize}
 
 \section{Notes for sigmakers}
 
 New sigmakers should send their ssh2 public key to \emph{luca at clamav.net}
 . Their public key will be added to rsyncX.clamav.net authorized\_keys
 (after being verified).
 
 Sigmakers can upload a new database to either rsync1.clamav.net or
 rsync2.clamav.net using a (scp|sftp|rsync)-only account. 
 
 The new database won't be available to other people immediately. First,
 sigmakers have to notify the rsyncX.clamav.net server that a new database
 is available. 
 
 Here is the step-by-step procedure to release a new database version
 and propagate it around the world:
 
 \begin{enumerate}
 \item Assume your ssh private key is \textasciitilde{}/.ssh/id\_rsa and
 you've just built a new daily.cvd. Assume you want to use rsync1.clamav.net
 \item In order to upload the new database, you have to run:\\
 \$ rsync -tcz --stats --progress -e 'ssh -i \textasciitilde{}/.ssh/id\_rsa'
 daily.cvd clamupload@rsync1.clamav.net:public\_html/
 \item Next, you need to notify rsync1.clamav.net that a new database is
 available:\\
 \$ ssh rsync1.clamav.net -i \textasciitilde{}/.ssh/id\_rsa -l clamavdb
 sleep 1
 \item rsync1.clamav.net will verify the digital signature of the newly uploaded
 database using \emph{sigtool -i}. If it finds an error, it will refuse
 to distribute the database to other mirrors.
 \item rsync1.clamav.net will copy the previously uploaded database to its
 rsync shared directory.
 \item rsync1.clamav.net will notify every mirror that a new database is
 available
 \item Every mirror will rsync its copy of the database from \emph{rsync1.clamav.net::clamavdb}
 (only mirrors can access the rsync server at rsync1.clamav.net, it's
 password protected)
 \end{enumerate}
 As a fallback, every three hours, either rsync1.clamav.net or rsync2.clamav.net
 force an update on every mirror. 
 
 If rsync1 can't reach rsync2 or viceversa, the automatic update doesn't
 take place. This is done to avoid propagating an old database.
 
 To avoid conflicts, sigmakers should use rsync1 by default and if
 it fails, switch to rsync2. Whenever a sigmaker uses rsync2, he should
 announce it on the clamav-team mailing-list so that every other sigmaker
 uses rsync2 too, until the issues with rsync1 are over. 
 
 
 \section{Mirror status}
 
 Every mirror is continously monitored to ensure that every ClamAV
 user gets the latest virus database.
 
 Every three hours we upload a file called \emph{timestamp} on every
 mirror. Every hour we choose a random mirror and check that \emph{timestamp}
 is fresh. If the file is one day old or unavailable, the mirror if
 marked as {}``old'' and the ClamAV team receive a warning. If the
 situation persists for two days, the mirror is temporarily removed
 from the list. 
 
 You can view the current status of every ClamAV database mirror at
 http://www.clamav.net/mirrors.html . 
 
 Please note that this page doesn't reflect how \emph{often} the database
 is propagated to mirrors. It just shows the trend of mirrors availability.
 \end{document}