Contents

  1. Policy on service provision
    1. dedicated servers to avoid interdependencies
    2. replicated services using different distributions
    3. replicated services using different power phases
    4. replicated services using different HW including virtual
  2. General setup
    1. Smaller footprint: removing RPMs
    2. iptables hardening: INPUT filtering
    3. iptables hardening: OUTPUT filtering
  3. www.* (http server)
    1. Server types
    2. Enabling https - mod_ssl / SSL / TLS
    3. www-dyn (dynamic server)
      1. user info (how to use the www-dyn server)
      2. phpMyAdmin setup (web management of mysql server)
      3. phpMyAdmin (web management of mysql server) create new database
      4. phpMyAdmin (web management of mysql server) checks
      5. tomcat
    4. www-dyns (dynamic system server)
      1. news - wordpress
        1. www-dyns service (pre 2011)
        2. www-news service (post 2011)
    5. www.lbtp (varnish accelerator)
      1. Removing a comment
      2. Removing a post
    6. www.ivc (32b - UG teaching)
    7. WiKi (Lab WiKi)
      1. WiKi backup
    8. SRGWiKi (SRG's WiKi)
    9. trac-c3d (arch HTTPS SVN server)
    10. trac-loki (arch HTTPS SVN server)
    11. sesame-wiki (DTG sesame WiKi)
    12. www-fluphone
    13. www-ecad
  4. svn (subversion server)
  5. mail-serv* (mail servers)
  6. lmserv-* (licence servers)
  7. slogin-serv* and ssh-relay*
    1. bundles: copy from another instance
    2. sshd OTPW
    3. condor submit service
  8. cups-serv* (CUPS servers)
  9. resolv* and dns-serv* (dns servers)
  10. NFS server
  11. mysql-serv*
    1. my.cnf
    2. mysql replication
      1. mysql replication: TODO: add archive only server
    3. mysql dumps
  12. monitoring - ping, snmptrap, dhcpleases, nagios, mrtg, netdisco
  13. syslog0
  14. radius servers
  15. standalone CPU servers (HOL)
  16. serial.srg
  17. gprs-router-1 (bluebird)
  18. exams machine
  19. publicdump CD/DVD writing service
  20. FootNotes

This is a dumping ground for setups for small services and other spacial machine setups, starting with generics ("an LM licence server") and going on to more specifics. See SysInfo/Specials for generic specials (such as how to setup a MadWiFi card).

Policy on service provision

dedicated servers to avoid interdependencies

A current objective is to move to the position in which services can be updated without causing other services to have problems. In the past with multiple services on one machine, upgrading one service might require a system upgrade which might break another service. By running services on dedicated systems (either physical machines or Xen domUs) such interactions should be avoided. Shared servers may be used for resilience (e.g. failure of Xen dom0s), as loss of one instance should not cause the service as a whole to be unavailable.

replicated services using different distributions

Where services can be replicated in some way (be it inherent [the protocol is designed for it, e.g. DHCP, NTP], transparent fail over [the clients try multiple servers without telling the user, e.g. ], automatic fail over [the code or a higher level wrapper looks for alternatives and tells the user, e.g. LPRng] , sys admin manual fail over [e.g. changing DNS entries or or explicit user selection), in cases where a service can be equally well provided by more than one Operating System, this should be done. If one instance is provided by a Xen domU, then another should be provided on a shared server. In some cases different releases of the same distribution may be used (e.g. Fedora Core: 3 vs 6), in others different distributions in the same family may be used (Redhat: FC vs CentOS), completely different distributions sharing basic common sources (gnu/Linux: Redhat vs SUSE) or completely different code bases (gnu/Linux vs Windows). In each case the extra complexity of management of and on diverse systems should be weighed against the extra diversity gained. In most cases, using the same family should allow the management to be the same, while protecting against problems such as a package upgrade breaking a service. Conversely, if there are a set of non replicated services which are all needed by clients (e.g. the ECAD flexlm servers) they should be put together so that they are either all available, or all not available.

replicated services using different power phases

Although not very likely to help much, where possible, replicated services should be on different power phases.

replicated services using different HW including virtual

Also unlikely, but using different hardware (e.g. onboard NIC) may reduce the chance of systematic failure.

Running as a virtual machine can improve the resilience in a number of ways. It makes "moving" a service in the case of HW failure much simpler. With machine migration and servers in multiple server rooms, it can cope with planned or unplanned problems in a room. With High Availability a service can be automatically restarted in case of a service, server, network or room fault.

General setup

If using a "std Lab FC6" setup (with SELinux set to "Enforcing") on a xen domU (possibly un-ticking "Office and Productivity" during the initial install which takes it down to 2GB on a 64b machine), include in /etc/user-config/bundles the lines

@R-server
@R-nonfs
@R-xen_domU

(omit the @R-xen_domU if not a domU, and make the nonfs be full if using NFS and/or LDAP). By not having access to the NFS service, dependencies are reduced.

Smaller footprint: removing RPMs

To reduce the disc footprint, consider removing all *.i386 RPMs on a 64b machine ("rpm -e $(rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}\n' -a|grep i.86$)"), and also (if not needed by the server) the following large RPMs by adding them to /etc/user-config/bundles

NOT_gimp
NOT_httpd
NOT_samba-common
NOT_mono-core
NOT_libgnome
# probably not needed, as R-server
NOT_php-common
NOT_CL-tetex

which takes it down to 1.8 GB on a 64b machine. Additional RPMs which can be removed to save disc or memory space (the ones marked on by default run demons) include (add more)

and chkconfig can be used to disable demons for

iptables hardening: INPUT filtering

Consider restricting SSH access to being from VLAN 100 or within the lab, by adding a "-s" flag, e.g.

-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -s 128.232.0.0/20 -j ACCEPT

iptables hardening: OUTPUT filtering

As they are dedicated servers, they normally do not need to make outgoing calls to many servers, so it is possible to tighten down the iptables rules for outgoing connections. /etc/sysconfig/iptables could have lines such as:

:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
:RH-Firewall-1-OUTPUT - [0:0]
-A OUTPUT -j RH-Firewall-1-OUTPUT
# Output chain
-A RH-Firewall-1-OUTPUT -o lo -j ACCEPT
-A RH-Firewall-1-OUTPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
#2 Allow HTTP(S) anywhere ?
-A RH-Firewall-1-OUTPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-OUTPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
#2 Allow DNS to Lab resolvers only ?
-A RH-Firewall-1-OUTPUT -d 128.232.1.0/30 -m state --state NEW -m udp -p udp --dport 53 -j ACCEPT
-A RH-Firewall-1-OUTPUT -d 128.232.1.0/30 -m state --state NEW -m tcp -p tcp --dport 53 -j ACCEPT
#1 Allow SMTP to cl servers only
-A RH-Firewall-1-OUTPUT -d 128.232.0.14/31 -m state --state NEW -m tcp -p tcp --dport 25 -j ACCEPT
#1 Allow RWHO packets
-A RH-Firewall-1-OUTPUT -d 128.232.15.255 -m state --state NEW -m udp -p udp --dport 513 -j ACCEPT
#1 Log a few rejected packets per minute, then drop through to ":OUTPUT DROP" above
-A RH-Firewall-1-OUTPUT  -m limit --limit 10/min -j LOG --log-prefix "ipt: Rej "

If LDAP / NFS access to elmer is needed, also add before the LOG line:

#5 Allow NFS to elmer
-A RH-Firewall-1-OUTPUT -d 128.232.1.222 -m state --state NEW -m udp -p udp --dport 111 -j ACCEPT
-A RH-Firewall-1-OUTPUT -d 128.232.1.222 -m state --state NEW -m tcp -p tcp --dport 111 -j ACCEPT
-A RH-Firewall-1-OUTPUT -d 128.232.1.222 -m state --state NEW -m udp -p udp --dport 2049 -j ACCEPT
-A RH-Firewall-1-OUTPUT -d 128.232.1.222 -m state --state NEW -m tcp -p tcp --dport 2049 -j ACCEPT
-A RH-Firewall-1-OUTPUT -d 128.232.1.222 -m state --state NEW -m tcp -p tcp --dport 4046 -j ACCEPT
#2 Allow LDAP to Lab servers (0.41-0.43)
-A RH-Firewall-1-OUTPUT -d 128.232.0.40/30 -m state --state NEW -m udp -p udp --dport 389 -j ACCEPT
-A RH-Firewall-1-OUTPUT -d 128.232.0.40/30 -m state --state NEW -m tcp -p tcp --dport 389 -j ACCEPT

(could use "-m multiport --dports 111,2049,4096", but that loses info as to which is used how often)

www.* (http server)

The "NOT_httpd" and "NOT_php-common" above should be omitted, as appropriate. As such, the httpd RPM should be installed by default, but not auto started. On Redhat systems, run "chkconfig httpd on" to cause it to auto start, and "service httpd start" to start it. At this point connecting to the server locally should give a std page. To allow connections from other machines, if on a Redhat system "HTTP" was not selected during firstboot add an extra service to allow through the firewall, run system-config-securitylevel and enable http and/or https as appropriate. Alternatively manually HACK the iptables configuration to include something like

-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT

before the final REJECT rule.

On a debian system using ufw, allow access for "Apache", "Apache Secure" or "Apache Full" as appropriate, e.g.

cl-asuser ufw allow "Apache Full"; cl-asuser ufw status

To set up as a more Lab Std server, add "@R-httpd" to /etc/user-config/bundles to add extra RPMs such as mod_ucam_webauth2 to enable raven, install /etc/httpd/conf/aacookiekey, copy over the relevant *.VH from /anfs/www/admin/sample-httpd/conf/ into /etc/httpd/conf.d/. Then restart using "cl-asuser service httpd graceful" and it should be using the new configuration. If using raven through a proxy server, the code needs to be edited.

Once everything is shown to work, copy back the configuration files in conf.d/ to /anfs/www/admin/sample-httpd/conf/conf.d- $machine. It should be setup owned by an appropriate user and/or group so that it can be kept up to date (it is also saved in ownfiles).

As of 2010/05, the distributions' configuration convention is followed. Thus Ubuntu uses /etc/apache2/ as the base configuration directory, with files $module.load and $module.conf in mods-available/ with links or copies in mods-enabled/ for enabled modules, and modules in /usr/lib/apache2/modules/ using commands such as "sudo a2enmod ssl", and similarly enable sites "sudo a2ensite default-ssl". Non standard files are in the configuration tree, e.g. Raven keys are in webauth_keys/ and the AACookieKey is in conf.d/local-aacookiekey which is automatically Included, and should be readable only by user root.

Although this page is about Small Servers, most of the above applies equally to non domU web servers such as www.cl.cam.ac.uk

Server types

There are a number of considerations when deciding which type of web server to use.

The standard server types are:

Enabling https - mod_ssl / SSL / TLS

On services where server authentication or traffic encryption is needed, https can be used instead of http. This makes caching and HTTP 1.1 hostname based VirtualHosts impossible, and requires a second copy of the VirtualHost information, so should only be used when needed.

Start by installing mod_ssl, restarting ("graceful" seems not to be sufficient) the server, and testing that the auto-generated self signed certificate works.

cl-asuser yum install mod_ssl -y
cl-asuser service httpd restart
elinks https://localhost/

Installing the RPM should do all the tailoring needed. slacksite have the minimal configuration as

<IfDefine SSL>
<VirtualHost _default_:443>
SSLEngine on
SSLCertificateFile    /usr/local/apache/etc/ssl.crt/server.crt
SSLCertificateKeyFile /usr/local/apache/etc/ssl.key/server.pem
SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown
CustomLog /usr/local/apache/var/log/ssl_request_log \
        "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
</VirtualHost>
</IfDefine>

If there is local tailoring and both http and https: access is required, generate files such as $service.VH

<VirtualHost    128.232.0.20:80>
Include conf.d/cl.VH-base
</VirtualHost>
<VirtualHost 128.232.1.20:443>
Include conf.d/cl.VH-base
Include conf.d/cl.ssl-base
</VirtualHost>

$service.VH-base

ServerName      www.cl.cam.ac.uk
...

and $service.ssl-base

# $Header: ...ssl-base,v 1.1 2007/06/05 09:57:21 pb22 Exp $
# File "Include"d by VHs which want to use SSL
# Generated by stripping comments etc from /etc/httpd/conf.d/ssl.conf
# and adding "Tailor" section and "SSLCertificateChainFile".
# Tailor these bits per host <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
SSLCertificateFile /etc/pki/tls/certs/www-dyn.crt
SSLCertificateKeyFile /etc/pki/tls/private/www-dyn.key
# Tailor these bits per host <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
# cl.cam.ac.uk addition for JANET certs <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
# (cd /etc/pki/tls/certs && sudo wget http://secure.globalsign.net/cacert/sureserverEDU.pem)
SSLCertificateChainFile /etc/pki/tls/certs/sureserverEDU.pem
# cl.cam.ac.uk addition for JANET certs >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
ErrorLog logs/ssl_error_log
TransferLog logs/ssl_access_log
LogLevel warn
SSLEngine on
SSLProtocol all -SSLv2
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
<Files ~ "\.(cgi|shtml|phtml|php3?)$">
    SSLOptions +StdEnvVars
</Files>
<Directory "/var/www/cgi-bin">
    SSLOptions +StdEnvVars
</Directory>
SetEnvIf User-Agent ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0
CustomLog logs/ssl_request_log \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

It's probably best to leave the auto-generated key alone, and generate a new one. See Chapter 3 of the CS course notes for some background. Set the service name (replace foo with the required name), generate a key, inspect it, generate a CSR, and inspect it:

export serv=foo
openssl genrsa -out $serv.key 1024
openssl rsa -in $serv.key -noout -text
( echo GB
  echo England
  echo Cambridge
  echo The University of Cambridge
  echo Computer Laboratory
  echo $serv.cl.cam.ac.uk
  echo
  echo
  echo
) |
openssl req -new -key $serv.key -out $serv.csr ; echo
openssl req -noout -text -in $serv.csr

If it's just for private / local / testing, or to check things out while the certificate is being signed by a CA, self-sign the cert, inspect it, and install it

openssl x509 -req -days 60 -in $serv.csr -signkey $serv.key -out $serv.crt
openssl x509 -in $serv.crt -noout -text
sudo cp -p $serv.crt /etc/pki/tls/certs/
sudo cp -p $serv.key /etc/pki/tls/private/

and update the apache configuration (e.g. /etc/httpd/conf.d/ssl.conf) to use it.

To get it signed via the CS, use their form. The JANET SCS can not be used for commercial use, e.g. to buy things or make donations, but should otherwise be used as it's cheaper (free as of 2007). Select "JANET SCS", "New certificate", "Apache-ModSSL, other OpenSSL-based servers", and click on "Next page >>". Paste in the full CSR, select the required lifetime, and click on "Next page>>". Give your full name and phone number, " webmaster@cl.cam.ac.uk " as the email address, then click on "Next page>>". It should return a 10 character password -- they say to keep this safely in case it might be needed - it is not in the normal case. The supplied email address will be sent a CC: of the email to the CS asking for conformation. Do check the contents, but do not reply to confirm it - the CS have to do that. They say the cert should be available in 1-3 days, but in practice it's an hour or so. save the .pem file in /etc/pki/tls/certs/ with a .crt extension. To allow chaining, add sureserverEDU.pem e.g. "SSLCertificateChainFile /etc/pki/tls/certs/sureserverEDU.pem", having fetched it using

(cd /etc/pki/tls/certs && sudo wget http://secure.globalsign.net/cacert/sureserverEDU.pem)

If a particular page is to be accessed only using https: (e.g. it includes a password) add entries along the lines of

<VirtualHost www.ivc.cl.cam.ac.uk:80>
Redirect        /ivc/login.pl https://www.ivc.cl.cam.ac.uk/ivc/login.pl
</VirtualHost>

(there should be a simpler way to just say "use https:") to ensure that the http: version is redirected to the https: version, and as a safety check

# If the redirect above fails, reject insecure login calls
<Files ~ "login.pl$">
    SSLRequireSSL
</Files>

www-dyn (dynamic server)

The dynamic servers (www-dyn, www-dyn2, etc) should provide a uniform platform for users' private scripts. For resilience, they run different distributions, so if one fails, try another. They supply pages using un-checked user supplied scripts, so are hardened to reduce the impact if one is compromised. Outgoing connections are blocked, with access to the mysql servers enabled, for user pages to use the server, and for phpMyAdmin to manage to server. The machines do not have netapp_netgroup so do not have general access to elmer. Instead, they have caller IP based access to just the directories they need, /auto/userfiles/$USER/dynamic_html. As the default httpd configuration is based around access to user's $HOME, the information supplied by LDAP is mapped to the relevant directory by having a one line /etc/auto.master

/home /etc/auto.home

with /etc/auto.home such as

*       -rw,nosuid,noac,bg,vers=3,tcp,timeo=600,rsize=32768,wsize=32768,hard,intr elmer:/vol/userfiles/&/dynamic_html

This then allows users to setup public_html in their "home directory". It, and the parent directory, needs to be writable only by the user.

Creating a /etc/httpd/conf.d/default.VH such as

# This is meant to be a single service, so allow any name (e.g. ssh tunnel)
<VirtualHost    _default_>
# Allow access to ~USER (which is HACKed using autofs) with CGI, but not PHP?
AddHandler cgi-script .cgi
<IfModule mod_userdir.c>
    UserDir public_html
<Directory /home/*/public_html>
    AddHandler cgi-script .cgi  # use suEXEC
    suPHP_AddHandler php5-script # use suPHP
    AllowOverride FileInfo AuthConfig Limit
    Options MultiViews Indexes SymLinksIfOwnerMatch ExecCGI
    <Limit GET POST OPTIONS>
        Order allow,deny
        Allow from all
    </Limit>
    <LimitExcept GET POST OPTIONS>
        Order deny,allow
        Deny from all
    </LimitExcept>
</Directory>
</IfModule>
</VirtualHost>

With suexec installed, it is automatically used for all CGI scripts in UserDir.

Load the mod_suphp RPM and patch /etc/suphp.conf to restrict the docroot to /home (can't do /home/*/public_html), disable checks that all files are in DOCUMENT_ROOT, and allow users above 100 (rather than 500).

--- /etc/suphp.conf-FC6 2007-05-16 19:08:37.000000000 +0100
+++ /etc/suphp.conf     2007-05-16 19:04:47.000000000 +0100
@@ -11,2 +11,2 @@
-;Path all scripts have to be in
-docroot=/
+;Path all scripts have to be in: cl.cam.ac.uk: was /: set check_vhost_docroot=false
+docroot=/home/
@@ -23,2 +23,2 @@
-;Check wheter script is within DOCUMENT_ROOT
-check_vhost_docroot=true
+;Check wheter script is within DOCUMENT_ROOT: cl.cam.ac.uk: was true: files are in /home/
+check_vhost_docroot=false
@@ -35,2 +35,2 @@
-; Minimum UID
-min_uid=500
+; Minimum UID: cl.cam.ac.uk: was 500: we have users 100-500
+min_uid=100

SELinux has problems, so it needs a module such as:

module httpddyn 1.0;
require {
        type mysqld_port_t;
        type httpd_t;
        type tmp_t;
        type nfs_t;
        class tcp_socket name_connect;
        class lnk_file { read getattr };
        class dir { read search getattr };
        class file { ioctl getattr read execute execute_no_trans };
}
#============= httpd_t ==============
allow httpd_t mysqld_port_t:tcp_socket name_connect;
allow httpd_t nfs_t:dir { read search getattr };
allow httpd_t nfs_t:file { read getattr execute ioctl execute_no_trans };
allow httpd_t nfs_t:lnk_file { read getattr };
allow httpd_t tmp_t:file { read getattr };

It also appears that /usr/sbin/suexec needs to be recompiled (e.g. in /usr/groups/linux/extra-packages/httpd) with AP_UID_MIN set to 100, and be user_u:object_r:sbin_t rather than .system_u:object_r:httpd_suexec_exec_t:s0 as restorecon sets it.

To allow users to access their error log entries, add a line to /etc/sudoers

ALL     ALL=NOPASSWD: /usr/bin/http-error

and uncomment the line "Defaults requiretty".

user info (how to use the www-dyn server)

As described above, access to the fileserver is restricted, and the machine uses a non standard $HOME (/auto/userfiles/$USER/dynamic_html/), and thus a non-standard ~/public_html. On a standard Lab machine (e.g. your Linux workstation or one of the slogin.cl.cam.ac.uk machines) create the $HOME directory mentioned above, and then within that the directory public_html. Within the hour, the system should have noticed the creation, and it should be possible to access http://www-dyn.cl.cam.ac.uk/~$USER (replacing $USER with your CRSID) and get a page saying "Index of /~$USER", with just a link to the parent directory (the Fedora test page). Ensure that all the directories are only user writable, not group writable.

As scripts are run as the user, related files should normally reside in the user's "home" directory on the machine. As ~/public_html is visible via the web server, "private" files should be kept in other directories in the user's home directory.

Access to the log files relating to a user is available using the script http-error under sudo, e.g. create a CGI script public_html/error_log.cgi such as:

#! /bin/sh
# Log the last 20 lines of error log for the calling user, in simple HTML format
echo "Content-type: text/html"; echo
sudo http-error --html --last 20 --file error_log

If you have errors which don't show up in the error log, you can try looking at all available logs, e.g.

#! /bin/sh
# Log the last 20 lines of the logs for the calling user, in simple HTML format
echo "Content-type: text/html"; echo
for f in suexec. error_ access_ ssl_error_ ssl_request_ ssl_access_
do echo "<p><i>... ${f}log</i></p>"; sudo http-error --html --last 20 --file ${f}log; done

phpMyAdmin setup (web management of mysql server)

Install the phpMyAdmin RPM. To tailor it, "Quick Install" says: "you must manually create a folder config in the phpMyAdmin directory" -- this refers to /usr/share/phpMyAdmin/ and NOT /etc/phpMyAdmin/. Use scripts/setup.php to setup a new server (e.g. "mysql-serv0.cl.cam.ac.uk") and set "Authentication type" to "cookie", press "Update" and "Save" (and remember to copy it back when finished).

phpMyAdmin (web management of mysql server) create new database

From the home page, click on "Privileges", then "Add a new User", add a new user "$user _ $package", in "Database for user" select "Create database with same name and grant all privileges". Tell the user the password, and say to change it. For users with many databases, add a user "$user", and in "Database for user" and select "Grant all privileges on wildcard name (username_%)" allowing the user to manage all that user's databases.

phpMyAdmin (web management of mysql server) checks

Home -> Privileges shows the User view with the Global priveleges. Most users should have access from a specific machine, www-dyn%.cl.cam.ac.uk if just access from the dynamic servers, or %.cl.cam.ac.uk if they need access from all hosts. Users should have Password Yes, Global privileges USAGE, Grant No'. Exceptions include:

tomcat

A user requested tomcat, so the tomcat5 RPM was installed. It listens by default on 8080 and AJP on 8009, which are blocked. As such, set up a redirect by editing /etc/httpd/conf.d/proxy_ajp.conf

ProxyPass /~ig206/tomcat5 ajp://localhost:8009/

As it seems to be a single user application, rather than sharable by all users, it was removed.

www-dyns (dynamic system server)

www-dyns provides Lab and group services similar to www-dyn. It does not have filer access. New directories are setup by emailing sys-admin.

news - wordpress

The "news" section of the Lab page was provided by wordpress on www-dyns, using the CNAME www-news, but as of 2011/01, www-news is a separate VM.

www-dyns service (pre 2011)

On www-dyns: The source files reside on the server in /var/www/html in the directory news/ and are loaded (and upgraded as necessary) using svn, e.g. to load 2.6.2 then upgrade to 2.6.3:

svn checkout http://svn.automattic.com/wordpress/tags/2.6.2 news
svn switch   http://svn.automattic.com/wordpress/tags/2.6.3

Create news/wp-content/uploads/ to store local images.

To add Better Feed, download wp_ozh_betterfeed.zip, and as per the included readme.txt file, unpack it in news/wp-content/plugins/, move wp_ozh_betterfeed.php and inc from ozh-better-feed/ and activate it via the WordPress 'Plugins' menu.

To add Advanced Category Excluder, download advanced-category-excluder.1.4.2.zip, and as per the included README.txt file, unpack it in news/wp-content/plugins/. Alternatively in wp-content/plugins/ use

svn co http://plugins.svn.wordpress.org/advanced-category-excluder/tags/1.4.2/ advanced-category-excluder

Activate it via the WordPress 'Plugins' menu.

Using phyMyAdmin: Create a database www_news.

Using its web interface: click on "Create a Configuration File", and fill in "Database Name" and "User Name" as www_news, the "Password", "Database Host" as mysql-serv0.cl.cam.ac.uk, and the "Table Prefix" as wp_news_, then click on Submit. It should report "All right sparky! You've made it through this part of the installation. WordPress can now communicate with your database. If you are ready, time now to…" - click ok "Run the Install". This requires no priviledges -- anyone can do it! Set the "Blog Title" to "Computer Laboratory News", "Your E-mail" appropriately, leave the "Allow my blog to appear in search engines like Google and Technorati." box ticked, and click on "Install "WordPress". This creates user admin, displays the password, and has a "Log In" button. Click on it to get to "Dashboard", click on the user name at the TRHC (admin), then click on "Authors and Users", then add a number of CRSIDs. Logout as admin, login using the CRSID, then delete admin.

Upgrade(rt#75780): Dump the database ("cd /a/misc-nosnap1/mysql-serv && DB=www_news bash README.cl"), snapshot the VM, and as above

cd /var/www/html/news
sudo svn sw http://svn.automattic.com/wordpress/tags/2.7.1

and then visit upgrade.php to update the database.

Maintenance: arrange that wp-content/uploads and the database are backed up. The latter can be done on a machine with access to mysql-serv0 (it will prompt for the password) using

mysqldump -uwww_news -p -hmysql-serv0 www_news

User admin: connect to dashboard, click on Users, and go to the "Add New User". Use the CRSID as the Username, fill in "First Name", "Last Name", E-mail, leave Website blank, put in a Password, ast the Role to Editor (or something else?) and click on Add User.

www-news service (post 2011)

In 2011/01 a dedicated Ubuntu 10.04 LTS VM was generated for www-news on the appliance VLAN -- see rt#72507.

wordpress was added to /etc/user-config/bundles and @R-httpd was uncommented to enable raven. Access to port 80 was enabled using "cl-asuser ufw allow from 128.232.0.0/17 to any app Apache". On mysql-serv0 access was enabled by adding to /etc/sysconfig/iptables the line

-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s 128.232.19.15  --dport 3306 -j ACCEPT

and on mysql-serv1 the line

mysql, 3306, 128.232.19.15, appliance: www-news (rt#72507)

was added to /etc/firestarter/inbound/allow-service and the tables reloaded using "cl-asuser invoke-rc.d firestarter restart".To enable access in the router, /usr/groups/netmaint/Cisco/access-lists had lines added in vlan291-in to allow access to the mysql servers

permit  tcp @www-news/H @mysql-serv0/H eq 3306  # allow www-news to mysql-serv[01]
permit  tcp @www-news/H @mysql-serv1/H eq 3306  # allow www-news to mysql-serv[01]

and access to the ssh and web ports enabled from within the Lab by adding to vlan291-out the lines

permit  tcp @$clnet @www-news/H eq 22           # allow ssh
permit  tcp @$clnet @www-news/H eq www          # allow WWW

www access is only needed by the machines which may act as proxies, but until those are nailed down, both iptables and the router allow access from any Lab machine.

www.lbtp (varnish accelerator)

lightbluetouchpaper uses the varnish accelerator on port 80 (configured in /etc/sysconfig/varnish) to speed up the backend httpd (which itself uses mysql) on port 8080 (configured in /etc/varnish/vcl.conf). The sysconfig changes are to make varnish the front end, to increase responsiveness, and reduce the delays before new posts are seen:

VARNISH_LISTEN_PORT=80
VARNISH_MIN_WORKER_THREADS=50
VARNISH_TTL=10

It was heavily themed, but after a CRACK in 2007/10, it was upgraded to wordpress 2.3, only minimal changes were made to get it working. Notes are in the svn repository file:///usr/groups/security/svn/blog

If there is a urgent need to remove a comment or post, such that the normal course of action would be to disable the site, the CL system administrators have permission to use the credentials in /root/wordpress-emergency-admin.txt to hide the comment/post until the user administrators (Markus Kuhn, Richard Clayton, Saar Drimer, Steven Murdoch) can be contacted.

Removing a comment

The comment will then be removed from public view

Removing a post

The post will then be removed from public view

www.ivc (32b - UG teaching)

The IVC system has lived on various misc boxes to which swm11 had access, until one required move coincided with acquiring a pair of new servers, so bedat was redeployed. Not a xen system, as it might require serious resources and the user had the HW.

It used a different distribution before (latest was Ubuntu), and was found to work best in 32b mode.

The perl-CGI-Session RPM (alternative is to use CPAN) needed to be shoehorned in.

swm11 managed to setup most of the rest (with a few "chgrp"'s done by sys-admin). Needed a /etc/httpd/conf.d/ivc.conf of

# SWM additions for IVC
AddHandler cgi-script .pl
<Directory "/var/www/html/ivc">
        AllowOverride None
        Options +ExecCGI +Includes +FollowSymLinks +Indexes
        Order allow,deny
        Allow from all
</Directory>
# post edit, restart web server using:
#   cl-asuser service httpd graceful

It lives in the DMZ with external WWW access, and internal ssh access.

smw11 requested that the CGI scripts gave a 10 second limit, so add

RLimitCPU 9 11

WiKi (Lab WiKi)

For installation instructions for the main Lab WiKi see /anfs/www/moin/README.cl.1.5. As of 2008/09 the old service name www.wiki is hosted on www-wiki using a pseudo IP alias. Two SELinux modules are required, to allow the WiKi to send email on page changes, and to allow the Opera Reading Group to work

module CLhttpdmail 1.0;
require {
        type httpd_sys_script_t;
        class tcp_socket create;
}
allow httpd_sys_script_t self:tcp_socket create;

module CLwikisetrlimit 1.0;
require {
        type httpd_sys_script_t;
        class process setrlimit;
}
allow httpd_sys_script_t self:process setrlimit;

WiKi backup

The master copy of the data resides on the root FS of the server. As this is a Virtual Machine, this is on an iSCSI LUN on the filer. This means that it is fairly robust and may be snapped, but in general, recovery is non trivial. A copy of the data is maintained in /anfs/www/cl-wiki/ by the script /anfs/www/cl-wiki/slurp-data (which can be run at any time to force a sync, and should produce no output, to avoid cron sending email when it works OK) being linked to from a suitable /etc/cron.* directory, e.g. radyr:/etc/cron.daily/, with a "safety" instance on toton as well.

As of 2009/07, it is run as root on client and server, but it may well be possible to run it as apache at both ends if a suitable ssh user key is set up. Also, as each version of a page is a new immutable file, its fits well with the filer's snapshot system, as there is no churn in the actual data. As such, it might be sensibe to increase the frequency of runs for pages to be hourly, while leaving the log files and such like as being daily (at least until we have the feature to spot common data blocks enabled on the filer, when everything could be hourly).

SRGWiKi (SRG's WiKi)

The SRG WiKi originally ran on stanier, which was virtualised as a xen domU, and then died. In 2007/12 it was resurrected in the form of srgwiki.cl.cam.ac.uk, a Lab Managed 64b Fedora 8 domU, initially running on paperboy running Ubuntu 7.10. The disc image was generated on an F8 box, copied over to the net to the dom0, the disc image mounted, and the partition copied to the 4GB LV lv_stanier on the VG vg_XenU, which is /vol/voli/xeno/iap10-xeno-0 on elmer, presented as LUN 0 of the group iap10-xeno-roots. Pygrub is used to boot the srgwiki domain, which allows the kernels/initrds to be placed in /boot, same as on non-virtualised systems. The domain config is in /etc/xen/srgwiki.conf on paperboy. In 2009/08 paperboy died, and the SRG didn't have anyone who knew anything about it (rt#55045). The root partition from the filer was copied to a machine in the XenE pool.

Create /var/www/cgi-bin/wikiconfig.py from /usr/share/moin/config/wikiconfig.py and under class Config(DefaultConfig): include

    from MoinMoin.auth import http
    auth = [http]
    user_autocreate = True

    acl_rights_before = u"NetosGroup:read,write,delete,revert,admin All:"

to use raven auth, avoid needing to manually create users, and set the default ACL to allow netos users full access, but others no access. Also set sitename, uncomment page_front_page, set data_dir to /var/www/html/srgwiki/data/, data_underlay_dir to /var/www/html/srgwiki/underlay, url_prefix to a null string, and uncomment u'%(page_front_page)s'. Create a /etc/httpd/conf.d/srgwiki.conf

ScriptAlias /wiki "/var/www/cgi-bin/moin.cgi"
LoadModule ucam_webauth_module modules/mod_ucam_webauth.so

AACookieKey "srgwikiauth"
<Directory "/">
  AuthType Ucam-WebAuth
  SetEnv AUTH_TYPE Basic
# AAForceAuthType Basic # if later mod_ucam_webauth is used
  Require valid-user
</Directory>

and create a page NetosGroup containing all the members to have general accesss.

trac-c3d (arch HTTPS SVN server)

See info on use of trac with proxied raven access.

Install a standard 64b CentOS 5.1, enable @R-httpd and add subversion.x86_64 and trac to /etc/user-config/bundles. Enable incoming ssh from 128.232.0.0/17 only and add HTTPS (443) in /etc/sysconfig/iptables. Create /etc/pki/tls/certs/trac-c3d.crt and /etc/pki/tls/private/trac-c3d.key. Create a FS, label it, and mount at a suitable point (/var/www/trac/ ??). See the Comp Arch page for more details.

trac-loki (arch HTTPS SVN server)

This was generated by cloning the trac-c3d server.

sesame-wiki (DTG sesame WiKi)

On the upgrade of the sesame-wiki server from CentOS 5.2 to 5.3, it started to serve null pages, and /var/log/httpd/error_log reported "Can't open perl script "/var/www/twiki/bin/view": Permission denied", due to /var/www/twiki being a link to /mnt/data/twiki_4_2_3/, causing it to be labelled system_u:object_r:mnt_t rather than system_u:system_r:httpd_sys_script_t, causing an error

avc:  denied { search } for  pid=1434 comm="view" name="mnt" dev=xvda1 ino=950 scontext=system_u:system_r:httpd_sys_script_t:s0 tcontext=system_u:object_r:mnt_t:s0 tclass=dir

Adding an semodule "allow httpd_sys_script_t mnt_t:dir search;" appears to be sufficient to fix it.

The partition is mounted in /mnt to ensure that the file system is backed up.

www-fluphone

Install a standard 64b CentOS 5 host. Enable restricted sec=sys NFS access by putting "NFS_sys_limited" in hosts.props and allowing access to /vol/vol3/grp-sr11 with "rw=@cl_hosts:www-fluphone," in /usr/groups/admin/netapp/conf-elmer/exports. Restrict the automounter by commenting out "+auto.master" in /etc/auto.master and add the LDAP map for /usr/groups/fluphone-data using

/auto/groups ldap:ldap.cl.cam.ac.uk:nisMapName=autofs.group,dc=cl,dc=cam,dc=ac,dc=uk

Login uses kerberos as usual, but user $HOMEs are local.

www-ecad

Starting from a std ubuntu 10.04 amd_64 minimal server server with OpenSSH server installed, add a std ubuntu apache2 and mysql-server. Create partitions for /var/www/ecad and /var/lib/mysql, and set them up

cl-asuser cfdisk /dev/xvdb
cl-asuser cfdisk /dev/xvdc
cl-asuser mkfs -t ext4 -s 1 -j -b 4096 -i 16000 -L www-ecad_www   /dev/xvdb1
cl-asuser mkfs -t ext4 -s 1 -j -b 4096 -i 16000 -L www-ecad_mysql /dev/xvdc1
sudo mkdir /var/www/ecad
sudo vi /etc/fstab
sudo mv /var/lib/mysql /var/lib/mysql-
sudo mkdir /var/lib/mysql
sudo chmod 700 /var/lib/mysql
sudo chown mysql.mysql /var/lib/mysql
cl-asuser mount -a
sudo cp -pvr /var/lib/mysql-/. /var/lib/mysql/.

Additional partition created for /usr/groups/ecad to allow ecad tool Modelsim to be copied (around 2.7GB worth of installation) from filer without incurring path problems.

Packaged added to /etc/user-config/bundles (plus emacs from earlier in the bundles file):

g++
rsync
tcsh
ia32-libs
graphviz
php5
php5-cli
php5-dbg
php5-mysql

svn (subversion server)

See SubVersion for information.

mail-serv* (mail servers)

A standard server has exim (to allow local email caching), but an MTA should have exim config and tables to do user email delivery, NFS sec=sys to allow user email delivery, and separate partitions for the spool (/var/spool/exim) and the logs (/var/spool/exim-log linked to from /var/spool/exim/msglog) - if the spool fails, at least details of the lost email is known.

As of 2010, we are moving to all MTAs being the same: 32 bit CentOS 5, running on a mix of virtual servers and bare metal. As the machines need NFS sec=sys access, they are on a "server" VLAN, which has port 25 open outgoing to the internet, and incoming from the expected CS fron end machines.

Install a std server (std install, but untick the Gnome bundle), uncomment "@R-B-full" in /etc/user-config/bundles, add "@F-mta", uncomment "@F-full" and comment out "@F-krb5-nfs" in /etc/user-config/patches, to setup the basics. Bare metal machines should have "NOT_CL-anacron-update-system" so that they are not automatically updated, in case a duff RPM breaks all the MTAs, and should be on a UPS. Virtual machines should have snapshots on the system disc to allow it to be rolled back in case of problems, and made "Highly Available" so that they are restarted if there are problems.

Setup filesystems (e.g. /var/spool/exim and /var/spool/exim-input) of at least 1GB each, with a link from /var/spool/exim/log to /var/spool/exim-log so that spooled email and logs are held on separate filing systems, so that even if the data partition is lost, logs of the messages might be available.

Configure exim and setup table copying and auto generation using (what?).

When fully tested, find an unused IP address with SMTP access from the internet, assign it by creating a file named something like /etc/sysconfig/network-scripts/ifcfg-eth0:mta1 (at some point, consider IPv6) containing

. /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0:15
IPADDR=128.232.0.15

lmserv-* (licence servers)

See License Servers page for content formerly here.

slogin-serv* and ssh-relay*

The slogin servers are not really "small" servers, providing a general slogin service. There are several of them, normally running a range of "similarish" distributions, such that most users should not care which one they get. As an example, they might run CentOS 5, Fedora Core 6 and Fedora 7 which are all fairly similar, but are unlikely all to be "broken" at the same time by a new RPM being made available.

As the official ssh / slogin external services, these machines should be hardened where feasible, without overly inconveniencing the users.

bundles: copy from another instance

The extra RPMs to be added can be copied from another slogin server. As there is no user admin, additions have to be made by COs, and a balance has to be set between including the kitchen sink (which may make the machine less secure, and may confuse simple users). They remove unneeded code, add generally useful RPMs, and RPMs which have been specifically requested. e.g.

sshd OTPW

See instrctions for sshd OTPW for machines which need it.

condor submit service

As per condor page add to /opt/condor-6.8.3/LOCAL/condor_config.local the lines

# 9699 wasn't enough to run 48 vms ..
HIGHPORT              = 9799
## cl.cam.ac.uk: may at some point move to HA submit server. Until then, this is harmless
MASTER_HA_LIST = SCHEDD
SPOOL = /usr/groups/linux/condor-queue/$(HOSTNAME)
HA_LOCK_URL = file:/usr/groups/linux/condor-queue/$(HOSTNAME)
## cl.cam.ac.uk: run a "no memory" startd
MEMORY  = 4

and open more ports in /etc/sysconfig/iptables

#2 condor submit server
-A RH-Firewall-1-INPUT -p udp -m udp --dport 9600:9799 -s 128.232.0.0/17 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 9600:9799 -s 128.232.0.0/17 -j ACCEPT

cups-serv* (CUPS servers)

Enable @F-cl-cups-serv in /etc/user-config/patches and@R-cupsd in /etc/user-config/bundles, and ensure dvips is available. Enable the server using "cl-asuser chkconfig cups on". To allow dvi to be printed, the SELinux module CLcupsd from CL-selinux-policy-targeted-cupsd may be needed -- if things still fail, ensure setroubleshoot is loaded, and run the sealert commands recommended in /var/log/messages, select the "avc: " lines, and process /usr/share/selinux/targeted/CLcupsd.samp.

/etc/sysconfig/iptables (and ip6tables as appropriate) should already allow the ipp port (631), e.g.

-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT

It appears to run OK in 128MB on a domU.

resolv* and dns-serv* (dns servers)

Add the bind-chroot RPM under Redhat, or bind9 package under Ubuntu. Ensure that the listening ports are open by adding to /etc/sysconfig/iptables (and/or ip6tables) the lines:

#2 cl.cam.ac.uk: this machine is a NS, so allow the traffic. Restrict sorce if a local resolver only
# See http://www.wiki.cl.cam.ac.uk/clwiki/SysInfo/SmallServers#dns
-A RH-Firewall-1-INPUT -p udp -m udp -s 128.232.0.0/17 -d 128.232.1.0/29 --dport 53 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp -s 128.232.0.0/17 -d 128.232.1.0/29 --dport 53 -j ACCEPT

or

#2 cl.cam.ac.uk: this machine is a NS, so allow the traffic. Restrict sorce if a local resolver only
# See http://www.wiki.cl.cam.ac.uk/clwiki/SysInfo/SmallServers#dns
-A RH-Firewall-1-INPUT -p udp -m udp -s 2001:630:200:4570::/64 -d 2001:630:200:4570::d:0/125 --dport 53 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp -s 2001:630:200:4570::/64 -d 2001:630:200:4570::d:0/125 --dport 53 -j ACCEPT

or under ubuntu run ONE of the commands below

cl-asuser ufw allow Bind9
cl-asuser ufw allow from 128.232.0.0/17 to any app Bind9; cl-asuser ufw allow from 2001:630:212:200::/56 to any app Bind9
cl-asuser ufw allow from 128.232.0.0/17 to any port domain; cl-asuser ufw allow from 2001:630:212:200::/56 to any port domain

It appears that SELinux may need to be disabled on some systems ("SELINUX=permissive" in /etc/sysconfig/selinux or add a suitable module), otherwise it logs

audit(1169502708.276:11): avc:  denied  { name_bind } for  pid=2378 comm="named" src=10053 scontext=root:system_r:named_t:s0 tcontext=system_u:object_r:port_t:s0 tclass=udp_socket

and fails (works OK under Fedora [89] and CentOS 5.[01]).

For Redhat, set the pseudo IP address using /etc/sysconfig/network-scripts/ifcfg-eth0:$name for ipv4 or add a suitable IPV6ADDR_SECONDARIES for IPv6, and in /var/named/chroot/ (or / if not chroot'ed) install etc/named.conf (which includes other files such as etc/named.cl.slaves) and populate var/named/data/ on dns-serv* hosts (resolv* should fetch what they need).

For Ubuntu, /etc/network/interfaces needs something such as

auto eth0:3
iface eth0:3 inet static
         address 128.232.1.3
         netmask 255.255.240.0
iface eth1 inet6 auto
        post-up  ip -f inet6 addr add 2001:630:212:200::d:2 dev eth1 preferred_lft 0 || true
        post-up  ip -f inet6 addr add 2001:630:212:267::d:2 dev eth1 preferred_lft 0 || true
        pre-down ip -f inet6 addr del 2001:630:212:267::d:2 dev eth1 || true
        pre-down ip -f inet6 addr del 2001:630:212:200::d:2 dev eth1 || true

Configuration is being moved towards using the CL-named-server package which creates a standard set of local files in /var/named/chroot/etc/named.d/ along with a patch file P-named.conf-dns or P-named.conf-resolver to patch a "standard" named.conf for local use. There are also sample files in named.d/named.conf-*. Possible patches include:

Note that the dns setup causes bind to crash under CentOS 5.2.

A simple resolver appears to run in 80MB.

On Ubuntu systems, /etc/apparmor.d/usr.sbin.named tends to allow read access to /etc/bind/** and read-write access to /var/lib/bind/** and /var/cache/bind/**. To make the std Lab system fit under standard Ubuntu this should be put into a package and installed BEFORE the cl-named-server package

sudo mkdir -p /var/cache/bind/{slaves,data} /var/named/chroot/etc
sudo ln -s /etc/bind /var/named/chroot/etc/named.d

This should be a standard Lab patch:

type=resolver # or dns
if ! grep named.d /etc/bind/named.conf.local
then cd /etc/bind
     test ! -e named.conf.local-preP &&
        sudo mv named.conf.local named.conf.local-preP &&
        sudo cp named.conf.local-preP named.conf.local
     sudo sh -c "echo 'include \"/etc/named.d/$type\";' >> named.conf.local"
fi

and similarly add to /etc/bind/named.conf.options the line

include "/etc/named.d/resolver";

NFS server

To allow a machine to server NFS files, it needs to allow incoming RPC, mount and NFS requests. RPC uses port 111, NFS nearly always uses port 2049, but the mount port needs to be fixed using "MOUNTD_PORT=2061" in /etc/sysconfig/nfs or "RPCMOUNTDOPTS="--port 2061"" in /etc/default/nfs-kernel-server. The ports then need to be enabled in /etc/sysconfig/iptables

#6 cl.cam.ac.uk: NFS server -- set "MOUNTD_PORT=2061" in /etc/sysconfig/nfs
-A RH-Firewall-1-INPUT -p udp -m udp --dport 111 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 111 -j ACCEPT

-A RH-Firewall-1-INPUT -p udp -m udp --dport 2049 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 2049 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 2061 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 2061 -j ACCEPT
#

or on Ubuntu systems, for access from within the Lab, use the commands

ufw allow from 128.232.0.0/17 to any port 111
ufw allow from 128.232.0.0/17 to any port 2049
ufw allow from 128.232.0.0/17 to any port 2061

mysql-serv*

See Installing MySQL on Linux for some background. When doing a standard Fedora "small server" install, do not remove "mysql". Setup the server

(cd /etc/user-config&&echo P-my.cnf-deb-allow-network>>patches&&echo mysql-server>>bundles;cl-update-system)
cl-asuser service mysqld  start; cl-asuser chkconfig mysqld on; # redhat only (mysqld)
cl-asuser service mysql restart; # debian only (mysql)
read -p 'Password for mysql root : ' -s RPASSWD; echo
mysqladmin -u root password $RPASSWD
mysqladmin -u root -h $HOSTNAME.cl.cam.ac.uk password $RPASSWD

where the password used was the 'physical access' one. The root password is set per connection type, e.g. unix socket and TCP above. Allow access to the service from applicable hosts (e.g. 128.232.0.0/20) at the TCP level, for redhat, add to /etc/sysconfig/iptables:

# This is a mysql server, so allow normal TCP access from within the Lab
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s 128.232.0.0/20 --dport 3306 -j ACCEPT
# ... and DMZ: www-2mc, www-emohunter, centos-ipd21 (rt#44529)
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s 128.232.104.27 --dport 3306 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s 128.232.104.29 --dport 3306 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s 128.232.104.58 --dport 3306 -j ACCEPT

and for debian using firestarter, to /etc/firestarter/inbound/allow-service add

mysql, 3306, 128.232.0.0/20, Internal mysql only
mysql, 3306, 128.232.104.27, DMZ: www-2mc
mysql, 3306, 128.232.104.29, DMZ: www-emohunter
mysql, 3306, 128.232.104.58, DMZ: centos-ipd21 (rt#44529)

Create a separate partition to contain /var/lib/mysql/, on redhat "restorecon -v -R" it to keep SELinux happy, and copy over the old contents. Access is also restricted by the server itself, using GRANT and such like (which update mysql.user etc). To allow nagios to probe a service, grant password free access to a null database; to enable management by phpMyAdmin access need to be granted to the web servers; e.g.

mysqladmin -u root --password="$RPASSWD" create nagios
for h in 0 1; do
echo "GRANT usage ON nagios.* TO nagios@'nagios-serv$h.cl.cam.ac.uk';" | mysql -u root --password="$RPASSWD"
echo "GRANT SUPER,REPLICATION CLIENT ON *.* TO 'nagios'@'nagios-serv1.cl.cam.ac.uk';" | mysql -u root --password="$RPASSWD"
done
for h in "" 2 3; do
echo "GRANT ALL PRIVILEGES ON *.* TO 'root'@'www-dyn$h.cl.cam.ac.uk' IDENTIFIED BY '$RPASSWD' WITH GRANT OPTION;" | mysql -u root --password="$RPASSWD"
done

Make sure the databases are dumped frequently, possibly separately (to allow individual ones to be reloaded) and the whole lot (to ensure noting is omitted) using "mysqldump -u root --password="$RPASSWD" -A". A suitable volume might be elmer:/vol/vol6/ which has a short snapvault history.

my.cnf

Under Redhat the server is configured using /etc/my.cnf whereas debian uses /etc/mysql/my.cnf - the former is used below.

Under Redhat it has a final line "!includedir /etc/mysql.d/" whereas debian uses /etc/mysql/conf.d/ - the former is used below.

For local edits, create a /etc/mysql.d/cl.cnf with a [mysqld] to which can be added tweaks below

# See http://www.wiki.cl.cam.ac.uk/clwiki/SysInfo/SmallServers#mysql
[mysqld]

To try to avoid some of the problems with ever growing ibdata1 files, add the line innodb_file_per_table in the [mysqld] section

innodb_file_per_table

dmg36 requested that max_allowed_packet be at least 6MB, so under redhat (debian already has a default of 16M) add in [mysqld]

# rt#23086 to allow 5.9MB files for dmg36
max_allowed_packet=8M

mysql replication

To enable replication (e.g. in master-master mode), server-id needs to be set in my.cnf, and on the master server log-bin. Consider where to put the binary logs (/var/log/mysql). It is good for it not to be on the root FS, as this may fill up and be replaced. It is also good for it not to be on the same FS as the main database files, allowing recent changes to be re done if the main files need to be restored. If /var/log/mysql is a symbolic link, on systems with SELinux, check that it is set up appropriately, e.g. /etc/selinux/targeted/contexts/files/file_contexts.local has

/var/log/mysql system_u:object_r:lib_t:s0

It is also useful to set auto-increment-increment to the number of servers, which each having a different value for auto-increment-offset so that any auto increments are unique whichever server is used. On slaves, set report-host to the server name so that "SHOW SLAVE HOSTS;" is helpful. On a master, consider setting expire_logs_days and max_binlog_size to limit the log file sizes.

On the master mysql-serv$M, to allow slave mysql-serv$S access, use

GRANT REPLICATION SLAVE ON *.* TO 'slave$S-user$M'@'mysql-serv$S.cl.cam.ac.uk' IDENTIFIED BY '$SLPW';

To copy an existing database ("LOAD DATA FROM MASTER" is deprecated), note the current log position using "SHOW MASTER STATUS;", dump the database on the master, restore it on the slave, "STOP SLAVE", load the database, then set the master using the log position, e.g.

CHANGE MASTER TO MASTER_HOST='mysql-serv0', MASTER_USER='slave1-user0', MASTER_PASSWORD='$SLPW', MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=903;

This stores the information in master.info which is resonably secure.

To check that the slave is working, on the master run

SHOW MASTER STATUS;
SHOW SLAVE HOSTS;

and on the slave run

SHOW SLAVE STATUS\G;

and check that there are entries such as

             Slave_IO_State: Waiting for master to send event
                Master_Host: mysql-serv0
                Master_User: slave1-user0
            Master_Log_File: mysql-bin.000002
        Read_Master_Log_Pos: 98
           Slave_IO_Running: Yes
          Slave_SQL_Running: Yes
      Seconds_Behind_Master: 0

and that Master_Log_File and Read_Master_Log_Pos match with the master.

To restore local access (e.g. needed to start and stop the server) by local user debian-sys-maint on a slave debian server to a non debian master, on the master run

echo "GRANT ALL PRIVILEGES ON *.* TO 'debian-sys-maint'@'localhost' IDENTIFIED BY '$DSMPW' WITH GRANT OPTION;" | mysql -u root -p=$RPASSWD

where $SDMPW is the string in /etc/mysql/debian.cnf on the slave1.

Create /etc/mysql.d/cl-repl.cnf with

# $Header: $
# See http://www.wiki.cl.cam.ac.uk/clwiki/SysInfo/SmallServers#mysql
# cl.cam.ac.uk tweaks for redhat mysql-serv0 for replication
[mysqld]
replicate-same-server-id= 0
# auto-increment-increment is the number of servers
auto-increment-increment= 2

# server-id is mysql-serv<N> is <N>+1
server-id             = 1
# auto-increment-offset is mysql-serv<N> is <N>+1
auto-increment-offset = 1
report-host           = mysql-serv0

# Needed on Redhat only - debian my.cnf has 10 day expire and 100M binlog
expire_logs_days = 10
max_binlog_size = 40M

# Set on Redhat
log-bin = /var/log/mysql/mysql-bin.log
relay-log       = mysql-serv0-relay-bin

or

# $Header: $
# See http://www.wiki.cl.cam.ac.uk/clwiki/SysInfo/SmallServers#mysql
# cl.cam.ac.uk tweaks for debian mysql-serv1 for replication
[mysqld]
# Listen on any address - debian only
bind-address    = 0

replicate-same-server-id= 0
# auto-increment-increment is the number of servers
auto-increment-increment= 2

# server-id is mysql-serv<N> is <N>+1
server-id             = 2
# auto-increment-offset is mysql-serv<N> is <N>+1
auto-increment-offset = 2
report-host           = mysql-serv1

If a database is trashing the binary log, it can be disabled until the user is contacted, by using "binlog-ignore-db = ''jra40_graph''"

mysql replication: TODO: add archive only server

crontab.gen arranges to rsync a copy of the raw database files to /usr/groups/linux/mysql/backup/ every quarter of an hour on each of the omnipotent servers (maybe set to use --backup-dir as the volume is snapshoted anyway?). This stores the raw database files on a snapshot'ed filer volume, allowing rollback at hourly granularity. The same could be achieved using a replication client which cannot be accessed by any clients, running with NFS access to the filer. This would ensure continuous updates, rather than polled, and the updates may cause less churn for the filer.

However, it is more complex a system and open to systematic failure of all the mysql servers.

mysql dumps

Replications improves resilience in case of server problems, but does not protect against user or admin errors. As such, users should take dumps of their databases at appropriate intervals. Occassional dumps are taken to /a/misc-nosnap1/mysql-serv/ but these should not be relied on by users.

The complete database can be rebuilt using

mysql -u root --password -h $serv < $file

or just a single database can be selected by editing the file.

Changes for $database since the last dump (e.g. at 2009-10-27 08:37:00) can be selected from the binary log using commands such as

sudo mysqlbinlog --database=$database mysql-bin.000$n   --start-datetime="2009-10-27 08:37:00"

monitoring - ping, snmptrap, dhcpleases, nagios, mrtg, netdisco

There is a separate page for monitoring tools such as ping, snmptrap, nagios, mrtg and disco.

syslog0

The syslog server is used to centrally log information, e.g. from the Cisco Routers and the mail servers. Most modern syslogd daemons only listen for local packets, so /etc/sysconfig/syslog needs to have "-r" added to SYSLOGD_OPTIONS, and /etc/sysconfig/iptables needs lines such as

#1 syslog server
-A RH-Firewall-1-INPUT -p udp -m udp --dport 514 -s 128.232.0.0/17 -d 128.232.0.37 -j ACCEPT

Ensure that the log files are retained and compressed by adding /var/log/local0.log to the list of files to process in /etc/logrotate.d/syslog, and the lines

    #5 cl.cam.ac.uk additions to keep logs on syslog0.cl.cam.ac.uk <<<<<<
    compress
    delaycompress
    daily
    rotate 1000
    olddir .KEEP
    #5 cl.cam.ac.uk additions to keep logs on syslog0.cl.cam.ac.uk >>>>>>

and in the postrotate script add

        #1 BUG: mode is NOT copied from existing file. Don't use "create 644" as just want local0.log <<<<<<
        /bin/chmod a+r /var/log/local0.log
        #1 BUG: mode is NOT copied from existing file. Don't use "create 644" as just want local0.log >>>>>>

to compress the last but one file (for space reasons, but keep the last one for cron Cisco processing below), process daily (there is quite a lot of traffic) and keep 3 years worth in a "hidden" directory .KEEP which has to be manually created. There is no "compressifnonempty" flag, so should "ifempty" be used? Should gzip or bzip2 be used? Put the Cisco Router (and any other local0 traffic there may be) into its own file by appending ";local0.none" to the *.info line for /var/log/messages in /etc/syslog.conf and add the new lines

#1 cl.cam.ac.uk: Save local0 (Cisco router) messages also to local0.log <<<<
local0.*                    -/var/log/local0.log
#1 cl.cam.ac.uk: Save local0 (Cisco router) messages also to local0.log >>>>

Setup a cron job to process the current log (and the previous one, in case it has cycled since last cron run) (the one on huntingdon was as user maj1) e.g.

0 * * * * /usr/groups/netmaint/Cisco/ports-up-down -q /var/log/.KEEP/local0.log-1  /var/log/local0.log
5 * * * * /usr/groups/netmaint/Cisco/port-vlan-summary -q

This needs the perl-DBD-Sybase RPM, so add it to /etc/user-config/bundles. Note that the logs have to be readable, so, e.g. run once "sudo chmod a+r /var/log/local0.log". Using "logrotate -f" (on a stripped down /etc/logrotate.d/syslog with just a /var/log/local0.log entry) or otherwise, create an initial /var/log/.KEEP/local0.log-1 so that the cron job above does not die.

radius servers

RADIUS has a pair of services: auth to authenticate the client and acct to log the usage. The ports used to be 1645 and 1646, but have moved to 1812 and 1813. We use Linux radius servers for accounting stansted.net's PPTP VPN service using radius-acct0 (e.g. meldreth) and radius-acct1 (e.g. XenE client). The authentication service runs on a Windows server (as of 2010/08/16, kingscross).

As of 2007/03 we use the TurboLinux 32b radiusd-cistron RPM (e.g. 1.6.6-6). The local configuration files are in /etc/raddb/, clients, dictionary, huntgroups and naslist from /usr/groups/netmaint/RADIUS/. See rt#64224 for some discussions. /etc/sysconfig/iptables should have en entry sch as:

-A RH-Firewall-1-INPUT -p udp -m udp --dport 1812:1813 -s 128.232.0.0/17 -j ACCEPT

Note that the service is not used very frequently, so do not expect natural traffic to test any changes.

On stansted.net add the keys as

radius-server host 128.232.0.38 auth-port 1812 acct-port 1813 key 7 xxxxxxxxxxxxxxxx
radius-server host 128.232.0.39 auth-port 1812 acct-port 1813 key 7 xxxxxxxxxxxxxxxx

Set up "aaa group server radius radius-accounting" to have the servers in the required order.

server 128.232.0.38 auth-port 1812 acct-port 1813
server 128.232.0.39 auth-port 1812 acct-port 1813

so that it tries .38 first, but if that fails, tries .39 as well. The logs are put in /var/log/ such as

standalone CPU servers (HOL)

In some cases completely standalone systems (accessible only using the console) are of use, e.g. HOL systems which cannot generate an image. Such systems need no networking, so no network listeners, system updates, etc. The filing system is less than 600M, and can be mounted read-only so that saved images can be reloaded later (to generate an ownfiles tar, attach a 100K file as xvdb and run "BASEDIR=/dev/shm/t cl-make-ownfiles -> /dev/xvdb"). The kernel memory use can be reduced to around 20M by using /etc/sysconfig/modules/udev-stw.modules to remove pcspkr ehci-hcd ohci-hcd uhci-hcd rather than adding extra modules.

A second build was based on a tailored FC6 install, disabling all packages other than basic, then removing RPMs. Until xen supports 32b PV kernels under a 64b hypervisor, to allow a machine to run under either, a 64b kernel and initrd is need in the domU's /boot. Force in the xen disc and network drivers, and tell it to omit the dom0's kernel modules which are not needed, e.g.

mkinitrd --with xennet --with xenblk ---with iptable_filter -rootdev=/dev/xvda1 --builtin=scsi_mod --builtin=ehci-hcd --builtin=ohci-hcd --builtin=uhci-hcd --builtin=xor --builtin=raid1 --builtin=ata_piix --builtin=libata --builtin=sd_mod --builtin=raid456 --builtin=scsi_transport_spi --builtin=aic7xxx --builtin=qla1280 --builtin=megaraid_mm --builtin=megaraid_mbox -v /boot/domU/initrd-2.6.20-1.2948.fc6xen_64-jdb-ext3  2.6.20-1.2948.fc6xen

With the network (but not NFS) available and 32b and 64b kernek and initrd (but not kernel RPM), it needs around 600MB of disk, and runs in 16MB (it's safer to give it 32MB).

The network is switched off completely by not having a vif= line in the domU config files. The above iptable_filter is sufficient for an unsubtle /etc/sysconfig/iptables of

:INPUT DROP [0:0]
-A RH-Firewall-1-INPUT -i lo -j ACCEPT

serial.srg

To allow the SRG to setup their serial server (on FC8 fletcher), sys admin needs to grant them write access to /etc/xinetd.d/ (to add nsplit and xcons') and /etc/services (to add nsplit on 12000-12862/tcp and xcons01-xcons20 on 13010-13200/tcp) -- see rt#47208. They also need to install the xinetd RPM, and add iptables rules to allow local machines to access the services. Root also needs to install /home/gm281/Unison/Exports/Documents/PhD/Infrastructure/identity-srg-bmc in /etc/ssh/ and change it to be user xenod, which should be added to the local /etc/passwd file, e.g.

xenod:!:94:141:Xen Daemon:/usr/groups/xeno/users/xenod:/dev/null

See /usr/groups/xeno/scripts/xencons and in /usr/groups/netos/ see systems/serial/etc/nslitd, sources/nsplitd/nsplitd.c', systems/serial/etc/xcons' and sources/nsplitd/xcons.c'.

gprs-router-1 (bluebird)

GPRS traffic is routed via a machine on a dedicated VLAN (128.232.48/28) to allow any intercepts to be made. This machine requiresn a pseudo interface, e.g. by having /etc/sysconfig/network-scripts/ifcfg-eth0:gprs-router-1 contain

. /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0:60
IPADDR=128.232.96.60

and needs to allow IP forwarding, e.g. by /etc/sysctl.conf containing

net.ipv4.ip_forward = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.ip_always_defrag = 0

exams machine

The exams machine is normally not connected to the network, in a locked room which has a non suite key.

It's an FC6 machine using DHCP and "@R-nonfs" (and NOT_autofs, NOT_sendmail and NOT_nfs-utils) in /etc/user-config/bundles to work standalone. Unneeded RPMs have to be manually removed. /etc/user-config/patches should have @F-nonfs to avoid applying inappropriate patches.

As it is standalone, it does not use LDAP or KRB5, so the timeouts in /etc./ldap.conf should be reduced to 1 second, and all referefnces to ldap removed from /etc/nsswitch.conf. All usere need local $HOME and entries in /etc/{passwd,shadow,group}.

It has a locally attached printer (larch: HP LaserJet 4000NT with Duplex Unit) which needs to be setup under CUPS, and it needs the CL-cups-serv RPM loaded.

publicdump CD/DVD writing service

Start with a standard FC6 install.

To ensure that the documentation is correct, add to /etc/sudoers:

# publicdump special
ALL     ALL=NOPASSWD: /usr/bin/cdrecord,/usr/bin/cl-writedvdimage,/usr/bin/writedvdimage

to allow the documented commands, and make writedvdimage available without the "cl-" prefix:

ln -s cl-writedvdimage /usr/bin/writedvdimage

To add CD/DVD related RPMs, add to /etc/user-config/bundles the lines:

k3b
CL-writedvdimage

FootNotes

  • 1 debian-sys-maint probably does not need Create_view_priv Show_view_priv Create_routine_priv Alter_routine_priv Create_user_priv

SysInfo/SmallServers (last edited 2014-01-04 19:34:32 by PieteBrooks)