Archive

Posts Tagged ‘Apache’

OpenVZ or Xen ?

January 30th, 2009

Warning: file_get_contents(http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService&AWSAccessKeyId=1TJ8QTQ6ZFCVAJ3X1T02&AssociateTag=ii0c3-20&Operation=ItemSearch&SearchIndex=Books&ResponseGroup=Small,Images&Keywords=Apache) [function.file-get-contents]: failed to open stream: HTTP request failed! HTTP/1.1 400 Bad Request in /home/manusia2/public_html/wp-content/plugins/amazonfeed/php/amazonfeed.class.php on line 271

OpenVZ:
Advantages: allows overselling. Very light weight. Can accommodate more Virtual Machines in a server.

Disadvantage: There is no per vps swap.

Why this is important:

OpenVZ will KILL your application if it goes beyond the limit, and this can cause some trouble. There are people out there who want to host oracle on a 64MB vps, and with such customers, using openVZ will lead to constant application crashes, which ultimately will be blamed on the provider. (This is actually something that is common with openvz/virtuozzo hosting in general; you can check some threads at wht).

With Xen, each vps has its own swap, and thus you get an EXACT dedicated server like environment, but with lesser resources. So here, the customers applications will NOT crash, but rather it will become slower. Also, majority of the applications, like apache, spamassassin expects a lot of memory, and openVZ makes memory a very valuable commodity.

So generally my recommendation is that: For friendly customers use openVZ, and use a lot of burst memory. For not-so-friendly customers, use Xen. And that is why we are providing transparent migration. You can start a customer on openVZ, and see how it works out, and if he is getting too many application crashes, you can move him to the SAME configuration on Xen, and he should be able to do fine, though his application would be slower.

Tags: Apache, burst, Memory, OpenVZ, oracle, swap, Virtuozzo, Virtuozzo, xen

Related posts

Virtuozzo , , , , , , ,

Mod_Security.. what is and how to ?

January 17th, 2009

    What Is ModSecurity?

    ModSecurity is a web application firewall that can work either embedded or as a reverse proxy. It provides protection from a range of attacks against web applications and allows for HTTP traffic monitoring, logging and real-time analysis.

    It is also an open source project that aims to make the web application firewall technology available to everyone.

    Licensing

    ModSecurity is available under two licenses. Users can choose to use the software under the terms of the GNU General Public License version 2 (licence text is included with the distribution), as an Open Source / Free Software product. A range of commercial licenses is also available, together with a range of commercial support contracts. For more information on commercial licensing please contact Breach Security.

    Note

    ModSecurity, mod_security, ModSecurity Pro, and ModSecurity Core Rules are trademarks or registered trademarks of Breach Security, Inc.

Tags: Apache, firewall, General, modsecurity, protection, security, software

Related posts

General , , , , ,

FreeBSD Handbook - Chapter 30 Firewalls

January 6th, 2009

Contributed by Joseph J. Barbish. Converted to SGML and updated by Brad Davis.


30.1 Introduction

Firewalls make it possible to filter incoming and outgoing traffic that flows through your system. A firewall can use one or more sets of “rules” to inspect the network packets as they come in or go out of your network connections and either allows the traffic through or blocks it. The rules of a firewall can inspect one or more characteristics of the packets, including but not limited to the protocol type, the source or destination host address, and the source or destination port.

Firewalls can greatly enhance the security of a host or a network. They can be used to do one or more of the following things:

  • To protect and insulate the applications, services and machines of your internal network from unwanted traffic coming in from the public Internet.

  • To limit or disable access from hosts of the internal network to services of the public Internet.

  • To support network address translation (NAT), which allows your internal network to use private IP addresses and share a single connection to the public Internet (either with a single IP address or by a shared pool of automatically assigned public addresses).

After reading this chapter, you will know:

  • How to properly define packet filtering rules.

  • The differences between the firewalls built into FreeBSD.

  • How to use and configure the OpenBSD PF firewall.

  • How to use and configure IPFILTER.

  • How to use and configure IPFW.

Before reading this chapter, you should:

  • Understand basic FreeBSD and Internet concepts.


30.2 Firewall Concepts

There are two basic ways to create firewall rulesets: “inclusive” or “exclusive”. An exclusive firewall allows all traffic through except for the traffic matching the ruleset. An inclusive firewall does the reverse. It only allows traffic matching the rules through and blocks everything else.

Inclusive firewalls are generally safer than exclusive firewalls because they significantly reduce the risk of allowing unwanted traffic to pass through the firewall.

Security can be tightened further using a “stateful firewall”. With a stateful firewall the firewall keeps track of which connections are opened through the firewall and will only allow traffic through which either matches an existing connection or opens a new one. The disadvantage of a stateful firewall is that it can be vulnerable to Denial of Service (DoS) attacks if a lot of new connections are opened very fast. With most firewalls it is possible to use a combination of stateful and non-stateful behavior to make an optimal firewall for the site.


30.3 Firewall Packages

FreeBSD has three different firewall packages built into the base system. They are: IPFILTER (also known as IPF), IPFIREWALL (also known as IPFW), and OpenBSD’s PacketFilter (also known as PF). FreeBSD also has two built in packages for traffic shaping (basically controlling bandwidth usage): altq(4) and dummynet(4). Dummynet has traditionally been closely tied with IPFW, and ALTQ with PF. Traffic shaping for IPFILTER can currently be done with IPFILTER for NAT and filtering and IPFW with dummynet(4) or by using PF with ALTQ. IPFW, and PF all use rules to control the access of packets to and from your system, although they go about it different ways and have different rule syntaxes.

The reason that FreeBSD has multiple built in firewall packages is that different people have different requirements and preferences. No single firewall package is the best.

The author prefers IPFILTER because its stateful rules are much less complicated to use in a NAT environment and it has a built in ftp proxy that simplifies the rules to allow secure outbound FTP usage.

Since all firewalls are based on inspecting the values of selected packet control fields, the creator of the firewall rulesets must have an understanding of how TCP/IP works, what the different values in the packet control fields are and how these values are used in a normal session conversation. For a good explanation go to: http://www.ipprimer.com/overview.cfm.


30.4 The OpenBSD Packet Filter (PF) and ALTQ

Revised and updated by John Ferrell.

As of July 2003 the OpenBSD firewall software application known as PF was ported to FreeBSD and made available in the FreeBSD Ports Collection. Released in 2004, FreeBSD 5.3 was the first release that contained PF as an integrated part of the base system. PF is a complete, full-featured firewall that has optional support for ALTQ (Alternate Queuing). ALTQ provides Quality of Service (QoS) functionality.

The OpenBSD Project does an outstanding job of maintaining the PF FAQ. As such, this section of the Handbook will focus on PF as it pertains to FreeBSD while providing some general information regarding usage. For detailed usage information please refer to the PF FAQ.

More information about PF for FreeBSD can be found at http://pf4freebsd.love2party.net/.


30.4.1 Using the PF loadable kernel module

Since the release of FreeBSD 5.3, PF has been included in the basic install as a separate run time loadable module. The system will dynamically load the PF kernel module when the rc.conf(5) statement pf_enable="YES" is present. However, the PF module will not load if the system cannot find a PF ruleset configuration file. The default location is /etc/pf.conf. If your PF ruleset is located somewhere else put pf_rules="/path/pf.rules" to your /etc/rc.conf configuration file to specify the location.

Note: As of FreeBSD 7.0 the sample pf.conf that was in /etc/ has been moved to /usr/share/examples/pf/. For FreeBSD versions prior to 7.0 there is an /etc/pf.conf by default.

The PF module can also be loaded manually from the command line:

# kldload pf.ko

The loadable module was created with pflog(4) enabled which provides support for logging. If you need other PF features you will need to compile PF support into the kernel.


30.4.2 PF kernel options

While it is not necessary that you compile PF support into the FreeBSD kernel, you may want to do so to take advantage of one of PF’s advanced features that is not included in the loadable module, namely pfsync(4), which is a pseudo-device that exposes certain changes to the state table used by PF. It can be paired with carp(4) to create failover firewalls using PF. More information on CARP can be found in chapter 29 of the handbook.

The PF kernel options can be found in /usr/src/sys/conf/NOTES and are reproduced below:

device pf
device pflog
device pfsync

The device pf option enables support for the “Packet Filter” firewall (pf(4)).

The device pflog option enables the optional pflog(4) pseudo network device which can be used to log traffic to a bpf(4) descriptor. The pflogd(8) daemon can be used to store the logging information to disk.

The device pfsync option enables the optional pfsync(4) pseudo-network device that is used to monitor “state changes”.


30.4.3 Available rc.conf Options

The following rc.conf(5) statements configure PF and pflog(4) at boot:

pf_enable="YES"                 # Enable PF (load module if required)
pf_rules="/etc/pf.conf"         # rules definition file for pf
pf_flags=""                     # additional flags for pfctl startup
pflog_enable="YES"              # start pflogd(8)
pflog_logfile="/var/log/pflog"  # where pflogd should store the logfile
pflog_flags=""                  # additional flags for pflogd startup

If you have a LAN behind this firewall and have to forward packets for the computers on the LAN or want to do NAT, you will need the following option as well:

gateway_enable="YES"            # Enable as LAN gateway

30.4.4 Creating Filtering Rules

PF reads its configuration rules from pf.conf(5) (/etc/pf.conf by default) and it modifies, drops, or passes packets according to the rules or definitions specified there. The FreeBSD installation includes several sample files located in /usr/share/examples/pf/. Please refer to the PF FAQ for complete coverage of PF rulesets.

Warning: When browsing the PF FAQ, please keep in mind that different versions of FreeBSD contain different versions of PF:

  • FreeBSD 5.XPF is at OpenBSD 3.5

  • FreeBSD 6.XPF is at OpenBSD 3.7

  • FreeBSD 7.XPF is at OpenBSD 4.1

The FreeBSD packet filter mailing list is a good place to ask questions about configuring and running the PF firewall. Do not forget to check the mailing list archives before asking questions!


30.4.5 Working with PF

Use pfctl(8) to control PF. Below are some useful commands (be sure to review the pfctl(8) man page for all available options):

Command

Purpose

pfctl -e

Enable PF

pfctl -d

Disable PF

pfctl -F all -f /etc/pf.conf

Flush all rules (nat, filter, state, table, etc.) and reload from the file /etc/pf.conf

pfctl -s [ rules | nat | state ]

Report on the filter rules, nat rules, or state table

pfctl -vnf /etc/pf.conf

Check /etc/pf.conf for errors, but do not load ruleset


30.4.6 Enabling ALTQ

ALTQ is only available by compiling support for it into the FreeBSD kernel. ALTQ is not supported by all of the available network card drivers. Please see the altq(4) manual page for a list of drivers that are supported in your release of FreeBSD.

The following kernel options will enable ALTQ and add additional functionality:

options         ALTQ
options         ALTQ_CBQ        # Class Bases Queuing (CBQ)
options         ALTQ_RED        # Random Early Detection (RED)
options         ALTQ_RIO        # RED In/Out
options         ALTQ_HFSC       # Hierarchical Packet Scheduler (HFSC)
options         ALTQ_PRIQ       # Priority Queuing (PRIQ)
options         ALTQ_NOPCC      # Required for SMP build

options ALTQ enables the ALTQ framework.

options ALTQ_CBQ enables Class Based Queuing (CBQ). CBQ allows you to divide a connection’s bandwidth into different classes or queues to prioritize traffic based on filter rules.

options ALTQ_RED enables Random Early Detection (RED). RED is used to avoid network congestion. RED does this by measuring the length of the queue and comparing it to the minimum and maximum thresholds for the queue. If the queue is over the maximum all new packets will be dropped. True to its name, RED drops packets from different connections randomly.

options ALTQ_RIO enables Random Early Detection In and Out.

options ALTQ_HFSC enables the Hierarchical Fair Service Curve Packet Scheduler. For more information about HFSC see: http://www-2.cs.cmu.edu/~hzhang/HFSC/main.html.

options ALTQ_PRIQ enables Priority Queuing (PRIQ). PRIQ will always pass traffic that is in a higher queue first.

options ALTQ_NOPCC enables SMP support for ALTQ. This option is required on SMP systems.


30.5 The IPFILTER (IPF) Firewall

Note: This section is work in progress. The contents might not be accurate at all times.

The author of IPFILTER is Darren Reed. IPFILTER is not operating system dependent: it is an open source application and has been ported to FreeBSD, NetBSD, OpenBSD, SunOS, HP/UX, and Solaris operating systems. IPFILTER is actively being supported and maintained, with updated versions being released regularly.

IPFILTER is based on a kernel-side firewall and NAT mechanism that can be controlled and monitored by userland interface programs. The firewall rules can be set or deleted with the ipf(8) utility. The NAT rules can be set or deleted with the ipnat(1) utility. The ipfstat(8) utility can print run-time statistics for the kernel parts of IPFILTER. The ipmon(8) program can log IPFILTER actions to the system log files.

IPF was originally written using a rule processing logic of “the last matching rule wins” and used only stateless type of rules. Over time IPF has been enhanced to include a “quick” option and a stateful “keep state” option which drastically modernized the rules processing logic. IPF’s official documentation covers the legacy rule coding parameters and the legacy rule file processing logic. The modernized functions are only included as additional options, completely understating their benefits in producing a far superior secure firewall.

The instructions contained in this section are based on using rules that contain the “quick” option and the stateful “keep state” option. This is the basic framework for coding an inclusive firewall rule set.

An inclusive firewall only allows packets matching the rules to pass through. This way you can control what services can originate behind the firewall destined for the public Internet and also control the services which can originate from the public Internet accessing your private network. Everything else is blocked and logged by default design. Inclusive firewalls are much, much more secure than exclusive firewall rule sets and is the only rule set type covered herein.

For detailed explanation of the legacy rules processing method see: http://www.obfuscation.org/ipf/ipf-howto.html#TOC_1 and http://coombs.anu.edu.au/~avalon/ip-filter.html.

The IPF FAQ is at http://www.phildev.net/ipf/index.html.

A searchable archive of the open-source IPFilter mailing list is available at http://marc.theaimsgroup.com/?l=ipfilter.


30.5.1 Enabling IPF

IPF is included in the basic FreeBSD install as a separate run time loadable module. The system will dynamically load the IPF kernel loadable module when the rc.conf statement ipfilter_enable="YES" is used. The loadable module was created with logging enabled and the default pass all options. You do not need to compile IPF into the FreeBSD kernel just to change the default to block all, you can do that by just coding a block all rule at the end of your rule set.


30.5.2 Kernel options

It is not a mandatory requirement that you enable IPF by compiling the following options into the FreeBSD kernel. It is only presented here as background information. Compiling IPF into the kernel causes the loadable module to never be used.

Sample kernel config IPF option statements are in the /usr/src/sys/conf/NOTES kernel source and are reproduced here:

options IPFILTER
options IPFILTER_LOG
options IPFILTER_DEFAULT_BLOCK

options IPFILTER enables support for the “IPFILTER” firewall.

options IPFILTER_LOG enables the option to have IPF log traffic by writing to the ipl packet logging pseudo–device for every rule that has the log keyword.

options IPFILTER_DEFAULT_BLOCK changes the default behavior so any packet not matching a firewall pass rule gets blocked.

These settings will take effect only after you have built and installed a kernel with them set.


30.5.3 Available rc.conf Options

You need the following statements in /etc/rc.conf to activate IPF at boot time:

ipfilter_enable="YES"             # Start ipf firewall
ipfilter_rules="/etc/ipf.rules"   # loads rules definition text file
ipmon_enable="YES"                # Start IP monitor log
ipmon_flags="-Ds"                 # D = start as daemon
                                  # s = log to syslog
                                  # v = log tcp window, ack, seq
                                  # n = map IP & port to names

If you have a LAN behind this firewall that uses the reserved private IP address ranges, then you need to add the following to enable NAT functionality:

gateway_enable="YES"              # Enable as LAN gateway
ipnat_enable="YES"                # Start ipnat function
ipnat_rules="/etc/ipnat.rules"    # rules definition file for ipnat

30.5.4 IPF

The ipf command is used to load your rules file. Normally you create a file containing your custom rules and use this command to replace in mass the currently running firewall internal rules:

# ipf -Fa -f /etc/ipf.rules

-Fa means flush all internal rules tables.

-f means this is the file to read for the rules to load.

This gives you the ability to make changes to your custom rules file, run the above IPF command, and thus update the running firewall with a fresh copy of all the rules without having to reboot the system. This method is very convenient for testing new rules as the procedure can be executed as many times as needed.

See the ipf(8) manual page for details on the other flags available with this command.

The ipf(8) command expects the rules file to be a standard text file. It will not accept a rules file written as a script with symbolic substitution.

There is a way to build IPF rules that utilizes the power of script symbolic substitution. For more information, see Section 30.5.9.


30.5.5 IPFSTAT

The default behavior of ipfstat(8) is to retrieve and display the totals of the accumulated statistics gathered as a result of applying the user coded rules against packets going in and out of the firewall since it was last started, or since the last time the accumulators were reset to zero by the ipf -Z command.

See the ipfstat(8) manual page for details.

The default ipfstat(8) command output will look something like this:

input packets: blocked 99286 passed 1255609 nomatch 14686 counted 0
 output packets: blocked 4200 passed 1284345 nomatch 14687 counted 0
 input packets logged: blocked 99286 passed 0
 output packets logged: blocked 0 passed 0
 packets logged: input 0 output 0
 log failures: input 3898 output 0
 fragment state(in): kept 0 lost 0
 fragment state(out): kept 0 lost 0
 packet state(in): kept 169364 lost 0
 packet state(out): kept 431395 lost 0
 ICMP replies: 0 TCP RSTs sent: 0
 Result cache hits(in): 1215208 (out): 1098963
 IN Pullups succeeded: 2 failed: 0
 OUT Pullups succeeded: 0 failed: 0
 Fastroute successes: 0 failures: 0
 TCP cksum fails(in): 0 (out): 0
 Packet log flags set: (0)

When supplied with either -i for inbound or -o for outbound, it will retrieve and display the appropriate list of filter rules currently installed and in use by the kernel.

ipfstat -in displays the inbound internal rules table with rule number.

ipfstat -on displays the outbound internal rules table with the rule number.

The output will look something like this:

@1 pass out on xl0 from any to any
@2 block out on dc0 from any to any
@3 pass out quick on dc0 proto tcp/udp from any to any keep state

ipfstat -ih displays the inbound internal rules table, prefixing each rule with a count of how many times the rule was matched.

ipfstat -oh displays the outbound internal rules table, prefixing each rule with a count of how many times the rule was matched.

The output will look something like this:

2451423 pass out on xl0 from any to any
354727 block out on dc0 from any to any
430918 pass out quick on dc0 proto tcp/udp from any to any keep state

One of the most important functions of the ipfstat command is the -t flag which displays the state table in a way similar to the way top(1) shows the FreeBSD running process table. When your firewall is under attack this function gives you the ability to identify, drill down to, and see the attacking packets. The optional sub-flags give the ability to select the destination or source IP, port, or protocol that you want to monitor in real time. See the ipfstat(8) manual page for details.


30.5.6 IPMON

In order for ipmon to work properly, the kernel option IPFILTER_LOG must be turned on. This command has two different modes that it can be used in. Native mode is the default mode when you type the command on the command line without the -D flag.

Daemon mode is for when you want to have a continuous system log file available so that you can review logging of past events. This is how FreeBSD and IPFILTER are configured to work together. FreeBSD has a built in facility to automatically rotate system logs. That is why outputting the log information to syslogd is better than the default of outputting to a regular file. In the default rc.conf file you see the ipmon_flags statement uses the -Ds flags:

ipmon_flags="-Ds" # D = start as daemon
                  # s = log to syslog
                  # v = log tcp window, ack, seq
                  # n = map IP & port to names

The benefits of logging are obvious. It provides the ability to review, after the fact, information such as which packets had been dropped, what addresses they came from and where they were going. These all give you a significant edge in tracking down attackers.

Even with the logging facility enabled, IPF will not generate any rule logging on its own. The firewall administrator decides what rules in the rule set he wants to log and adds the log keyword to those rules. Normally only deny rules are logged.

It is very customary to include a default deny everything rule with the log keyword included as your last rule in the rule set. This way you get to see all the packets that did not match any of the rules in the rule set.


30.5.7 IPMON Logging

Syslogd uses its own special method for segregation of log data. It uses special groupings called “facility” and “level”. IPMON in -Ds mode uses security as the “facility” name. All IPMON logged data goes to security The following levels can be used to further segregate the logged data if desired:

LOG_INFO - packets logged using the "log" keyword as the action rather than pass or block.
LOG_NOTICE - packets logged which are also passed
LOG_WARNING - packets logged which are also blocked
LOG_ERR - packets which have been logged and which can be considered short

To setup IPFILTER to log all data to /var/log/ipfilter.log, you will need to create the file. The following command will do that:

# touch /var/log/ipfilter.log

The syslog function is controlled by definition statements in the /etc/syslog.conf file. The syslog.conf file offers considerable flexibility in how syslog will deal with system messages issued by software applications like IPF.

Add the following statement to /etc/syslog.conf:

security.* /var/log/ipfilter.log

The security.* means to write all the logged messages to the coded file location.

To activate the changes to /etc/syslog.conf you can reboot or bump the syslog task into re-reading /etc/syslog.conf by running /etc/rc.d/syslogd reload

Do not forget to change /etc/newsyslog.conf to rotate the new log you just created above.


30.5.8 The Format of Logged Messages

Messages generated by ipmon consist of data fields separated by white space. Fields common to all messages are:

  1. The date of packet receipt.

  2. The time of packet receipt. This is in the form HH:MM:SS.F, for hours, minutes, seconds, and fractions of a second (which can be several digits long).

  3. The name of the interface the packet was processed on, e.g. dc0.

  4. The group and rule number of the rule, e.g. @0:17.

These can be viewed with ipfstat -in.

  1. The action: p for passed, b for blocked, S for a short packet, n did not match any rules, L for a log rule. The order of precedence in showing flags is: S, p, b, n, L. A capital P or B means that the packet has been logged due to a global logging setting, not a particular rule.

  2. The addresses. This is actually three fields: the source address and port (separated by a comma), the -> symbol, and the destination address and port. 209.53.17.22,80 -> 198.73.220.17,1722.

  3. PR followed by the protocol name or number, e.g. PR tcp.

  4. len followed by the header length and total length of the packet, e.g. len 20 40.

If the packet is a TCP packet, there will be an additional field starting with a hyphen followed by letters corresponding to any flags that were set. See the ipmon(8) manual page for a list of letters and their flags.

If the packet is an ICMP packet, there will be two fields at the end, the first always being “ICMP”, and the next being the ICMP message and sub-message type, separated by a slash, e.g. ICMP 3/3 for a port unreachable message.


30.5.9 Building the Rule Script with Symbolic Substitution

Some experienced IPF users create a file containing the rules and code them in a manner compatible with running them as a script with symbolic substitution. The major benefit of doing this is that you only have to change the value associated with the symbolic name and when the script is run all the rules containing the symbolic name will have the value substituted in the rules. Being a script, you can use symbolic substitution to code frequently used values and substitute them in multiple rules. You will see this in the following example.

The script syntax used here is compatible with the sh, csh, and tcsh shells.

Symbolic substitution fields are prefixed with a dollar sign: $.

Symbolic fields do not have the $ prefix.

The value to populate the symbolic field must be enclosed with double quotes (").

Start your rule file with something like this:

############# Start of IPF rules script ########################

oif="dc0"            # name of the outbound interface
odns="192.0.2.11"    # ISP's DNS server IP address
myip="192.0.2.7"     # my static IP address from ISP
ks="keep state"
fks="flags S keep state"

# You can choose between building /etc/ipf.rules file
# from this script or running this script "as is".
#
# Uncomment only one line and comment out another.
#
# 1) This can be used for building /etc/ipf.rules:
#cat > /etc/ipf.rules << EOF
#
# 2) This can be used to run script "as is":
/sbin/ipf -Fa -f - << EOF

# Allow out access to my ISP's Domain name server.
pass out quick on $oif proto tcp from any to $odns port = 53 $fks
pass out quick on $oif proto udp from any to $odns port = 53 $ks

# Allow out non-secure standard www function
pass out quick on $oif proto tcp from $myip to any port = 80 $fks

# Allow out secure www function https over TLS SSL
pass out quick on $oif proto tcp from $myip to any port = 443 $fks
EOF
################## End of IPF rules script ########################

That is all there is to it. The rules are not important in this example; how the symbolic substitution fields are populated and used are. If the above example was in a file named /etc/ipf.rules.script, you could reload these rules by entering the following command:

# sh /etc/ipf.rules.script

There is one problem with using a rules file with embedded symbolics: IPF does not understand symbolic substitution, and cannot read such scripts directly.

This script can be used in one of two ways:

  • Uncomment the line that begins with cat, and comment out the line that begins with /sbin/ipf. Place ipfilter_enable="YES" into /etc/rc.conf as usual, and run script once after each modification to create or update /etc/ipf.rules.

  • Disable IPFILTER in system startup scripts by adding ipfilter_enable="NO" (this is default value) into /etc/rc.conf file.

    Add a script like the following to your /usr/local/etc/rc.d/ startup directory. The script should have an obvious name like ipf.loadrules.sh. The .sh extension is mandatory.

    #!/bin/sh
    sh /etc/ipf.rules.script

    The permissions on this script file must be read, write, execute for owner root.

    # chmod 700 /usr/local/etc/rc.d/ipf.loadrules.sh

Now, when your system boots, your IPF rules will be loaded.


30.5.10 IPF Rule Sets

A rule set is a group of ipf rules coded to pass or block packets based on the values contained in the packet. The bi-directional exchange of packets between hosts comprises a session conversation. The firewall rule set processes the packet two times, once on its arrival from the public Internet host and again as it leaves for its return trip back to the public Internet host. Each TCP/IP service (i.e. telnet, www, mail, etc.) is predefined by its protocol, source and destination IP address, or the source and destination port number. This is the basic selection criteria used to create rules which will pass or block services.

IPF was originally written using a rules processing logic of “the last matching rule wins” and used only stateless rules. Over time IPF has been enhanced to include a “quick” option and a stateful “keep state” option which drastically modernized the rule processing logic.

The instructions contained in this section are based on using rules that contain the “quick” option and the stateful “keep state” option. This is the basic framework for coding an inclusive firewall rule set.

An inclusive firewall only allows services matching the rules through. This way you can control what services can originate behind the firewall destined for the public Internet and also control the services which can originate from the public Internet accessing your private network. Everything else is blocked and logged by default design. Inclusive firewalls are much, much securer than exclusive firewall rule sets and is the only rule set type covered herein.

Warning: When working with the firewall rules, be very careful. Some configurations will lock you out of the server. To be on the safe side, you may wish to consider performing the initial firewall configuration from the local console rather than doing it remotely e.g. via ssh.


30.5.11 Rule Syntax

The rule syntax presented here has been simplified to only address the modern stateful rule context and “first matching rule wins” logic. For the complete legacy rule syntax description see the ipf(8) manual page.

A # character is used to mark the start of a comment and may appear at the end of a rule line or on its own line. Blank lines are ignored.

Rules contain keywords. These keywords have to be coded in a specific order from left to right on the line. Keywords are identified in bold type. Some keywords have sub-options which may be keywords themselves and also include more sub-options. Each of the headings in the below syntax has a bold section header which expands on the content.

ACTION IN-OUT OPTIONS SELECTION STATEFUL PROTO SRC_ADDR,DST_ADDR OBJECT PORT_NUM TCP_FLAG STATEFUL

ACTION = block | pass

IN-OUT = in | out

OPTIONS = log | quick | on interface-name

SELECTION = proto value | source/destination IP | port = number | flags flag-value

PROTO = tcp/udp | udp | tcp | icmp

SRC_ADD,DST_ADDR = all | from object to object

OBJECT = IP address | any

PORT_NUM = port number

TCP_FLAG = S

STATEFUL = keep state


30.5.11.1 ACTION

The action indicates what to do with the packet if it matches the rest of the filter rule. Each rule must have a action. The following actions are recognized:

block indicates that the packet should be dropped if the selection parameters match the packet.

pass indicates that the packet should exit the firewall if the selection parameters match the packet.


30.5.11.2 IN-OUT

A mandatory requirement is that each filter rule explicitly state which side of the I/O it is to be used on. The next keyword must be either in or out and one or the other has to be coded or the rule will not pass syntax checks.

in means this rule is being applied against an inbound packet which has just been received on the interface facing the public Internet.

out means this rule is being applied against an outbound packet destined for the interface facing the public Internet.


30.5.11.3 OPTIONS

Note: These options must be used in the order shown here.

log indicates that the packet header will be written to the ipl log (as described in the LOGGING section below) if the selection parameters match the packet.

quick indicates that if the selection parameters match the packet, this rule will be the last rule checked, allowing a “short-circuit” path to avoid processing any following rules for this packet. This option is a mandatory requirement for the modernized rules processing logic.

on indicates the interface name to be incorporated into the selection parameters. Interface names are as displayed by ifconfig(8). Using this option, the rule will only match if the packet is going through that interface in the specified direction (in/out). This option is a mandatory requirement for the modernized rules processing logic.

When a packet is logged, the headers of the packet are written to the IPL packet logging pseudo-device. Immediately following the log keyword, the following qualifiers may be used (in this order):

body indicates that the first 128 bytes of the packet contents will be logged after the headers.

first If the log keyword is being used in conjunction with a “keep state” option, it is recommended that this option is also applied so that only the triggering packet is logged and not every packet which thereafter matches the “keep state” information.


30.5.11.4 SELECTION

The keywords described in this section are used to describe attributes of the packet to be interrogated when determining whether rules match or not. There is a keyword subject, and it has sub-option keywords, one of which has to be selected. The following general-purpose attributes are provided for matching, and must be used in this order:


30.5.11.5 PROTO

proto is the subject keyword and must be coded along with one of its corresponding keyword sub-option values. The value allows a specific protocol to be matched against. This option is a mandatory requirement for the modernized rules processing logic.

tcp/udp | udp | tcp | icmp or any protocol names found in /etc/protocols are recognized and may be used. The special protocol keyword tcp/udp may be used to match either a TCP or a UDP packet, and has been added as a convenience to save duplication of otherwise identical rules.


30.5.11.6 SRC_ADDR/DST_ADDR

The all keyword is essentially a synonym for “from any to any” with no other match parameters.

from src to dst: the from and to keywords are used to match against IP addresses. Rules must specify BOTH source and destination parameters. any is a special keyword that matches any IP address. Examples of use: “from any to any” or “from 0.0.0.0/0 to any” or “from any to 0.0.0.0/0” or “from 0.0.0.0 to any” or “from any to 0.0.0.0”.

IP addresses may be specified as a dotted IP address numeric form/mask-length, or as single dotted IP address numeric form.

There is no way to match ranges of IP addresses which do not express themselves easily as mask-length. See this web page for help on writing mask-length: http://jodies.de/ipcalc.


30.5.11.7 PORT

If a port match is included, for either or both of source and destination, then it is only applied to TCP and UDP packets. When composing port comparisons, either the service name from /etc/services or an integer port number may be used. When the port appears as part of the from object, it matches the source port number; when it appears as part of the to object, it matches the destination port number. The use of the port option with the to object is a mandatory requirement for the modernized rules processing logic. Example of use: “from any to any port = 80”

Port comparisons may be done in a number of forms, with a number of comparison operators, or port ranges may be specified.

port "=" | "!=" | "<" | ">" | "<=" | ">=" | "eq" | "ne" | "lt" | "gt" | "le" | "ge".

To specify port ranges, port "<>" | "><"

Warning: Following the source and destination matching parameters, the following two parameters are mandatory requirements for the modernized rules processing logic.


30.5.11.8 TCP_FLAG

Flags are only effective for TCP filtering. The letters represents one of the possible flags that can be interrogated in the TCP packet header.

The modernized rules processing logic uses the flags S parameter to identify the tcp session start request.


30.5.11.9 STATEFUL

keep state indicates that on a pass rule, any packets that match the rules selection parameters should activate the stateful filtering facility.

Note: This option is a mandatory requirement for the modernized rules processing logic.


30.5.12 Stateful Filtering

Stateful filtering treats traffic as a bi-directional exchange of packets comprising a session conversation. When activated, keep-state dynamically generates internal rules for each anticipated packet being exchanged during the bi-directional session conversation. It has the interrogation abilities to determine if the session conversation between the originating sender and the destination are following the valid procedure of bi-directional packet exchange. Any packets that do not properly fit the session conversation template are automatically rejected as impostors.

Keep state will also allow ICMP packets related to a TCP or UDP session through. So if you get ICMP type 3 code 4 in response to some web surfing allowed out by a keep state rule, they will be automatically allowed in. Any packet that IPF can be certain is part of an active session, even if it is a different protocol, will be let in.

What happens is:

Packets destined to go out the interface connected to the public Internet are first checked against the dynamic state table, if the packet matches the next expected packet comprising in a active session conversation, then it exits the firewall and the state of the session conversation flow is updated in the dynamic state table, the remaining packets get checked against the outbound rule set.

Packets coming in to the interface connected to the public Internet are first checked against the dynamic state table, if the packet matches the next expected packet comprising a active session conversation, then it exits the firewall and the state of the session conversation flow is updated in the dynamic state table, the remaining packets get checked against the inbound rule set.

When the conversation completes it is removed from the dynamic state table.

Stateful filtering allows you to focus on blocking/passing new sessions. If the new session is passed, all its subsequent packets will be allowed through automatically and any impostors automatically rejected. If a new session is blocked, none of its subsequent packets will be allowed through. Stateful filtering has technically advanced interrogation abilities capable of defending against the flood of different attack methods currently employed by attackers.


30.5.13 Inclusive Rule Set Example

The following rule set is an example of how to code a very secure inclusive type of firewall. An inclusive firewall only allows services matching pass rules through and blocks all other by default. All firewalls have at the minimum two interfaces which have to have rules to allow the firewall to function.

All UNIX flavored systems including FreeBSD are designed to use interface lo0 and IP address 127.0.0.1 for internal communication within the operating system. The firewall rules must contain rules to allow free unmolested movement of these special internally used packets.

The interface which faces the public Internet is the one where you place your rules to authorize and control access out to the public Internet and access requests arriving from the public Internet. This can be your user PPP tun0 interface or your NIC that is connected to your DSL or cable modem.

In cases where one or more NICs are cabled to private LANs behind the firewall, those interfaces must have a rule coded to allow free unmolested movement of packets originating from those LAN interfaces.

The rules should be first organized into three major sections: all the free unmolested interfaces, the public interface outbound, and the public interface inbound.

The rules in each of the public interface sections should have the most frequently matched rules placed before less commonly matched rules, with the last rule in the section blocking and logging all packets on that interface and direction.

The Outbound section in the following rule set only contains ‘pass’ rules which contain selection values that uniquely identify the service that is authorized for public Internet access. All the rules have the ‘quick’, ‘on’, ‘proto’, ‘port’, and ‘keep state’ option coded. The ‘proto tcp’ rules have the ‘flag’ option included to identify the session start request as the triggering packet to activate the stateful facility.

The Inbound section has all the blocking of undesirable packets first, for two different reasons. The first is that these things being blocked may be part of an otherwise valid packet which may be allowed in by the later authorized service rules. The second reason is that by having a rule that explicitly blocks selected packets that I receive on an infrequent basis and that I do not want to see in the log, they will not be caught by the last rule in the section which blocks and logs all packets which have fallen through the rules. The last rule in the section which blocks and logs all packets is how you create the legal evidence needed to prosecute the people who are attacking your system.

Another thing you should take note of, is there is no response returned for any of the undesirable stuff, their packets just get dropped and vanish. This way the attacker has no knowledge if his packets have reached your system. The less the attackers can learn about your system, the more time they must invest before actually doing something bad. The inbound ‘nmap OS fingerprint’ attempts rule I log the first occurrence because this is something a attacker would do.

Any time you see log messages on a rule with ‘log first’. You should do an ipfstat -hio command to see the number of times the rule has been matched so you know if you are being flooded, i.e. under attack.

When you log packets with port numbers you do not recognize, look it up in /etc/services or go to http://www.securitystats.com/tools/portsearch.php and do a port number lookup to find what the purpose of that port number is.

Check out this link for port numbers used by Trojans http://www.simovits.com/trojans/trojans.html.

The following rule set is a complete very secure ‘inclusive’ type of firewall rule set that I have used on my system. You can not go wrong using this rule set for your own. Just comment out any pass rules for services that you do not want to authorize.

If you see messages in your log that you want to stop seeing just add a block rule in the inbound section.

You have to change the dc0 interface name in every rule to the interface name of the Nic card that connects your system to the public Internet. For user PPP it would be tun0.

Add the following statements to /etc/ipf.rules:

#################################################################
# No restrictions on Inside LAN Interface for private network
# Not needed unless you have LAN
#################################################################

#pass out quick on xl0 all
#pass in quick on xl0 all

#################################################################
# No restrictions on Loopback Interface
#################################################################
pass in quick on lo0 all
pass out quick on lo0 all

#################################################################
# Interface facing Public Internet (Outbound Section)
# Interrogate session start requests originating from behind the
# firewall on the private network
# or from this gateway server destine for the public Internet.
#################################################################

# Allow out access to my ISP's Domain name server.
# xxx must be the IP address of your ISP's DNS.
# Dup these lines if your ISP has more than one DNS server
# Get the IP addresses from /etc/resolv.conf file
pass out quick on dc0 proto tcp from any to xxx port = 53 flags S keep state
pass out quick on dc0 proto udp from any to xxx port = 53 keep state

# Allow out access to my ISP's DHCP server for cable or DSL networks.
# This rule is not needed for 'user ppp' type connection to the
# public Internet, so you can delete this whole group.
# Use the following rule and check log for IP address.
# Then put IP address in commented out rule & delete first rule
pass out log quick on dc0 proto udp from any to any port = 67 keep state
#pass out quick on dc0 proto udp from any to z.z.z.z port = 67 keep state

# Allow out non-secure standard www function
pass out quick on dc0 proto tcp from any to any port = 80 flags S keep state

# Allow out secure www function https over TLS SSL
pass out quick on dc0 proto tcp from any to any port = 443 flags S keep state

# Allow out send & get email function
pass out quick on dc0 proto tcp from any to any port = 110 flags S keep state
pass out quick on dc0 proto tcp from any to any port = 25 flags S keep state

# Allow out Time
pass out quick on dc0 proto tcp from any to any port = 37 flags S keep state

# Allow out nntp news
pass out quick on dc0 proto tcp from any to any port = 119 flags S keep state

# Allow out gateway & LAN users non-secure FTP ( both passive & active modes)
# This function uses the IPNAT built in FTP proxy function coded in
# the nat rules file to make this single rule function correctly.
# If you want to use the pkg_add command to install application packages
# on your gateway system you need this rule.
pass out quick on dc0 proto tcp from any to any port = 21 flags S keep state

# Allow out secure FTP, Telnet, and SCP
# This function is using SSH (secure shell)
pass out quick on dc0 proto tcp from any to any port = 22 flags S keep state

# Allow out non-secure Telnet
pass out quick on dc0 proto tcp from any to any port = 23 flags S keep state

# Allow out FBSD CVSUP function
pass out quick on dc0 proto tcp from any to any port = 5999 flags S keep state

# Allow out ping to public Internet
pass out quick on dc0 proto icmp from any to any icmp-type 8 keep state

# Allow out whois for LAN PC to public Internet
pass out quick on dc0 proto tcp from any to any port = 43 flags S keep state

# Block and log only the first occurrence of everything
# else that's trying to get out.
# This rule enforces the block all by default logic.
block out log first quick on dc0 all

#################################################################
# Interface facing Public Internet (Inbound Section)
# Interrogate packets originating from the public Internet
# destine for this gateway server or the private network.
#################################################################

# Block all inbound traffic from non-routable or reserved address spaces
block in quick on dc0 from 192.168.0.0/16 to any    #RFC 1918 private IP
block in quick on dc0 from 172.16.0.0/12 to any     #RFC 1918 private IP
block in quick on dc0 from 10.0.0.0/8 to any        #RFC 1918 private IP
block in quick on dc0 from 127.0.0.0/8 to any       #loopback
block in quick on dc0 from 0.0.0.0/8 to any         #loopback
block in quick on dc0 from 169.254.0.0/16 to any    #DHCP auto-config
block in quick on dc0 from 192.0.2.0/24 to any      #reserved for docs
block in quick on dc0 from 204.152.64.0/23 to any   #Sun cluster interconnect
block in quick on dc0 from 224.0.0.0/3 to any       #Class D & E multicast

##### Block a bunch of different nasty things. ############
# That I do not want to see in the log

# Block frags
block in quick on dc0 all with frags

# Block short tcp packets
block in quick on dc0 proto tcp all with short

# block source routed packets
block in quick on dc0 all with opt lsrr
block in quick on dc0 all with opt ssrr

# Block nmap OS fingerprint attempts
# Log first occurrence of these so I can get their IP address
block in log first quick on dc0 proto tcp from any to any flags FUP

# Block anything with special options
block in quick on dc0 all with ipopts

# Block public pings
block in quick on dc0 proto icmp all icmp-type 8

# Block ident
block in quick on dc0 proto tcp from any to any port = 113

# Block all Netbios service. 137=name, 138=datagram, 139=session
# Netbios is MS/Windows sharing services.
# Block MS/Windows hosts2 name server requests 81
block in log first quick on dc0 proto tcp/udp from any to any port = 137
block in log first quick on dc0 proto tcp/udp from any to any port = 138
block in log first quick on dc0 proto tcp/udp from any to any port = 139
block in log first quick on dc0 proto tcp/udp from any to any port = 81

# Allow traffic in from ISP's DHCP server. This rule must contain
# the IP address of your ISP's DHCP server as it's the only
# authorized source to send this packet type. Only necessary for
# cable or DSL configurations. This rule is not needed for
# 'user ppp' type connection to the public Internet.
# This is the same IP address you captured and
# used in the outbound section.
pass in quick on dc0 proto udp from z.z.z.z to any port = 68 keep state

# Allow in standard www function because I have apache server
pass in quick on dc0 proto tcp from any to any port = 80 flags S keep state

# Allow in non-secure Telnet session from public Internet
# labeled non-secure because ID/PW passed over public Internet as clear text.
# Delete this sample group if you do not have telnet server enabled.
#pass in quick on dc0 proto tcp from any to any port = 23 flags S keep state

# Allow in secure FTP, Telnet, and SCP from public Internet
# This function is using SSH (secure shell)
pass in quick on dc0 proto tcp from any to any port = 22 flags S keep state

# Block and log only first occurrence of all remaining traffic
# coming into the firewall. The logging of only the first
# occurrence stops a .denial of service. attack targeted
# at filling up your log file space.
# This rule enforces the block all by default logic.
block in log first quick on dc0 all
################### End of rules file #####################################

30.5.14 NAT

NAT stands for Network Address Translation. To those familiar with Linux, this concept is called IP Masquerading; NAT and IP Masquerading are the same thing. One of the many things the IPF NAT function enables is the ability to have a private Local Area Network (LAN) behind the firewall sharing a single ISP assigned IP address on the public Internet.

You may ask why would someone want to do this. ISPs normally assign a dynamic IP address to their non-commercial users. Dynamic means that the IP address can be different each time you dial in and log on to your ISP, or for cable and DSL modem users when you power off and then power on your modems you can get assigned a different IP address. This IP address is how you are known to the public Internet.

Now lets say you have five PCs at home and each one needs Internet access. You would have to pay your ISP for an individual Internet account for each PC and have five phone lines.

With NAT you only need a single account with your ISP, then cable your other four PCs to a switch and the switch to the NIC in your FreeBSD system which is going to service your LAN as a gateway. NAT will automatically translate the private LAN IP address for each separate PC on the LAN to the single public IP address as it exits the firewall bound for the public Internet. It also does the reverse translation for returning packets.

NAT is most often accomplished without the approval, or knowledge, of your ISP and in most cases is grounds for your ISP terminating your account if found out. Commercial users pay a lot more for their Internet connection and usually get assigned a block of static IP address which never change. The ISP also expects and consents to their Commercial customers using NAT for their internal private LANs.

There is a special range of IP addresses reserved for NATed private LAN IP address. According to RFC 1918, you can use the following IP ranges for private nets which will never be routed directly to the public Internet:

Start IP 10.0.0.0

-

Ending IP 10.255.255.255

Start IP 172.16.0.0

-

Ending IP 172.31.255.255

Start IP 192.168.0.0

-

Ending IP 192.168.255.255


30.5.15 IPNAT

NAT rules are loaded by using the ipnat command. Typically the NAT rules are stored in /etc/ipnat.rules. See ipnat(1) for details.

When changing the NAT rules after NAT has been started, make your changes to the file containing the NAT rules, then run ipnat command with the -CF flags to delete the internal in use NAT rules and flush the contents of the translation table of all active entries.

To reload the NAT rules issue a command like this:

# ipnat -CF -f /etc/ipnat.rules

To display some statistics about your NAT, use this command:

# ipnat -s

To list the NAT table’s current mappings, use this command:

# ipnat -l

To turn verbose mode on, and display information relating to rule processing and active rules/table entries:

# ipnat -v

30.5.16 IPNAT Rules

NAT rules are very flexible and can accomplish many different things to fit the needs of commercial and home users.

The rule syntax presented here has been simplified to what is most commonly used in a non-commercial environment. For a complete rule syntax description see the ipnat(5) manual page.

The syntax for a NAT rule looks something like this:

map IF LAN_IP_RANGE -> PUBLIC_ADDRESS

The keyword map starts the rule.

Replace IF with the external interface.

The LAN_IP_RANGE is what your internal clients use for IP Addressing, usually this is something like 192.168.1.0/24.

The PUBLIC_ADDRESS can either be the external IP address or the special keyword 0/32, which means to use the IP address assigned to IF.


30.5.17 How NAT works

A packet arrives at the firewall from the LAN with a public destination. It passes through the outbound filter rules, NAT gets his turn at the packet and applies its rules top down, first matching rule wins. NAT tests each of its rules against the packets interface name and source IP address. When a packets interface name matches a NAT rule then the [source IP address, i.e. private LAN IP address] of the packet is checked to see if it falls within the IP address range specified to the left of the arrow symbol on the NAT rule. On a match the packet has its source IP address rewritten with the public IP address obtained by the 0/32 keyword. NAT posts a entry in its internal NAT table so when the packet returns from the public Internet it can be mapped back to its original private IP address and then passed to the filter rules for processing.


30.5.18 Enabling IPNAT

To enable IPNAT add these statements to /etc/rc.conf.

To enable your machine to route traffic between interfaces:

gateway_enable="YES"

To start IPNAT automatically each time:

ipnat_enable="YES"

To specify where to load the IPNAT rules from:

ipnat_rules="/etc/ipnat.rules"

30.5.19 NAT for a very large LAN

For networks that have large numbers of PC’s on the LAN or networks with more than a single LAN, the process of funneling all those private IP addresses into a single public IP address becomes a resource problem that may cause problems with the same port numbers being used many times across many NATed LAN PC’s, causing collisions. There are two ways to relieve this resource problem.


30.5.19.1 Assigning Ports to Use

A normal NAT rule would look like:

map dc0 192.168.1.0/24 -> 0/32

In the above rule the packet’s source port is unchanged as the packet passes through IPNAT. By adding the portmap keyword you can tell IPNAT to only use source ports in a range. For example the following rule will tell IPNAT to modify the source port to be within that range:

map dc0 192.168.1.0/24 -> 0/32 portmap tcp/udp 20000:60000

Additionally we can make things even easier by using the auto keyword to tell IPNAT to determine by itself which ports are available to use:

map dc0 192.168.1.0/24 -> 0/32 portmap tcp/udp auto

30.5.19.2 Using a pool of public addresses

In very large LANs there comes a point where there are just too many LAN addresses to fit into a single public address. If a block of public IP addresses is available, you can use these addresses as a “pool”, and let IPNAT pick one of the public IP addresses as packet-addresses are mapped on their way out.

For example, instead of mapping all packets through a single public IP address, as in:

map dc0 192.168.1.0/24 -> 204.134.75.1

A range of public IP addresses can be specified either with a netmask:

map dc0 192.168.1.0/24 -> 204.134.75.0/255.255.255.0

or using CIDR notation:

map dc0 192.168.1.0/24 -> 204.134.75.0/24

30.5.20 Port Redirection

A very common practice is to have a web server, email server, database server and DNS server each segregated to a different PC on the LAN. In this case the traffic from these servers still have to be NATed, but there has to be some way to direct the inbound traffic to the correct LAN PCs. IPNAT has the redirection facilities of NAT to solve this problem. Lets say you have your web server on LAN address 10.0.10.25 and your single public IP address is 20.20.20.5 you would code the rule like this:

rdr dc0 20.20.20.5/32 port 80 -> 10.0.10.25 port 80

or:

rdr dc0 0.0.0.0/0 port 80 -> 10.0.10.25 port 80

or for a LAN DNS Server on LAN address of 10.0.10.33 that needs to receive public DNS requests:

rdr dc0 20.20.20.5/32 port 53 -> 10.0.10.33 port 53 udp

30.5.21 FTP and NAT

FTP is a dinosaur left over from the time before the Internet as it is known today, when research universities were leased lined together and FTP was used to share files among research Scientists. This was a time when data security was not a consideration. Over the years the FTP protocol became buried into the backbone of the emerging Internet and its username and password being sent in clear text was never changed to address new security concerns. FTP has two flavors, it can run in active mode or passive mode. The difference is in how the data channel is acquired. Passive mode is more secure as the data channel is acquired be the ordinal ftp session requester. For a real good explanation of FTP and the different modes see http://www.slacksite.com/other/ftp.html.


30.5.21.1 IPNAT Rules

IPNAT has a special built in FTP proxy option which can be specified on the NAT map rule. It can monitor all outbound packet traffic for FTP active or passive start session requests and dynamically create temporary filter rules containing only the port number really in use for the data channel. This eliminates the security risk FTP normally exposes the firewall to from having large ranges of high order port numbers open.

This rule will handle all the traffic for the internal LAN:

map dc0 10.0.10.0/29 -> 0/32 proxy port 21 ftp/tcp

This rule handles the FTP traffic from the gateway:

map dc0 0.0.0.0/0 -> 0/32 proxy port 21 ftp/tcp

This rule handles all non-FTP traffic from the internal LAN:

map dc0 10.0.10.0/29 -> 0/32

The FTP map rule goes before our regular map rule. All packets are tested against the first rule from the top. Matches on interface name, then private LAN source IP address, and then is it a FTP packet. If all that matches then the special FTP proxy creates temp filter rules to let the FTP session packets pass in and out, in addition to also NATing the FTP packets. All LAN packets that are not FTP do not match the first rule and fall through to the third rule and are tested, matching on interface and source IP, then are NATed.


30.5.21.2 IPNAT FTP Filter Rules

Only one filter rule is needed for FTP if the NAT FTP proxy is used.

Without the FTP Proxy you will need the following three rules:

# Allow out LAN PC client FTP to public Internet
# Active and passive modes
pass out quick on rl0 proto tcp from any to any port = 21 flags S keep state

# Allow out passive mode data channel high order port numbers
pass out quick on rl0 proto tcp from any to any port > 1024 flags S keep state

# Active mode let data channel in from FTP server
pass in quick on rl0 proto tcp from any to any port = 20 flags S keep state

30.6 IPFW

The IPFIREWALL (IPFW) is a FreeBSD sponsored firewall software application authored and maintained by FreeBSD volunteer staff members. It uses the legacy stateless rules and a legacy rule coding technique to achieve what is referred to as Simple Stateful logic.

The IPFW sample rule set (found in /etc/rc.firewall and /etc/rc.firewall6) in the standard FreeBSD install is rather simple and it is not expected that it used directly without modifications. The example does not use stateful filtering, which is beneficial in most setups, so it will not be used as base for this section.

The IPFW stateless rule syntax is empowered with technically sophisticated selection capabilities which far surpasses the knowledge level of the customary firewall installer. IPFW is targeted at the professional user or the advanced technical computer hobbyist who have advanced packet selection requirements. A high degree of detailed knowledge into how different protocols use and create their unique packet header information is necessary before the power of the IPFW rules can be unleashed. Providing that level of explanation is out of the scope of this section of the handbook.

IPFW is composed of seven components, the primary component is the kernel firewall filter rule processor and its integrated packet accounting facility, the logging facility, the ‘divert’ rule which triggers the NAT facility, and the advanced special purpose facilities, the dummynet traffic shaper facilities, the ‘fwd rule’ forward facility, the bridge facility, and the ipstealth facility. IPFW supports both IPv4 and IPv6.


30.6.1 Enabling IPFW

IPFW is included in the basic FreeBSD install as a separate run time loadable module. The system will dynamically load the kernel module when the rc.conf statement firewall_enable="YES" is used. You do not need to compile IPFW into the FreeBSD kernel unless you want NAT function enabled.

After rebooting your system with firewall_enable="YES" in rc.conf the following white highlighted message is displayed on the screen as part of the boot process:

ipfw2 initialized, divert disabled, rule-based forwarding disabled, default to deny, logging disabled

The loadable module does have logging ability compiled in. To enable logging and set the verbose logging limit, there is a knob you can set in /etc/sysctl.conf by adding these statements, logging will be enabled on future reboots:

net.inet.ip.fw.verbose=1
net.inet.ip.fw.verbose_limit=5

30.6.2 Kernel Options

It is not a mandatory requirement that you enable IPFW by compiling the following options into the FreeBSD kernel unless you need NAT function. It is presented here as background information.

options    IPFIREWALL

This option enables IPFW as part of the kernel

options    IPFIREWALL_VERBOSE

Enables logging of packets that pass through IPFW and have the ‘log’ keyword specified in the rule set.

options    IPFIREWALL_VERBOSE_LIMIT=5

Limits the number of packets logged through syslogd(8) on a per entry basis. You may wish to use this option in hostile environments which you want to log firewall activity. This will close a possible denial of service attack via syslog flooding.

options    IPFIREWALL_DEFAULT_TO_ACCEPT

This option will allow everything to pass through the firewall by default, which is a good idea when you are first setting up your firewall.

options    IPDIVERT

This enables the use of NAT functionality.

Note: If you do not include IPFIREWALL_DEFAULT_TO_ACCEPT or set your rules to allow incoming packets you will block all packets going to and from this machine.


30.6.3 /etc/rc.conf Options

Enable the firewall:

firewall_enable="YES"

To select one of the default firewall types provided by FreeBSD, select one by reading the /etc/rc.firewall file and place it in the following:

firewall_type="open"

Available values for this setting are:

  • open — pass all traffic.

  • client — will protect only this machine.

  • simple — protect the whole network.

  • closed — entirely disables IP traffic except for the loopback interface.

  • UNKNOWN — disables the loading of firewall rules.

  • filename — absolute path of file containing firewall rules.

It is possible to use two different ways to load custom rules for ipfw firewall. One is by setting firewall_type variable to absolute path of file, which contains firewall rules without any command-line options for ipfw(8) itself. A simple example of ruleset file can be following:

add block in  all
add block out all

On the other hand, it is possible to set firewall_script variable to absolute path of executable script that includes ipfw commands being executed at system boot time. A valid ruleset script that would be equivalent to the ruleset file shown above would be following:

#!/bin/sh

ipfw -q flush

ipfw add block in  all
ipfw add block out all

Note: If firewall_type is set to either client or simple, the default rules found in /etc/rc.firewall should be reviewed to fit to the configuration of the given machine. Also note that the examples used in this chapter expect that the firewall_script is set to /etc/ipfw.rules.

Enable logging:

firewall_logging="YES"

Warning: The only thing that the firewall_logging variable will do is setting the net.inet.ip.fw.verbose sysctl variable to the value of 1 (see Section 30.6.1). There is no rc.conf variable to set log limitations, but it can be set via sysctl variable, manually or from the /etc/sysctl.conf file:

net.inet.ip.fw.verbose_limit=5

If your machine is acting as a gateway, i.e. providing Network Address Translation (NAT) via natd(8), please refer to Section 31.9 for information regarding the required /etc/rc.conf options.


30.6.4 The IPFW Command

The ipfw command is the normal vehicle for making manual single rule additions or deletions to the firewall active internal rules while it is running. The problem with using this method is once your system is shutdown or halted all the rules you added or changed or deleted are lost. Writing all your rules in a file and using that file to load the rules at boot time, or to replace in mass the currently running firewall rules with changes you made to the files content is the recommended method used here.

The ipfw command is still a very useful to display the running firewall rules to the console screen. The IPFW accounting facility dynamically creates a counter for each rule that counts each packet that matches the rule. During the process of testing a rule, listing the rule with its counter is the one of the ways of determining if the rule is functioning.

To list all the rules in sequence:

# ipfw list

To list all the rules with a time stamp of when the last time the rule was matched:

# ipfw -t list

To list the accounting information, packet count for matched rules along with the rules themselves. The first column is the rule number, followed by the number of outgoing matched packets, followed by the number of incoming matched packets, and then the rule itself.

# ipfw -a list

List the dynamic rules in addition to the static rules:

# ipfw -d list

Also show the expired dynamic rules:

# ipfw -d -e list

Zero the counters:

# ipfw zero

Zero the counters for just rule NUM:

# ipfw zero NUM

30.6.5 IPFW Rule Sets

A rule set is a group of ipfw rules coded to allow or deny packets based on the values contained in the packet. The bi-directional exchange of packets between hosts comprises a session conversation. The firewall rule set processes the packet twice: once on its arrival from the public Internet host and again as it leaves for its return trip back to the public Internet host. Each tcp/ip service (i.e. telnet, www, mail, etc.) is predefined by its protocol, and port number. This is the basic selection criteria used to create rules which will allow or deny services.

When a packet enters the firewall it is compared against the first rule in the rule set and progress one rule at a time moving from top to bottom of the set in ascending rule number sequence order. When the packet matches a rule selection parameters, the rules action field value is executed and the search of the rule set terminates for that packet. This is referred to as “the first match wins” search method. If the packet does not match any of the rules, it gets caught by the mandatory ipfw default rule, number 65535 which denies all packets and discards them without any reply back to the originating destination.

Note: The search continues after count, skipto and tee rules.

The instructions contained here are based on using rules that contain the stateful ‘keep state’, ‘limit’, ‘in’/'out’, and via options. This is the basic framework for coding an inclusive type firewall rule set.

An inclusive firewall only allows services matching the rules through. This way you can control what services can originate behind the firewall destine for the public Internet and also control the services which can originate from the public Internet accessing your private network. Everything else is denied by default design. Inclusive firewalls are much, much more secure than exclusive firewall rule sets and is the only rule set type covered here in.

Warning: When working with the firewall rules be careful, you can end up locking your self out.


30.6.5.1 Rule Syntax

The rule syntax presented here has been simplified to what is necessary to create a standard inclusive type firewall rule set. For a complete rule syntax description see the ipfw(8) manual page.

Rules contain keywords: these keywords have to be coded in a specific order from left to right on the line. Keywords are identified in bold type. Some keywords have sub-options which may be keywords them selves and also include more sub-options.

# is used to mark the start of a comment and may appear at the end of a rule line or on its own lines. Blank lines are ignored.

CMD RULE_NUMBER ACTION LOGGING SELECTION STATEFUL


30.6.5.1.1 CMD

Each new rule has to be prefixed with add to add the rule to the internal table.


30.6.5.1.2 RULE_NUMBER

Each rule has to have a rule number to go with it.


30.6.5.1.3 ACTION

A rule can be associated with one of the following actions, which will be executed when the packet matches the selection criterion of the rule.

allow | accept | pass | permit

These all mean the same thing which is to allow packets that match the rule to exit the firewall rule processing. The search terminates at this rule.

check-state

Checks the packet against the dynamic rules table. If a match is found, execute the action associated with the rule which generated this dynamic rule, otherwise move to the next rule. The check-state rule does not have selection criterion. If no check-state rule is present in the rule set, the dynamic rules table is checked at the first keep-state or limit rule.

deny | drop

Both words mean the same thing which is to discard packets that match this rule. The search terminates.


30.6.5.1.4 Logging

log or logamount

When a packet matches a rule with the log keyword, a message will be logged to syslogd with a facility name of SECURITY. The logging only occurs if the number of packets logged so far for that particular rule does not exceed the logamount parameter. If no logamount is specified, the limit is taken from the sysctl variable net.inet.ip.fw.verbose_limit. In both cases, a value of zero removes the logging limit. Once the limit is reached, logging can be re-enabled by clearing the logging counter or the packet counter for that rule, see the ipfw reset log command.

Note: Logging is done after all other packet matching conditions have been successfully verified, and before performing the final action (accept, deny) on the packet. It is up to you to decide which rules you want to enable logging on.


30.6.5.1.5 Selection

The keywords described in this section are used to describe attributes of the packet to be interrogated when determining whether rules match the packet or not. The following general-purpose attributes are provided for matching, and must be used in this order:

udp | tcp | icmp

or any protocol names found in /etc/protocols are recognized and may be used. The value specified is protocol to be matched against. This is a mandatory requirement.

from src to dst

The from and to keywords are used to match against IP addresses. Rules must specify BOTH source and destination parameters. any is a special keyword that matches any IP address. me is a special keyword that matches any IP address configured on an interface in your FreeBSD system to represent the PC the firewall is running on (i.e. this box) as in ‘from me to any’ or ‘from any to me’ or ‘from 0.0.0.0/0 to any’ or ‘from any to 0.0.0.0/0′ or ‘from 0.0.0.0 to any’ or ‘from any to 0.0.0.0′ or ‘from me to 0.0.0.0′. IP addresses are specified as a dotted IP address numeric form/mask-length, or as single dotted IP address numeric form. This is a mandatory requirement. See this link for help on writing mask-lengths. http://jodies.de/ipcalc

port number

For protocols which support port numbers (such as TCP and UDP). It is mandatory that you code the port number of the service you want to match on. Service names (from /etc/services) may be used instead of numeric port values.

in | out

Matches incoming or outgoing packets, respectively. The in and out are keywords and it is mandatory that you code one or the other as part of your rule matching criterion.

via IF

Matches packets going through the interface specified by exact name. The via keyword causes the interface to always be checked as part of the match process.

setup

This is a mandatory keyword that identifies the session start request for TCP packets.

keep-state

This is a mandatory keyword. Upon a match, the firewall will create a dynamic rule, whose default behavior is to match bidirectional traffic between source and destination IP/port using the same protocol.

limit {src-addr | src-port | dst-addr | dst-port}

The firewall will only allow N connections with the same set of parameters as specified in the rule. One or more of source and destination addresses and ports can be specified. The ‘limit’ and ‘keep-state’ can not be used on same rule. Limit provides the same stateful function as ‘keep-state’ plus its own functions.


30.6.5.2 Stateful Rule Option

Stateful filtering treats traffic as a bi-directional exchange of packets comprising a session conversation. It has the interrogation abilities to determine if the session conversation between the originating sender and the destination are following the valid procedure of bi-directional packet exchange. Any packets that do not properly fit the session conversation template are automatically rejected as impostors.

‘check-state’ is used to identify where in the IPFW rules set the packet is to be tested against the dynamic rules facility. On a match the packet exits the firewall to continue on its way and a new rule is dynamic created for the next anticipated packet being exchanged during this bi-directional session conversation. On a no match the packet advances to the next rule in the rule set for testing.

The dynamic rules facility is vulnerable to resource depletion from a SYN-flood attack which would open a huge number of dynamic rules. To counter this attack, FreeBSD added another new option named limit. This option is used to limit the number of simultaneous session conversations by interrogating the rules source or destinations fields as directed by the limit option and using the packet’s IP address found there, in a search of the open dynamic rules counting the number of times this rule and IP address combination occurred, if this count is greater that the value specified on the limit option, the packet is discarded.


30.6.5.3 Logging Firewall Messages

The benefits of logging are obvious: it provides the ability to review after the fact the rules you activated logging on which provides information like, what packets had been dropped, what addresses they came from, where they were going, giving you a significant edge in tracking down attackers.

Even with the logging facility enabled, IPFW will not generate any rule logging on it’s own. The firewall administrator decides what rules in the rule set he wants to log and adds the log verb to those rules. Normally only deny rules are logged, like the deny rule for incoming ICMP pings. It is very customary to duplicate the ipfw default deny everything rule with the log verb included as your last rule in the rule set. This way you get to see all the packets that did not match any of the rules in the rule set.

Logging is a two edged sword, if you are not careful, you can lose yourself in the over abundance of log data and fill your disk up with growing log files. DoS attacks that fill up disk drives is one of the oldest attacks around. These log message are not only written to syslogd, but also are displayed on the root console screen and soon become very annoying.

The IPFIREWALL_VERBOSE_LIMIT=5 kernel option limits the number of consecutive messages sent to the system logger syslogd, concerning the packet matching of a given rule. When this option is enabled in the kernel, the number of consecutive messages concerning a particular rule is capped at the number specified. There is nothing to be gained from 200 log messages saying the same identical thing. For instance, five consecutive messages concerning a particular rule would be logged to syslogd, the remainder identical consecutive messages would be counted and posted to the syslogd with a phrase like this:

last message repeated 45 times

All logged packets messages are written by default to /var/log/security file, which is defined in the /etc/syslog.conf file.


30.6.5.4 Building a Rule Script

Most experienced IPFW users create a file containing the rules and code them in a manner compatible with running them as a script. The major benefit of doing this is the firewall rules can be refreshed in mass without the need of rebooting the system to activate the new rules. This method is very convenient in testing new rules as the procedure can be executed as many times as needed. Being a script, you can use symbolic substitution to code frequent used values and substitution them in multiple rules. You will see this in the following example.

The script syntax used here is compatible with the ’sh’, ‘csh’, ‘tcsh’ shells. Symbolic substitution fields are prefixed with a dollar sign $. Symbolic fields do not have the $ prefix. The value to populate the Symbolic field must be enclosed to "double quotes".

Start your rules file like this:

############### start of example ipfw rules script #############
#
ipfw -q -f flush       # Delete all rules
# Set defaults
oif="tun0"             # out interface
odns="192.0.2.11"      # ISP's DNS server IP address
cmd="ipfw -q add "     # build rule prefix
ks="keep-state"        # just too lazy to key this each time
$cmd 00500 check-state
$cmd 00502 deny all from any to any frag
$cmd 00501 deny tcp from any to any established
$cmd 00600 allow tcp from any to any 80 out via $oif setup $ks
$cmd 00610 allow tcp from any to $odns 53 out via $oif setup $ks
$cmd 00611 allow udp from any to $odns 53 out via $oif $ks
################### End of example ipfw rules script ############

That is all there is to it. The rules are not important in this example, how the Symbolic substitution field are populated and used are.

If the above example was in /etc/ipfw.rules file, you could reload these rules by entering on the command line.

# sh /etc/ipfw.rules

The /etc/ipfw.rules file could be located anywhere you want and the file could be named any thing you would like.

The same thing could also be accomplished by running these commands by hand:

# ipfw -q -f flush
# ipfw -q add check-state
# ipfw -q add deny all from any to any frag
# ipfw -q add deny tcp from any to any established
# ipfw -q add allow tcp from any to any 80 out via tun0 setup keep-state
# ipfw -q add allow tcp from any to 192.0.2.11 53 out via tun0 setup keep-state
# ipfw -q add 00611 allow udp from any to 192.0.2.11 53 out via tun0 keep-state

30.6.5.5 Stateful Ruleset

The following non-NATed rule set is an example of how to code a very secure ‘inclusive’ type of firewall. An inclusive firewall only allows services matching pass rules through and blocks all other by default. All firewalls have at the minimum two interfaces which have to have rules to allow the firewall to function.

All UNIX flavored operating systems, FreeBSD included, are designed to use interface lo0 and IP address 127.0.0.1 for internal communication with in the operating system. The firewall rules must contain rules to allow free unmolested movement of these special internally used packets.

The interface which faces the public Internet, is the one which you code your rules to authorize and control access out to the public Internet and access requests arriving from the public Internet. This can be your ppp tun0 interface or your NIC that is connected to your DSL or cable modem.

In cases where one or more than one NIC are connected to a private LANs behind the firewall, those interfaces must have rules coded to allow free unmolested movement of packets originating from those LAN interfaces.

The rules should be first organized into three major sections, all the free unmolested interfaces, public interface outbound, and the public interface inbound.

The order of the rules in each of the public interface sections should be in order of the most used rules being placed before less often used rules with the last rule in the section being a block log all packets on that interface and direction.

The Outbound section in the following rule set only contains ‘allow’ rules which contain selection values that uniquely identify the service that is authorized for public Internet access. All the rules have the, proto, port, in/out, via and keep state option coded. The ‘proto tcp’ rules have the ’setup’ option included to identify the start session request as the trigger packet to be posted to the keep state stateful table.

The Inbound section has all the blocking of undesirable packets first for two different reasons. First is these things being blocked may be part of an otherwise valid packet which may be allowed in by the later authorized service rules. Second reason is that by having a rule that explicitly blocks selected packets that I receive on an infrequent bases and do not want to see in the log, this keeps them from being caught by the last rule in the section which blocks and logs all packets which have fallen through the rules. The last rule in the section which blocks and logs all packets is how you create the legal evidence needed to prosecute the people who are attacking your system.

Another thing you should take note of, is there is no response returned for any of the undesirable stuff, their packets just get dropped and vanish. This way the attackers has no knowledge if his packets have reached your system. The less the attackers can learn about your system the more secure it is. When you log packets with port numbers you do not recognize, look the numbers up in /etc/services/ or go to http://www.securitystats.com/tools/portsearch.php and do a port number lookup to find what the purpose of that port number is. Check out this link for port numbers used by Trojans: http://www.simovits.com/trojans/trojans.html.


30.6.5.6 An Example Inclusive Ruleset

The following non-NATed rule set is a complete inclusive type ruleset. You can not go wrong using this rule set for you own. Just comment out any pass rules for services you do not want. If you see messages in your log that you want to stop seeing just add a deny rule in the inbound section. You have to change the ‘dc0′ interface name in every rule to the interface name of the NIC that connects your system to the public Internet. For user ppp it would be ‘tun0′.

You will see a pattern in the usage of these rules.

  • All statements that are a request to start a session to the public Internet use keep-state.

  • All the authorized services that originate from the public Internet have the limit option to stop flooding.

  • All rules use in or out to clarify direction.

  • All rules use via interface name to specify the interface the packet is traveling over.

The following rules go into /etc/ipfw.rules.

################ Start of IPFW rules file ###############################
# Flush out the list before we begin.
ipfw -q -f flush

# Set rules command prefix
cmd="ipfw -q add"
pif="dc0"     # public interface name of NIC
              # facing the public Internet

#################################################################
# No restrictions on Inside LAN Interface for private network
# Not needed unless you have LAN.
# Change xl0 to your LAN NIC interface name
#################################################################
#$cmd 00005 allow all from any to any via xl0

#################################################################
# No restrictions on Loopback Interface
#################################################################
$cmd 00010 allow all from any to any via lo0

#################################################################
# Allow the packet through if it has previous been added to the
# the "dynamic" rules table by a allow keep-state statement.
#################################################################
$cmd 00015 check-state

#################################################################
# Interface facing Public Internet (Outbound Section)
# Interrogate session start requests originating from behind the
# firewall on the private network or from this gateway server
# destine for the public Internet.
#################################################################

# Allow out access to my ISP's Domain name server.
# x.x.x.x must be the IP address of your ISP.s DNS
# Dup these lines if your ISP has more than one DNS server
# Get the IP addresses from /etc/resolv.conf file
$cmd 00110 allow tcp from any to x.x.x.x 53 out via $pif setup keep-state
$cmd 00111 allow udp from any to x.x.x.x 53 out via $pif keep-state

# Allow out access to my ISP's DHCP server for cable/DSL configurations.
# This rule is not needed for .user ppp. connection to the public Internet.
# so you can delete this whole group.
# Use the following rule and check log for IP address.
# Then put IP address in commented out rule & delete first rule
$cmd 00120 allow log udp from any to any 67 out via $pif keep-state
#$cmd 00120 allow udp from any to x.x.x.x 67 out via $pif keep-state

# Allow out non-secure standard www function
$cmd 00200 allow tcp from any to any 80 out via $pif setup keep-state

# Allow out secure www function https over TLS SSL
$cmd 00220 allow tcp from any to any 443 out via $pif setup keep-state

# Allow out send & get email function
$cmd 00230 allow tcp from any to any 25 out via $pif setup keep-state
$cmd 00231 allow tcp from any to any 110 out via $pif setup keep-state

# Allow out FBSD (make install & CVSUP) functions
# Basically give user root "GOD" privileges.
$cmd 00240 allow tcp from me to any out via $pif setup keep-state uid root

# Allow out ping
$cmd 00250 allow icmp from any to any out via $pif keep-state

# Allow out Time
$cmd 00260 allow tcp from any to any 37 out via $pif setup keep-state

# Allow out nntp news (i.e. news groups)
$cmd 00270 allow tcp from any to any 119 out via $pif setup keep-state

# Allow out secure FTP, Telnet, and SCP
# This function is using SSH (secure shell)
$cmd 00280 allow tcp from any to any 22 out via $pif setup keep-state

# Allow out whois
$cmd 00290 allow tcp from any to any 43 out via $pif setup keep-state

# deny and log everything else that.s trying to get out.
# This rule enforces the block all by default logic.
$cmd 00299 deny log all from any to any out via $pif

#################################################################
# Interface facing Public Internet (Inbound Section)
# Interrogate packets originating from the public Internet
# destine for this gateway server or the private network.
#################################################################

# Deny all inbound traffic from non-routable reserved address spaces
$cmd 00300 deny all from 192.168.0.0/16 to any in via $pif  #RFC 1918 private IP
$cmd 00301 deny all from 172.16.0.0/12 to any in via $pif     #RFC 1918 private IP
$cmd 00302 deny all from 10.0.0.0/8 to any in via $pif          #RFC 1918 private IP
$cmd 00303 deny all from 127.0.0.0/8 to any in via $pif        #loopback
$cmd 00304 deny all from 0.0.0.0/8 to any in via $pif            #loopback
$cmd 00305 deny all from 169.254.0.0/16 to any in via $pif   #DHCP auto-config
$cmd 00306 deny all from 192.0.2.0/24 to any in via $pif       #reserved for docs
$cmd 00307 deny all from 204.152.64.0/23 to any in via $pif  #Sun cluster interconnect
$cmd 00308 deny all from 224.0.0.0/3 to any in via $pif         #Class D & E multicast

# Deny public pings
$cmd 00310 deny icmp from any to any in via $pif

# Deny ident
$cmd 00315 deny tcp from any to any 113 in via $pif

# Deny all Netbios service. 137=name, 138=datagram, 139=session
# Netbios is MS/Windows sharing services.
# Block MS/Windows hosts2 name server requests 81
$cmd 00320 deny tcp from any to any 137 in via $pif
$cmd 00321 deny tcp from any to any 138 in via $pif
$cmd 00322 deny tcp from any to any 139 in via $pif
$cmd 00323 deny tcp from any to any 81 in via $pif

# Deny any late arriving packets
$cmd 00330 deny all from any to any frag in via $pif

# Deny ACK packets that did not match the dynamic rule table
$cmd 00332 deny tcp from any to any established in via $pif

# Allow traffic in from ISP's DHCP server. This rule must contain
# the IP address of your ISP.s DHCP server as it.s the only
# authorized source to send this packet type.
# Only necessary for cable or DSL configurations.
# This rule is not needed for .user ppp. type connection to
# the public Internet. This is the same IP address you captured
# and used in the outbound section.
#$cmd 00360 allow udp from any to x.x.x.x 67 in via $pif keep-state

# Allow in standard www function because I have apache server
$cmd 00400 allow tcp from any to me 80 in via $pif setup limit src-addr 2

# Allow in secure FTP, Telnet, and SCP from public Internet
$cmd 00410 allow tcp from any to me 22 in via $pif setup limit src-addr 2

# Allow in non-secure Telnet session from public Internet
# labeled non-secure because ID & PW are passed over public
# Internet as clear text.
# Delete this sample group if you do not have telnet server enabled.
$cmd 00420 allow tcp from any to me 23 in via $pif setup limit src-addr 2

# Reject & Log all incoming connections from the outside
$cmd 00499 deny log all from any to any in via $pif

# Everything else is denied by default
# deny and log all packets that fell through to see what they are
$cmd 00999 deny log all from any to any
################ End of IPFW rules file ###############################

30.6.5.7 An Example NAT and Stateful Ruleset

There are some additional configuration statements that need to be enabled to activate the NAT function of IPFW. The kernel source needs ‘option IPDIVERT’ statement added to the other IPFIREWALL statements compiled into a custom kernel.

In addition to the normal IPFW options in /etc/rc.conf, the following are needed.

natd_enable="YES"                   # Enable NATD function
natd_interface="rl0"                # interface name of public Internet NIC
natd_flags="-dynamic -m"            # -m = preserve port numbers if possible

Utilizing stateful rules with divert natd rule (Network Address Translation) greatly complicates the rule set coding logic. The positioning of the check-state, and ‘divert natd’ rules in the rule set becomes very critical. This is no longer a simple fall-through logic flow. A new action type is used, called ’skipto’. To use the skipto command it is mandatory that you number each rule so you know exactly where the skipto rule number is you are really jumping to.

The following is an uncommented example of one coding method, selected here to explain the sequence of the packet flow through the rule sets.

The processing flow starts with the first rule from the top of the rule file and progress one rule at a time deeper into the file until the end is reach or the packet being tested to the selection criteria matches and the packet is released out of the firewall. It is important to take notice of the location of rule numbers 100 101, 450, 500, and 510. These rules control the translation of the outbound and inbound packets so their entries in the keep-state dynamic table always register the private LAN IP address. Next notice that all the allow and deny rules specified the direction the packet is going (IE outbound or inbound) and the interface. Also notice that all the start outbound session requests all skipto rule 500 for the network address translation.

Lets say a LAN user uses their web browser to get a web page. Web pages use port 80 to communicate over. So the packet enters the firewall, It does not match 100 because it is headed out not in. It passes rule 101 because this is the first packet so it has not been posted to the keep-state dynamic table yet. The packet finally comes to rule 125 a matches. It is outbound through the NIC facing the public Internet. The packet still has it’s source IP address as a private LAN IP address. On the match to this rule, two actions take place. The keep-state option will post this rule into the keep-state dynamic rules table and the specified action is executed. The action is part of the info posted to the dynamic table. In this case it is "skipto rule 500". Rule 500 NATs the packet IP address and out it goes. Remember this, this is very important. This packet makes its way to the destination and returns and enters the top of the rule set. This time it does match rule 100 and has it destination IP address mapped back to its corresponding LAN IP address. It then is processed by the check-state rule, it’s found in the table as an existing session conversation and released to the LAN. It goes to the LAN PC that sent it and a new packet is sent requesting another segment of the data from the remote server. This time it gets checked by the check-state rule and its outbound entry is found, the associated action, ’skipto 500′, is executed. The packet jumps to rule 500 gets NATed and released on it’s way out.

On the inbound side, everything coming in that is part of an existing session conversation is being automatically handled by the check-state rule and the properly placed divert natd rules. All we have to address is denying all the bad packets and only allowing in the authorized services. Lets say there is a apache server running on the firewall box and we want people on the public Internet to be able to access the local web site. The new inbound start request packet matches rule 100 and its IP address is mapped to LAN IP for the firewall box. The packet is them matched against all the nasty things we want to check for and finally matches against rule 425. On a match two things occur. The packet rule is posted to the keep-state dynamic table but this time any new session requests originating from that source IP address is limited to 2. This defends against DoS attacks of service running on the specified port number. The action is allow so the packet is released to the LAN. On return the check-state rule recognizes the packet as belonging to an existing session conversation sends it to rule 500 for NATing and released to outbound interface.

Example Ruleset #1:

#!/bin/sh
cmd="ipfw -q add"
skip="skipto 500"
pif=rl0
ks="keep-state"
good_tcpo="22,25,37,43,53,80,443,110,119"

ipfw -q -f flush

$cmd 002 allow all from any to any via xl0  # exclude LAN traffic
$cmd 003 allow all from any to any via lo0  # exclude loopback traffic

$cmd 100 divert natd ip from any to any in via $pif
$cmd 101 check-state

# Authorized outbound packets
$cmd 120 $skip udp from any to xx.168.240.2 53 out via $pif $ks
$cmd 121 $skip udp from any to xx.168.240.5 53 out via $pif $ks
$cmd 125 $skip tcp from any to any $good_tcpo out via $pif setup $ks
$cmd 130 $skip icmp from any to any out via $pif $ks
$cmd 135 $skip udp from any to any 123 out via $pif $ks

# Deny all inbound traffic from non-routable reserved address spaces
$cmd 300 deny all from 192.168.0.0/16  to any in via $pif  #RFC 1918 private IP
$cmd 301 deny all from 172.16.0.0/12   to any in via $pif  #RFC 1918 private IP
$cmd 302 deny all from 10.0.0.0/8      to any in via $pif  #RFC 1918 private IP
$cmd 303 deny all from 127.0.0.0/8     to any in via $pif  #loopback
$cmd 304 deny all from 0.0.0.0/8       to any in via $pif  #loopback
$cmd 305 deny all from 169.254.0.0/16  to any in via $pif  #DHCP auto-config
$cmd 306 deny all from 192.0.2.0/24    to any in via $pif  #reserved for docs
$cmd 307 deny all from 204.152.64.0/23 to any in via $pif  #Sun cluster
$cmd 308 deny all from 224.0.0.0/3     to any in via $pif  #Class D & E multicast

# Authorized inbound packets
$cmd 400 allow udp from xx.70.207.54 to any 68 in $ks
$cmd 420 allow tcp from any to me 80 in via $pif setup limit src-addr 1

$cmd 450 deny log ip from any to any

# This is skipto location for outbound stateful rules
$cmd 500 divert natd ip from any to any out via $pif
$cmd 510 allow ip from any to any

######################## end of rules  ##################

The following is pretty much the same as above, but uses a self documenting coding style full of description comments to help the inexperienced IPFW rule writer to better understand what the rules are doing.

Example Ruleset #2:

#!/bin/sh
################ Start of IPFW rules file ###############################
# Flush out the list before we begin.
ipfw -q -f flush

# Set rules command prefix
cmd="ipfw -q add"
skip="skipto 800"
pif="rl0"     # public interface name of NIC
              # facing the public Internet

#################################################################
# No restrictions on Inside LAN Interface for private network
# Change xl0 to your LAN NIC interface name
#################################################################
$cmd 005 allow all from any to any via xl0

#################################################################
# No restrictions on Loopback Interface
#################################################################
$cmd 010 allow all from any to any via lo0

#################################################################
# check if packet is inbound and nat address if it is
#################################################################
$cmd 014 divert natd ip from any to any in via $pif

#################################################################
# Allow the packet through if it has previous been added to the
# the "dynamic" rules table by a allow keep-state statement.
#################################################################
$cmd 015 check-state

#################################################################
# Interface facing Public Internet (Outbound Section)
# Interrogate session start requests originating from behind the
# firewall on the private network or from this gateway server
# destine for the public Internet.
#################################################################

# Allow out access to my ISP's Domain name server.
# x.x.x.x must be the IP address of your ISP's DNS
# Dup these lines if your ISP has more than one DNS server
# Get the IP addresses from /etc/resolv.conf file
$cmd 020 $skip tcp from any to x.x.x.x 53 out via $pif setup keep-state

# Allow out access to my ISP's DHCP server for cable/DSL configurations.
$cmd 030 $skip udp from any to x.x.x.x 67 out via $pif keep-state

# Allow out non-secure standard www function
$cmd 040 $skip tcp from any to any 80 out via $pif setup keep-state

# Allow out secure www function https over TLS SSL
$cmd 050 $skip tcp from any to any 443 out via $pif setup keep-state

# Allow out send & get email function
$cmd 060 $skip tcp from any to any 25 out via $pif setup keep-state
$cmd 061 $skip tcp from any to any 110 out via $pif setup keep-state

# Allow out FreeBSD (make install & CVSUP) functions
# Basically give user root "GOD" privileges.
$cmd 070 $skip tcp from me to any out via $pif setup keep-state uid root

# Allow out ping
$cmd 080 $skip icmp from any to any out via $pif keep-state

# Allow out Time
$cmd 090 $skip tcp from any to any 37 out via $pif setup keep-state

# Allow out nntp news (i.e. news groups)
$cmd 100 $skip tcp from any to any 119 out via $pif setup keep-state

# Allow out secure FTP, Telnet, and SCP
# This function is using SSH (secure shell)
$cmd 110 $skip tcp from any to any 22 out via $pif setup keep-state

# Allow out whois
$cmd 120 $skip tcp from any to any 43 out via $pif setup keep-state

# Allow ntp time server
$cmd 130 $skip udp from any to any 123 out via $pif keep-state

#################################################################
# Interface facing Public Internet (Inbound Section)
# Interrogate packets originating from the public Internet
# destine for this gateway server or the private network.
#################################################################

# Deny all inbound traffic from non-routable reserved address spaces
$cmd 300 deny all from 192.168.0.0/16  to any in via $pif  #RFC 1918 private IP
$cmd 301 deny all from 172.16.0.0/12   to any in via $pif  #RFC 1918 private IP
$cmd 302 deny all from 10.0.0.0/8      to any in via $pif  #RFC 1918 private IP
$cmd 303 deny all from 127.0.0.0/8     to any in via $pif  #loopback
$cmd 304 deny all from 0.0.0.0/8       to any in via $pif  #loopback
$cmd 305 deny all from 169.254.0.0/16  to any in via $pif  #DHCP auto-config
$cmd 306 deny all from 192.0.2.0/24    to any in via $pif  #reserved for docs
$cmd 307 deny all from 204.152.64.0/23 to any in via $pif  #Sun cluster
$cmd 308 deny all from 224.0.0.0/3     to any in via $pif  #Class D & E multicast

# Deny ident
$cmd 315 deny tcp from any to any 113 in via $pif

# Deny all Netbios service. 137=name, 138=datagram, 139=session
# Netbios is MS/Windows sharing services.
# Block MS/Windows hosts2 name server requests 81
$cmd 320 deny tcp from any to any 137 in via $pif
$cmd 321 deny tcp from any to any 138 in via $pif
$cmd 322 deny tcp from any to any 139 in via $pif
$cmd 323 deny tcp from any to any 81  in via $pif

# Deny any late arriving packets
$cmd 330 deny all from any to any frag in via $pif

# Deny ACK packets that did not match the dynamic rule table
$cmd 332 deny tcp from any to any established in via $pif

# Allow traffic in from ISP's DHCP server. This rule must contain
# the IP address of your ISP's DHCP server as it's the only
# authorized source to send this packet type.
# Only necessary for cable or DSL configurations.
# This rule is not needed for 'user ppp' type connection to
# the public Internet. This is the same IP address you captured
# and used in the outbound section.
$cmd 360 allow udp from x.x.x.x to any 68 in via $pif keep-state

# Allow in standard www function because I have Apache server
$cmd 370 allow tcp from any to me 80 in via $pif setup limit src-addr 2

# Allow in secure FTP, Telnet, and SCP from public Internet
$cmd 380 allow tcp from any to me 22 in via $pif setup limit src-addr 2

# Allow in non-secure Telnet session from public Internet
# labeled non-secure because ID & PW are passed over public
# Internet as clear text.
# Delete this sample group if you do not have telnet server enabled.
$cmd 390 allow tcp from any to me 23 in via $pif setup limit src-addr 2

# Reject & Log all unauthorized incoming connections from the public Internet
$cmd 400 deny log all from any to any in via $pif

# Reject & Log all unauthorized out going connections to the public Internet
$cmd 450 deny log all from any to any out via $pif

# This is skipto location for outbound stateful rules
$cmd 800 divert natd ip from any to any out via $pif
$cmd 801 allow ip from any to any

# Everything else is denied by default
# deny and log all packets that fell through to see what they are
$cmd 999 deny log all from any to any
################ End of IPFW rules file ###############################
Tags: account, Apache, archive, bsd, cron, database, domain, domain name, email, freebsd, FreeBSD Handbook, ftp, netbsd, openbsd, password, php, pop, software, ssh, ssl, telnet, trojan

Related posts

FreeBSD Handbook , , , , , , , , , , , , , , , , , , , ,

FreeBSD Handbook - Chapter 29 Network Servers

January 6th, 2009

Reorganized by Murray Stokely.

29.1 Synopsis

This chapter will cover some of the more frequently used network services on UNIX systems. We will cover how to install, configure, test, and maintain many different types of network services. Example configuration files are included throughout this chapter for you to benefit from.

After reading this chapter, you will know:

  • How to manage the inetd daemon.

  • How to set up a network file system.

  • How to set up a network information server for sharing user accounts.

  • How to set up automatic network settings using DHCP.

  • How to set up a domain name server.

  • How to set up the Apache HTTP Server.

  • How to set up a File Transfer Protocol (FTP) Server.

  • How to set up a file and print server for Windows clients using Samba.

  • How to synchronize the time and date, and set up a time server, with the NTP protocol.

  • How to configure the standard logging daemon, syslogd, to accept logs from remote hosts.

Before reading this chapter, you should:

  • Understand the basics of the /etc/rc scripts.

  • Be familiar with basic network terminology.

  • Know how to install additional third-party software (Chapter 4).


29.2 The inetd “Super-Server”

Contributed by Chern Lee. Updated for FreeBSD 6.1-RELEASE by The FreeBSD Documentation Project.

29.2.1 Overview

inetd(8) is sometimes referred to as the “Internet Super-Server” because it manages connections for several services. When a connection is received by inetd, it determines which program the connection is destined for, spawns the particular process and delegates the socket to it (the program is invoked with the service socket as its standard input, output and error descriptors). Running inetd for servers that are not heavily used can reduce the overall system load, when compared to running each daemon individually in stand-alone mode.

Primarily, inetd is used to spawn other daemons, but several trivial protocols are handled directly, such as chargen, auth, and daytime.

This section will cover the basics in configuring inetd through its command-line options and its configuration file, /etc/inetd.conf.


29.2.2 Settings

inetd is initialized through the rc(8) system. The inetd_enable option is set to NO by default, but may be turned on by sysinstall during installation, depending on the configuration chosen by the user. Placing:

inetd_enable="YES"

or

inetd_enable="NO"

into /etc/rc.conf will enable or disable inetd starting at boot time. The command:

# /etc/rc.d/inetd rcvar

can be run to display the current effective setting.

Additionally, different command-line options can be passed to inetd via the inetd_flags option.


29.2.3 Command-Line Options

Like most server daemons, inetd has a number of options that it can be passed in order to modify its behaviour. The full list of options reads:

inetd [-d] [-l] [-w] [-W] [-c maximum] [-C rate] [-a address | hostname] [-p filename] [-R rate] [-s maximum] [configuration file]

Options can be passed to inetd using the inetd_flags option in /etc/rc.conf. By default, inetd_flags is set to -wW -C 60, which turns on TCP wrapping for inetd’s services, and prevents any single IP address from requesting any service more than 60 times in any given minute.

Novice users may be pleased to note that these parameters usually do not need to be modified, although we mention the rate-limiting options below as they be useful should you find that you are receiving an excessive amount of connections. A full list of options can be found in the inetd(8) manual.

-c maximum

Specify the default maximum number of simultaneous invocations of each service; the default is unlimited. May be overridden on a per-service basis with the max-child parameter.

-C rate

Specify the default maximum number of times a service can be invoked from a single IP address in one minute; the default is unlimited. May be overridden on a per-service basis with the max-connections-per-ip-per-minute parameter.

-R rate

Specify the maximum number of times a service can be invoked in one minute; the default is 256. A rate of 0 allows an unlimited number of invocations.

-s maximum

Specify the maximum number of times a service can be invoked from a single IP address at any one time; the default is unlimited. May be overridden on a per-service basis with the max-child-per-ip parameter.


29.2.4 inetd.conf

Configuration of inetd is done via the file /etc/inetd.conf.

When a modification is made to /etc/inetd.conf, inetd can be forced to re-read its configuration file by running the command:

Example 29-1. Reloading the inetd configuration file

# /etc/rc.d/inetd reload

Each line of the configuration file specifies an individual daemon. Comments in the file are preceded by a “#”. The format of each entry in /etc/inetd.conf is as follows:

service-name
socket-type
protocol
{wait|nowait}[/max-child[/max-connections-per-ip-per-minute[/max-child-per-ip]]]
user[:group][/login-class]
server-program
server-program-arguments

An example entry for the ftpd(8) daemon using IPv4 might read:

ftp     stream  tcp     nowait  root    /usr/libexec/ftpd       ftpd -l
service-name

This is the service name of the particular daemon. It must correspond to a service listed in /etc/services. This determines which port inetd must listen to. If a new service is being created, it must be placed in /etc/services first.

socket-type

Either stream, dgram, raw, or seqpacket. stream must be used for connection-based, TCP daemons, while dgram is used for daemons utilizing the UDP transport protocol.

protocol

One of the following:

Protocol

Explanation

tcp, tcp4

TCP IPv4

udp, udp4

UDP IPv4

tcp6

TCP IPv6

udp6

UDP IPv6

tcp46

Both TCP IPv4 and v6

udp46

Both UDP IPv4 and v6

{wait|nowait}[/max-child[/max-connections-per-ip-per-minute[/max-child-per-ip]]]

wait|nowait indicates whether the daemon invoked from inetd is able to handle its own socket or not. dgram socket types must use the wait option, while stream socket daemons, which are usually multi-threaded, should use nowait. wait usually hands off multiple sockets to a single daemon, while nowait spawns a child daemon for each new socket.

The maximum number of child daemons inetd may spawn can be set using the max-child option. If a limit of ten instances of a particular daemon is needed, a /10 would be placed after nowait. Specifying /0 allows an unlimited number of children

In addition to max-child, two other options which limit the maximum connections from a single place to a particular daemon can be enabled. max-connections-per-ip-per-minute limits the number of connections from any particular IP address per minutes, e.g. a value of ten would limit any particular IP address connecting to a particular service to ten attempts per minute. max-child-per-ip limits the number of children that can be started on behalf on any single IP address at any moment. These options are useful to prevent intentional or unintentional excessive resource consumption and Denial of Service (DoS) attacks to a machine.

In this field, either of wait or nowait is mandatory. max-child, max-connections-per-ip-per-minute and max-child-per-ip are optional.

A stream-type multi-threaded daemon without any max-child, max-connections-per-ip-per-minute or max-child-per-ip limits would simply be: nowait.

The same daemon with a maximum limit of ten daemons would read: nowait/10.

The same setup with a limit of twenty connections per IP address per minute and a maximum total limit of ten child daemons would read: nowait/10/20.

These options are utilized by the default settings of the fingerd(8) daemon, as seen here:

finger stream  tcp     nowait/3/10 nobody /usr/libexec/fingerd fingerd -s

Finally, an example of this field with a maximum of 100 children in total, with a maximum of 5 for any one IP address would read: nowait/100/0/5.

user

This is the username that the particular daemon should run as. Most commonly, daemons run as the root user. For security purposes, it is common to find some servers running as the daemon user, or the least privileged nobody user.

server-program

The full path of the daemon to be executed when a connection is received. If the daemon is a service provided by inetd internally, then internal should be used.

server-program-arguments

This works in conjunction with server-program by specifying the arguments, starting with argv[0], passed to the daemon on invocation. If mydaemon -d is the command line, mydaemon -d would be the value of server-program-arguments. Again, if the daemon is an internal service, use internal here.


29.2.5 Security

Depending on the choices made at install time, many of inetd’s services may be enabled by default. If there is no apparent need for a particular daemon, consider disabling it. Place a “#” in front of the daemon in question in /etc/inetd.conf, and then reload the inetd configuration. Some daemons, such as fingerd, may not be desired at all because they provide information that may be useful to an attacker.

Some daemons are not security-conscious and have long, or non-existent, timeouts for connection attempts. This allows an attacker to slowly send connections to a particular daemon, thus saturating available resources. It may be a good idea to place max-connections-per-ip-per-minute, max-child or max-child-per-ip limitations on certain daemons if you find that you have too many connections.

By default, TCP wrapping is turned on. Consult the hosts_access(5) manual page for more information on placing TCP restrictions on various inetd invoked daemons.


29.2.6 Miscellaneous

daytime, time, echo, discard, chargen, and auth are all internally provided services of inetd.

The auth service provides identity network services, and is configurable to a certain degree, whilst the others are simply on or off.

Consult the inetd(8) manual page for more in-depth information.


29.3 Network File System (NFS)

Reorganized and enhanced by Tom Rhodes. Written by Bill Swingle.

Among the many different file systems that FreeBSD supports is the Network File System, also known as NFS. NFS allows a system to share directories and files with others over a network. By using NFS, users and programs can access files on remote systems almost as if they were local files.

Some of the most notable benefits that NFS can provide are:

  • Local workstations use less disk space because commonly used data can be stored on a single machine and still remain accessible to others over the network.

  • There is no need for users to have separate home directories on every network machine. Home directories could be set up on the NFS server and made available throughout the network.

  • Storage devices such as floppy disks, CDROM drives, and Zip® drives can be used by other machines on the network. This may reduce the number of removable media drives throughout the network.


29.3.1 How NFS Works

NFS consists of at least two main parts: a server and one or more clients. The client remotely accesses the data that is stored on the server machine. In order for this to function properly a few processes have to be configured and running.

The server has to be running the following daemons:

Daemon

Description

nfsd

The NFS daemon which services requests from the NFS clients.

mountd

The NFS mount daemon which carries out the requests that nfsd(8) passes on to it.

rpcbind

This daemon allows NFS clients to discover which port the NFS server is using.

The client can also run a daemon, known as nfsiod. The nfsiod daemon services the requests from the NFS server. This is optional, and improves performance, but is not required for normal and correct operation. See the nfsiod(8) manual page for more information.


29.3.2 Configuring NFS

NFS configuration is a relatively straightforward process. The processes that need to be running can all start at boot time with a few modifications to your /etc/rc.conf file.

On the NFS server, make sure that the following options are configured in the /etc/rc.conf file:

rpcbind_enable="YES"
nfs_server_enable="YES"
mountd_flags="-r"

mountd runs automatically whenever the NFS server is enabled.

On the client, make sure this option is present in /etc/rc.conf:

nfs_client_enable="YES"

The /etc/exports file specifies which file systems NFS should export (sometimes referred to as “share”). Each line in /etc/exports specifies a file system to be exported and which machines have access to that file system. Along with what machines have access to that file system, access options may also be specified. There are many such options that can be used in this file but only a few will be mentioned here. You can easily discover other options by reading over the exports(5) manual page.

Here are a few example /etc/exports entries:

The following examples give an idea of how to export file systems, although the settings may be different depending on your environment and network configuration. For instance, to export the /cdrom directory to three example machines that have the same domain name as the server (hence the lack of a domain name for each) or have entries in your /etc/hosts file. The -ro flag makes the exported file system read-only. With this flag, the remote system will not be able to write any changes to the exported file system.

/cdrom -ro host1 host2 host3

The following line exports /home to three hosts by IP address. This is a useful setup if you have a private network without a DNS server configured. Optionally the /etc/hosts file could be configured for internal hostnames; please review hosts(5) for more information. The -alldirs flag allows the subdirectories to be mount points. In other words, it will not mount the subdirectories but permit the client to mount only the directories that are required or needed.

/home  -alldirs  10.0.0.2 10.0.0.3 10.0.0.4

The following line exports /a so that two clients from different domains may access the file system. The -maproot=root flag allows the root user on the remote system to write data on the exported file system as root. If the -maproot=root flag is not specified, then even if a user has root access on the remote system, he will not be able to modify files on the exported file system.

/a  -maproot=root  host.example.com box.example.org

In order for a client to access an exported file system, the client must have permission to do so. Make sure the client is listed in your /etc/exports file.

In /etc/exports, each line represents the export information for one file system to one host. A remote host can only be specified once per file system, and may only have one default entry. For example, assume that /usr is a single file system. The following /etc/exports would be invalid:

# Invalid when /usr is one file system
/usr/src   client
/usr/ports client

One file system, /usr, has two lines specifying exports to the same host, client. The correct format for this situation is:

/usr/src /usr/ports  client

The properties of one file system exported to a given host must all occur on one line. Lines without a client specified are treated as a single host. This limits how you can export file systems, but for most people this is not an issue.

The following is an example of a valid export list, where /usr and /exports are local file systems:

# Export src and ports to client01 and client02, but only
# client01 has root privileges on it
/usr/src /usr/ports -maproot=root    client01
/usr/src /usr/ports               client02
# The client machines have root and can mount anywhere
# on /exports. Anyone in the world can mount /exports/obj read-only
/exports -alldirs -maproot=root      client01 client02
/exports/obj -ro

The mountd daemon must be forced to recheck the /etc/exports file whenever it has been modified, so the changes can take effect. This can be accomplished either by sending a HUP signal to the running daemon:

# kill -HUP `cat /var/run/mountd.pid`

or by invoking the mountd rc(8) script with the appropriate parameter:

# /etc/rc.d/mountd onereload

Please refer to Section 11.7 for more information about using rc scripts.

Alternatively, a reboot will make FreeBSD set everything up properly. A reboot is not necessary though. Executing the following commands as root should start everything up.

On the NFS server:

# rpcbind
# nfsd -u -t -n 4
# mountd -r

On the NFS client:

# nfsiod -n 4

Now everything should be ready to actually mount a remote file system. In these examples the server’s name will be server and the client’s name will be client. If you only want to temporarily mount a remote file system or would rather test the configuration, just execute a command like this as root on the client:

# mount server:/home /mnt

This will mount the /home directory on the server at /mnt on the client. If everything is set up correctly you should be able to enter /mnt on the client and see all the files that are on the server.

If you want to automatically mount a remote file system each time the computer boots, add the file system to the /etc/fstab file. Here is an example:

server:/home   /mnt    nfs rw  0   0

The fstab(5) manual page lists all the available options.


29.3.3 Locking

Some applications (e.g. mutt) require file locking to operate correctly. In the case of NFS, rpc.lockd can be used for file locking. To enable it, add the following to the /etc/rc.conf file on both client and server (it is assumed that the NFS client and server are configured already):

rpc_lockd_enable="YES"
rpc_statd_enable="YES"

Start the application by using:

# /etc/rc.d/lockd start
# /etc/rc.d/statd start

If real locking between the NFS clients and NFS server is not required, it is possible to let the NFS client do locking locally by passing -L to mount_nfs(8). Refer to the mount_nfs(8) manual page for further details.


29.3.4 Practical Uses

NFS has many practical uses. Some of the more common ones are listed below:

  • Set several machines to share a CDROM or other media among them. This is cheaper and often a more convenient method to install software on multiple machines.

  • On large networks, it might be more convenient to configure a central NFS server in which to store all the user home directories. These home directories can then be exported to the network so that users would always have the same home directory, regardless of which workstation they log in to.

  • Several machines could have a common /usr/ports/distfiles directory. That way, when you need to install a port on several machines, you can quickly access the source without downloading it on each machine.


29.3.5 Automatic Mounts with amd

Contributed by Wylie Stilwell. Rewritten by Chern Lee.

amd(8) (the automatic mounter daemon) automatically mounts a remote file system whenever a file or directory within that file system is accessed. Filesystems that are inactive for a period of time will also be automatically unmounted by amd. Using amd provides a simple alternative to permanent mounts, as permanent mounts are usually listed in /etc/fstab.

amd operates by attaching itself as an NFS server to the /host and /net directories. When a file is accessed within one of these directories, amd looks up the corresponding remote mount and automatically mounts it. /net is used to mount an exported file system from an IP address, while /host is used to mount an export from a remote hostname.

An access to a file within /host/foobar/usr would tell amd to attempt to mount the /usr export on the host foobar.

Example 29-2. Mounting an Export with amd

You can view the available mounts of a remote host with the showmount command. For example, to view the mounts of a host named foobar, you can use:

% showmount -e foobar
Exports list on foobar:
/usr                               10.10.10.0
/a                                 10.10.10.0
% cd /host/foobar/usr

As seen in the example, the showmount shows /usr as an export. When changing directories to /host/foobar/usr, amd attempts to resolve the hostname foobar and automatically mount the desired export.

amd can be started by the startup scripts by placing the following lines in /etc/rc.conf:

amd_enable="YES"

Additionally, custom flags can be passed to amd from the amd_flags option. By default, amd_flags is set to:

amd_flags="-a /.amd_mnt -l syslog /host /etc/amd.map /net /etc/amd.map"

The /etc/amd.map file defines the default options that exports are mounted with. The /etc/amd.conf file defines some of the more advanced features of amd.

Consult the amd(8) and amd.conf(5) manual pages for more information.


29.3.6 Problems Integrating with Other Systems

Contributed by John Lind.

Certain Ethernet adapters for ISA PC systems have limitations which can lead to serious network problems, particularly with NFS. This difficulty is not specific to FreeBSD, but FreeBSD systems are affected by it.

The problem nearly always occurs when (FreeBSD) PC systems are networked with high-performance workstations, such as those made by Silicon Graphics, Inc., and Sun Microsystems, Inc. The NFS mount will work fine, and some operations may succeed, but suddenly the server will seem to become unresponsive to the client, even though requests to and from other systems continue to be processed. This happens to the client system, whether the client is the FreeBSD system or the workstation. On many systems, there is no way to shut down the client gracefully once this problem has manifested itself. The only solution is often to reset the client, because the NFS situation cannot be resolved.

Though the “correct” solution is to get a higher performance and capacity Ethernet adapter for the FreeBSD system, there is a simple workaround that will allow satisfactory operation. If the FreeBSD system is the server, include the option -w=1024 on the mount from the client. If the FreeBSD system is the client, then mount the NFS file system with the option -r=1024. These options may be specified using the fourth field of the fstab entry on the client for automatic mounts, or by using the -o parameter of the mount(8) command for manual mounts.

It should be noted that there is a different problem, sometimes mistaken for this one, when the NFS servers and clients are on different networks. If that is the case, make certain that your routers are routing the necessary UDP information, or you will not get anywhere, no matter what else you are doing.

In the following examples, fastws is the host (interface) name of a high-performance workstation, and freebox is the host (interface) name of a FreeBSD system with a lower-performance Ethernet adapter. Also, /sharedfs will be the exported NFS file system (see exports(5)), and /project will be the mount point on the client for the exported file system. In all cases, note that additional options, such as hard or soft and bg may be desirable in your application.

Examples for the FreeBSD system (freebox) as the client in /etc/fstab on freebox:

fastws:/sharedfs /project nfs rw,-r=1024 0 0

As a manual mount command on freebox:

# mount -t nfs -o -r=1024 fastws:/sharedfs /project

Examples for the FreeBSD system as the server in /etc/fstab on fastws:

freebox:/sharedfs /project nfs rw,-w=1024 0 0

As a manual mount command on fastws:

# mount -t nfs -o -w=1024 freebox:/sharedfs /project

Nearly any 16-bit Ethernet adapter will allow operation without the above restrictions on the read or write size.

For anyone who cares, here is what happens when the failure occurs, which also explains why it is unrecoverable. NFS typically works with a “block” size of 8 K (though it may do fragments of smaller sizes). Since the maximum Ethernet packet is around 1500 bytes, the NFS “block” gets split into multiple Ethernet packets, even though it is still a single unit to the upper-level code, and must be received, assembled, and acknowledged as a unit. The high-performance workstations can pump out the packets which comprise the NFS unit one right after the other, just as close together as the standard allows. On the smaller, lower capacity cards, the later packets overrun the earlier packets of the same unit before they can be transferred to the host and the unit as a whole cannot be reconstructed or acknowledged. As a result, the workstation will time out and try again, but it will try again with the entire 8 K unit, and the process will be repeated, ad infinitum.

By keeping the unit size below the Ethernet packet size limitation, we ensure that any complete Ethernet packet received can be acknowledged individually, avoiding the deadlock situation.

Overruns may still occur when a high-performance workstations is slamming data out to a PC system, but with the better cards, such overruns are not guaranteed on NFS “units”. When an overrun occurs, the units affected will be retransmitted, and there will be a fair chance that they will be received, assembled, and acknowledged.


29.4 Network Information System (NIS/YP)

Written by Bill Swingle. Enhanced by Eric Ogren and Udo Erdelhoff.

29.4.1 What Is It?

NIS, which stands for Network Information Services, was developed by Sun Microsystems to centralize administration of UNIX (originally SunOS) systems. It has now essentially become an industry standard; all major UNIX like systems (Solaris, HP-UX, AIX®, Linux, NetBSD, OpenBSD, FreeBSD, etc) support NIS.

NIS was formerly known as Yellow Pages, but because of trademark issues, Sun changed the name. The old term (and yp) is still often seen and used.

It is a RPC-based client/server system that allows a group of machines within an NIS domain to share a common set of configuration files. This permits a system administrator to set up NIS client systems with only minimal configuration data and add, remove or modify configuration data from a single location.

It is similar to the Windows NT® domain system; although the internal implementation of the two are not at all similar, the basic functionality can be compared.


29.4.2 Terms/Processes You Should Know

There are several terms and several important user processes that you will come across when attempting to implement NIS on FreeBSD, whether you are trying to create an NIS server or act as an NIS client:

Term

Description

NIS domainname

An NIS master server and all of its clients (including its slave servers) have a NIS domainname. Similar to an Windows NT domain name, the NIS domainname does not have anything to do with DNS.

rpcbind

Must be running in order to enable RPC (Remote Procedure Call, a network protocol used by NIS). If rpcbind is not running, it will be impossible to run an NIS server, or to act as an NIS client.

ypbind

“Binds” an NIS client to its NIS server. It will take the NIS domainname from the system, and using RPC, connect to the server. ypbind is the core of client-server communication in an NIS environment; if ypbind dies on a client machine, it will not be able to access the NIS server.

ypserv

Should only be running on NIS servers; this is the NIS server process itself. If ypserv(8) dies, then the server will no longer be able to respond to NIS requests (hopefully, there is a slave server to take over for it). There are some implementations of NIS (but not the FreeBSD one), that do not try to reconnect to another server if the server it used before dies. Often, the only thing that helps in this case is to restart the server process (or even the whole server) or the ypbind process on the client.

rpc.yppasswdd

Another process that should only be running on NIS master servers; this is a daemon that will allow NIS clients to change their NIS passwords. If this daemon is not running, users will have to login to the NIS master server and change their passwords there.


29.4.3 How Does It Work?

There are three types of hosts in an NIS environment: master servers, slave servers, and clients. Servers act as a central repository for host configuration information. Master servers hold the authoritative copy of this information, while slave servers mirror this information for redundancy. Clients rely on the servers to provide this information to them.

Information in many files can be shared in this manner. The master.passwd, group, and hosts files are commonly shared via NIS. Whenever a process on a client needs information that would normally be found in these files locally, it makes a query to the NIS server that it is bound to instead.


29.4.3.1 Machine Types
  • A NIS master server. This server, analogous to a Windows NT primary domain controller, maintains the files used by all of the NIS clients. The passwd, group, and other various files used by the NIS clients live on the master server.

    Note: It is possible for one machine to be an NIS master server for more than one NIS domain. However, this will not be covered in this introduction, which assumes a relatively small-scale NIS environment.

  • NIS slave servers. Similar to the Windows NT backup domain controllers, NIS slave servers maintain copies of the NIS master’s data files. NIS slave servers provide the redundancy, which is needed in important environments. They also help to balance the load of the master server: NIS Clients always attach to the NIS server whose response they get first, and this includes slave-server-replies.

  • NIS clients. NIS clients, like most Windows NT workstations, authenticate against the NIS server (or the Windows NT domain controller in the Windows NT workstations case) to log on.


29.4.4 Using NIS/YP

This section will deal with setting up a sample NIS environment.


29.4.4.1 Planning

Let us assume that you are the administrator of a small university lab. This lab, which consists of 15 FreeBSD machines, currently has no centralized point of administration; each machine has its own /etc/passwd and /etc/master.passwd. These files are kept in sync with each other only through manual intervention; currently, when you add a user to the lab, you must run adduser on all 15 machines. Clearly, this has to change, so you have decided to convert the lab to use NIS, using two of the machines as servers.

Therefore, the configuration of the lab now looks something like:

Machine name

IP address

Machine role

ellington

10.0.0.2

NIS master

coltrane

10.0.0.3

NIS slave

basie

10.0.0.4

Faculty workstation

bird

10.0.0.5

Client machine

cli[1-11]

10.0.0.[6-17]

Other client machines

If you are setting up a NIS scheme for the first time, it is a good idea to think through how you want to go about it. No matter what the size of your network, there are a few decisions that need to be made.


29.4.4.1.1 Choosing a NIS Domain Name

This might not be the “domainname” that you are used to. It is more accurately called the “NIS domainname”. When a client broadcasts its requests for info, it includes the name of the NIS domain that it is part of. This is how multiple servers on one network can tell which server should answer which request. Think of the NIS domainname as the name for a group of hosts that are related in some way.

Some organizations choose to use their Internet domainname for their NIS domainname. This is not recommended as it can cause confusion when trying to debug network problems. The NIS domainname should be unique within your network and it is helpful if it describes the group of machines it represents. For example, the Art department at Acme Inc. might be in the “acme-art” NIS domain. For this example, assume you have chosen the name test-domain.

However, some operating systems (notably SunOS) use their NIS domain name as their Internet domain name. If one or more machines on your network have this restriction, you must use the Internet domain name as your NIS domain name.


29.4.4.1.2 Physical Server Requirements

There are several things to keep in mind when choosing a machine to use as a NIS server. One of the unfortunate things about NIS is the level of dependency the clients have on the server. If a client cannot contact the server for its NIS domain, very often the machine becomes unusable. The lack of user and group information causes most systems to temporarily freeze up. With this in mind you should make sure to choose a machine that will not be prone to being rebooted regularly, or one that might be used for development. The NIS server should ideally be a stand alone machine whose sole purpose in life is to be an NIS server. If you have a network that is not very heavily used, it is acceptable to put the NIS server on a machine running other services, just keep in mind that if the NIS server becomes unavailable, it will affect all of your NIS clients adversely.


29.4.4.2 NIS Servers

The canonical copies of all NIS information are stored on a single machine called the NIS master server. The databases used to store the information are called NIS maps. In FreeBSD, these maps are stored in /var/yp/[domainname] where [domainname] is the name of the NIS domain being served. A single NIS server can support several domains at once, therefore it is possible to have several such directories, one for each supported domain. Each domain will have its own independent set of maps.

NIS master and slave servers handle all NIS requests with the ypserv daemon. ypserv is responsible for receiving incoming requests from NIS clients, translating the requested domain and map name to a path to the corresponding database file and transmitting data from the database back to the client.


29.4.4.2.1 Setting Up a NIS Master Server

Setting up a master NIS server can be relatively straight forward, depending on your needs. FreeBSD comes with support for NIS out-of-the-box. All you need is to add the following lines to /etc/rc.conf, and FreeBSD will do the rest for you.

  1. nisdomainname="test-domain"

    This line will set the NIS domainname to test-domain upon network setup (e.g. after reboot).

  2. nis_server_enable="YES"

    This will tell FreeBSD to start up the NIS server processes when the networking is next brought up.

  3. nis_yppasswdd_enable="YES"

    This will enable the rpc.yppasswdd daemon which, as mentioned above, will allow users to change their NIS password from a client machine.

Note: Depending on your NIS setup, you may need to add further entries. See the section about NIS servers that are also NIS clients, below, for details.

Now, all you have to do is to run the command /etc/netstart as superuser. It will set up everything for you, using the values you defined in /etc/rc.conf.


29.4.4.2.2 Initializing the NIS Maps

The NIS maps are database files, that are kept in the /var/yp directory. They are generated from configuration files in the /etc directory of the NIS master, with one exception: the /etc/master.passwd file. This is for a good reason, you do not want to propagate passwords to your root and other administrative accounts to all the servers in the NIS domain. Therefore, before we initialize the NIS maps, you should:

# cp /etc/master.passwd /var/yp/master.passwd
# cd /var/yp
# vi master.passwd

You should remove all entries regarding system accounts (bin, tty, kmem, games, etc), as well as any accounts that you do not want to be propagated to the NIS clients (for example root and any other UID 0 (superuser) accounts).

Note: Make sure the /var/yp/master.passwd is neither group nor world readable (mode 600)! Use the chmod command, if appropriate.

When you have finished, it is time to initialize the NIS maps! FreeBSD includes a script named ypinit to do this for you (see its manual page for more information). Note that this script is available on most UNIX Operating Systems, but not on all. On Digital UNIX/Compaq Tru64 UNIX it is called ypsetup. Because we are generating maps for an NIS master, we are going to pass the -m option to ypinit. To generate the NIS maps, assuming you already performed the steps above, run:

ellington# ypinit -m test-domain
Server Type: MASTER Domain: test-domain
Creating an YP server will require that you answer a few questions.
Questions will all be asked at the beginning of the procedure.
Do you want this procedure to quit on non-fatal errors? [y/n: n] n
Ok, please remember to go back and redo manually whatever fails.
If you don't, something might not work.
At this point, we have to construct a list of this domains YP servers.
rod.darktech.org is already known as master server.
Please continue to add any slave servers, one per line. When you are
done with the list, type a <control D>.
master server   :  ellington
next host to add:  coltrane
next host to add:  ^D
The current list of NIS servers looks like this:
ellington
coltrane
Is this correct?  [y/n: y] y

[..output from map generation..]

NIS Map update completed.
ellington has been setup as an YP master server without any errors.

ypinit should have created /var/yp/Makefile from /var/yp/Makefile.dist. When created, this file assumes that you are operating in a single server NIS environment with only FreeBSD machines. Since test-domain has a slave server as well, you must edit /var/yp/Makefile:

ellington# vi /var/yp/Makefile

You should comment out the line that says

NOPUSH = "True"

(if it is not commented out already).


29.4.4.2.3 Setting up a NIS Slave Server

Setting up an NIS slave server is even more simple than setting up the master. Log on to the slave server and edit the file /etc/rc.conf as you did before. The only difference is that we now must use the -s option when running ypinit. The -s option requires the name of the NIS master be passed to it as well, so our command line looks like:

coltrane# ypinit -s ellington test-domain

Server Type: SLAVE Domain: test-domain Master: ellington

Creating an YP server will require that you answer a few questions.
Questions will all be asked at the beginning of the procedure.

Do you want this procedure to quit on non-fatal errors? [y/n: n]  n

Ok, please remember to go back and redo manually whatever fails.
If you don't, something might not work.
There will be no further questions. The remainder of the procedure
should take a few minutes, to copy the databases from ellington.
Transferring netgroup...
ypxfr: Exiting: Map successfully transferred
Transferring netgroup.byuser...
ypxfr: Exiting: Map successfully transferred
Transferring netgroup.byhost...
ypxfr: Exiting: Map successfully transferred
Transferring master.passwd.byuid...
ypxfr: Exiting: Map successfully transferred
Transferring passwd.byuid...
ypxfr: Exiting: Map successfully transferred
Transferring passwd.byname...
ypxfr: Exiting: Map successfully transferred
Transferring group.bygid...
ypxfr: Exiting: Map successfully transferred
Transferring group.byname...
ypxfr: Exiting: Map successfully transferred
Transferring services.byname...
ypxfr: Exiting: Map successfully transferred
Transferring rpc.bynumber...
ypxfr: Exiting: Map successfully transferred
Transferring rpc.byname...
ypxfr: Exiting: Map successfully transferred
Transferring protocols.byname...
ypxfr: Exiting: Map successfully transferred
Transferring master.passwd.byname...
ypxfr: Exiting: Map successfully transferred
Transferring networks.byname...
ypxfr: Exiting: Map successfully transferred
Transferring networks.byaddr...
ypxfr: Exiting: Map successfully transferred
Transferring netid.byname...
ypxfr: Exiting: Map successfully transferred
Transferring hosts.byaddr...
ypxfr: Exiting: Map successfully transferred
Transferring protocols.bynumber...
ypxfr: Exiting: Map successfully transferred
Transferring ypservers...
ypxfr: Exiting: Map successfully transferred
Transferring hosts.byname...
ypxfr: Exiting: Map successfully transferred

coltrane has been setup as an YP slave server without any errors.
Don't forget to update map ypservers on ellington.

You should now have a directory called /var/yp/test-domain. Copies of the NIS master server’s maps should be in this directory. You will need to make sure that these stay updated. The following /etc/crontab entries on your slave servers should do the job:

20      *       *       *       *       root   /usr/libexec/ypxfr passwd.byname
21      *       *       *       *       root   /usr/libexec/ypxfr passwd.byuid

These two lines force the slave to sync its maps with the maps on the master server. Although these entries are not mandatory, since the master server attempts to ensure any changes to its NIS maps are communicated to its slaves and because password information is vital to systems depending on the server, it is a good idea to force the updates. This is more important on busy networks where map updates might not always complete.

Now, run the command /etc/netstart on the slave server as well, which again starts the NIS server.


29.4.4.3 NIS Clients

An NIS client establishes what is called a binding to a particular NIS server using the ypbind daemon. ypbind checks the system’s default domain (as set by the domainname command), and begins broadcasting RPC requests on the local network. These requests specify the name of the domain for which ypbind is attempting to establish a binding. If a server that has been configured to serve the requested domain receives one of the broadcasts, it will respond to ypbind, which will record the server’s address. If there are several servers available (a master and several slaves, for example), ypbind will use the address of the first one to respond. From that point on, the client system will direct all of its NIS requests to that server. ypbind will occasionally “ping” the server to make sure it is still up and running. If it fails to receive a reply to one of its pings within a reasonable amount of time, ypbind will mark the domain as unbound and begin broadcasting again in the hopes of locating another server.


29.4.4.3.1 Setting Up a NIS Client

Setting up a FreeBSD machine to be a NIS client is fairly straightforward.

  1. Edit the file /etc/rc.conf and add the following lines in order to set the NIS domainname and start ypbind upon network startup:

    nisdomainname="test-domain"
    nis_client_enable="YES"
  2. To import all possible password entries from the NIS server, remove all user accounts from your /etc/master.passwd file and use vipw to add the following line to the end of the file:

    +:::::::::

    Note: This line will afford anyone with a valid account in the NIS server’s password maps an account. There are many ways to configure your NIS client by changing this line. See the netgroups section below for more information. For more detailed reading see O’Reilly’s book on Managing NFS and NIS.

    Note: You should keep at least one local account (i.e. not imported via NIS) in your /etc/master.passwd and this account should also be a member of the group wheel. If there is something wrong with NIS, this account can be used to log in remotely, become root, and fix things.

  3. To import all possible group entries from the NIS server, add this line to your /etc/group file:

    +:*::

After completing these steps, you should be able to run ypcat passwd and see the NIS server’s passwd map.


29.4.5 NIS Security

In general, any remote user can issue an RPC to ypserv(8) and retrieve the contents of your NIS maps, provided the remote user knows your domainname. To prevent such unauthorized transactions, ypserv(8) supports a feature called “securenets” which can be used to restrict access to a given set of hosts. At startup, ypserv(8) will attempt to load the securenets information from a file called /var/yp/securenets.

Note: This path varies depending on the path specified with the -p option. This file contains entries that consist of a network specification and a network mask separated by white space. Lines starting with “#” are considered to be comments. A sample securenets file might look like this:

# allow connections from local host -- mandatory
127.0.0.1     255.255.255.255
# allow connections from any host
# on the 192.168.128.0 network
192.168.128.0 255.255.255.0
# allow connections from any host
# between 10.0.0.0 to 10.0.15.255
# this includes the machines in the testlab
10.0.0.0      255.255.240.0

If ypserv(8) receives a request from an address that matches one of these rules, it will process the request normally. If the address fails to match a rule, the request will be ignored and a warning message will be logged. If the /var/yp/securenets file does not exist, ypserv will allow connections from any host.

The ypserv program also has support for Wietse Venema’s TCP Wrapper package. This allows the administrator to use the TCP Wrapper configuration files for access control instead of /var/yp/securenets.

Note: While both of these access control mechanisms provide some security, they, like the privileged port test, are vulnerable to “IP spoofing” attacks. All NIS-related traffic should be blocked at your firewall.

Servers using /var/yp/securenets may fail to serve legitimate NIS clients with archaic TCP/IP implementations. Some of these implementations set all host bits to zero when doing broadcasts and/or fail to observe the subnet mask when calculating the broadcast address. While some of these problems can be fixed by changing the client configuration, other problems may force the retirement of the client systems in question or the abandonment of /var/yp/securenets.

Using /var/yp/securenets on a server with such an archaic implementation of TCP/IP is a really bad idea and will lead to loss of NIS functionality for large parts of your network.

The use of the TCP Wrapper package increases the latency of your NIS server. The additional delay may be long enough to cause timeouts in client programs, especially in busy networks or with slow NIS servers. If one or more of your client systems suffers from these symptoms, you should convert the client systems in question into NIS slave servers and force them to bind to themselves.


29.4.6 Barring Some Users from Logging On

In our lab, there is a machine basie that is supposed to be a faculty only workstation. We do not want to take this machine out of the NIS domain, yet the passwd file on the master NIS server contains accounts for both faculty and students. What can we do?

There is a way to bar specific users from logging on to a machine, even if they are present in the NIS database. To do this, all you must do is add -username to the end of the /etc/master.passwd file on the client machine, where username is the username of the user you wish to bar from logging in. This should preferably be done using vipw, since vipw will sanity check your changes to /etc/master.passwd, as well as automatically rebuild the password database when you finish editing. For example, if we wanted to bar user bill from logging on to basie we would:

basie# vipw
[add -bill to the end, exit]
vipw: rebuilding the database...
vipw: done

basie# cat /etc/master.passwd

root:[password]:0:0::0:0:The super-user:/root:/bin/csh
toor:[password]:0:0::0:0:The other super-user:/root:/bin/sh
daemon:*:1:1::0:0:Owner of many system processes:/root:/sbin/nologin
operator:*:2:5::0:0:System &:/:/sbin/nologin
bin:*:3:7::0:0:Binaries Commands and Source,,,:/:/sbin/nologin
tty:*:4:65533::0:0:Tty Sandbox:/:/sbin/nologin
kmem:*:5:65533::0:0:KMem Sandbox:/:/sbin/nologin
games:*:7:13::0:0:Games pseudo-user:/usr/games:/sbin/nologin
news:*:8:8::0:0:News Subsystem:/:/sbin/nologin
man:*:9:9::0:0:Mister Man Pages:/usr/share/man:/sbin/nologin
bind:*:53:53::0:0:Bind Sandbox:/:/sbin/nologin
uucp:*:66:66::0:0:UUCP pseudo-user:/var/spool/uucppublic:/usr/libexec/uucp/uucico
xten:*:67:67::0:0:X-10 daemon:/usr/local/xten:/sbin/nologin
pop:*:68:6::0:0:Post Office Owner:/nonexistent:/sbin/nologin
nobody:*:65534:65534::0:0:Unprivileged user:/nonexistent:/sbin/nologin
+:::::::::
-bill

basie#

29.4.7 Using Netgroups

Contributed by Udo Erdelhoff.

The method shown in the previous section works reasonably well if you need special rules for a very small number of users and/or machines. On larger networks, you will forget to bar some users from logging onto sensitive machines, or you may even have to modify each machine separately, thus losing the main benefit of NIS: centralized administration.

The NIS developers’ solution for this problem is called netgroups. Their purpose and semantics can be compared to the normal groups used by UNIX file systems. The main differences are the lack of a numeric ID and the ability to define a netgroup by including both user accounts and other netgroups.

Netgroups were developed to handle large, complex networks with hundreds of users and machines. On one hand, this is a Good Thing if you are forced to deal with such a situation. On the other hand, this complexity makes it almost impossible to explain netgroups with really simple examples. The example used in the remainder of this section demonstrates this problem.

Let us assume that your successful introduction of NIS in your laboratory caught your superiors’ interest. Your next job is to extend your NIS domain to cover some of the other machines on campus. The two tables contain the names of the new users and new machines as well as brief descriptions of them.

User Name(s)

Description

alpha, beta

Normal employees of the IT department

charlie, delta

The new apprentices of the IT department

echo, foxtrott, golf, …

Ordinary employees

able, baker, …

The current interns

Machine Name(s)

Description

war, death, famine, pollution

Your most important servers. Only the IT employees are allowed to log onto these machines.

pride, greed, envy, wrath, lust, sloth

Less important servers. All members of the IT department are allowed to login onto these machines.

one, two, three, four, …

Ordinary workstations. Only the real employees are allowed to use these machines.

trashcan

A very old machine without any critical data. Even the intern is allowed to use this box.

If you tried to implement these restrictions by separately blocking each user, you would have to add one -user line to each system’s passwd for each user who is not allowed to login onto that system. If you forget just one entry, you could be in trouble. It may be feasible to do this correctly during the initial setup, however you will eventually forget to add the lines for new users during day-to-day operations. After all, Murphy was an optimist.

Handling this situation with netgroups offers several advantages. Each user need not be handled separately; you assign a user to one or more netgroups and allow or forbid logins for all members of the netgroup. If you add a new machine, you will only have to define login restrictions for netgroups. If a new user is added, you will only have to add the user to one or more netgroups. Those changes are independent of each other: no more “for each combination of user and machine do…” If your NIS setup is planned carefully, you will only have to modify exactly one central configuration file to grant or deny access to machines.

The first step is the initialization of the NIS map netgroup. FreeBSD’s ypinit(8) does not create this map by default, but its NIS implementation will support it once it has been created. To create an empty map, simply type

ellington# vi /var/yp/netgroup

and start adding content. For our example, we need at least four netgroups: IT employees, IT apprentices, normal employees and interns.

IT_EMP  (,alpha,test-domain)    (,beta,test-domain)
IT_APP  (,charlie,test-domain)  (,delta,test-domain)
USERS   (,echo,test-domain)     (,foxtrott,test-domain) \
        (,golf,test-domain)
INTERNS (,able,test-domain)     (,baker,test-domain)

IT_EMP, IT_APP etc. are the names of the netgroups. Each bracketed group adds one or more user accounts to it. The three fields inside a group are:

  1. The name of the host(s) where the following items are valid. If you do not specify a hostname, the entry is valid on all hosts. If you do specify a hostname, you will enter a realm of darkness, horror and utter confusion.

  2. The name of the account that belongs to this netgroup.

  3. The NIS domain for the account. You can import accounts from other NIS domains into your netgroup if you are one of the unlucky fellows with more than one NIS domain.

Each of these fields can contain wildcards. See netgroup(5) for details.

Note: Netgroup names longer than 8 characters should not be used, especially if you have machines running other operating systems within your NIS domain. The names are case sensitive; using capital letters for your netgroup names is an easy way to distinguish between user, machine and netgroup names.

Some NIS clients (other than FreeBSD) cannot handle netgroups with a large number of entries. For example, some older versions of SunOS start to cause trouble if a netgroup contains more than 15 entries. You can circumvent this limit by creating several sub-netgroups with 15 users or less and a real netgroup that consists of the sub-netgroups:

BIGGRP1  (,joe1,domain)  (,joe2,domain)  (,joe3,domain) [...]
BIGGRP2  (,joe16,domain)  (,joe17,domain) [...]
BIGGRP3  (,joe31,domain)  (,joe32,domain)
BIGGROUP  BIGGRP1 BIGGRP2 BIGGRP3

You can repeat this process if you need more than 225 users within a single netgroup.

Activating and distributing your new NIS map is easy:

ellington# cd /var/yp
ellington# make

This will generate the three NIS maps netgroup, netgroup.byhost and netgroup.byuser. Use ypcat(1) to check if your new NIS maps are available:

ellington% ypcat -k netgroup
ellington% ypcat -k netgroup.byhost
ellington% ypcat -k netgroup.byuser

The output of the first command should resemble the contents of /var/yp/netgroup. The second command will not produce output if you have not specified host-specific netgroups. The third command can be used to get the list of netgroups for a user.

The client setup is quite simple. To configure the server war, you only have to start vipw(8) and replace the line

+:::::::::

with

+@IT_EMP:::::::::

Now, only the data for the users defined in the netgroup IT_EMP is imported into war’s password database and only these users are allowed to login.

Unfortunately, this limitation also applies to the ~ function of the shell and all routines converting between user names and numerical user IDs. In other words, cd ~user will not work, ls -l will show the numerical ID instead of the username and find . -user joe -print will fail with “No such user”. To fix this, you will have to import all user entries without allowing them to login onto your servers.

This can be achieved by adding another line to /etc/master.passwd. This line should contain:

+:::::::::/sbin/nologin, meaning “Import all entries but replace the shell with /sbin/nologin in the imported entries”. You can replace any field in the passwd entry by placing a default value in your /etc/master.passwd.

Warning: Make sure that the line +:::::::::/sbin/nologin is placed after +@IT_EMP:::::::::. Otherwise, all user accounts imported from NIS will have /sbin/nologin as their login shell.

After this change, you will only have to change one NIS map if a new employee joins the IT department. You could use a similar approach for the less important servers by replacing the old +::::::::: in their local version of /etc/master.passwd with something like this:

+@IT_EMP:::::::::
+@IT_APP:::::::::
+:::::::::/sbin/nologin

The corresponding lines for the normal workstations could be:

+@IT_EMP:::::::::
+@USERS:::::::::
+:::::::::/sbin/nologin

And everything would be fine until there is a policy change a few weeks later: The IT department starts hiring interns. The IT interns are allowed to use the normal workstations and the less important servers; and the IT apprentices are allowed to login onto the main servers. You add a new netgroup IT_INTERN, add the new IT interns to this netgroup and start to change the configuration on each and every machine… As the old saying goes: “Errors in centralized planning lead to global mess”.

NIS’ ability to create netgroups from other netgroups can be used to prevent situations like these. One possibility is the creation of role-based netgroups. For example, you could create a netgroup called BIGSRV to define the login restrictions for the important servers, another netgroup called SMALLSRV for the less important servers and a third netgroup called USERBOX for the normal workstations. Each of these netgroups contains the netgroups that are allowed to login onto these machines. The new entries for your NIS map netgroup should look like this:

BIGSRV    IT_EMP  IT_APP
SMALLSRV  IT_EMP  IT_APP  ITINTERN
USERBOX   IT_EMP  ITINTERN USERS

This method of defining login restrictions works reasonably well if you can define groups of machines with identical restrictions. Unfortunately, this is the exception and not the rule. Most of the time, you will need the ability to define login restrictions on a per-machine basis.

Machine-specific netgroup definitions are the other possibility to deal with the policy change outlined above. In this scenario, the /etc/master.passwd of each box contains two lines starting with “+”. The first of them adds a netgroup with the accounts allowed to login onto this machine, the second one adds all other accounts with /sbin/nologin as shell. It is a good idea to use the “ALL-CAPS” version of the machine name as the name of the netgroup. In other words, the lines should look like this:

+@BOXNAME:::::::::
+:::::::::/sbin/nologin

Once you have completed this task for all your machines, you will not have to modify the local versions of /etc/master.passwd ever again. All further changes can be handled by modifying the NIS map. Here is an example of a possible netgroup map for this scenario with some additional goodies:

# Define groups of users first
IT_EMP    (,alpha,test-domain)    (,beta,test-domain)
IT_APP    (,charlie,test-domain)  (,delta,test-domain)
DEPT1     (,echo,test-domain)     (,foxtrott,test-domain)
DEPT2     (,golf,test-domain)     (,hotel,test-domain)
DEPT3     (,india,test-domain)    (,juliet,test-domain)
ITINTERN  (,kilo,test-domain)     (,lima,test-domain)
D_INTERNS (,able,test-domain)     (,baker,test-domain)
#
# Now, define some groups based on roles
USERS     DEPT1   DEPT2     DEPT3
BIGSRV    IT_EMP  IT_APP
SMALLSRV  IT_EMP  IT_APP    ITINTERN
USERBOX   IT_EMP  ITINTERN  USERS
#
# And a groups for a special tasks
# Allow echo and golf to access our anti-virus-machine
SECURITY  IT_EMP  (,echo,test-domain)  (,golf,test-domain)
#
# machine-based netgroups
# Our main servers
WAR       BIGSRV
FAMINE    BIGSRV
# User india needs access to this server
POLLUTION  BIGSRV  (,india,test-domain)
#
# This one is really important and needs more access restrictions
DEATH     IT_EMP
#
# The anti-virus-machine mentioned above
ONE       SECURITY
#
# Restrict a machine to a single user
TWO       (,hotel,test-domain)
# [...more groups to follow]

If you are using some kind of database to manage your user accounts, you should be able to create the first part of the map with your database’s report tools. This way, new users will automatically have access to the boxes.

One last word of caution: It may not always be advisable to use machine-based netgroups. If you are deploying a couple of dozen or even hundreds of identical machines for student labs, you should use role-based netgroups instead of machine-based netgroups to keep the size of the NIS map within reasonable limits.


29.4.8 Important Things to Remember

There are still a couple of things that you will need to do differently now that you are in an NIS environment.

  • Every time you wish to add a user to the lab, you must add it to the master NIS server only, and you must remember to rebuild the NIS maps. If you forget to do this, the new user will not be able to login anywhere except on the NIS master. For example, if we needed to add a new user jsmith to the lab, we would:

    # pw useradd jsmith
    # cd /var/yp
    # make test-domain

    You could also run adduser jsmith instead of pw useradd jsmith.

  • Keep the administration accounts out of the NIS maps. You do not want to be propagating administrative accounts and passwords to machines that will have users that should not have access to those accounts.

  • Keep the NIS master and slave secure, and minimize their downtime. If somebody either hacks or simply turns off these machines, they have effectively rendered many people without the ability to login to the lab.

    This is the chief weakness of any centralized administration system. If you do not protect your NIS servers, you will have a lot of angry users!


29.4.9 NIS v1 Compatibility

FreeBSD’s ypserv has some support for serving NIS v1 clients. FreeBSD’s NIS implementation only uses the NIS v2 protocol, however other implementations include support for the v1 protocol for backwards compatibility with older systems. The ypbind daemons supplied with these systems will try to establish a binding to an NIS v1 server even though they may never actually need it (and they may persist in broadcasting in search of one even after they receive a response from a v2 server). Note that while support for normal client calls is provided, this version of ypserv does not handle v1 map transfer requests; consequently, it cannot be used as a master or slave in conjunction with older NIS servers that only support the v1 protocol. Fortunately, there probably are not any such servers still in use today.


29.4.10 NIS Servers That Are Also NIS Clients

Care must be taken when running ypserv in a multi-server domain where the server machines are also NIS clients. It is generally a good idea to force the servers to bind to themselves rather than allowing them to broadcast bind requests and possibly become bound to each other. Strange failure modes can result if one server goes down and others are dependent upon it. Eventually all the clients will time out and attempt to bind to other servers, but the delay involved can be considerable and the failure mode is still present since the servers might bind to each other all over again.

You can force a host to bind to a particular server by running ypbind with the -S flag. If you do not want to do this manually each time you reboot your NIS server, you can add the following lines to your /etc/rc.conf:

nis_client_enable="YES"    # run client stuff as well
nis_client_flags="-S NIS domain,server"

See ypbind(8) for further information.


29.4.11 Password Formats

One of the most common issues that people run into when trying to implement NIS is password format compatibility. If your NIS server is using DES encrypted passwords, it will only support clients that are also using DES. For example, if you have Solaris NIS clients in your network, then you will almost certainly need to use DES encrypted passwords.

To check which format your servers and clients are using, look at /etc/login.conf. If the host is configured to use DES encrypted passwords, then the default class will contain an entry like this:

default:\
    :passwd_format=des:\
    :copyright=/etc/COPYRIGHT:\
    [Further entries elided]

Other possible values for the passwd_format capability include blf and md5 (for Blowfish and MD5 encrypted passwords, respectively).

If you have made changes to /etc/login.conf, you will also need to rebuild the login capability database, which is achieved by running the following command as root:

# cap_mkdb /etc/login.conf

Note: The format of passwords already in /etc/master.passwd will not be updated until a user changes his password for the first time after the login capability database is rebuilt.

Next, in order to ensure that passwords are encrypted with the format that you have chosen, you should also check that the crypt_default in /etc/auth.conf gives precedence to your chosen password format. To do this, place the format that you have chosen first in the list. For example, when using DES encrypted passwords, the entry would be:

crypt_default  =   des blf md5

Having followed the above steps on each of the FreeBSD based NIS servers and clients, you can be sure that they all agree on which password format is used within your network. If you have trouble authenticating on an NIS client, this is a pretty good place to start looking for possible problems. Remember: if you want to deploy an NIS server for a heterogenous network, you will probably have to use DES on all systems because it is the lowest common standard.


29.5 Automatic Network Configuration (DHCP)

Written by Greg Sutter.

29.5.1 What Is DHCP?

DHCP, the Dynamic Host Configuration Protocol, describes the means by which a system can connect to a network and obtain the necessary information for communication upon that network. FreeBSD versions prior to 6.0 use the ISC (Internet Software Consortium) DHCP client (dhclient(8)) implementation. Later versions use the OpenBSD dhclient taken from OpenBSD 3.7. All information here regarding dhclient is for use with either of the ISC or OpenBSD DHCP clients. The DHCP server is the one included in the ISC distribution.


29.5.2 What This Section Covers

This section describes both the client-side components of the ISC and OpenBSD DHCP client and server-side components of the ISC DHCP system. The client-side program, dhclient, comes integrated within FreeBSD, and the server-side portion is available from the net/isc-dhcp3-server port. The dhclient(8), dhcp-options(5), and dhclient.conf(5) manual pages, in addition to the references below, are useful resources.


29.5.3 How It Works

When dhclient, the DHCP client, is executed on the client machine, it begins broadcasting requests for configuration information. By default, these requests are on UDP port 68. The server replies on UDP 67, giving the client an IP address and other relevant network information such as netmask, router, and DNS servers. All of this information comes in the form of a DHCP “lease” and is only valid for a certain time (configured by the DHCP server maintainer). In this manner, stale IP addresses for clients no longer connected to the network can be automatically reclaimed.

DHCP clients can obtain a great deal of information from the server. An exhaustive list may be found in dhcp-options(5).


29.5.4 FreeBSD Integration

FreeBSD fully integrates the ISC or OpenBSD DHCP client, dhclient (according to the FreeBSD version you run). DHCP client support is provided within both the installer and the base system, obviating the need for detailed knowledge of network configurations on any network that runs a DHCP server. dhclient has been included in all FreeBSD distributions since 3.2.

DHCP is supported by sysinstall. When configuring a network interface within sysinstall, the second question asked is: “Do you want to try DHCP configuration of the interface?”. Answering affirmatively will execute dhclient, and if successful, will fill in the network configuration information automatically.

There are two things you must do to have your system use DHCP upon startup:

  • Make sure that the bpf device is compiled into your kernel. To do this, add device bpf to your kernel configuration file, and rebuild the kernel. For more information about building kernels, see Chapter 8.

    The bpf device is already part of the GENERIC kernel that is supplied with FreeBSD, so if you do not have a custom kernel, you should not need to create one in order to get DHCP working.

    Note: For those who are particularly security conscious, you should be warned that bpf is also the device that allows packet sniffers to work correctly (although they still have to be run as root). bpf is required to use DHCP, but if you are very sensitive about security, you probably should not add bpf to your kernel in the expectation that at some point in the future you will be using DHCP.

  • Edit your /etc/rc.conf to include the following:

    ifconfig_fxp0="DHCP"

    Note: Be sure to replace fxp0 with the designation for the interface that you wish to dynamically configure, as described in Section 11.8.

    If you are using a different location for dhclient, or if you wish to pass additional flags to dhclient, also include the following (editing as necessary):

    dhclient_program="/sbin/dhclient"
    dhclient_flags=""

The DHCP server, dhcpd, is included as part of the net/isc-dhcp3-server port in the ports collection. This port contains the ISC DHCP server and documentation.


29.5.5 Files
  • /etc/dhclient.conf

    dhclient requires a configuration file, /etc/dhclient.conf. Typically the file contains only comments, the defaults being reasonably sane. This configuration file is described by the dhclient.conf(5) manual page.

  • /sbin/dhclient

    dhclient is statically linked and resides in /sbin. The dhclient(8) manual page gives more information about dhclient.

  • /sbin/dhclient-script

    dhclient-script is the FreeBSD-specific DHCP client configuration script. It is described in dhclient-script(8), but should not need any user modification to function properly.

  • /var/db/dhclient.leases

    The DHCP client keeps a database of valid leases in this file, which is written as a log. dhclient.leases(5) gives a slightly longer description.


29.5.6 Further Reading

The DHCP protocol is fully described in RFC 2131. An informational resource has also been set up at http://www.dhcp.org/.


29.5.7 Installing and Configuring a DHCP Server
29.5.7.1 What This Section Covers

This section provides information on how to configure a FreeBSD system to act as a DHCP server using the ISC (Internet Software Consortium) implementation of the DHCP server.

The server is not provided as part of FreeBSD, and so you will need to install the net/isc-dhcp3-server port to provide this service. See Chapter 4 for more information on using the Ports Collection.


29.5.7.2 DHCP Server Installation

In order to configure your FreeBSD system as a DHCP server, you will need to ensure that the bpf(4) device is compiled into your kernel. To do this, add device bpf to your kernel configuration file, and rebuild the kernel. For more information about building kernels, see Chapter 8.

The bpf device is already part of the GENERIC kernel that is supplied with FreeBSD, so you do not need to create a custom kernel in order to get DHCP working.

Note: Those who are particularly security conscious should note that bpf is also the device that allows packet sniffers to work correctly (although such programs still need privileged access). bpf is required to use DHCP, but if you are very sensitive about security, you probably should not include bpf in your kernel purely because you expect to use DHCP at some point in the future.

The next thing that you will need to do is edit the sample dhcpd.conf which was installed by the net/isc-dhcp3-server port. By default, this will be /usr/local/etc/dhcpd.conf.sample, and you should copy this to /usr/local/etc/dhcpd.conf before proceeding to make changes.


29.5.7.3 Configuring the DHCP Server

dhcpd.conf is comprised of declarations regarding subnets and hosts, and is perhaps most easily explained using an example :

option domain-name "example.com";(1)
option domain-name-servers 192.168.4.100;(2)
option subnet-mask 255.255.255.0;(3)

default-lease-time 3600;(4)
max-lease-time 86400;(5)
ddns-update-style none;(6)

subnet 192.168.4.0 netmask 255.255.255.0 {
  range 192.168.4.129 192.168.4.254;(7)
  option routers 192.168.4.1;(8)
}

host mailhost {
  hardware ethernet 02:03:04:05:06:07;(9)
  fixed-address mailhost.example.com;(10)
}
(1)
This option specifies the domain that will be provided to clients as the default search domain. See resolv.conf(5) for more information on what this means.
(2)
This option specifies a comma separated list of DNS servers that the client should use.
(3)
The netmask that will be provided to clients.
(4)
A client may request a specific length of time that a lease will be valid. Otherwise the server will assign a lease with this expiry value (in seconds).
(5)
This is the maximum length of time that the server will lease for. Should a client request a longer lease, a lease will be issued, although it will only be valid for max-lease-time seconds.
(6)
This option specifies whether the DHCP server should attempt to update DNS when a lease is accepted or released. In the ISC implementation, this option is required.
(7)
This denotes which IP addresses should be used in the pool reserved for allocating to clients. IP addresses between, and including, the ones stated are handed out to clients.
(8)
Declares the default gateway that will be provided to clients.
(9)
The hardware MAC address of a host (so that the DHCP server can recognize a host when it makes a request).
(10)
Specifies that the host should always be given the same IP address. Note that using a hostname is correct here, since the DHCP server will resolve the hostname itself before returning the lease information.

Once you have finished writing your dhcpd.conf, you should enable the DHCP server in /etc/rc.conf, i.e. by adding:

dhcpd_enable="YES"
dhcpd_ifaces="dc0"

Replace the dc0 interface name with the interface (or interfaces, separated by whitespace) that your DHCP server should listen on for DHCP client requests.

Then, you can proceed to start the server by issuing the following command:

# /usr/local/etc/rc.d/isc-dhcpd.sh start

Should you need to make changes to the configuration of your server in the future, it is important to note that sending a SIGHUP signal to dhcpd does not result in the configuration being reloaded, as it does with most daemons. You will need to send a SIGTERM signal to stop the process, and then restart it using the command above.


29.5.7.4 Files
  • /usr/local/sbin/dhcpd

    dhcpd is statically linked and resides in /usr/local/sbin. The dhcpd(8) manual page installed with the port gives more information about dhcpd.

  • /usr/local/etc/dhcpd.conf

    dhcpd requires a configuration file, /usr/local/etc/dhcpd.conf before it will start providing service to clients. This file needs to contain all the information that should be provided to clients that are being serviced, along with information regarding the operation of the server. This configuration file is described by the dhcpd.conf(5) manual page installed by the port.

  • /var/db/dhcpd.leases

    The DHCP server keeps a database of leases it has issued in this file, which is written as a log. The manual page dhcpd.leases(5), installed by the port gives a slightly longer description.

  • /usr/local/sbin/dhcrelay

    dhcrelay is used in advanced environments where one DHCP server forwards a request from a client to another DHCP server on a separate network. If you require this functionality, then install the net/isc-dhcp3-relay port. The dhcrelay(8) manual page provided with the port contains more detail.


29.6 Domain Name System (DNS)

Contributed by Chern Lee, Tom Rhodes, and Daniel Gerzo.

29.6.1 Overview

FreeBSD utilizes, by default, a version of BIND (Berkeley Internet Name Domain), which is the most common implementation of the DNS protocol. DNS is the protocol through which names are mapped to IP addresses, and vice versa. For example, a query for www.FreeBSD.org will receive a reply with the IP address of The FreeBSD Project’s web server, whereas, a query for ftp.FreeBSD.org will return the IP address of the corresponding FTP machine. Likewise, the opposite can happen. A query for an IP address can resolve its hostname. It is not necessary to run a name server to perform DNS lookups on a system.

FreeBSD currently comes with BIND9 DNS server software by default. Our installation provides enhanced security features, a new file system layout and automated chroot(8) configuration.

DNS is coordinated across the Internet through a somewhat complex system of authoritative root, Top Level Domain (TLD), and other smaller-scale name servers which host and cache individual domain information.

Currently, BIND is maintained by the Internet Software Consortium http://www.isc.org/.


29.6.2 Terminology

To understand this document, some terms related to DNS must be understood.

Term

Definition

Forward DNS

Mapping of hostnames to IP addresses.

Origin

Refers to the domain covered in a particular zone file.

named, BIND, name server

Common names for the BIND name server package within FreeBSD.

Resolver

A system process through which a machine queries a name server for zone information.

Reverse DNS

The opposite of forward DNS; mapping of IP addresses to hostnames.

Root zone

The beginning of the Internet zone hierarchy. All zones fall under the root zone, similar to how all files in a file system fall under the root directory.

Zone

An individual domain, subdomain, or portion of the DNS administered by the same authority.

Examples of zones:

  • . is the root zone.

  • org. is a Top Level Domain (TLD) under the root zone.

  • example.org. is a zone under the org. TLD.

  • 1.168.192.in-addr.arpa is a zone referencing all IP addresses which fall under the 192.168.1.* IP space.

As one can see, the more specific part of a hostname appears to its left. For example, example.org. is more specific than org., as org. is more specific than the root zone. The layout of each part of a hostname is much like a file system: the /dev directory falls within the root, and so on.


29.6.3 Reasons to Run a Name Server

Name servers usually come in two forms: an authoritative name server, and a caching name server.

An authoritative name server is needed when:

  • One wants to serve DNS information to the world, replying authoritatively to queries.

  • A domain, such as example.org, is registered and IP addresses need to be assigned to hostnames under it.

  • An IP address block requires reverse DNS entries (IP to hostname).

  • A backup or second name server, called a slave, will reply to queries.

A caching name server is needed when:

  • A local DNS server may cache and respond more quickly than querying an outside name server.

When one queries for www.FreeBSD.org, the resolver usually queries the uplink ISP’s name server, and retrieves the reply. With a local, caching DNS server, the query only has to be made once to the outside world by the caching DNS server. Every additional query will not have to look to the outside of the local network, since the information is cached locally.


29.6.4 How It Works

In FreeBSD, the BIND daemon is called named for obvious reasons.

File

Description

named(8)

The BIND daemon.

rndc(8)

Name server control utility.

/etc/namedb

Directory where BIND zone information resides.

/etc/namedb/named.conf

Configuration file of the daemon.

Depending on how a given zone is configured on the server, the files related to that zone can be found in the master, slave, or dynamic subdirectories of the /etc/namedb directory. These files contain the DNS information that will be given out by the name server in response to queries.


29.6.5 Starting BIND

Since BIND is installed by default, configuring it all is relatively simple.

The default named configuration is that of a basic resolving name server, ran in a chroot(8) environment. To start the server one time with this configuration, use the following command:

# /etc/rc.d/named forcestart

To ensure the named daemon is started at boot each time, put the following line into the /etc/rc.conf:

named_enable="YES"

There are obviously many configuration options for /etc/namedb/named.conf that are beyond the scope of this document. However, if you are interested in the startup options for named on FreeBSD, take a look at the named_* flags in /etc/defaults/rc.conf and consult the rc.conf(5) manual page. The Section 11.7 section is also a good read.


29.6.6 Configuration Files

Configuration files for named currently reside in /etc/namedb directory and will need modification before use, unless all that is needed is a simple resolver. This is where most of the configuration will be performed.


29.6.6.1 Using make-localhost

To configure a master zone for the localhost visit the /etc/namedb directory and run the following command:

# sh make-localhost

If all went well, a new file should exist in the master subdirectory. The filenames should be localhost.rev for the local domain name and localhost-v6.rev for IPv6 configurations. As the default configuration file, required information will be present in the named.conf file.


29.6.6.2 /etc/namedb/named.conf
// $FreeBSD$
//
// Refer to the named.conf(5) and named(8) man pages, and the documentation
// in /usr/share/doc/bind9 for more details.
//
// If you are going to set up an authoritative server, make sure you
// understand the hairy details of how DNS works.  Even with
// simple mistakes, you can break connectivity for affected parties,
// or cause huge amounts of useless Internet traffic.

options {
    directory   "/etc/namedb";
    pid-file    "/var/run/named/pid";
    dump-file   "/var/dump/named_dump.db";
    statistics-file "/var/stats/named.stats";

// If named is being used only as a local resolver, this is a safe default.
// For named to be accessible to the network, comment this option, specify
// the proper IP address, or delete this option.
    listen-on   { 127.0.0.1; };

// If you have IPv6 enabled on this system, uncomment this option for
// use as a local resolver.  To give access to the network, specify
// an IPv6 address, or the keyword "any".
//  listen-on-v6    { ::1; };

// In addition to the "forwarders" clause, you can force your name
// server to never initiate queries of its own, but always ask its
// forwarders only, by enabling the following line:
//
//  forward only;

// If you've got a DNS server around at your upstream provider, enter
// its IP address here, and enable the line below.  This will make you
// benefit from its cache, thus reduce overall DNS traffic in the Internet.
/*
    forwarders {
        127.0.0.1;
    };
*/

Just as the comment says, to benefit from an uplink’s cache, forwarders can be enabled here. Under normal circumstances, a name server will recursively query the Internet looking at certain name servers until it finds the answer it is looking for. Having this enabled will have it query the uplink’s name server (or name server provided) first, taking advantage of its cache. If the uplink name server in question is a heavily trafficked, fast name server, enabling this may be worthwhile.

Warning: 127.0.0.1 will not work here. Change this IP address to a name server at your uplink.

   /*
     * If there is a firewall between you and nameservers you want
     * to talk to, you might need to uncomment the query-source
     * directive below.  Previous versions of BIND always asked
     * questions using port 53, but BIND versions 8 and later
     * use a pseudo-random unprivileged UDP port by default.
     */
     // query-source address * port 53;
};

// If you enable a local name server, don't forget to enter 127.0.0.1
// first in your /etc/resolv.conf so this server will be queried.
// Also, make sure to enable it in /etc/rc.conf.

zone "." {
    type hint;
    file "named.root";
};

zone "0.0.127.IN-ADDR.ARPA" {
    type master;
    file "master/localhost.rev";
};

// RFC 3152
zone "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.IP6.ARPA" {
    type master;
    file "master/localhost-v6.rev";
};

// NB: Do not use the IP addresses below, they are faked, and only
// serve demonstration/documentation purposes!
//
// Example slave zone config entries.  It can be convenient to become
// a slave at least for the zone your own domain is in.  Ask
// your network administrator for the IP address of the responsible
// primary.
//
// Never forget to include the reverse lookup (IN-ADDR.ARPA) zone!
// (This is named after the first bytes of the IP address, in reverse
// order, with ".IN-ADDR.ARPA" appended.)
//
// Before starting to set up a primary zone, make sure you fully
// understand how DNS and BIND works.  There are sometimes
// non-obvious pitfalls.  Setting up a slave zone is simpler.
//
// NB: Don't blindly enable the examples below. :-)  Use actual names
// and addresses instead.

/* An example master zone
zone "example.net" {
    type master;
    file "master/example.net";
};
*/

/* An example dynamic zone
key "exampleorgkey" {
    algorithm hmac-md5;
    secret "sf87HJqjkqh8ac87a02lla==";
};
zone "example.org" {
    type master;
    allow-update {
        key "exampleorgkey";
    };
    file "dynamic/example.org";
};
*/

/* Examples of forward and reverse slave zones
zone "example.com" {
    type slave;
    file "slave/example.com";
    masters {
        192.168.1.1;
    };
};
zone "1.168.192.in-addr.arpa" {
    type slave;
    file "slave/1.168.192.in-addr.arpa";
    masters {
        192.168.1.1;
    };
};
*/

In named.conf, these are examples of slave entries for a forward and reverse zone.

For each new zone served, a new zone entry must be added to named.conf.

For example, the simplest zone entry for example.org can look like:

zone "example.org" {
    type master;
    file "master/example.org";
};

The zone is a master, as indicated by the type statement, holding its zone information in /etc/namedb/master/example.org indicated by the file statement.

zone "example.org" {
    type slave;
    file "slave/example.org";
};

In the slave case, the zone information is transferred from the master name server for the particular zone, and saved in the file specified. If and when the master server dies or is unreachable, the slave name server will have the transferred zone information and will be able to serve it.


29.6.6.3 Zone Files

An example master zone file for example.org (existing within /etc/namedb/master/example.org) is as follows:

$TTL 3600        ; 1 hour
example.org.    IN      SOA      ns1.example.org. admin.example.org. (
                                2006051501      ; Serial
                                10800           ; Refresh
                                3600            ; Retry
                                604800          ; Expire
                                86400           ; Minimum TTL
                        )

; DNS Servers
                IN      NS      ns1.example.org.
                IN      NS      ns2.example.org.

; MX Records
                IN      MX 10   mx.example.org.
                IN      MX 20   mail.example.org.

                IN      A       192.168.1.1

; Machine Names
localhost       IN      A       127.0.0.1
ns1             IN      A       192.168.1.2
ns2             IN      A       192.168.1.3
mx              IN      A       192.168.1.4
mail            IN      A       192.168.1.5

; Aliases
www             IN      CNAME   @

Note that every hostname ending in a “.” is an exact hostname, whereas everything without a trailing “.” is referenced to the origin. For example, www is translated into www.origin. In our fictitious zone file, our origin is example.org., so www would translate to www.example.org.

The format of a zone file follows:

recordname      IN recordtype   value

The most commonly used DNS records:

SOA

start of zone authority

NS

an authoritative name server

A

a host address

CNAME

the canonical name for an alias

MX

mail exchanger

PTR

a domain name pointer (used in reverse DNS)

example.org. IN SOA ns1.example.org. admin.example.org. (
                        2006051501      ; Serial
                        10800           ; Refresh after 3 hours
                        3600            ; Retry after 1 hour
                        604800          ; Expire after 1 week
                        86400 )         ; Minimum TTL of 1 day
example.org.

the domain name, also the origin for this zone file.

ns1.example.org.

the primary/authoritative name server for this zone.

admin.example.org.

the responsible person for this zone, email address with “@” replaced. (<admin@example.org> becomes admin.example.org)

2006051501

the serial number of the file. This must be incremented each time the zone file is modified. Nowadays, many admins prefer a yyyymmddrr format for the serial number. 2006051501 would mean last modified 05/15/2006, the latter 01 being the first time the zone file has been modified this day. The serial number is important as it alerts slave name servers for a zone when it is updated.

       IN NS           ns1.example.org.

This is an NS entry. Every name server that is going to reply authoritatively for the zone must have one of these entries.

localhost       IN      A       127.0.0.1
ns1             IN      A       192.168.1.2
ns2             IN      A       192.168.1.3
mx              IN      A       192.168.1.4
mail            IN      A       192.168.1.5

The A record indicates machine names. As seen above, ns1.example.org would resolve to 192.168.1.2.

                IN      A       192.168.1.1

This line assigns IP address 192.168.1.1 to the current origin, in this case example.org.

www             IN CNAME        @

The canonical name record is usually used for giving aliases to a machine. In the example, www is aliased to the “master” machine which name equals to domain name example.org (192.168.1.1). CNAMEs can be used to provide alias hostnames, or round robin one hostname among multiple machines.

               IN MX   10      mail.example.org.

The MX record indicates which mail servers are responsible for handling incoming mail for the zone. mail.example.org is the hostname of the mail server, and 10 being the priority of that mail server.

One can have several mail servers, with priorities of 10, 20 and so on. A mail server attempting to deliver to example.org would first try the highest priority MX (the record with the lowest priority number), then the second highest, etc, until the mail can be properly delivered.

For in-addr.arpa zone files (reverse DNS), the same format is used, except with PTR entries instead of A or CNAME.

$TTL 3600

1.168.192.in-addr.arpa. IN SOA ns1.example.org. admin.example.org. (
                        2006051501      ; Serial
                        10800           ; Refresh
                        3600            ; Retry
                        604800          ; Expire
                        3600 )          ; Minimum

        IN      NS      ns1.example.org.
        IN      NS      ns2.example.org.

1       IN      PTR     example.org.
2       IN      PTR     ns1.example.org.
3       IN      PTR     ns2.example.org.
4       IN      PTR     mx.example.org.
5       IN      PTR     mail.example.org.

This file gives the proper IP address to hostname mappings of our above fictitious domain.


29.6.7 Caching Name Server

A caching name server is a name server that is not authoritative for any zones. It simply asks queries of its own, and remembers them for later use. To set one up, just configure the name server as usual, omitting any inclusions of zones.


29.6.8 Security

Although BIND is the most common implementation of DNS, there is always the issue of security. Possible and exploitable security holes are sometimes found.

While FreeBSD automatically drops named into a chroot(8) environment; there are several other security mechanisms in place which could help to lure off possible DNS service attacks.

It is always good idea to read CERT’s security advisories and to subscribe to the FreeBSD security notifications mailing list to stay up to date with the current Internet and FreeBSD security issues.

Tip: If a problem arises, keeping sources up to date and having a fresh build of named would not hurt.


29.6.9 Further Reading

BIND/named manual pages: rndc(8) named(8) named.conf(5)

  • Official ISC BIND Page

  • Official ISC BIND Forum

  • BIND9 FAQ

  • O’Reilly DNS and BIND 5th Edition

  • RFC1034 - Domain Names - Concepts and Facilities

  • RFC1035 - Domain Names - Implementation and Specification


29.7 Apache HTTP Server

Contributed by Murray Stokely.


29.7.1 Overview

FreeBSD is used to run some of the busiest web sites in the world. The majority of web servers on the Internet are using the Apache HTTP Server. Apache software packages should be included on your FreeBSD installation media. If you did not install Apache when you first installed FreeBSD, then you can install it from the www/apache13 or www/apache22 port.

Once Apache has been installed successfully, it must be configured.

Note: This section covers version 1.3.X of the Apache HTTP Server as that is the most widely used version for FreeBSD. Apache 2.X introduces many new technologies but they are not discussed here. For more information about Apache 2.X, please see http://httpd.apache.org/.


29.7.2 Configuration

The main Apache HTTP Server configuration file is installed as /usr/local/etc/apache/httpd.conf on FreeBSD. This file is a typical UNIX text configuration file with comment lines beginning with the # character. A comprehensive description of all possible configuration options is outside the scope of this book, so only the most frequently modified directives will be described here.

ServerRoot "/usr/local"

This specifies the default directory hierarchy for the Apache installation. Binaries are stored in the bin and sbin subdirectories of the server root, and configuration files are stored in etc/apache.

ServerAdmin you@your.address

The address to which problems with the server should be emailed. This address appears on some server-generated pages, such as error documents.

ServerName www.example.com

ServerName allows you to set a host name which is sent back to clients for your server if it is different to the one that the host is configured with (i.e., use www instead of the host’s real name).

DocumentRoot "/usr/local/www/data"

DocumentRoot: The directory out of which you will serve your documents. By default, all requests are taken from this directory, but symbolic links and aliases may be used to point to other locations.

It is always a good idea to make backup copies of your Apache configuration file before making changes. Once you are satisfied with your initial configuration you are ready to start running Apache.


29.7.3 Running Apache

Apache does not run from the inetd super server as many other network servers do. It is configured to run standalone for better performance for incoming HTTP requests from client web browsers. A shell script wrapper is included to make starting, stopping, and restarting the server as simple as possible. To start up Apache for the first time, just run:

# /usr/local/sbin/apachectl start

You can stop the server at any time by typing:

# /usr/local/sbin/apachectl stop

After making changes to the configuration file for any reason, you will need to restart the server:

# /usr/local/sbin/apachectl restart

To restart Apache without aborting current connections, run:

# /usr/local/sbin/apachectl graceful

Additional information available at apachectl(8) manual page.

To launch Apache at system startup, add the following line to /etc/rc.conf:

apache_enable="YES"

or for Apache 2.2:

apache22_enable="YES"

If you would like to supply additional command line options for the Apache httpd program started at system boot, you may specify them with an additional line in rc.conf:

apache_flags=""

Now that the web server is running, you can view your web site by pointing a web browser to http://localhost/. The default web page that is displayed is /usr/local/www/data/index.html.


29.7.4 Virtual Hosting

Apache supports two different types of Virtual Hosting. The first method is Name-based Virtual Hosting. Name-based virtual hosting uses the clients HTTP/1.1 headers to figure out the hostname. This allows many different domains to share the same IP address.

To setup Apache to use Name-based Virtual Hosting add an entry like the following to your httpd.conf:

NameVirtualHost *

If your webserver was named www.domain.tld and you wanted to setup a virtual domain for www.someotherdomain.tld then you would add the following entries to httpd.conf:

<VirtualHost *>
ServerName www.domain.tld
DocumentRoot /www/domain.tld
</VirtualHost>

<VirtualHost *>
ServerName www.someotherdomain.tld
DocumentRoot /www/someotherdomain.tld
</VirtualHost>

Replace the addresses with the addresses you want to use and the path to the documents with what you are using.

For more information about setting up virtual hosts, please consult the official Apache documentation at: http://httpd.apache.org/docs/vhosts/.


29.7.5 Apache Modules

There are many different Apache modules available to add functionality to the basic server. The FreeBSD Ports Collection provides an easy way to install Apache together with some of the more popular add-on modules.


29.7.5.1 mod_ssl

The mod_ssl module uses the OpenSSL library to provide strong cryptography via the Secure Sockets Layer (SSL v2/v3) and Transport Layer Security (TLS v1) protocols. This module provides everything necessary to request a signed certificate from a trusted certificate signing authority so that you can run a secure web server on FreeBSD.

If you have not yet installed Apache, then a version of Apache 1.3.X that includes mod_ssl may be installed with the www/apache13-modssl port. SSL support is also available for Apache 2.X in the www/apache22 port, where it is enabled by default.


29.7.5.2 Language Bindings

There are Apache modules for most major scripting languages. These modules typically make it possible to write Apache modules entirely in a scripting language. They are also often used as a persistent interpreter embedded into the server that avoids the overhead of starting an external interpreter and the startup-time penalty for dynamic websites, as described in the next section.


29.7.6 Dynamic Websites

In the last decade, more businesses have turned to the Internet in order to enhance their revenue and increase exposure. This has also increased the need for interactive web content. While some companies, such as Microsoft, have introduced solutions into their proprietary products, the open source community answered the call. Modern options for dynamic web content include Django, Ruby on Rails, mod_perl, and mod_php.


29.7.6.1 Django

Django is a BSD licensed framework designed to allow developers to write high performance, elegant web applications quickly. It provides an object-relational mapper so that data types are developed as Python objects, and a rich dynamic database-access API is provided for those objects without the developer ever having to write SQL. It also provides an extensible template system so that the logic of the application is separated from the HTML presentation.

Django depends on mod_python, Apache, and an SQL database engine of your choice. The FreeBSD Port will install all of these pre-requisites for you with the appropriate flags.

Example 29-3. Installing Django with Apache2, mod_python3, and PostgreSQL

# cd /usr/ports/www/py-django; make all install clean -DWITH_MOD_PYTHON3 -DWITH_POSTGRESQL

Once Django and these pre-requisites are installed, you will need to create a Django project directory and then configure Apache to use the embedded Python interpreter to call your application for specific URLs on your site.

Example 29-4. Apache Configuration for Django/mod_python

You will need to add a line to the apache httpd.conf file to configure Apache to pass requests for certain URLs to your web application:

<Location "/">
    SetHandler python-program
    PythonPath "['/dir/to/your/django/packages/'] + sys.path"
    PythonHandler django.core.handlers.modpython
    SetEnv DJANGO_SETTINGS_MODULE mysite.settings
    PythonAutoReload On
    PythonDebug On
</Location>

29.7.6.2 Ruby on Rails

Ruby on Rails is another open source web framework that provides a full development stack and is optimized to make web developers more productive and capable of writing powerful applications quickly. It can be installed easily from the ports system.

# cd /usr/ports/www/rubygem-rails; make all install clean

29.7.6.3 mod_perl

The Apache/Perl integration project brings together the full power of the Perl programming language and the Apache HTTP Server. With the mod_perl module it is possible to write Apache modules entirely in Perl. In addition, the persistent interpreter embedded in the server avoids the overhead of starting an external interpreter and the penalty of Perl start-up time.

mod_perl is available a few different ways. To use mod_perl remember that mod_perl 1.0 only works with Apache 1.3 and mod_perl 2.0 only works with Apache 2.X. mod_perl 1.0 is available in www/mod_perl and a statically compiled version is available in www/apache13-modperl. mod_perl 2.0 is available in www/mod_perl2.


29.7.6.4 mod_php

Written by Tom Rhodes.

PHP, also known as “PHP: Hypertext Preprocessor” is a general-purpose scripting language that is especially suited for Web development. Capable of being embedded into HTML its syntax draws upon C, Java, and Perl with the intention of allowing web developers to write dynamically generated webpages quickly.

To gain support for PHP5 for the Apache web server, begin by installing the lang/php5 port.

If the lang/php5 port is being installed for the first time, available OPTIONS will be displayed automatically. If a menu is not displayed, i.e. because the lang/php5 port has been installed some time in the past, it is always possible to bring the options dialog up again by running:

# make config

in the port directory.

In the options dialog, check the APACHE option to build mod_php5 as a loadable module for the Apache web server.

Note: A lot of sites are still using PHP4 for various reasons (i.e. compatibility issues or already deployed web applications). If the mod_php4 is needed instead of mod_php5, then please use the lang/php4 port. The lang/php4 port supports many of the configuration and build-time options of the lang/php5 port.

This will install and configure the modules required to support dynamic PHP applications. Check to ensure the following sections have been added to /usr/local/etc/apache/httpd.conf:

LoadModule php5_module        libexec/apache/libphp5.so
AddModule mod_php5.c
    <IfModule mod_php5.c>
        DirectoryIndex index.php index.html
    </IfModule>
    <IfModule mod_php5.c>
        AddType application/x-httpd-php .php
        AddType application/x-httpd-php-source .phps
    </IfModule>

Once completed, a simple call to the apachectl command for a graceful restart is needed to load the PHP module:

# apachectl graceful

For future upgrades of PHP, the make config command will not be required; the selected OPTIONS are saved automatically by the FreeBSD Ports framework.

The PHP support in FreeBSD is extremely modular so the base install is very limited. It is very easy to add support using the lang/php5-extensions port. This port provides a menu driven interface to PHP extension installation. Alternatively, individual extensions can be installed using the appropriate port.

For instance, to add support for the MySQL database server to PHP5, simply install the databases/php5-mysql port.

After installing an extension, the Apache server must be reloaded to pick up the new configuration changes:

# apachectl graceful

29.8 File Transfer Protocol (FTP)

Contributed by Murray Stokely.


29.8.1 Overview

The File Transfer Protocol (FTP) provides users with a simple way to transfer files to and from an FTP server. FreeBSD includes FTP server software, ftpd, in the base system. This makes setting up and administering an FTP server on FreeBSD very straightforward.


29.8.2 Configuration

The most important configuration step is deciding which accounts will be allowed access to the FTP server. A normal FreeBSD system has a number of system accounts used for various daemons, but unknown users should not be allowed to log in with these accounts. The /etc/ftpusers file is a list of users disallowed any FTP access. By default, it includes the aforementioned system accounts, but it is possible to add specific users here that should not be allowed access to FTP.

You may want to restrict the access of some users without preventing them completely from using FTP. This can be accomplished with the /etc/ftpchroot file. This file lists users and groups subject to FTP access restrictions. The ftpchroot(5) manual page has all of the details so it will not be described in detail here.

If you would like to enable anonymous FTP access to your server, then you must create a user named ftp on your FreeBSD system. Users will then be able to log on to your FTP server with a username of ftp or anonymous and with any password (by convention an email address for the user should be used as the password). The FTP server will call chroot(2) when an anonymous user logs in, to restrict access to only the home directory of the ftp user.

There are two text files that specify welcome messages to be displayed to FTP clients. The contents of the file /etc/ftpwelcome will be displayed to users before they reach the login prompt. After a successful login, the contents of the file /etc/ftpmotd will be displayed. Note that the path to this file is relative to the login environment, so the file ~ftp/etc/ftpmotd would be displayed for anonymous users.

Once the FTP server has been configured properly, it must be enabled in /etc/inetd.conf. All that is required here is to remove the comment symbol “#” from in front of the existing ftpd line :

ftp    stream  tcp nowait  root    /usr/libexec/ftpd   ftpd -l

As explained in Example 29-1, the inetd configuration must be reloaded after this configuration file is changed. Please refer to Section 29.2.2 for details on enabling inetd on your system.

Alternatively, ftpd can also be started as a stand-alone server. In this case, it is sufficient to set the appropriate variable in /etc/rc.conf:

ftpd_enable="YES"

After setting the above variable, the stand-alone server will be started at the next reboot, or it can be started manually by executing the following command as root:

# /etc/rc.d/ftpd start

You can now log on to your FTP server by typing:

% ftp localhost

29.8.3 Maintaining

The ftpd daemon uses syslog(3) to log messages. By default, the system log daemon will put messages related to FTP in the /var/log/xferlog file. The location of the FTP log can be modified by changing the following line in /etc/syslog.conf:

ftp.info      /var/log/xferlog

Be aware of the potential problems involved with running an anonymous FTP server. In particular, you should think twice about allowing anonymous users to upload files. You may find that your FTP site becomes a forum for the trade of unlicensed commercial software or worse. If you do need to allow anonymous FTP uploads, then you should set up the permissions so that these files can not be read by other anonymous users until they have been reviewed.


29.9 File and Print Services for Microsoft Windows clients (Samba)

Contributed by Murray Stokely.


29.9.1 Overview

Samba is a popular open source software package that provides file and print services for Microsoft Windows clients. Such clients can connect to and use FreeBSD filespace as if it was a local disk drive, or FreeBSD printers as if they were local printers.

Samba software packages should be included on your FreeBSD installation media. If you did not install Samba when you first installed FreeBSD, then you can install it from the net/samba3 port or package.


29.9.2 Configuration

A default Samba configuration file is installed as /usr/local/share/examples/samba/smb.conf.default. This file must be copied to /usr/local/etc/smb.conf and customized before Samba can be used.

The smb.conf file contains runtime configuration information for Samba, such as definitions of the printers and “file system shares” that you would like to share with Windows clients. The Samba package includes a web based tool called swat which provides a simple way of configuring the smb.conf file.


29.9.2.1 Using the Samba Web Administration Tool (SWAT)

The Samba Web Administration Tool (SWAT) runs as a daemon from inetd. Therefore, the following line in /etc/inetd.conf should be uncommented before swat can be used to configure Samba:

swat   stream  tcp     nowait/400      root    /usr/local/sbin/swat    swat

As explained in Example 29-1, the inetd configuration must be reloaded after this configuration file is changed.

Once swat has been enabled in inetd.conf, you can use a browser to connect to http://localhost:901. You will first have to log on with the system root account.

Once you have successfully logged on to the main Samba configuration page, you can browse the system documentation, or begin by clicking on the Globals tab. The Globals section corresponds to the variables that are set in the [global] section of /usr/local/etc/smb.conf.


29.9.2.2 Global Settings

Whether you are using swat or editing /usr/local/etc/smb.conf directly, the first directives you are likely to encounter when configuring Samba are:

workgroup

NT Domain-Name or Workgroup-Name for the computers that will be accessing this server.

netbios name

This sets the NetBIOS name by which a Samba server is known. By default it is the same as the first component of the host’s DNS name.

server string

This sets the string that will be displayed with the net view command and some other networking tools that seek to display descriptive text about the server.


29.9.2.3 Security Settings

Two of the most important settings in /usr/local/etc/smb.conf are the security model chosen, and the backend password format for client users. The following directives control these options:

security

The two most common options here are security = share and security = user. If your clients use usernames that are the same as their usernames on your FreeBSD machine then you will want to use user level security. This is the default security policy and it requires clients to first log on before they can access shared resources.

In share level security, client do not need to log onto the server with a valid username and password before attempting to connect to a shared resource. This was the default security model for older versions of Samba.

passdb backend

Samba has several different backend authentication models. You can authenticate clients with LDAP, NIS+, a SQL database, or a modified password file. The default authentication method is smbpasswd, and that is all that will be covered here.

Assuming that the default smbpasswd backend is used, the /usr/local/private/smbpasswd file must be created to allow Samba to authenticate clients. If you would like to give your UNIX user accounts access from Windows clients, use the following command:

# smbpasswd -a username

Note: Since Samba 3.0.23c, the actual directory for authentication files is /usr/local/etc/samba. The recommended backend is now tdbsam, and the following command should be used to add user accounts:

# pdbedit -a -u username

Please see the Official Samba HOWTO for additional information about configuration options. With the basics outlined here, you should have everything you need to start running Samba.


29.9.3 Starting Samba

The net/samba3 port adds a new startup script, which can be used to control Samba. To enable this script, so that it can be used for example to start, stop or restart Samba, add the following line to the /etc/rc.conf file:

samba_enable="YES"

Or, for fine grain control:

nmbd_enable="YES"
smbd_enable="YES"

Note: This will also configure Samba to automatically start at system boot time.

It is possible then to start Samba at any time by typing:

# /usr/local/etc/rc.d/samba start
Starting SAMBA: removing stale tdbs :
Starting nmbd.
Starting smbd.

Please refer to Section 11.7 for more information about using rc scripts.

Samba actually consists of three separate daemons. You should see that both the nmbd and smbd daemons are started by the samba script. If you enabled winbind name resolution services in smb.conf, then you will also see that the winbindd daemon is started.

You can stop Samba at any time by typing :

# /usr/local/etc/rc.d/samba stop

Samba is a complex software suite with functionality that allows broad integration with Microsoft Windows networks. For more information about functionality beyond the basic installation described here, please see http://www.samba.org.


29.10 Clock Synchronization with NTP

Contributed by Tom Hukins.


29.10.1 Overview

Over time, a computer’s clock is prone to drift. The Network Time Protocol (NTP) is one way to ensure your clock stays accurate.

Many Internet services rely on, or greatly benefit from, computers’ clocks being accurate. For example, a web server may receive requests to send a file if it has been modified since a certain time. In a local area network environment, it is essential that computers sharing files from the same file server have synchronized clocks so that file timestamps stay consistent. Services such as cron(8) also rely on an accurate system clock to run commands at the specified times.

FreeBSD ships with the ntpd(8) NTP server which can be used to query other NTP servers to set the clock on your machine or provide time services to others.


29.10.2 Choosing Appropriate NTP Servers

In order to synchronize your clock, you will need to find one or more NTP servers to use. Your network administrator or ISP may have set up an NTP server for this purpose–check their documentation to see if this is the case. There is an online list of publicly accessible NTP servers which you can use to find an NTP server near to you. Make sure you are aware of the policy for any servers you choose, and ask for permission if required.

Choosing several unconnected NTP servers is a good idea in case one of the servers you are using becomes unreachable or its clock is unreliable. ntpd(8) uses the responses it receives from other servers intelligently–it will favor unreliable servers less than reliable ones.


29.10.3 Configuring Your Machine

29.10.3.1 Basic Configuration

If you only wish to synchronize your clock when the machine boots up, you can use ntpdate(8). This may be appropriate for some desktop machines which are frequently rebooted and only require infrequent synchronization, but most machines should run ntpd(8).

Using ntpdate(8) at boot time is also a good idea for machines that run ntpd(8). The ntpd(8) program changes the clock gradually, whereas ntpdate(8) sets the clock, no matter how great the difference between a machine’s current clock setting and the correct time.

To enable ntpdate(8) at boot time, add ntpdate_enable="YES" to /etc/rc.conf. You will also need to specify all servers you wish to synchronize with and any flags to be passed to ntpdate(8) in ntpdate_flags.


29.10.3.2 General Configuration

NTP is configured by the /etc/ntp.conf file in the format described in ntp.conf(5). Here is a simple example:

server ntplocal.example.com prefer
server timeserver.example.org
server ntp2a.example.net

driftfile /var/db/ntp.drift

The server option specifies which servers are to be used, with one server listed on each line. If a server is specified with the prefer argument, as with ntplocal.example.com, that server is preferred over other servers. A response from a preferred server will be discarded if it differs significantly from other servers’ responses, otherwise it will be used without any consideration to other responses. The prefer argument is normally used for NTP servers that are known to be highly accurate, such as those with special time monitoring hardware.

The driftfile option specifies which file is used to store the system clock’s frequency offset. The ntpd(8) program uses this to automatically compensate for the clock’s natural drift, allowing it to maintain a reasonably correct setting even if it is cut off from all external time sources for a period of time.

The driftfile option specifies which file is used to store information about previous responses from the NTP servers you are using. This file contains internal information for NTP. It should not be modified by any other process.


29.10.3.3 Controlling Access to Your Server

By default, your NTP server will be accessible to all hosts on the Internet. The restrict option in /etc/ntp.conf allows you to control which machines can access your server.

If you want to deny all machines from accessing your NTP server, add the following line to /etc/ntp.conf:

restrict default ignore

Note: This will also prevent access from your server to any servers listed in your local configuration. If you need to synchronise your NTP server with an external NTP server you should allow the specific server. See the ntp.conf(5) manual for more information.

If you only want to allow machines within your own network to synchronize their clocks with your server, but ensure they are not allowed to configure the server or used as peers to synchronize against, add

restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

instead, where 192.168.1.0 is an IP address on your network and 255.255.255.0 is your network’s netmask.

/etc/ntp.conf can contain multiple restrict options. For more details, see the Access Control Support subsection of ntp.conf(5).


29.10.4 Running the NTP Server

To ensure the NTP server is started at boot time, add the line ntpd_enable="YES" to /etc/rc.conf. If you wish to pass additional flags to ntpd(8), edit the ntpd_flags parameter in /etc/rc.conf.

To start the server without rebooting your machine, run ntpd being sure to specify any additional parameters from ntpd_flags in /etc/rc.conf. For example:

# ntpd -p /var/run/ntpd.pid

29.10.5 Using ntpd with a Temporary Internet Connection

The ntpd(8) program does not need a permanent connection to the Internet to function properly. However, if you have a temporary connection that is configured to dial out on demand, it is a good idea to prevent NTP traffic from triggering a dial out or keeping the connection alive. If you are using user PPP, you can use filter directives in /etc/ppp/ppp.conf. For example:

 set filter dial 0 deny udp src eq 123
 # Prevent NTP traffic from initiating dial out
 set filter dial 1 permit 0 0
 set filter alive 0 deny udp src eq 123
 # Prevent incoming NTP traffic from keeping the connection open
 set filter alive 1 deny udp dst eq 123
 # Prevent outgoing NTP traffic from keeping the connection open
 set filter alive 2 permit 0/0 0/0

For more details see the PACKET FILTERING section in ppp(8) and the examples in /usr/share/examples/ppp/.

Note: Some Internet access providers block low-numbered ports, preventing NTP from functioning since replies never reach your machine.


29.10.6 Further Information

Documentation for the NTP server can be found in /usr/share/doc/ntp/ in HTML format.


29.11 Remote Host Logging with syslogd

Contributed by Tom Rhodes.

Interacting with system logs is a crucial aspect of both security and system administration. Monitoring the log files of multiple hosts can get very unwieldy when these hosts are distributed across medium or large networks, or when they are parts of various different types of networks. In these cases, configuring remote logging may make the whole process a lot more comfortable.

Centralized logging to a specific logging host can reduce some of the administrative burden of log file administration. Log file aggregation, merging and rotation can be configured in one location, using the native tools of FreeBSD, such as syslogd(8) and newsyslog(8). In the following example configuration, host A, named logserv.example.com, will collect logging information for the local network. Host B, named logclient.example.com will pass logging information to the server system. In live configurations, both hosts require proper forward and reverse DNS or entries in /etc/hosts. Otherwise, data will be rejected by the server.


29.11.1 Log Server Configuration

Log servers are machines configured to accept logging information from remote hosts. In most cases this is to ease configuration, in other cases it may just be a better administration move. Regardless of reason, there are a few requirements before continuing.

A properly configured logging server has met the following minimal requirements:

  • The firewall ruleset allows for UDP to be passed on port 514 on both the client and server;

  • syslogd has been configured to accept remote messages from client machines;

  • The syslogd server and all client machines must have valid entries for both forward and reverse DNS, or be properly configured in /etc/hosts.

To configure the log server, the client must be listed in /etc/syslog.conf, and the logging facility must be specified:

+logclient.example.com
*.*     /var/log/logclient.log

Note: More information on various supported and available facilities may be found in the syslog.conf(5) manual page.

Once added, all facility messages will be logged to the file specified previously, /var/log/logclient.log.

The server machine must also have the following listing placed inside /etc/rc.conf:

syslogd_enable="YES"
syslogd_flags="-a logclient.example.com -vv"

The first option will enable the syslogd daemon on boot up, and the second option allows data from the specified client to be accepted on this server. The latter part, using -vv, will increase the verbosity of logged messages. This is extremely useful for tweaking facilities as administrators are able to see what type of messages are being logged under which facility.

Multiple -a options may be specified to allow logging from multiple clients. IP addresses and whole netblocks may also be specified, see the syslog(3) manual page for a full list of possible options.

Finally, the log file should be created. The method used does not matter, but touch(1) works great for situations such as this:

# touch /var/log/logclient.log

At this point, the syslogd daemon should be restarted and verified:

# /etc/rc.d/syslogd restart
# pgrep syslog

If a PID is returned, the server has been restarted successfully, and client configuration may begin. If the server has not restarted, consult the /var/log/messages log for any output.


29.11.2 Log Client Configuration

A logging client is a machine which sends log information to a logging server in addition to keeping local copies.

Similar to log servers, clients must also meet a few minimum requirements:

  • syslogd(8) must be configured to send messages of specific types to a log server, which must accept them;

  • The firewall must allow UDP packets through on port 514;

  • Both forward and reverse DNS must be configured or have proper entries in the /etc/hosts.

Client configuration is a bit more relaxed when compared to that of the servers. The client machine must have the following listing placed inside /etc/rc.conf:

syslogd_enable="YES"
syslogd_flags="-s -vv"

As before, these entries will enable the syslogd daemon on boot up, and increases the verbosity of logged messages. The -s option prevents logs from being accepted by this client from other hosts.

Facilities describe the system part for which a message is generated. For an example, ftp and ipfw are both facilities. When log messages are generated for those two services, they will normally include those two utilities in any log messages. Facilities are accompanied with a priority or level, which is used to mark how important a log message is. The most common will be the warning and info. Please refer to the syslog(3) manual page for a full list of available facilities and priorities.

The logging server must be defined in the client’s /etc/syslog.conf. In this instance, the @ symbol is used to send logging data to a remote server and would look similar to the following entry:

*.*        @logserv.example.com

Once added, syslogd must be restarted for the changes to take effect:

# /etc/rc.d/syslogd restart

To test that log messages are being sent across the network, use logger(1) on the client to send a message to syslogd:

# logger "Test message from logclient"

This message should now exist both in /var/log/messages on the client, and /var/log/logclient.log on the log server.


29.11.3 Debugging Log Servers

In certain cases, debugging may be required if messages are not being received on the log server. There are several reasons this may occur; however, the most common two are network connection issues and DNS issues. To test these cases, ensure both hosts are able to reach one another using the hostname specified in /etc/rc.conf. If this appears to be working properly, an alternation to the syslogd_flags option in /etc/rc.conf will be required.

In the following example, /var/log/logclient.log is empty, and the /var/log/messages files indicate no reason for the failure. To increase debugging output, change the syslogd_flags option to look like the following example, and issue a restart:

syslogd_flags="-d -a logclien.example.com -vv"
# /etc/rc.d/syslogd restart

Debugging data similar to the following will flash on the screen immediately after the restart:

logmsg: pri 56, flags 4, from logserv.example.com, msg syslogd: restart
syslogd: restarted
logmsg: pri 6, flags 4, from logserv.example.com, msg syslogd: kernel boot file is /boot/kernel/kernel
Logging to FILE /var/log/messages
syslogd: kernel boot file is /boot/kernel/kernel
cvthname(192.168.1.10)
validate: dgram from IP 192.168.1.10, port 514, name logclient.example.com;
rejected in rule 0 due to name mismatch.

It appears obvious the messages are being rejected due to a name mismatch. After reviewing the configuration bit by bit, it appears a typo in the following /etc/rc.conf line has an issue:

syslogd_flags="-d -a logclien.example.com -vv"

The line should contain logclient, not logclien. After the proper alterations are made, a restart is issued with expected results:

# /etc/rc.d/syslogd restart
logmsg: pri 56, flags 4, from logserv.example.com, msg syslogd: restart
syslogd: restarted
logmsg: pri 6, flags 4, from logserv.example.com, msg syslogd: kernel boot file is /boot/kernel/kernel
syslogd: kernel boot file is /boot/kernel/kernel
logmsg: pri 166, flags 17, from logserv.example.com,
msg Dec 10 20:55:02 <syslog.err> logserv.example.com syslogd: exiting on signal 2
cvthname(192.168.1.10)
validate: dgram from IP 192.168.1.10, port 514, name logclient.example.com;
accepted in rule 0.
logmsg: pri 15, flags 0, from logclient.example.com, msg Dec 11 02:01:28 trhodes: Test message 2
Logging to FILE /var/log/logclient.log
Logging to FILE /var/log/messages

At this point, the messages are being properly received and placed in the correct file.


29.11.4 Security Considerations

As with any network service, security requirements should be considered before implementing this configuration. At times, log files may contain sensitive data about services enabled on the local host, user accounts, and configuration data. Network data sent from the client to the server will not be encrypted nor password protected. If a need for encryption exists, it might be possible to use security/stunnel, which will transmit data over an encrypted tunnel.

Local security is also an issue. Log files are not encrypted during use or after log rotation. Local users may access these files to gain additional insight on system configuration. In those cases, setting proper permissions on these files will be critical. The newsyslog(8) utility supports setting permissions on newly created and rotated log files. Setting log files to mode 600 should prevent any unwanted snooping by local users.

Tags: account, analog, Apache, backup, bsd, cron, database, domain, domain name, email, forwarder, freebsd, FreeBSD Handbook, ftp, inetd, manage, mysql, netbsd, openbsd, password, php, pop, postgresql, raw, software, ssl, virus

Related posts

FreeBSD Handbook , , , , , , , , , , , , , , , , , , , , , , , , ,

FreeBSD Handbook - Chapter 24 Updating and Upgrading FreeBSD

January 6th, 2009

Restructured, reorganized, and parts updated by Jim Mock. Original work by Jordan Hubbard, Poul-Henning Kamp, John Polstra, and Nik Clayton.

24.1 Synopsis

FreeBSD is under constant development between releases. Some people prefer to use the officially released versions, while others prefer to keep in sync with the latest developments. However, even official releases are often updated with security and other critical fixes. Regardless of the version used, FreeBSD provides all necessary tools to keep your system updated, and also allows for easy upgrades between versions. This chapter will help you decide if you want to track the development system, or stick with one of the released versions. The basic tools for keeping your system up to date are also presented.

After reading this chapter, you will know:

  • What utilities may be used to update the system and the Ports Collection.

  • How to keep your system up to date with freebsd-update, CVSup, CVS, or CTM.

  • How to compare the state of an installed system against a known pristine copy.

  • The difference between the two development branches: FreeBSD-STABLE and FreeBSD-CURRENT.

  • How to rebuild and reinstall the entire base system with make buildworld (etc).

Before reading this chapter, you should:

  • Properly set up your network connection (Chapter 31).

  • Know how to install additional third-party software (Chapter 4).

Note: Throughout this chapter, the cvsup command is used to obtain and update FreeBSD sources. To use it, you will need to install a port or package like net/cvsup-without-gui. If you are using FreeBSD 6.2-RELEASE or later, you may wish to substitute this with csup(1), which is now part of the base system.


24.2 FreeBSD Update

Written by Tom Rhodes. Based on notes provided by Colin Percival.

Applying security patches is an important part of maintaining computer software, especially the operating system. For the longest time on FreeBSD this process was not an easy one. Patches had to be applied to the source code, the code rebuilt into binaries, and then the binaries had to be re-installed.

This is no longer the case as FreeBSD now includes a utility simply called freebsd-update. This utility provides two separate functions. First, it allows for binary security and errata updates to be applied to the FreeBSD base system without the build and install requirements. Second, the utility supports minor and major release upgrades.

Note: Binary updates are available for all architectures and releases currently supported by the security team; however, some features, such as the FreeBSD operating system upgrades, require the latest release of freebsd-update(8) and FreeBSD 6.3 or greater. Before updating to a new release, the current release announcements should be reviewed as they may contain important information pertinent to the desired release. These announcements may be viewed at the following link: http://www.FreeBSD.org/releases/.

If a crontab utilizing the features of freebsd-update exists, it must be disabled before the following operation is started.


24.2.1 The Configuration File

Some users may wish to tweak the configuration file, allowing better control of the process. The options are very well documented, but the following few may require a bit more explanation:

# Components of the base system which should be kept updated.
Components src world kernel

This parameter controls what parts of FreeBSD will be kept up to date. The default is to update the source code, the entire base system, and the kernel. Components are the same as those available during the install, for instance, adding "world/games" here would allow game patches to be applied. Using "src/bin" would allow the source code in src/bin to be updated.

The best option is to leave this at the default as changing it to include specific items will require the user to list every item they prefer to be updated. This could have disastrous consequences as source code and binaries may become out of sync.

# Paths which start with anything matching an entry in an IgnorePaths
# statement will be ignored.
IgnorePaths

Add paths, such as /bin or /sbin to leave these specific directories untouched during the update process. This option may be used to prevent freebsd-update from overwriting local modifications.

# Paths which start with anything matching an entry in an UpdateIfUnmodified
# statement will only be updated if the contents of the file have not been
# modified by the user (unless changes are merged; see below).
UpdateIfUnmodified /etc/ /var/ /root/ /.cshrc /.profile

Update configuration files in the specified directories only if they have not been modified. Any changes made by the user will invalidate the automatic updating of these files. There is another option, KeepModifiedMetadata, which will instruct freebsd-update to save the changes during the merge.

# When upgrading to a new FreeBSD release, files which match MergeChanges
# will have any local changes merged into the version from the new release.
MergeChanges /etc/ /var/named/etc/

List of directories with configuration files that freebsd-update should attempt merges in. The file merge process is a series of diff(1) patches similar to mergemaster(8) with fewer options, the merges are either accepted, open an editor, or freebsd-update will abort. When in doubt, backup /etc and just accept the merges. See Section 24.6.11.1 for more information about the mergemaster command.

# Directory in which to store downloaded updates and temporary
# files used by FreeBSD Update.
# WorkDir /var/db/freebsd-update

This directory is where all patches and temporary files will be placed. In cases where the user is doing a version upgrade, this location should have a least a gigabyte of disk space available.

# When upgrading between releases, should the list of Components be
# read strictly (StrictComponents yes) or merely as a list of components
# which *might* be installed of which FreeBSD Update should figure out
# which actually are installed and upgrade those (StrictComponents no)?
# StrictComponents no

When set to yes, freebsd-update will assume that the Components list is complete and will not attempt to make changes outside of the list. Effectively, freebsd-update will attempt to update every file which belongs to the Components list.


24.2.2 Security Patches

Security patches are stored on a remote machine and may be downloaded and installed using the following command:

# freebsd-update fetch
# freebsd-update install

If any kernel patches have been applied the system will need a reboot. If all went well the system should be patched and freebsd-update may be ran as a nightly cron(8) job. An entry in /etc/crontab would be sufficient to accomplish this task:

@daily                                  root    freebsd-update cron

This entry states that once every day, the freebsd-update will be ran. In this way, using the cron argument, freebsd-update will only check if updates exist. If patches exist, they will automatically be downloaded to the local disk but not applied. The root user will be sent an email so they may install them manually.

If anything went wrong, freebsd-update has the ability to roll back the last set of changes with the following command:

# freebsd-update rollback

Once complete, the system should be restarted if the kernel or any kernel modules were modified. This will allow FreeBSD to load the new binaries into memory.

The freebsd-update utility can automatically update the GENERIC kernel only. If a custom kernel is in use, it will have to be rebuilt and reinstalled after freebsd-update finishes installing the rest of the updates. However, freebsd-update will detect and update the GENERIC kernel in /boot/GENERIC (if it exists), even if it is not the current (running) kernel of the system.

Note: It is a good idea to always keep a copy of the GENERIC kernel in /boot/GENERIC. It will be helpful in diagnosing a variety of problems, and in performing version upgrades using freebsd-update as described in Section 24.2.3.

Unless the default configuration in /etc/freebsd-update.conf has been changed, freebsd-update will install the updated kernel sources along with the rest of the updates. Rebuilding and reinstalling your new custom kernel can then be performed in the usual way.

Note: The updates distributed via freebsd-update, do not always involve the kernel. It will not be necessary to rebuild your custom kernel if the kernel sources have not been modified by the execution of freebsd-update install. However, freebsd-update will always update the /usr/src/sys/conf/newvers.sh file. The current patch level (as indicated by the -p number reported by uname -r) is obtained from this file. Rebuilding your custom kernel, even if nothing else changed, will allow uname(1) to accurately report the current patch level of the system. This is particularly helpful when maintaining multiple systems, as it allows for a quick assessment of the updates installed in each one.


24.2.3 Major and Minor Upgrades

This process will remove old object files and libraries which will break most third party applications. It is recommended that all installed ports either be removed and re-installed or upgraded later using the ports-mgmt/portupgrade utility. Most users will want to run a test build using the following command:

# portupgrade -af

This will ensure everything will be re-installed correctly. Note that setting the BATCH environment variable to yes will answer yes to any prompts during this process, removing the need for manual intervention during the build process.

If a custom kernel is in use, the upgrade process is slightly more involved. A copy of the GENERIC kernel is needed, and it should be placed in /boot/GENERIC. If the GENERIC kernel is not already present in the system, it may be obtained using one of the following methods:

  • If a custom kernel has only been built once, the kernel in /boot/kernel.old is actually the GENERIC one. Simply rename this directory to /boot/GENERIC.

  • Assuming physical access to the machine is possible, a copy of the GENERIC kernel can be installed from the CD-ROM media. Insert your installation disc and use the following commands:

    # mount /cdrom
    # cd /cdrom/X.Y-RELEASE/kernels
    # ./install.sh GENERIC

    Replace X.Y-RELEASE with the actual version of the release you are using. The GENERIC kernel will be installed in /boot/GENERIC by default.

  • Failing all the above, the GENERIC kernel may be rebuilt and installed from the sources:

    # cd /usr/src
    # env DESTDIR=/boot/GENERIC make kernel
    # mv /boot/GENERIC/boot/kernel/* /boot/GENERIC
    # rm -rf /boot/GENERIC/boot

    For this kernel to be picked up as GENERIC by freebsd-update, the GENERIC configuration file must not have been modified in any way. It is also suggested that it is built without any other special options (preferably with an empty /etc/make.conf).

Rebooting to the GENERIC kernel is not required at this stage.

Major and minor version updates may be performed by providing freebsd-update with a release version target, for example, the following command will update to FreeBSD 6.4:

# freebsd-update -r 6.4-RELEASE upgrade

After the command has been received, freebsd-update will evaluate the configuration file and current system in an attempt to gather the information necessary to update the system. A screen listing will display what components have been detected and what components have not been detected. For example:

Looking up update.FreeBSD.org mirrors... 1 mirrors found.
Fetching metadata signature for 6.3-RELEASE from update1.FreeBSD.org... done.
Fetching metadata index... done.
Inspecting system... done.

The following components of FreeBSD seem to be installed:
kernel/smp src/base src/bin src/contrib src/crypto src/etc src/games
src/gnu src/include src/krb5 src/lib src/libexec src/release src/rescue
src/sbin src/secure src/share src/sys src/tools src/ubin src/usbin
world/base world/info world/lib32 world/manpages

The following components of FreeBSD do not seem to be installed:
kernel/generic world/catpages world/dict world/doc world/games
world/proflibs

Does this look reasonable (y/n)? y

At this point, freebsd-update will attempt to download all files required for the upgrade. In some cases, the user may be prompted with questions regarding what to install or how to proceed.

When using a custom kernel, the above step will produce a warning similar to the following:

WARNING: This system is running a "MYKERNEL" kernel, which is not a
kernel configuration distributed as part of FreeBSD 6.3-RELEASE.
This kernel will not be updated: you MUST update the kernel manually
before running "/usr/sbin/freebsd-update install"

This warning may be safely ignored at this point. The updated GENERIC kernel will be used as an intermediate step in the upgrade process.

After all patches have been downloaded to the local system, they will then be applied. This process may take a while depending on the speed and workload of the machine. Configuration files will then be merged — this part of the process requires some user intervention as a file may be merged or an editor may appear on screen for a manual merge. The results of every successful merge will be shown to the user as the process continues. A failed or ignored merge will cause the process to abort. Users may wish to make a backup of /etc and manually merge important files, such as master.passwd or group at a later time.

Note: The system is not being altered yet, all patching and merging is happening in another directory. When all patches have been applied successfully, all configuration files have been merged and it seems the process will go smoothly, the changes will need to be committed by the user.

Once this process is complete, the upgrade may be committed to disk using the following command.

# freebsd-update install

The kernel and kernel modules will be patched first. At this point the machine must be rebooted. If the system was running with a custom kernel, use the nextboot(8) command to set the kernel for the next boot to /boot/GENERIC (which was updated):

# nextboot -k GENERIC

Warning: Before rebooting with the GENERIC kernel, make sure it contains all drivers required for your system to boot properly (and connect to the network, if the machine that is being updated is accessed remotely). In particular, if the previously running custom kernel contained built-in functionality usually provided by kernel modules, make sure to temporarily load these modules into the GENERIC kernel using the /boot/loader.conf facility. You may also wish to disable non-essential services, disk and network mounts, etc. until the upgrade process is complete.

The machine should now be restarted with the updated kernel:

# shutdown -r now

Once the system has come back online, freebsd-update will need to be started again. The state of the process has been saved and thus, freebsd-update will not start from the beginning, but will remove all old shared libraries and object files. To continue to this stage, issue the following command:

# freebsd-update install

Note: Depending on whether any libraries version numbers got bumped, there may only be two install phases instead of three.

All third party software will now need to be rebuilt and re-installed. This is required as installed software may depend on libraries which have been removed during the upgrade process. The ports-mgmt/portupgrade command may be used to automate this process. The following commands may be used to begin this process:

# portupgrade -f ruby
# rm /var/db/pkg/pkgdb.db
# portupgrade -f ruby18-bdb
# rm /var/db/pkg/pkgdb.db /usr/ports/INDEX-*.db
# portupgrade -af

Once this has completed, finish the upgrade process with a final call to freebsd-update. Issue the following command to tie up all loose ends in the upgrade process:

# freebsd-update install

If the GENERIC kernel was temporarily used, this is the time to build and install a new custom kernel in the usual way.

Reboot the machine into the new FreeBSD version. The process is complete.


24.2.4 System State Comparison

The freebsd-update utility may be used to test the state of the installed FreeBSD version against a known good copy. This option evaluates the current version of system utilities, libraries, and configuration files. To begin the comparison, issue the following command:

# freebsd-update IDS >> outfile.ids

Warning: While the command name is IDS it should in no way be a replacement for an intrusion detection system such as security/snort. As freebsd-update stores data on disk, the possibility of tampering is evident. While this possibility may be reduced by using the kern.securelevel setting and storing the freebsd-update data on a read only file system when not in use, a better solution would be to compare the system against a secure disk, such as a DVD or securely stored external USB disk device.

The system will now be inspected, and a list of files along with their sha256(1) hash values, both the known value in the release and the current installed value, will be printed. This is why the output has been sent to the outfile.ids file. It scrolls by too quickly for eye comparisons, and soon it fills up the console buffer.

These lines are also extremely long, but the output format may be parsed quite easily. For instance, to obtain a list of all files different from those in the release, issue the following command:

# cat outfile.ids | awk '{ print $1 }' | more
/etc/master.passwd
/etc/motd
/etc/passwd
/etc/pf.conf

This output has been truncated, many more files exist. Some of these files have natural modifications, the /etc/passwd has been modified because users have been added to the system. In some cases, there may be other files, such as kernel modules, which differ as freebsd-update may have updated them. To exclude specific files or directories, add them to the IDSIgnorePaths option in /etc/freebsd-update.conf.

This system may be used as part of an elaborate upgrade method, aside from the previously discussed version.


24.3 Portsnap: A Ports Collection Update Tool

Written by Tom Rhodes. Based on notes provided by Colin Percival.

The base system of FreeBSD includes a utility for updating the Ports Collection too: the portsnap(8) utility. Upon execution, it will connect to a remote site, verify the secure key, and download a new copy of the Ports Collection. The key is used to verify the integrity of all downloaded files, ensuring they have not been modified in-flight. To download the latest Ports Collection files, issue the following command:

# portsnap fetch
Looking up portsnap.FreeBSD.org mirrors... 3 mirrors found.
Fetching snapshot tag from portsnap1.FreeBSD.org... done.
Fetching snapshot metadata... done.
Updating from Wed Aug  6 18:00:22 EDT 2008 to Sat Aug 30 20:24:11 EDT 2008.
Fetching 3 metadata patches.. done.
Applying metadata patches... done.
Fetching 3 metadata files... done.
Fetching 90 patches.....10....20....30....40....50....60....70....80....90. done.
Applying patches... done.
Fetching 133 new ports or files... done.

What this example shows is that portsnap(8) has found and verified several patches to the current ports data. This also indicates that the utility was run previously, if it was a first time run, the collection would have simply been downloaded.

When portsnap(8) successfully completes a fetch operation, the Ports Collection and subsequent patches exist on the local system that have passed verification. The updated files may be installed by typing:

# portsnap extract
/usr/ports/.cvsignore
/usr/ports/CHANGES
/usr/ports/COPYRIGHT
/usr/ports/GIDs
/usr/ports/KNOBS
/usr/ports/LEGAL
/usr/ports/MOVED
/usr/ports/Makefile
/usr/ports/Mk/bsd.apache.mk
/usr/ports/Mk/bsd.autotools.mk
/usr/ports/Mk/bsd.cmake.mk
...

The process is now complete, and applications may be installed or upgraded using the updated Ports Collection.

To run both processes consecutively, issue the following command:

# portsnap fetch update

24.4 Tracking a Development Branch

There are two development branches to FreeBSD: FreeBSD-CURRENT and FreeBSD-STABLE. This section will explain a bit about each and describe how to keep your system up-to-date with each respective tree. FreeBSD-CURRENT will be discussed first, then FreeBSD-STABLE.


24.4.1 Staying Current with FreeBSD

As you read this, keep in mind that FreeBSD-CURRENT is the “bleeding edge” of FreeBSD development. FreeBSD-CURRENT users are expected to have a high degree of technical skill, and should be capable of solving difficult system problems on their own. If you are new to FreeBSD, think twice before installing it.


24.4.1.1 What Is FreeBSD-CURRENT?

FreeBSD-CURRENT is the latest working sources for FreeBSD. This includes work in progress, experimental changes, and transitional mechanisms that might or might not be present in the next official release of the software. While many FreeBSD developers compile the FreeBSD-CURRENT source code daily, there are periods of time when the sources are not buildable. These problems are resolved as expeditiously as possible, but whether or not FreeBSD-CURRENT brings disaster or greatly desired functionality can be a matter of which exact moment you grabbed the source code in!


24.4.1.2 Who Needs FreeBSD-CURRENT?

FreeBSD-CURRENT is made available for 3 primary interest groups:

  1. Members of the FreeBSD community who are actively working on some part of the source tree and for whom keeping “current” is an absolute requirement.

  2. Members of the FreeBSD community who are active testers, willing to spend time solving problems in order to ensure that FreeBSD-CURRENT remains as sane as possible. These are also people who wish to make topical suggestions on changes and the general direction of FreeBSD, and submit patches to implement them.

  3. Those who merely wish to keep an eye on things, or to use the current sources for reference purposes (e.g. for reading, not running). These people also make the occasional comment or contribute code.


24.4.1.3 What Is FreeBSD-CURRENT Not?
  1. A fast-track to getting pre-release bits because you heard there is some cool new feature in there and you want to be the first on your block to have it. Being the first on the block to get the new feature means that you are the first on the block to get the new bugs.

  2. A quick way of getting bug fixes. Any given version of FreeBSD-CURRENT is just as likely to introduce new bugs as to fix existing ones.

  3. In any way “officially supported”. We do our best to help people genuinely in one of the 3 “legitimate” FreeBSD-CURRENT groups, but we simply do not have the time to provide tech support. This is not because we are mean and nasty people who do not like helping people out (we would not even be doing FreeBSD if we were). We simply cannot answer hundreds messages a day and work on FreeBSD! Given the choice between improving FreeBSD and answering lots of questions on experimental code, the developers opt for the former.


24.4.1.4 Using FreeBSD-CURRENT
  1. Join the freebsd-current and the svn-src-head lists. This is not just a good idea, it is essential. If you are not on the freebsd-current list, you will not see the comments that people are making about the current state of the system and thus will probably end up stumbling over a lot of problems that others have already found and solved. Even more importantly, you will miss out on important bulletins which may be critical to your system’s continued health.

    The svn-src-head list will allow you to see the commit log entry for each change as it is made, along with any pertinent information on possible side-effects.

    To join these lists, or one of the others available go to http://lists.FreeBSD.org/mailman/listinfo and click on the list that you wish to subscribe to. Instructions on the rest of the procedure are available there. If you are interested in tracking changes for the whole source tree, we would recommend subscribing to the svn-src-all list.

  2. Grab the sources from a FreeBSD mirror site. You can do this in one of two ways:

    1. Use the cvsup program with the supfile named standard-supfile available from /usr/share/examples/cvsup. This is the most recommended method, since it allows you to grab the entire collection once and then only what has changed from then on. Many people run cvsup from cron and keep their sources up-to-date automatically. You have to customize the sample supfile above, and configure cvsup for your environment.

      Note: The sample standard-supfile is intended for tracking a specific security branch of FreeBSD, and not FreeBSD-CURRENT. You will need to edit this file and replace the following line:

      *default release=cvs tag=RELENG_X_Y

      With this one:

      *default release=cvs tag=.

      For a detailed explanation of usable tags, please refer to the Handbook’s CVS Tags section.

    2. Use the CTM facility. If you have very bad connectivity (high price connections or only email access) CTM is an option. However, it is a lot of hassle and can give you broken files. This leads to it being rarely used, which again increases the chance of it not working for fairly long periods of time. We recommend using CVSup for anybody with a 9600 bps modem or faster connection.

  3. If you are grabbing the sources to run, and not just look at, then grab all of FreeBSD-CURRENT, not just selected portions. The reason for this is that various parts of the source depend on updates elsewhere, and trying to compile just a subset is almost guaranteed to get you into trouble.

    Before compiling FreeBSD-CURRENT, read the Makefile in /usr/src carefully. You should at least install a new kernel and rebuild the world the first time through as part of the upgrading process. Reading the FreeBSD-CURRENT mailing list and /usr/src/UPDATING will keep you up-to-date on other bootstrapping procedures that sometimes become necessary as we move toward the next release.

  4. Be active! If you are running FreeBSD-CURRENT, we want to know what you have to say about it, especially if you have suggestions for enhancements or bug fixes. Suggestions with accompanying code are received most enthusiastically!


24.4.2 Staying Stable with FreeBSD
24.4.2.1 What Is FreeBSD-STABLE?

FreeBSD-STABLE is our development branch from which major releases are made. Changes go into this branch at a different pace, and with the general assumption that they have first gone into FreeBSD-CURRENT for testing. This is still a development branch, however, and this means that at any given time, the sources for FreeBSD-STABLE may or may not be suitable for any particular purpose. It is simply another engineering development track, not a resource for end-users.


24.4.2.2 Who Needs FreeBSD-STABLE?

If you are interested in tracking or contributing to the FreeBSD development process, especially as it relates to the next “point” release of FreeBSD, then you should consider following FreeBSD-STABLE.

While it is true that security fixes also go into the FreeBSD-STABLE branch, you do not need to track FreeBSD-STABLE to do this. Every security advisory for FreeBSD explains how to fix the problem for the releases it affects [13], and tracking an entire development branch just for security reasons is likely to bring in a lot of unwanted changes as well.

Although we endeavor to ensure that the FreeBSD-STABLE branch compiles and runs at all times, this cannot be guaranteed. In addition, while code is developed in FreeBSD-CURRENT before including it in FreeBSD-STABLE, more people run FreeBSD-STABLE than FreeBSD-CURRENT, so it is inevitable that bugs and corner cases will sometimes be found in FreeBSD-STABLE that were not apparent in FreeBSD-CURRENT.

For these reasons, we do not recommend that you blindly track FreeBSD-STABLE, and it is particularly important that you do not update any production servers to FreeBSD-STABLE without first thoroughly testing the code in your development environment.

If you do not have the resources to do this then we recommend that you run the most recent release of FreeBSD, and use the binary update mechanism to move from release to release.


24.4.2.3 Using FreeBSD-STABLE
  1. Join the freebsd-stable list. This will keep you informed of build-dependencies that may appear in FreeBSD-STABLE or any other issues requiring special attention. Developers will also make announcements in this mailing list when they are contemplating some controversial fix or update, giving the users a chance to respond if they have any issues to raise concerning the proposed change.

    Join the relevant SVN list for the branch you are tracking. For example, if you are tracking the 7-STABLE branch, join the svn-src-stable-7 list. This will allow you to view the commit log entry for each change as it is made, along with any pertinent information on possible side-effects.

    To join these lists, or one of the others available go to http://lists.FreeBSD.org/mailman/listinfo and click on the list that you wish to subscribe to. Instructions on the rest of the procedure are available there. If you are interested in tracking changes for the whole source tree, we would recommend subscribing to the svn-src-all list.

  2. If you are going to install a new system and want it to run monthly snapshot built from FreeBSD-STABLE, please check the Snapshots web page for more information. Alternatively, it is possible to install the most recent FreeBSD-STABLE release from the mirror sites and follow the instructions below to upgrade your system to the most up to date FreeBSD-STABLE source code.

    If you are already running a previous release of FreeBSD and wish to upgrade via sources then you can easily do so from FreeBSD mirror site. This can be done in one of two ways:

    1. Use the cvsup program with the supfile named stable-supfile from the directory /usr/share/examples/cvsup. This is the most recommended method, since it allows you to grab the entire collection once and then only what has changed from then on. Many people run cvsup from cron to keep their sources up-to-date automatically. You have to customize the sample supfile above, and configure cvsup for your environment.

    2. Use the CTM facility. If you do not have a fast and inexpensive connection to the Internet, this is the method you should consider using.

  3. Essentially, if you need rapid on-demand access to the source and communications bandwidth is not a consideration, use cvsup or ftp. Otherwise, use CTM.

  4. Before compiling FreeBSD-STABLE, read the Makefile in /usr/src carefully. You should at least install a new kernel and rebuild the world the first time through as part of the upgrading process. Reading the FreeBSD-STABLE mailing list and /usr/src/UPDATING will keep you up-to-date on other bootstrapping procedures that sometimes become necessary as we move toward the next release.


24.5 Synchronizing Your Source

There are various ways of using an Internet (or email) connection to stay up-to-date with any given area of the FreeBSD project sources, or all areas, depending on what interests you. The primary services we offer are Anonymous CVS, CVSup, and CTM.

Warning: While it is possible to update only parts of your source tree, the only supported update procedure is to update the entire tree and recompile both userland (i.e., all the programs that run in user space, such as those in /bin and /sbin) and kernel sources. Updating only part of your source tree, only the kernel, or only userland will often result in problems. These problems may range from compile errors to kernel panics or data corruption.

Anonymous CVS and CVSup use the pull model of updating sources. In the case of CVSup the user (or a cron script) invokes the cvsup program, and it interacts with a cvsupd server somewhere to bring your files up-to-date. The updates you receive are up-to-the-minute and you get them when, and only when, you want them. You can easily restrict your updates to the specific files or directories that are of interest to you. Updates are generated on the fly by the server, according to what you have and what you want to have. Anonymous CVS is quite a bit more simplistic than CVSup in that it is just an extension to CVS which allows it to pull changes directly from a remote CVS repository. CVSup can do this far more efficiently, but Anonymous CVS is easier to use.

CTM, on the other hand, does not interactively compare the sources you have with those on the master archive or otherwise pull them across. Instead, a script which identifies changes in files since its previous run is executed several times a day on the master CTM machine, any detected changes being compressed, stamped with a sequence-number and encoded for transmission over email (in printable ASCII only). Once received, these “CTM deltas” can then be handed to the ctm_rmail(1) utility which will automatically decode, verify and apply the changes to the user’s copy of the sources. This process is far more efficient than CVSup, and places less strain on our server resources since it is a push rather than a pull model.

There are other trade-offs, of course. If you inadvertently wipe out portions of your archive, CVSup will detect and rebuild the damaged portions for you. CTM will not do this, and if you wipe some portion of your source tree out (and do not have it backed up) then you will have to start from scratch (from the most recent CVS “base delta”) and rebuild it all with CTM or, with Anonymous CVS, simply delete the bad bits and resync.


24.6 Rebuilding “world”

Once you have synchronized your local source tree against a particular version of FreeBSD (FreeBSD-STABLE, FreeBSD-CURRENT, and so on) you can then use the source tree to rebuild the system.

Make a Backup: It cannot be stressed enough how important it is to make a backup of your system before you do this. While rebuilding the world is (as long as you follow these instructions) an easy task to do, there will inevitably be times when you make mistakes, or when mistakes made by others in the source tree render your system unbootable.

Make sure you have taken a backup. And have a fixit floppy or bootable CD at hand. You will probably never have to use it, but it is better to be safe than sorry!

Subscribe to the Right Mailing List: The FreeBSD-STABLE and FreeBSD-CURRENT branches are, by their nature, in development. People that contribute to FreeBSD are human, and mistakes occasionally happen.

Sometimes these mistakes can be quite harmless, just causing your system to print a new diagnostic warning. Or the change may be catastrophic, and render your system unbootable or destroy your file systems (or worse).

If problems like these occur, a “heads up” is posted to the appropriate mailing list, explaining the nature of the problem and which systems it affects. And an “all clear” announcement is posted when the problem has been solved.

If you try to track FreeBSD-STABLE or FreeBSD-CURRENT and do not read the FreeBSD-STABLE mailing list or the FreeBSD-CURRENT mailing list respectively, then you are asking for trouble.

Do not use make world: A lot of older documentation recommends using make world for this. Doing that skips some important steps and should only be used if you are sure of what you are doing. For almost all circumstances make world is the wrong thing to do, and the procedure described here should be used instead.


24.6.1 The Canonical Way to Update Your System

To update your system, you should check /usr/src/UPDATING for any pre-buildworld steps necessary for your version of the sources and then use the following procedure:

# cd /usr/src
# make buildworld
# make buildkernel
# make installkernel
# shutdown -r now

Note: There are a few rare cases when an extra run of mergemaster -p is needed before the buildworld step. These are described in UPDATING. In general, though, you can safely omit this step if you are not updating across one or more major FreeBSD versions.

After installkernel finishes successfully, you should boot in single user mode (i.e. using boot -s from the loader prompt). Then run:

# mount -a -t ufs
# mergemaster -p
# cd /usr/src
# make installworld
# mergemaster
# reboot

Read Further Explanations: The sequence described above is only a short resume to help you getting started. You should however read the following sections to clearly understand each step, especially if you want to use a custom kernel configuration.


24.6.2 Read /usr/src/UPDATING

Before you do anything else, read /usr/src/UPDATING (or the equivalent file wherever you have a copy of the source code). This file should contain important information about problems you might encounter, or specify the order in which you might have to run certain commands. If UPDATING contradicts something you read here, UPDATING takes precedence.

Important: Reading UPDATING is not an acceptable substitute for subscribing to the correct mailing list, as described previously. The two requirements are complementary, not exclusive.


24.6.3 Check /etc/make.conf

Examine the files /usr/share/examples/etc/make.conf and /etc/make.conf. The first contains some default defines - most of which are commented out. To make use of them when you rebuild your system from source, add them to /etc/make.conf. Keep in mind that anything you add to /etc/make.conf is also used every time you run make, so it is a good idea to set them to something sensible for your system.

A typical user will probably want to copy the CFLAGS and NO_PROFILE lines found in /usr/share/examples/etc/make.conf to /etc/make.conf and uncomment them.

Examine the other definitions (COPTFLAGS, NOPORTDOCS and so on) and decide if they are relevant to you.


24.6.4 Update the Files in /etc

The /etc directory contains a large part of your system’s configuration information, as well as scripts that are run at system startup. Some of these scripts change from version to version of FreeBSD.

Some of the configuration files are also used in the day to day running of the system. In particular, /etc/group.

There have been occasions when the installation part of make installworld has expected certain usernames or groups to exist. When performing an upgrade it is likely that these users or groups did not exist. This caused problems when upgrading. In some cases make buildworld will check to see if these users or groups exist.

An example of this is when the smmsp user was added. Users had the installation process fail for them when mtree(8) was trying to create /var/spool/clientmqueue.

The solution is to run mergemaster(8) in pre-buildworld mode by providing the -p option. This will compare only those files that are essential for the success of buildworld or installworld. If your old version of mergemaster does not support -p, use the new version in the source tree when running for the first time:

# cd /usr/src/usr.sbin/mergemaster
# ./mergemaster.sh -p

Tip: If you are feeling particularly paranoid, you can check your system to see which files are owned by the group you are renaming or deleting:

# find / -group GID -print

will show all files owned by group GID (which can be either a group name or a numeric group ID).


24.6.5 Drop to Single User Mode

You may want to compile the system in single user mode. Apart from the obvious benefit of making things go slightly faster, reinstalling the system will touch a lot of important system files, all the standard system binaries, libraries, include files and so on. Changing these on a running system (particularly if you have active users on the system at the time) is asking for trouble.

Another method is to compile the system in multi-user mode, and then drop into single user mode for the installation. If you would like to do it this way, simply hold off on the following steps until the build has completed. You can postpone dropping to single user mode until you have to installkernel or installworld.

As the superuser, you can execute:

# shutdown now

from a running system, which will drop it to single user mode.

Alternatively, reboot the system, and at the boot prompt, select the “single user” option. The system will then boot single user. At the shell prompt you should then run:

# fsck -p
# mount -u /
# mount -a -t ufs
# swapon -a

This checks the file systems, remounts / read/write, mounts all the other UFS file systems referenced in /etc/fstab and then turns swapping on.

Note: If your CMOS clock is set to local time and not to GMT (this is true if the output of the date(1) command does not show the correct time and zone), you may also need to run the following command:

# adjkerntz -i

This will make sure that your local time-zone settings get set up correctly — without this, you may later run into some problems.


24.6.6 Remove /usr/obj

As parts of the system are rebuilt they are placed in directories which (by default) go under /usr/obj. The directories shadow those under /usr/src.

You can speed up the make buildworld process, and possibly save yourself some dependency headaches by removing this directory as well.

Some files below /usr/obj may have the immutable flag set (see chflags(1) for more information) which must be removed first.

# cd /usr/obj
# chflags -R noschg *
# rm -rf *

24.6.7 Recompile the Base System
24.6.7.1 Saving the Output

It is a good idea to save the output you get from running make(1) to another file. If something goes wrong you will have a copy of the error message. While this might not help you in diagnosing what has gone wrong, it can help others if you post your problem to one of the FreeBSD mailing lists.

The easiest way to do this is to use the script(1) command, with a parameter that specifies the name of the file to save all output to. You would do this immediately before rebuilding the world, and then type exit when the process has finished.

# script /var/tmp/mw.out
Script started, output file is /var/tmp/mw.out
# make TARGET
... compile, compile, compile ...
# exit
Script done, ...

If you do this, do not save the output in /tmp. This directory may be cleared next time you reboot. A better place to store it is in /var/tmp (as in the previous example) or in root’s home directory.


24.6.7.2 Compile the Base System

You must be in the /usr/src directory:

# cd /usr/src

(unless, of course, your source code is elsewhere, in which case change to that directory instead).

To rebuild the world you use the make(1) command. This command reads instructions from the Makefile, which describes how the programs that comprise FreeBSD should be rebuilt, the order in which they should be built, and so on.

The general format of the command line you will type is as follows:

# make -x -DVARIABLE target

In this example, -x is an option that you would pass to make(1). See the make(1) manual page for an example of the options you can pass.

-DVARIABLE passes a variable to the Makefile. The behavior of the Makefile is controlled by these variables. These are the same variables as are set in /etc/make.conf, and this provides another way of setting them.

# make -DNO_PROFILE target

is another way of specifying that profiled libraries should not be built, and corresponds with the

NO_PROFILE=    true     #    Avoid compiling profiled libraries

line in /etc/make.conf.

target tells make(1) what you want to do. Each Makefile defines a number of different “targets”, and your choice of target determines what happens.

Some targets are listed in the Makefile, but are not meant for you to run. Instead, they are used by the build process to break out the steps necessary to rebuild the system into a number of sub-steps.

Most of the time you will not need to pass any parameters to make(1), and so your command like will look like this:

# make target

Where target will be one of many build options. The first target should always be buildworld.

As the names imply, buildworld builds a complete new tree under /usr/obj, and installworld, another target, installs this tree on the current machine.

Having separate options is very useful for two reasons. First, it allows you to do the build safe in the knowledge that no components of your running system will be affected. The build is “self hosted”. Because of this, you can safely run buildworld on a machine running in multi-user mode with no fear of ill-effects. It is still recommended that you run the installworld part in single user mode, though.

Secondly, it allows you to use NFS mounts to upgrade multiple machines on your network. If you have three machines, A, B and C that you want to upgrade, run make buildworld and make installworld on A. B and C should then NFS mount /usr/src and /usr/obj from A, and you can then run make installworld to install the results of the build on B and C.

Although the world target still exists, you are strongly encouraged not to use it.

Run

# make buildworld

It is possible to specify a -j option to make which will cause it to spawn several simultaneous processes. This is most useful on multi-CPU machines. However, since much of the compiling process is IO bound rather than CPU bound it is also useful on single CPU machines.

On a typical single-CPU machine you would run:

# make -j4 buildworld

make(1) will then have up to 4 processes running at any one time. Empirical evidence posted to the mailing lists shows this generally gives the best performance benefit.

If you have a multi-CPU machine and you are using an SMP configured kernel try values between 6 and 10 and see how they speed things up.


24.6.7.3 Timings

Many factors influence the build time, but fairly recent machines may only take a one or two hours to build the FreeBSD-STABLE tree, with no tricks or shortcuts used during the process. A FreeBSD-CURRENT tree will take somewhat longer.


24.6.8 Compile and Install a New Kernel

To take full advantage of your new system you should recompile the kernel. This is practically a necessity, as certain memory structures may have changed, and programs like ps(1) and top(1) will fail to work until the kernel and source code versions are the same.

The simplest, safest way to do this is to build and install a kernel based on GENERIC. While GENERIC may not have all the necessary devices for your system, it should contain everything necessary to boot your system back to single user mode. This is a good test that the new system works properly. After booting from GENERIC and verifying that your system works you can then build a new kernel based on your normal kernel configuration file.

On FreeBSD it is important to build world before building a new kernel.

Note: If you want to build a custom kernel, and already have a configuration file, just use KERNCONF=MYKERNEL like this:

# cd /usr/src
# make buildkernel KERNCONF=MYKERNEL
# make installkernel KERNCONF=MYKERNEL

Note that if you have raised kern.securelevel above 1 and you have set either the noschg or similar flags to your kernel binary, you might find it necessary to drop into single user mode to use installkernel. Otherwise you should be able to run both these commands from multi user mode without problems. See init(8) for details about kern.securelevel and chflags(1) for details about the various file flags.


24.6.9 Reboot into Single User Mode

You should reboot into single user mode to test the new kernel works. Do this by following the instructions in Section 24.6.5.


24.6.10 Install the New System Binaries

If you were building a version of FreeBSD recent enough to have used make buildworld then you should now use installworld to install the new system binaries.

Run

# cd /usr/src
# make installworld

Note: If you specified variables on the make buildworld command line, you must specify the same variables in the make installworld command line. This does not necessarily hold true for other options; for example, -j must never be used with installworld.

For example, if you ran:

# make -DNO_PROFILE buildworld

you must install the results with:

# make -DNO_PROFILE installworld

otherwise it would try to install profiled libraries that had not been built during the make buildworld phase.


24.6.11 Update Files Not Updated by make installworld

Remaking the world will not update certain directories (in particular, /etc, /var and /usr) with new or changed configuration files.

The simplest way to update these files is to use mergemaster(8), though it is possible to do it manually if you would prefer to do that. Regardless of which way you choose, be sure to make a backup of /etc in case anything goes wrong.


24.6.11.1 mergemaster

Contributed by Tom Rhodes.

The mergemaster(8) utility is a Bourne script that will aid you in determining the differences between your configuration files in /etc, and the configuration files in the source tree /usr/src/etc. This is the recommended solution for keeping the system configuration files up to date with those located in the source tree.

To begin simply type mergemaster at your prompt, and watch it start going. mergemaster will then build a temporary root environment, from / down, and populate it with various system configuration files. Those files are then compared to the ones currently installed in your system. At this point, files that differ will be shown in diff(1) format, with the + sign representing added or modified lines, and - representing lines that will be either removed completely, or replaced with a new line. See the diff(1) manual page for more information about the diff(1) syntax and how file differences are shown.

mergemaster(8) will then show you each file that displays variances, and at this point you will have the option of either deleting the new file (referred to as the temporary file), installing the temporary file in its unmodified state, merging the temporary file with the currently installed file, or viewing the diff(1) results again.

Choosing to delete the temporary file will tell mergemaster(8) that we wish to keep our current file unchanged, and to delete the new version. This option is not recommended, unless you see no reason to change the current file. You can get help at any time by typing ? at the mergemaster(8) prompt. If the user chooses to skip a file, it will be presented again after all other files have been dealt with.

Choosing to install the unmodified temporary file will replace the current file with the new one. For most unmodified files, this is the best option.

Choosing to merge the file will present you with a text editor, and the contents of both files. You can now merge them by reviewing both files side by side on the screen, and choosing parts from both to create a finished product. When the files are compared side by side, the l key will select the left contents and the r key will select contents from your right. The final output will be a file consisting of both parts, which can then be installed. This option is customarily used for files where settings have been modified by the user.

Choosing to view the diff(1) results again will show you the file differences just like mergemaster(8) did before prompting you for an option.

After mergemaster(8) is done with the system files you will be prompted for other options. mergemaster(8) may ask if you want to rebuild the password file and will finish up with an option to remove left-over temporary files.


24.6.11.2 Manual Update

If you wish to do the update manually, however, you cannot just copy over the files from /usr/src/etc to /etc and have it work. Some of these files must be “installed” first. This is because the /usr/src/etc directory is not a copy of what your /etc directory should look like. In addition, there are files that should be in /etc that are not in /usr/src/etc.

If you are using mergemaster(8) (as recommended), you can skip forward to the next section.

The simplest way to do this by hand is to install the files into a new directory, and then work through them looking for differences.

Backup Your Existing /etc: Although, in theory, nothing is going to touch this directory automatically, it is always better to be sure. So copy your existing /etc directory somewhere safe. Something like:

# cp -Rp /etc /etc.old

-R does a recursive copy, -p preserves times, ownerships on files and suchlike.

You need to build a dummy set of directories to install the new /etc and other files into. /var/tmp/root is a reasonable choice, and there are a number of subdirectories required under this as well.

# mkdir /var/tmp/root
# cd /usr/src/etc
# make DESTDIR=/var/tmp/root distrib-dirs distribution

This will build the necessary directory structure and install the files. A lot of the subdirectories that have been created under /var/tmp/root are empty and should be deleted. The simplest way to do this is to:

# cd /var/tmp/root
# find -d . -type d | xargs rmdir 2>/dev/null

This will remove all empty directories. (Standard error is redirected to /dev/null to prevent the warnings about the directories that are not empty.)

/var/tmp/root now contains all the files that should be placed in appropriate locations below /. You now have to go through each of these files, determining how they differ with your existing files.

Note that some of the files that will have been installed in /var/tmp/root have a leading “.”. At the time of writing the only files like this are shell startup files in /var/tmp/root/ and /var/tmp/root/root/, although there may be others (depending on when you are reading this). Make sure you use ls -a to catch them.

The simplest way to do this is to use diff(1) to compare the two files:

# diff /etc/shells /var/tmp/root/etc/shells

This will show you the differences between your /etc/shells file and the new /var/tmp/root/etc/shells file. Use these to decide whether to merge in changes that you have made or whether to copy over your old file.

Name the New Root Directory (/var/tmp/root) with a Time Stamp, so You Can Easily Compare Differences Between Versions: Frequently rebuilding the world means that you have to update /etc frequently as well, which can be a bit of a chore.

You can speed this process up by keeping a copy of the last set of changed files that you merged into /etc. The following procedure gives one idea of how to do this.

  1. Make the world as normal. When you want to update /etc and the other directories, give the target directory a name based on the current date. If you were doing this on the 14th of February 1998 you could do the following:

    # mkdir /var/tmp/root-19980214
    # cd /usr/src/etc
    # make DESTDIR=/var/tmp/root-19980214 \
        distrib-dirs distribution
  2. Merge in the changes from this directory as outlined above.

    Do not remove the /var/tmp/root-19980214 directory when you have finished.

  3. When you have downloaded the latest version of the source and remade it, follow step 1. This will give you a new directory, which might be called /var/tmp/root-19980221 (if you wait a week between doing updates).

  4. You can now see the differences that have been made in the intervening week using diff(1) to create a recursive diff between the two directories:

    # cd /var/tmp
    # diff -r root-19980214 root-19980221

    Typically, this will be a much smaller set of differences than those between /var/tmp/root-19980221/etc and /etc. Because the set of differences is smaller, it is easier to migrate those changes across into your /etc directory.

  5. You can now remove the older of the two /var/tmp/root-* directories:

    # rm -rf /var/tmp/root-19980214
  6. Repeat this process every time you need to merge in changes to /etc.

You can use date(1) to automate the generation of the directory names:

# mkdir /var/tmp/root-`date "+%Y%m%d"`

24.6.12 Rebooting

You are now done. After you have verified that everything appears to be in the right place you can reboot the system. A simple shutdown(8) should do it:

# shutdown -r now

24.6.13 Finished

You should now have successfully upgraded your FreeBSD system. Congratulations.

If things went slightly wrong, it is easy to rebuild a particular piece of the system. For example, if you accidentally deleted /etc/magic as part of the upgrade or merge of /etc, the file(1) command will stop working. In this case, the fix would be to run:

# cd /usr/src/usr.bin/file
# make all install

24.6.14 Questions
24.6.14.1. Do I need to re-make the world for every change?
24.6.14.2. My compile failed with lots of signal 11 (or other signal number) errors. What has happened?
24.6.14.3. Can I remove /usr/obj when I have finished?
24.6.14.4. Can interrupted builds be resumed?
24.6.14.5. How can I speed up making the world?
24.6.14.6. What do I do if something goes wrong?

24.6.14.1. Do I need to re-make the world for every change?

There is no easy answer to this one, as it depends on the nature of the change. For example, if you just ran CVSup, and it has shown the following files as being updated:

src/games/cribbage/instr.c
src/games/sail/pl_main.c
src/release/sysinstall/config.c
src/release/sysinstall/media.c
src/share/mk/bsd.port.mk

it probably is not worth rebuilding the entire world. You could just go to the appropriate sub-directories and make all install, and that’s about it. But if something major changed, for example src/lib/libc/stdlib then you should either re-make the world, or at least those parts of it that are statically linked (as well as anything else you might have added that is statically linked).

At the end of the day, it is your call. You might be happy re-making the world every fortnight say, and let changes accumulate over that fortnight. Or you might want to re-make just those things that have changed, and be confident you can spot all the dependencies.

And, of course, this all depends on how often you want to upgrade, and whether you are tracking FreeBSD-STABLE or FreeBSD-CURRENT.

24.6.14.2. My compile failed with lots of signal 11 (or other signal number) errors. What has happened?

This is normally indicative of hardware problems. (Re)making the world is an effective way to stress test your hardware, and will frequently throw up memory problems. These normally manifest themselves as the compiler mysteriously dying on receipt of strange signals.

A sure indicator of this is if you can restart the make and it dies at a different point in the process.

In this instance there is little you can do except start swapping around the components in your machine to determine which one is failing.

24.6.14.3. Can I remove /usr/obj when I have finished?

The short answer is yes.

/usr/obj contains all the object files that were produced during the compilation phase. Normally, one of the first steps in the make buildworld process is to remove this directory and start afresh. In this case, keeping /usr/obj around after you have finished makes little sense, and will free up a large chunk of disk space (currently about 340 MB).

However, if you know what you are doing you can have make buildworld skip this step. This will make subsequent builds run much faster, since most of sources will not need to be recompiled. The flip side of this is that subtle dependency problems can creep in, causing your build to fail in odd ways. This frequently generates noise on the FreeBSD mailing lists, when one person complains that their build has failed, not realizing that it is because they have tried to cut corners.

24.6.14.4. Can interrupted builds be resumed?

This depends on how far through the process you got before you found a problem.

In general (and this is not a hard and fast rule) the make buildworld process builds new copies of essential tools (such as gcc(1), and make(1)) and the system libraries. These tools and libraries are then installed. The new tools and libraries are then used to rebuild themselves, and are installed again. The entire system (now including regular user programs, such as ls(1) or grep(1)) is then rebuilt with the new system files.

If you are at the last stage, and you know it (because you have looked through the output that you were storing) then you can (fairly safely) do:

... fix the problem ...
# cd /usr/src
# make -DNO_CLEAN all

This will not undo the work of the previous make buildworld.

If you see the message:

--------------------------------------------------------------
Building everything..
--------------------------------------------------------------

in the make buildworld output then it is probably fairly safe to do so.

If you do not see that message, or you are not sure, then it is always better to be safe than sorry, and restart the build from scratch.

24.6.14.5. How can I speed up making the world?

  • Run in single user mode.

  • Put the /usr/src and /usr/obj directories on separate file systems held on separate disks. If possible, put these disks on separate disk controllers.

  • Better still, put these file systems across multiple disks using the ccd(4) (concatenated disk driver) device.

  • Turn off profiling (set “NO_PROFILE=true” in /etc/make.conf). You almost certainly do not need it.

  • Also in /etc/make.conf, set CFLAGS to something like -O -pipe. The optimization -O2 is much slower, and the optimization difference between -O and -O2 is normally negligible. -pipe lets the compiler use pipes rather than temporary files for communication, which saves disk access (at the expense of memory).

  • Pass the -jn option to make(1) to run multiple processes in parallel. This usually helps regardless of whether you have a single or a multi processor machine.

  • The file system holding /usr/src can be mounted (or remounted) with the noatime option. This prevents the file system from recording the file access time. You probably do not need this information anyway.

    # mount -u -o noatime /usr/src

    Warning: The example assumes /usr/src is on its own file system. If it is not (if it is a part of /usr for example) then you will need to use that file system mount point, and not /usr/src.

  • The file system holding /usr/obj can be mounted (or remounted) with the async option. This causes disk writes to happen asynchronously. In other words, the write completes immediately, and the data is written to the disk a few seconds later. This allows writes to be clustered together, and can be a dramatic performance boost.

    Warning: Keep in mind that this option makes your file system more fragile. With this option there is an increased chance that, should power fail, the file system will be in an unrecoverable state when the machine restarts.

    If /usr/obj is the only thing on this file system then it is not a problem. If you have other, valuable data on the same file system then ensure your backups are fresh before you enable this option.

    # mount -u -o async /usr/obj

    Warning: As above, if /usr/obj is not on its own file system, replace it in the example with the name of the appropriate mount point.

24.6.14.6. What do I do if something goes wrong?

Make absolutely sure your environment has no extraneous cruft from earlier builds. This is simple enough.

# chflags -R noschg /usr/obj/usr
# rm -rf /usr/obj/usr
# cd /usr/src
# make cleandir
# make cleandir

Yes, make cleandir really should be run twice.

Then restart the whole process, starting with make buildworld.

If you still have problems, send the error and the output of uname -a to FreeBSD general questions mailing list. Be prepared to answer other questions about your setup!


24.7 Tracking for Multiple Machines

Contributed by Mike Meyer.

If you have multiple machines that you want to track the same source tree, then having all of them download sources and rebuild everything seems like a waste of resources: disk space, network bandwidth, and CPU cycles. It is, and the solution is to have one machine do most of the work, while the rest of the machines mount that work via NFS. This section outlines a method of doing so.


24.7.1 Preliminaries

First, identify a set of machines that is going to run the same set of binaries, which we will call a build set. Each machine can have a custom kernel, but they will be running the same userland binaries. From that set, choose a machine to be the build machine. It is going to be the machine that the world and kernel are built on. Ideally, it should be a fast machine that has sufficient spare CPU to run make buildworld and make buildkernel. You will also want to choose a machine to be the test machine, which will test software updates before they are put into production. This must be a machine that you can afford to have down for an extended period of time. It can be the build machine, but need not be.

All the machines in this build set need to mount /usr/obj and /usr/src from the same machine, and at the same point. Ideally, those are on two different drives on the build machine, but they can be NFS mounted on that machine as well. If you have multiple build sets, /usr/src should be on one build machine, and NFS mounted on the rest.

Finally make sure that /etc/make.conf and /etc/src.conf on all the machines in the build set agrees with the build machine. That means that the build machine must build all the parts of the base system that any machine in the build set is going to install. Also, each build machine should have its kernel name set with KERNCONF in /etc/make.conf, and the build machine should list them all in KERNCONF, listing its own kernel first. The build machine must have the kernel configuration files for each machine in /usr/src/sys/arch/conf if it is going to build their kernels.


24.7.2 The Base System

Now that all that is done, you are ready to build everything. Build the kernel and world as described in Section 24.6.7.2 on the build machine, but do not install anything. After the build has finished, go to the test machine, and install the kernel you just built. If this machine mounts /usr/src and /usr/obj via NFS, when you reboot to single user you will need to enable the network and mount them. The easiest way to do this is to boot to multi-user, then run shutdown now to go to single user mode. Once there, you can install the new kernel and world and run mergemaster just as you normally would. When done, reboot to return to normal multi-user operations for this machine.

After you are certain that everything on the test machine is working properly, use the same procedure to install the new software on each of the other machines in the build set.


24.7.3 Ports

The same ideas can be used for the ports tree. The first critical step is mounting /usr/ports from the same machine to all the machines in the build set. You can then set up /etc/make.conf properly to share distfiles. You should set DISTDIR to a common shared directory that is writable by whichever user root is mapped to by your NFS mounts. Each machine should set WRKDIRPREFIX to a local build directory. Finally, if you are going to be building and distributing packages, you should set PACKAGES to a directory similar to DISTDIR.

Tags: Apache, archive, backup, bsd, catch, cron, email, freebsd, FreeBSD Handbook, ftp, password, pop, software, ssl

Related posts

FreeBSD Handbook , , , , , , , , , , , ,