2015-01-11 05:04:51 +05:30
|
|
|
ndhc, Copyright (C) 2004-2015 Nicholas J. Kain.
|
2011-07-25 12:00:57 +05:30
|
|
|
See LICENSE for licensing information. In short: Two-clause / New BSD.
|
2010-11-12 14:32:18 +05:30
|
|
|
|
|
|
|
Requirements:
|
|
|
|
|
2011-07-25 12:00:57 +05:30
|
|
|
Linux kernel
|
2011-07-13 12:00:10 +05:30
|
|
|
GNU Make (tested: 3.82) or CMake (tested: 2.8)
|
2014-03-15 09:21:11 +05:30
|
|
|
Ragel (tested: 6.7)
|
2010-11-12 22:35:37 +05:30
|
|
|
|
2010-11-12 14:32:18 +05:30
|
|
|
INTRODUCTION
|
|
|
|
------------
|
|
|
|
|
2014-03-10 10:39:25 +05:30
|
|
|
ndhc is a multi-process, privilege-separated dhcp client. Each subprocess runs
|
|
|
|
with the minimal necessary privileges in order to perform its task. Currently,
|
2014-04-15 04:02:08 +05:30
|
|
|
ndhc consists of three subprocesses: the ndhc-master, ndhc-ifch, and
|
|
|
|
ndhc-sockd.
|
2010-11-12 14:32:18 +05:30
|
|
|
|
2014-03-19 10:16:54 +05:30
|
|
|
ndhc-master communicates with dhcp servers and handles the vagaries of the dhcp
|
2014-04-15 04:02:08 +05:30
|
|
|
client protocol. It runs as a non-root user inside a chroot. ndhc runs as a
|
|
|
|
normal user with no special privileges and is restricted to a chroot that
|
|
|
|
contains nothing more than a domain socket filesystem object (if using syslog),
|
|
|
|
a urandom device node, and a null device node.
|
|
|
|
|
|
|
|
ndhc-ifch handles interface change requests. It listens on a unix socket for
|
2014-03-19 10:16:54 +05:30
|
|
|
such requests. ndhc-ifch runs as a non-root user inside a chroot, and retains
|
|
|
|
only the power to configure network interfaces. ndhc-ifch automatically forks
|
|
|
|
from ndhc-master to perform its job.
|
2010-11-12 14:32:18 +05:30
|
|
|
|
2014-04-15 04:02:08 +05:30
|
|
|
ndhc-sockd plays a similar role to ndhc-ifch, but it instead has the ability to
|
|
|
|
bind to a low port, the ability to open a raw socket, and the ability to
|
|
|
|
communicate on broadcast channels. ndhc communicates with ndhc-sockd
|
|
|
|
over a unix socket, and the file descriptors that ndhc-sockd creates are
|
|
|
|
passed back to ndhc over the unix socket.
|
|
|
|
|
2011-07-27 17:13:09 +05:30
|
|
|
ndhc fully implements RFC5227's address conflict detection and defense. Great
|
|
|
|
care is taken to ensure that address conflicts will be detected, and ndhc also
|
|
|
|
has extensive support for address defense. Care is taken to prevent
|
|
|
|
unintentional ARP flooding under any circumstance.
|
2011-07-06 04:55:19 +05:30
|
|
|
|
|
|
|
ndhc also monitors hardware link status via netlink events and reacts
|
|
|
|
appropriately when interface carrier status changes or an interface is
|
2011-07-12 13:39:05 +05:30
|
|
|
explicitly deconfigured. This functionality can be useful on wired networks
|
|
|
|
when transient carrier downtimes occur (or cables are changed), but it is
|
|
|
|
particularly useful on wireless networks.
|
2010-11-12 14:32:18 +05:30
|
|
|
|
2011-07-27 17:13:09 +05:30
|
|
|
RFC3927's IPv4 Link Local Addressing is not supported. I have found v4 LLAs
|
|
|
|
to be more of an annoyance than a help. v6 LLAs work much better in practice.
|
|
|
|
|
2011-07-13 12:00:10 +05:30
|
|
|
FEATURES
|
|
|
|
--------
|
|
|
|
|
2014-03-10 10:39:25 +05:30
|
|
|
Privilege-separated. ndhc does not run as root after initial startup, and
|
2014-04-15 04:02:08 +05:30
|
|
|
capabilities are divided between the subprocesses. All processes run in a
|
2014-03-10 10:39:25 +05:30
|
|
|
chroot.
|
2011-07-13 12:00:10 +05:30
|
|
|
|
2014-04-15 04:02:08 +05:30
|
|
|
Robust. ndhc performs no runtime heap allocations -- malloc() (more
|
|
|
|
specifically, brk(), mmap(), etc) is never called after initialization (libc
|
2014-04-25 03:59:52 +05:30
|
|
|
behavior during initialization time will vary), and ndhc never performs
|
2014-04-15 04:02:08 +05:30
|
|
|
recursive calls and only stack-allocates fixed-length types, so stack depth is
|
|
|
|
bounded, too.
|
2011-07-13 12:00:10 +05:30
|
|
|
|
|
|
|
Active defense of IP address and IP collision avoidance. ndhc fully implements
|
|
|
|
RFC5227. It is capable of both a normal level of tenacity in defense, where
|
|
|
|
it will eventually back off and request a new lease if a peer won't relent
|
|
|
|
in the case of a conflict, and of relentlessly defending a lease forever. In
|
|
|
|
either mode, it rate-limits defense messages, so it can't be tricked into
|
|
|
|
flooding by a hostile peer or DHCP server, either.
|
|
|
|
|
2014-03-10 10:39:25 +05:30
|
|
|
Small. Both ndhc avoids unnecessary outside dependencies and is written in
|
2014-04-15 04:02:08 +05:30
|
|
|
plain C.
|
2011-07-13 12:00:10 +05:30
|
|
|
|
|
|
|
Fast. ndhc filters input using the BPF/LPF mechanism so that uninteresting
|
|
|
|
packets are dropped by the operating system before ndhc even sees the data.
|
|
|
|
ndhc also only listens to DHCP traffic when it's necessary.
|
|
|
|
|
|
|
|
Flexible. ndhc can request particular IPs, send user-specified client IDs,
|
2014-03-10 10:39:25 +05:30
|
|
|
write a file that contains the current lease IP, write PID files, etc.
|
|
|
|
|
|
|
|
Self-contained. ndhc does not exec other processes, or rely on the shell.
|
2015-01-11 05:04:51 +05:30
|
|
|
Further, ndhc relies on no external libraries aside from the system libc.
|
2011-07-13 12:00:10 +05:30
|
|
|
|
|
|
|
Aware of the hardware link status. If you disconnect an interface on which
|
|
|
|
ndhc is providing dhcp service, it will be aware. When the link status
|
|
|
|
returns, ndhc will fingerprint the reconnected network and make sure that it
|
|
|
|
corresponds to the one on which it has a lease. If the new network is
|
|
|
|
different, it will forget about the old lease and request a new one.
|
|
|
|
|
2010-11-12 14:32:18 +05:30
|
|
|
USAGE
|
|
|
|
-----
|
|
|
|
|
2014-03-10 10:39:25 +05:30
|
|
|
1) Compile and install ndhc.
|
2014-03-19 10:16:54 +05:30
|
|
|
a) make
|
2014-03-10 10:39:25 +05:30
|
|
|
b) Install the build/ndhc executable in a normal place. I would suggest
|
|
|
|
/usr/sbin or /usr/local/sbin.
|
2011-07-12 13:39:05 +05:30
|
|
|
|
2014-03-10 10:39:25 +05:30
|
|
|
1alt) Compile and install ndhc.
|
2010-11-12 22:35:37 +05:30
|
|
|
a) Create a build directory:
|
|
|
|
mkdir build && cd build
|
|
|
|
b) Create the makefiles:
|
|
|
|
cmake ..
|
2014-03-10 10:39:25 +05:30
|
|
|
c) Build ndhc:
|
2010-11-12 22:35:37 +05:30
|
|
|
make
|
2014-03-10 10:39:25 +05:30
|
|
|
d) Install the ndhc/ndhc executable in a normal place. I would suggest
|
|
|
|
/usr/sbin or /usr/local/sbin.
|
2010-11-12 14:32:18 +05:30
|
|
|
|
2014-03-10 10:39:25 +05:30
|
|
|
2) Time to create the jail in which ndhc will run.
|
|
|
|
a) Become root and create new group "ndhc".
|
2010-11-12 14:32:18 +05:30
|
|
|
|
2010-11-12 20:09:33 +05:30
|
|
|
$ su -
|
|
|
|
# umask 077
|
2014-03-10 10:39:25 +05:30
|
|
|
# groupadd ndhc
|
2010-11-12 20:09:33 +05:30
|
|
|
|
2014-03-19 10:16:54 +05:30
|
|
|
b) Create new users "dhcpifch" and "dhcp". The primary group of these
|
2014-03-10 10:39:25 +05:30
|
|
|
users should be "ndhc".
|
2010-11-12 14:32:18 +05:30
|
|
|
|
2014-04-15 04:02:08 +05:30
|
|
|
# useradd -d /var/lib/ndhc -s /sbin/nologin -g ndhc dhcpsockd
|
2014-03-19 10:16:54 +05:30
|
|
|
# useradd -d /var/lib/ndhc -s /sbin/nologin -g ndhc dhcpifch
|
2014-03-10 10:39:25 +05:30
|
|
|
# useradd -d /var/lib/ndhc -s /sbin/nologin -g ndhc dhcp
|
2010-11-12 20:09:33 +05:30
|
|
|
|
2014-03-19 10:16:54 +05:30
|
|
|
c) Create the state directory where DUIDs and IAIDs will be stored.
|
|
|
|
|
|
|
|
# mkdir /etc/ndhc
|
|
|
|
# chown root.root /etc/ndhc
|
|
|
|
# chmod 0755 /etc/ndhc
|
|
|
|
|
|
|
|
d) Create the jail directory and set its ownership properly.
|
2010-11-12 14:32:18 +05:30
|
|
|
|
2010-11-12 20:09:33 +05:30
|
|
|
# mkdir /var/lib/ndhc
|
2011-05-02 06:13:29 +05:30
|
|
|
# chown root.root /var/lib/ndhc
|
2010-11-12 20:09:33 +05:30
|
|
|
# chmod a+rx /var/lib/ndhc
|
2011-05-02 06:13:29 +05:30
|
|
|
# cd /var/lib/ndhc
|
|
|
|
# mkdir var
|
|
|
|
# mkdir var/state
|
|
|
|
# mkdir var/run
|
2014-03-10 10:39:25 +05:30
|
|
|
# chown -R dhcp.ndhc var
|
2011-05-02 06:13:29 +05:30
|
|
|
# chmod -R a+rx var
|
2011-06-10 23:37:03 +05:30
|
|
|
# chmod g+w var/run
|
2010-11-12 20:09:33 +05:30
|
|
|
|
2014-03-19 10:16:54 +05:30
|
|
|
e) Create a urandom device for ndhc to use within the jail.
|
2010-11-12 14:32:18 +05:30
|
|
|
|
2010-11-12 20:09:33 +05:30
|
|
|
# mkdir dev
|
|
|
|
# mknod dev/urandom c 1 9
|
2010-11-12 23:54:07 +05:30
|
|
|
# mknod dev/null c 1 3
|
2010-11-12 20:09:33 +05:30
|
|
|
# chown -R root.root dev
|
|
|
|
# chmod a+rx dev
|
|
|
|
# chmod a+r dev/urandom
|
2010-11-12 23:54:07 +05:30
|
|
|
# chmod a+rw dev/null
|
2010-11-12 14:32:18 +05:30
|
|
|
|
2014-03-19 10:16:54 +05:30
|
|
|
f) (optional) If you wish for logging to properly work, you
|
2010-11-12 20:09:33 +05:30
|
|
|
will need to properly configure your logging daemon so that it
|
|
|
|
opens a domain socket in the proper location within the jail.
|
|
|
|
Since this varies per-daemon, I cannot provide a general
|
|
|
|
configuration.
|
2010-11-12 14:32:18 +05:30
|
|
|
|
2014-03-19 10:16:54 +05:30
|
|
|
3) At this point the jail is usable; ndhc is ready to be used. An example
|
|
|
|
of invoking ndhc:
|
2010-11-12 14:32:18 +05:30
|
|
|
|
2014-04-25 03:59:52 +05:30
|
|
|
# ndhc -i wan0 -u dhcp -U dhcpifch -D dhcpsockd -C /var/lib/ndhc
|
2010-11-12 14:32:18 +05:30
|
|
|
|
2014-03-17 06:46:06 +05:30
|
|
|
4o) If you encounter problems, I suggest running ndhc in the foreground and
|
|
|
|
examining the printed output.
|
2010-11-12 14:32:18 +05:30
|
|
|
|
|
|
|
|
|
|
|
BEHAVIOR NOTES
|
|
|
|
--------------
|
|
|
|
|
2014-03-10 10:39:25 +05:30
|
|
|
ndhc does not enable updates of the local hostname and resolv.conf by default.
|
|
|
|
If you wish to enable these functions, use the --resolve (-R) and --hostname
|
|
|
|
(-H) flags. See ndhc --help.
|
2010-11-12 14:32:18 +05:30
|
|
|
|
2014-03-19 10:16:54 +05:30
|
|
|
STATE STORAGE NOTES
|
|
|
|
-------------------
|
|
|
|
|
|
|
|
ndhc requires a read/writable directory to store the DUID/IAID states. By
|
|
|
|
default this directory is /etc/ndhc. It exists outside the chroot. The DUID
|
|
|
|
will be stored in a single file, DUID. The IAIDs exist per-interface and are
|
|
|
|
stored in files with names similar to IAID-xx:xx:xx:xx:xx:xx, where the xx
|
|
|
|
values are replaced by the Ethernet hardware address of the interface.
|
|
|
|
|
|
|
|
If it is impossible to read or store the DUIDs or IAIDs, ndhc will
|
|
|
|
fail at start time before it performs any network activity or forks any
|
|
|
|
subprocesses.
|
|
|
|
|
|
|
|
If the host system lacks volatile storage, then a clientid should manually
|
2014-04-15 04:02:08 +05:30
|
|
|
be specified using the -I or --clientid command arguments.
|
2014-03-19 10:16:54 +05:30
|
|
|
|
|
|
|
RANDOMNESS NOTES
|
|
|
|
----------------
|
|
|
|
|
2015-01-11 05:04:51 +05:30
|
|
|
Each ndhc subprocess maintains a PCG PRNG that is uniquely seeded from the
|
|
|
|
kernel random device at startup. Each PRNG consumes 128 bits of entropy for
|
|
|
|
its initial state.
|
2014-03-19 10:16:54 +05:30
|
|
|
|
|
|
|
DHCP does not require cryptographic randomness, so this arrangement should
|
|
|
|
be more than sufficient to ensure proper UUIDs, assuming only that the
|
|
|
|
kernel random device is even minimally seeded with real entropy.
|
|
|
|
|
2010-11-12 14:32:18 +05:30
|
|
|
PORTING NOTES
|
|
|
|
-------------
|
|
|
|
|
2011-07-13 12:00:10 +05:30
|
|
|
DHCP clients aren't naturally very portable. It's necessary to perform a lot
|
|
|
|
of tasks that are platform-specific. ndhc is rather platform-dependent, and it
|
|
|
|
extensively uses Linux-specific features. Some of these features are also
|
|
|
|
available on the BSDs.
|
2011-07-12 13:39:05 +05:30
|
|
|
|
2014-03-10 10:39:25 +05:30
|
|
|
1) ndhc takes advantage of Linux capabilities so that it does not need full
|
2011-07-12 13:39:05 +05:30
|
|
|
root privileges. Capabilities were a proposed POSIX feature that was not made
|
|
|
|
part of the official standard, so any implemention that may exist will be
|
|
|
|
system-dependent.
|
|
|
|
|
2014-03-10 10:39:25 +05:30
|
|
|
2) ndhc configures network interfaces and routes. Interface and route
|
2014-03-17 06:46:06 +05:30
|
|
|
configuration is entirely non-portable.
|
2011-07-12 13:39:05 +05:30
|
|
|
|
2014-03-17 06:46:06 +05:30
|
|
|
3) ndhc uses netlink sockets extensively for fetching data, setting data,
|
|
|
|
and hardware link state change notification events.
|
2010-11-12 14:32:18 +05:30
|
|
|
|
2014-03-10 10:39:25 +05:30
|
|
|
4) ndhc uses the Berkeley Packet Filter / Linux Packet Filter interfaces to
|
2011-07-12 13:39:05 +05:30
|
|
|
drop unwanted packets in kernelspace. This functionality is available on
|
|
|
|
most modern unix systems, but it is not standard.
|
|
|
|
|
2014-03-10 10:39:25 +05:30
|
|
|
5) ndhc uses epoll() and signalfd(). These are Linux-specific.
|
2011-07-12 13:39:05 +05:30
|
|
|
|
2014-03-10 10:39:25 +05:30
|
|
|
6) Numerous socket options are used, and the AF_PACKET socket family is used
|
2011-07-12 13:39:05 +05:30
|
|
|
for raw sockets and ARP. These are largely Linux-specific, too.
|
|
|
|
|
2014-03-17 06:46:06 +05:30
|
|
|
7) ndhc can optionally use seccomp-filter to allow only a set of whitelisted
|
|
|
|
syscalls. This functionality is Linux-specific.
|
|
|
|
|
2011-07-12 13:39:05 +05:30
|
|
|
HISTORY
|
|
|
|
-------
|
|
|
|
|
|
|
|
I started writing ndhc back in 2004. My ISP at the time required a dhcp
|
|
|
|
client for connection authentication, and I was not comfortable with any
|
|
|
|
of the existing clients, which all ran as root and had colorful security
|
|
|
|
histories. DHCP is generally not a routed protocol, and lacks real
|
|
|
|
authentication mechanisms in real world deployments (some largely
|
|
|
|
abandoned RFCs for such behavior do exist), so no program existed to
|
|
|
|
fill the niche of a truly secure DHCP client.
|
|
|
|
|
|
|
|
My router/server at the time ran a custom Linux distro that was designed
|
|
|
|
for extreme security. A root privileged DHCP client would be nearly the
|
|
|
|
only root-owned process running on the machine, so I was highly motivated
|
|
|
|
to develop an alternative.
|
|
|
|
|
2014-03-10 10:39:25 +05:30
|
|
|
A separate ifchd was first written entirely from scratch. It did not take long
|
2014-03-19 10:16:54 +05:30
|
|
|
to write, since it was by design rather simple, and I was already familiar with
|
2014-03-10 10:39:25 +05:30
|
|
|
the quirks of Linux capabilities. That left me with the choice of adapting an
|
|
|
|
existing DHCP client or writing my own from scratch.
|
2011-07-12 13:39:05 +05:30
|
|
|
|
|
|
|
At the time, I just wanted something that would work, so my choice was to
|
|
|
|
adapt udhcpc to work with ifchd. udhcpc was chosen since it was intended to
|
|
|
|
be used with resource-constrained or embedded systems, and was thus very
|
|
|
|
small. ISC dhclient was another alternative, but it is an extremely large
|
|
|
|
program, and it would have been very hard to audit it for correctness.
|
|
|
|
|
2011-07-25 03:32:25 +05:30
|
|
|
udhcpc was not did not really fit my requirements well, since it was designed
|
|
|
|
to be small at all costs, sacrificing correctness when necessary. The code was
|
|
|
|
hard to follow, and had many quirks. Bounds-checking was rare, type aliasing
|
|
|
|
common, and state transitions were convoluted. Not all of the client was
|
|
|
|
asynchronous, and no precautions were taken against conflicting peers. ARP was
|
|
|
|
not used at all.
|
2011-07-12 13:39:05 +05:30
|
|
|
|
|
|
|
However, it was small. With a lot of work, I ripped out the script-calling
|
|
|
|
mechanisms and replaced them with ifchd requests. Bounds-checking was
|
|
|
|
aggressively (and somewhat hamfistedly) retrofitted into the code. It was
|
|
|
|
cleaned to a degree, and importantly it worked for connecting to my ISP.
|
|
|
|
|
|
|
|
Then I changed ISPs. My new ISP used PPPoE, not dhcp. Around the same time, I
|
|
|
|
also switched to using Gentoo rather than a hand-built distribution. I didn't
|
|
|
|
have time to maintain the old custom setup, and it was very hard keeping up
|
|
|
|
with library vulnerabilties in eg, zlib or openssl, and ensuring that all
|
|
|
|
installed binaries, dynamic and static, were updated. ndhc was abandoned for
|
|
|
|
many years. It wasn't needed on my server, and it was "too much effort" to
|
|
|
|
deviate from the stock distro dhcp clients on other machines.
|
|
|
|
|
|
|
|
Then, around 2008, I changed ISPs again. This time my new ISP used dhcp and
|
|
|
|
not PPPoE. So, after a few months, I decided to dust off the old ndhc/ifchd
|
|
|
|
project and adapt it to my modern standards and machines.
|
|
|
|
|
|
|
|
ifchd was in good shape and required little work. I ended up rewriting
|
|
|
|
ndhc. The only parts that remained from the original were the parts that
|
|
|
|
I had already rewritten before, and some of those were rewritten, too.
|
|
|
|
|
2014-03-10 10:39:25 +05:30
|
|
|
Eventually ifchd was rewritten to extensively use a Ragel-generated DFA-based
|
|
|
|
parser to make it easier to verify correct behavior for all possible inputs.
|
|
|
|
|
2014-04-25 03:59:52 +05:30
|
|
|
Quite a while later, I eventually merged ifchd into the same binary as ndhc
|
|
|
|
and instead rely on forking subprocesses and using socketpairs for IPC. This
|
2014-03-10 10:39:25 +05:30
|
|
|
brought a lot of simplifications, particularly for user configuration.
|
|
|
|
|
2015-02-13 12:18:00 +05:30
|
|
|
Afterwards, privilege separation was applied to the remaining capabilities,
|
2014-04-15 04:02:08 +05:30
|
|
|
creating the ndhc-sockd subprocess. After this change, the main ndhc
|
|
|
|
process runs completely unprivileged.
|
|
|
|
|
2011-07-12 13:39:05 +05:30
|
|
|
The end result is a modern DHCP client is largely RFC-compliant, except where
|
|
|
|
the RFCs dictate behavior that would be problematic, overly complex, useless,
|
|
|
|
or exploitable. DHCP is poorly specified, and real-world servers and clients
|
|
|
|
vary a lot from the RFCs, so these conditions are necessary for a useful
|
|
|
|
program.
|
|
|
|
|
|
|
|
Although ndhc's implementation and behavior are different, I have to credit
|
|
|
|
the idea of using netlink events to discover hardware link status transitions
|
|
|
|
to Stefan Rompf and his 'dhcpclient' program. The Linux netlink events that
|
|
|
|
are used are otherwise rather obscure and poorly documented, and I wouldn't
|
|
|
|
have known about them otherwise.
|
|
|
|
|
|
|
|
GRSECURITY NOTES
|
|
|
|
----------------
|
|
|
|
|
2014-03-10 10:39:25 +05:30
|
|
|
Make sure that CONFIG_GRKERNSEC_CHROOT_CAPS is disabled. Otherwise, ndhc will
|
2011-07-12 13:39:05 +05:30
|
|
|
lose its capabilities (in particular, the ability to reconfigure interfaces)
|
|
|
|
when it chroots.
|
|
|
|
|
2011-07-13 12:00:10 +05:30
|
|
|
DHCP PROTOCOL QUIRKS
|
|
|
|
--------------------
|
|
|
|
|
|
|
|
Send a packet that has an options field set to:
|
|
|
|
'DHCP-OPTION-OVERLOAD:3'
|
|
|
|
Then in the file and sname fields:
|
|
|
|
'DHCP-OPTION-OVERLOAD:3'
|
|
|
|
I suspect some bad dhcp programs will hang given this input.
|
|
|
|
|
|
|
|
DHCP explicitly specifies that there is no minimum lease time and also
|
|
|
|
specifies that the minimum default rebinding time is leasetime*0.875 and
|
|
|
|
the minimum default renewing time is leasetime*0.500. All times are relative
|
|
|
|
to the instant when the lease is bound and are specified in seconds. Taken
|
|
|
|
together, this means that a client strictly implementing the RFC should
|
|
|
|
accept a lease that either is perpetually rebinding (lease == 1s) or instantly
|
|
|
|
expires (lease == 0s). ndhc ignores the RFC and specifies a minimum lease
|
|
|
|
time of one minute.
|
|
|
|
|
|
|
|
Renew and rebind times are optionally specified and may take on any value.
|
|
|
|
This means that a malicious server could demand a rebind time before a renew
|
|
|
|
time, or make these times ridiculously short, or specify both times past
|
|
|
|
that of the lease duration. ndhc avoids all of this nonsense by simply
|
|
|
|
ignoring these options and using the default values specified by the RFC.
|
|
|
|
|
|
|
|
There are other quirks, but these are just several interesting ones that
|
|
|
|
immediately occur to me while I'm writing this document.
|
2011-07-12 13:39:05 +05:30
|
|
|
|