Go to file
Alejandro Colomar 1a0e13f94e Optimize csrand_uniform()
Use a different algorithm to minimize rejection.  This is essentially
the same algorithm implemented in the Linux kernel for
__get_random_u32_below(), but written in a more readable way, and
avoiding microopimizations that make it less readable.

Which (the Linux kernel implementation) is itself based on Daniel
Lemire's algorithm from "Fast Random Integer Generation in an Interval",
linked below.  However, I couldn't really understand that paper very
much, so I had to reconstruct the proofs from scratch, just from what I
could understand from the Linux kernel implementation source code.

I constructed some graphical explanation of how it works, and why it
is optimal, because I needed to visualize it to understand it.  It is
published in the GitHub pull request linked below.

Here goes a wordy explanation of why this algorithm based on
multiplication is better optimized than my original implementation based
on masking.

masking:

	It discards the extra bits of entropy that are not necessary for
	this operation.  This works as if dividing the entire space of
	possible csrand() values into smaller spaces of a size that is
	a smaller power of 2.  Each of those smaller spaces has a
	rejection band, so we get as many rejection bands as spaces
	there are.  For smaller values of 'n', the size of each
	rejection band is smaller, but having more rejection bands
	compensates for this, and results in the same inefficiency as
	for large values of 'n'.

multiplication:

	It divides the entire space of possible random numbers in
	chunks of size exactly 'n', so that there is only one rejection
	band that is the remainder of `2^64 % n`.  The worst case is
	still similar to the masking algorithm, a rejection band that is
	almost half the entire space (n = 2^63 + 1), but for lower
	values of 'n', by only having one small rejection band, it is
	much faster than the masking algorithm.

	This algorithm, however, has one caveat: the implementation
	is harder to read, since it relies on several bitwise tricky
	operations to perform operations like `2^64 % n`, `mult % 2^64`,
	and `mult / 2^64`.  And those operations are different depending
	on the number of bits of the maximum possible random number
	generated by the function.  This means that while this algorithm
	could also be applied to get uniform random numbers in the range
	[0, n-1] quickly from a function like rand(3), which only
	produces 31 bits of (non-CS) random numbers, it would need to be
	implemented differently.  However, that's not a concern for us,
	it's just a note so that nobody picks this code and expects it
	to just work with rand(3) (which BTW I tried for testing it, and
	got a bit confused until I realized this).

Finally, here's some light testing of this implementation, just to know
that I didn't goof it.  I pasted this function into a standalone
program, and run it many times to find if it has any bias (I tested also
to see how many iterations it performs, and it's also almost always 1,
but that test is big enough to not paste it here).

int main(int argc, char *argv[])
{
	printf("%lu\n", csrand_uniform(atoi(argv[1])));
}

$ seq 1 1000 | while read _; do ./a.out 3; done | grep 1 | wc -l
341
$ seq 1 1000 | while read _; do ./a.out 3; done | grep 1 | wc -l
339
$ seq 1 1000 | while read _; do ./a.out 3; done | grep 1 | wc -l
338
$ seq 1 1000 | while read _; do ./a.out 3; done | grep 2 | wc -l
336
$ seq 1 1000 | while read _; do ./a.out 3; done | grep 2 | wc -l
328
$ seq 1 1000 | while read _; do ./a.out 3; done | grep 2 | wc -l
335
$ seq 1 1000 | while read _; do ./a.out 3; done | grep 0 | wc -l
332
$ seq 1 1000 | while read _; do ./a.out 3; done | grep 0 | wc -l
331
$ seq 1 1000 | while read _; do ./a.out 3; done | grep 0 | wc -l
327

This isn't a complete test for a cryptographically-secure random number
generator, of course, but I leave that for interested parties.

Link: <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e9a688bcb19348862afe30d7c85bc37c4c293471>
Link: <https://github.com/shadow-maint/shadow/pull/624#discussion_r1059574358>
Link: <https://arxiv.org/abs/1805.10941>
Cc: "Jason A. Donenfeld" <Jason@zx2c4.com>
Cc: Cristian Rodríguez <crrodriguez@opensuse.org>
Cc: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Cc: Björn Esser <besser82@fedoraproject.org>
Cc: Yann Droneaud <ydroneaud@opteya.com>
Cc: Joseph Myers <joseph@codesourcery.com>
Cc: Sam James <sam@gentoo.org>
Cc: Serge Hallyn <serge@hallyn.com>
Cc: Iker Pedrosa <ipedrosa@redhat.com>
[Daniel Lemire: Added link to research paper in source code]
Cc: Daniel Lemire <daniel@lemire.me>
Signed-off-by: Alejandro Colomar <alx@kernel.org>
2023-01-27 21:48:37 -06:00
.builds CI: add libbsd and pkg-config dependencies 2022-11-28 09:07:41 -06:00
.github workflow: update checkout acton v2 to v3 2023-01-13 09:51:05 +01:00
contrib Fix typos 2023-01-26 22:44:39 -06:00
doc Remove traces of utmpx 2022-12-22 10:31:43 +01:00
docs fix spelling and unify whitespace 2021-08-18 18:06:02 +00:00
etc fix PAM service files --without-selinux 2022-03-04 08:51:20 -06:00
lib Add WIDTHOF() to get the width in bits 2023-01-27 21:48:37 -06:00
libmisc Optimize csrand_uniform() 2023-01-27 21:48:37 -06:00
libsubid Revert "Drop unused function subid_init()" 2022-08-21 13:20:56 -05:00
man Supporting vendor given -shells- configuration file 2023-01-26 22:45:32 -06:00
po updated Dutch translation 2022-09-27 16:01:31 -05:00
src Supporting vendor given -shells- configuration file 2023-01-26 22:45:32 -06:00
tests Add '62_usermod_remove_supplementary_groups' test case to test runner scripts (run_some) 2022-12-11 10:58:37 -06:00
.gitignore Show libsubid api version in subid.h 2021-12-05 08:02:57 -06:00
.travis.yml subids: support nsswitch 2021-04-16 21:02:37 -05:00
acinclude.m4 configure: replace obsolete autoconf macros 2022-05-10 09:55:18 +02:00
AUTHORS.md AUTHORS: improve markdown output 2022-03-18 16:10:51 -05:00
autogen.sh undo accidental autogen.sh commit: enable-shared 2021-11-27 14:56:03 -06:00
ChangeLog fix typo 2023-01-12 12:10:57 +01:00
configure.ac Supporting vendor given -shells- configuration file 2023-01-26 22:45:32 -06:00
COPYING Update licensing info 2021-12-23 19:36:50 -06:00
Makefile.am fix spelling and unify whitespace 2021-08-18 18:06:02 +00:00
NEWS fix typo 2023-01-12 12:10:57 +01:00
README Add README as symlink to README.md 2021-12-19 14:09:08 -06:00
README.md README: update content and format 2021-11-22 15:31:54 +01:00
SECURITY.md Add Christian Brauner to SECURITY.md 2021-10-25 14:26:37 -05:00
shadow.spec.in * shadow.spec.in: Fix the source (new FTP). 2008-08-31 17:30:45 +00:00
TODO fix spelling and unify whitespace 2021-08-18 18:06:02 +00:00

shadow-utils

Introduction

The shadow-utils package includes the necessary programs for converting UNIX password files to the shadow password format, plus programs for managing user and group accounts. The pwconv command converts passwords to the shadow password format. The pwunconv command unconverts shadow passwords and generates a passwd file (a standard UNIX password file). The pwck command checks the integrity of password and shadow files. The lastlog command prints out the last login times for all users. The useradd, userdel, and usermod commands are used for managing user accounts. The groupadd, groupdel, and groupmod commands are used for managing group accounts.

Sites

Contacts

There are several ways to contact us:

Mailing archives

Authors and maintainers

Authors and maintainers are listed in AUTHORS.md.