The 'PROC_FILL' flags, found in readproc.h, had become
almost unmanageable. The hex values were scattered all
over the map as new flags had been introduced. So this
commit resets all of them and will help ensure any new
flags don't duplicate some already existing hex value.
Signed-off-by: Jim Warner <james.warner@comcast.net>
When declaring a pointer there's usually a space after
the thing-pointed-to and no space between the asterisk
and the pointer-thingy itself. So this commit enforces
such conventions where needed on old library elements.
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch just brings 'PIDS_IO' source into alignment
with the names being used in that /proc/<pid>/io file.
[ i had my chance to fix them in a whitespace change ]
[ made in the patch referenced below, but i blew it! ]
Reference(s):
commit 2dcbe71f3b
Signed-off-by: Jim Warner <james.warner@comcast.net>
That recent addition of USS to the ps program prompted
this change. Rather than have it (and soon top) add 2
separate items to yield the desired value, we will let
our new library perform the arithmetic when necessary.
Outside of a little extra storage, there is no runtime
costs for such an extension. There is, however, a real
benefit to having such code in the library. Now should
callers choose to sort on this new field, results will
be guaranteed to be what was expected (i.e. accurate).
Signed-off-by: Jim Warner <james.warner@comcast.net>
The two special hugetlbfs items were misnamed. The TBL
reference (table) should be TLB (transaction lookaside
buffer). Besides, I never liked their position anyway!
[ and one macro argument tweak is being snuck in too ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
A couple of people have suggested that smaps_rollup be
added to the ps program and/or top program. This patch
is intended to set the stage for just such extensions.
There are currently 20 displayable items in the rollup
file. And newlib sometimes uses sscanf when populating
the target, sometimes hsearch and one customized gperf
approach. None of these fit well with the smaps items.
Thus, an approach using a simple table lookup was used
and, by disabling 1 code line, it could be made immune
from changes to the items order (unlike a sscanf call)
and doesn't carry the greater cost of a hsearch/gperf.
Note: The next patch will allow top to display some of
these new fields. Then, it'll be possible to determine
the colossal costs of accessing the smaps_rollup file.
Here is a small preview of just what you will discover
when using the command 'time top/top -d0 -n1000' while
configured with just two fields: PID + 1 memory field.
------------------------------------ as a regular user
with only PID + RES (statm)
real 0m2.605s
user 0m1.060s
sys 0m1.377s
with only PID + RSS (smaps)
real 0m26.397s 10x more costly
user 0m1.253s
sys 0m24.915s
----------------- as a root (thus smaps for all tasks)
with only PID + RES (statm)
real 0m2.651s
user 0m1.177s
sys 0m1.286s
with only PID + RSS (smaps)
real 0m33.040s 12x more costly
user 0m1.256s
sys 0m31.533s
Reference(s):
. ps: expose shared/private memory separately
https://gitlab.com/procps-ng/procps/-/issues/201
. top/ps: add support for PSS reporting
https://gitlab.com/procps-ng/procps/-/issues/112
Signed-off-by: Jim Warner <james.warner@comcast.net>
This commit is strictly cosmetic. It was an attempt to
normalize/standardize/alphabetize those #define/#undef
statements. Some missing #undef's were added plus some
comments regarding sources corrected/standardized too.
Signed-off-by: Jim Warner <james.warner@comcast.net>
The commit referenced below was well done but needed a
small whitespace tweak to preserve existing alignment.
Reference(s):
. io accounting added
commit a7afe06e6f
Signed-off-by: Jim Warner <james.warner@comcast.net>
This is a modification of MR !122 by @renit1609 to fit the new
library.
Problem statement:
The procps library has no PROC_FILLIO flag to
fetch the proc field "/proc/[pid]/io" data
process-wise.
IO Accounting is not included as part of procps.
Requirement:
We have a requirement to fetch process wise
IO utilization which can be used for monitoring.
When looking through the procps library, I see
that IO Accounting (/proc/[pid]/io) is not being
included as part of procps. There is no such
flag like PROC_FILLIO being included in readproc.h .
Solution:
While looking at the implementation done for
other proc fields, I used the spare bits in app code.
I renamed PROC_SPARE_1 as PROC_FILLIO, the spare bit from
PROC_SPARE_* and used it for fetching /proc/[pid]/io
data as part of the procps library similar to other
fields. I moved the PROC_SPARE_* bits each by 1 bit
to retain the spare bits. Meanwhile added the IO fields
in proc_t structure.
References:
procps-ng/procps!122procps-ng/procps#184
nl_langinfo and CODESET are undefined in a musl system. Instead of
uncondionally including langinfo.h, this change includes include/nls.h
which has the tests and work-arounds for systems that don't have these
features. This is similar to how other programs within procps include
langinfo.h via nls.h
References:
procps-ng/procps!130
Signed-off-by: Craig Small <csmall@dropbear.xyz>
This simple two line code change fixes an intermittent
bug whereby %CPU for parent(s) with collapsed children
could be vastly understated from those displayed under
the current 3.3.17 publicly available top & libprocps.
If one started several top instances in the background
using very a small delay interval (zero?), then if the
shell under which they were running was collapsed, you
would see similar %CPU results for both the libraries.
However, when running a demanding 'make' like a kernel
compile (especially if backed by fast processors and a
SSD), then newlib would generally show only 1/3 to 1/2
of the collapsed %CPU values that appeared for 3.3.17.
Of course, now that the bug has been swatted with this
commit the disparities between those results is easily
explained. Since newly created tasks never contributed
tics during the interval where they were created, only
with many short lived tasks would differences surface.
Signed-off-by: Jim Warner <james.warner@comcast.net>
Rather than run risks of more expensive and repetitive
address resolution, we will establish this local index
for a one time cost and avoid any potential gcc bloat.
[ this commit was made in pursuit of a bug involving ]
[ the distortion of elapsed task tics. but, it turns ]
[ out these changes had nothing to do with that bug. ]
[ however, the patch is being retained as desirable. ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
That old library defined this field as 'unsigned int'.
However, here it was known as a 'signed int'. Thus for
consistency we'll now also treat it as 'unsigned int'.
[ this commit was made in pursuit of a bug involving ]
[ the distortion of elapsed task tics. but, it turns ]
[ out these changes had nothing to do with that bug. ]
[ however, the patch is being retained as desirable. ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
After removing brackets from those 'derived' notations
I was surprised to discover that several origin/source
comments were wrong. So this patch fixes those errors.
[ along the way a couple enumerators were renamed to ]
[ better (i hope) reflect what they're representing. ]
[ that, in turn, also required a little rearranging. ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
None of the other four new api headers use brackets on
derived items. With this patch we normalize the fifth.
[ it makes for a cleaner, less confusing, appearance ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
This small change was a result of some experimentation
trading our current 'hsearch' hash scheme for 'gperf'.
I discovered that when the ':' character was a part of
each 'gperf' key, that generated search logic was more
complicated and thus slower. But without a ':', it was
a little cleaner/leaner and therefore slightly faster.
Assuming that the same trailing ':' *might* affect the
current 'hsearch' logic, to be safe we will remove it.
[ while the 'gperf' version will slightly outperform ]
[ an 'hsearch', too many ugly implementation details ]
[ were exposed which complicates future maintenance. ]
[ thus, we'll retain our current 'hsearch' approach. ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch will condense some logic in those functions
associated with the file input operations. The changes
will not, for the most part, alter any generated code.
More significantly (though not very) was the change to
two 'strtoul' calls. Since the returned 'endptr' value
isn't exploited, when that parm is set to NULL, we can
save one instruction on each side of such calls (wow).
Signed-off-by: Jim Warner <james.warner@comcast.net>
When #define QUICK_THREADS was introduced, for copying
some task data for a child thread, one proc_t pad byte
was used to mark, then later identify, those children.
Later the QUICK_THREADS was recycled as FALSE_THREADS,
and used for a different purpose, but a conditional in
the header file erroneously remained. Now, it is gone!
Reference(s):
. Jul, 2016 - QUICK_THREADS become FALSE_THREADS
commit c546d9dd44
. Aug, 2011 - QUICK_THREADS intruduced
commit bb4f08ba29
Signed-off-by: Jim Warner <james.warner@comcast.net>
What to call the new library?
Keep using libprocps wouldn't do, its a very different library from
the programs' point of view. It would also mean we could have some
clashes around the packages (two package names, same library name).
The ancient procps used libproc or libproc-a.b.c where a.b.c was the
package version. Kept the revision numbers down (it was always 0.0.0)
but the name of the library changed.
So if we use libproc-2 is there a clash with an ancient procps?
procps v 2.0.0 was around in 1999 so it was 22 years ago, also the
name of the library would have been libproc-2.0.0.so not libproc-2.so
so we're fine with that.
libproc-2 seems to fit, our second major re-work of the procps
library.
Signed-off-by: Craig Small <csmall@dropbear.xyz>
Since 'procps_uptime' will access the /proc filesystem
the <pids> 'new' guy should should protect against the
possibility /proc isn't mounted when 'boot_seconds' is
established. A zero is better than the negative value.
[ the only distortion would be to PIDS_TIME_ELAPSED. ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
With the way those 'extents_free_all' guys were coded,
there's no real need to check for a NULL this->extents
before calling 'em. That's how <stat> already does it.
Signed-off-by: Jim Warner <james.warner@comcast.net>
In addition to copyright date, the initial descriptive
line was changed from a generic statement to one which
reflects the specific portion of the proc/ filesystem.
[ such descriptions alternate between 'declarations' ]
[ (h files) & corresponding 'definitions' (c files). ]
Also, a few missing copyright attributions were added.
Signed-off-by: Jim Warner <james.warner@comcast.net>
This commit just ensures that the relatively expensive
ID to name conversions aren't performed unless they're
explicitly requested. It also internalizes those flags
that required the PROC_FILLSTATUS flag to also be set.
[ requiring a caller, in our case pids.c, to provide ]
[ two flags when a single field was the objective is ]
[ wrong & represents a future potential toe-stubber. ]
[ moreover, what's worse is that those two flags are ]
[ seemingly unrelated. but, without both, a SEGV can ]
[ can be expected when a result.str pointer is NULL. ]
[ by contrast, in the master branch those fields are ]
[ arrays which, when set to zeroes, produce an empty ]
[ string. So, there is no abend (but no name either) ]
[ when one of those two required flags were omitted. ]
[ and worth noting, in that branch it's not just one ]
[ caller required to observe a two flag requirement. ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
With the 4 header files removed in the previous patch,
this commit just changes all those obsolete references
to that new consolidated 'misc.h' header file instead.
Signed-off-by: Jim Warner <james.warner@comcast.net>
Prior to this patch, we had four separate header files
dealing with miscellaneous functions. Those four files
were documented in the single man page: procps_misc.3.
Now, we will have just a single header file documented
in a single man page (that is what's called symmetry).
[ and while we're at it, we will shorten that overly ]
[ long struct 'procps_namespaces' name to agree with ]
[ the function naming conventions, e.g. 'procps_ns'. ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
The preceding commit made that 'esc_all' function more
efficient by using direct pointer manipulation instead
of an indexed string reference approach within a loop.
This commit applies the same approach to the 'esc_ctl'
function. Now we'll save 12 more iterated instructions
while decreasing the function's code size by 43 bytes.
Signed-off-by: Jim Warner <james.warner@comcast.net>
While this patch has some cosmetic whitespace changes,
more importantly it makes that 'esc_all' function more
efficient. By abandoning the indexed loop approach for
a direct pointer manipulation, we will save 9 iterated
machine instructions, for a total of 33 bytes of code.
Signed-off-by: Jim Warner <james.warner@comcast.net>
Much of what was represented in the commit message for
the reference shown below was revisited in this patch.
It also means that the assertion in the last paragraph
of that message will only now be true with LANG unset.
[ and forget all the bullshit about not altering any ]
[ kernel supplied data. sometimes we must to avoid a ]
[ corrupt display due to a string we can not decode. ]
And while this commit still avoids the overhead of the
'mbrtowc', 'wcwidth' 'isprint, & 'iswprint' functions,
we achieve all the benefits with simple table lookups.
Plus such benefits are extended to additional strings.
For example, both PIDS_EXE and PIDS_CMD fields are now
also subject to being 'escaped'. If a program name did
contain multibyte characters, potential truncation may
corrupt it when it's squeezed into a 15/63 byte array.
Now, all future users of this new library only need to
deal with the disparities between string and printable
lengths. Such strings themselves are always printable.
[ the ps program now contains some unnecessary costs ]
[ with the duplicated former 'escape' functions. But ]
[ we retain that copied escape.c code for posterity. ]
[ besides, in a one-shot guy it's of little concern. ]
Note: Proper display of some multibyte strings was not
possible at the linux console. It would seem a concept
of zero length chars (like a 'combining acute accent')
is not recognized. Thus the display becomes corrupted.
But if utf8 decoding is disabled (via LANG=), then all
callers will now see '?', restoring correct alignment.
Reference(s):
. Dec 2020, newlib 'escape' logic refactored
commit a221b9084a
Signed-off-by: Jim Warner <james.warner@comcast.net>
This new library provides callers with pure strings or
string vectors. It is up to those callers to deal with
potential utf8 multibyte characters and any difference
between strlen and the corresponding printable widths.
So, it makes no sense for the library to go to all the
trouble of invoking those rather expensive 'mbrtowc' &
'wcwidth' functions to ultimately yield total 'cells'.
Thus, this patch will eliminate all the code and parms
that are involved with such possible multibyte issues.
[ Along the way we'll lose the ability to substitute ]
[ '?' for an invalid/unprintable multibyte sequence. ]
[ We will, however, replace ctrl chars with the '?'. ]
[ This presents no problem for that ps program since ]
[ it now duplicates all of the original escape code. ]
[ And, we'll no longer be executing that code twice! ]
[ As for the top program, it takes the position that ]
[ it is wrong to alter kernel supplied data. So with ]
[ potential invalid/unprintable stuff, he'll rely on ]
[ terminal emulators to properly handle such issues! ]
[ Besides, even using a proper multibyte string, not ]
[ all terminals generate the proper printable width. ]
[ This is especially true when it comes to an emoji. ]
[ And should callers chose not to be portable to all ]
[ locales by calling setlocale(LC_ALL, ""), they can ]
[ expect to see lots of "?", regardless of what this ]
[ library fixes in a faulty multibyte string anyway. ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
After 'errno' management was standardized, a couple of
fields were added to the <pids> api. Only 1 (PIDS_EXE)
involved dynamic memory and, unfortunately, it did not
conform to that expected normalized ENOMEM convention.
Reference(s):
. Jun 2018, added 'exe' to library
commit ad4269f118
. Nov 2017, standardized 'errno' management
commit 06be33b43e
Signed-off-by: Jim Warner <james.warner@comcast.net>
Under the above #define this commit now also addresses
2 additional possible toe stubbers involving 'select'.
If some readproc.h constants were uncoupled from their
pids.h enumerators a 'make check-lib' will now detect.
Signed-off-by: Jim Warner <james.warner@comcast.net>
If a hash results report is output (via UNREF_RPTHASH)
a portion is devoted to occupied table entries ordered
by depth. There is a possibility that some depths will
not be found among existing occupied table entries and
to avoid any confusion probably should not be printed.
[ to illustrate the potential for confusion prior to ]
[ this patch, force a very small table size (like 8) ]
[ & then trigger the procps_pids_unref() eoj report. ]
So this patch ensures only 'in use' entries are shown.
[ admittedly, all of the remaining logic in the loop ]
[ could/should be subordinate to this new 'if' test, ]
[ but we will keep the change to a minimum. besides, ]
[ there's no harm subtracting/adding a zero numdepth ]
[ especially since the chance of a zero is very low. ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch just raises the size of the hash table used
to calculate elapsed task stuff. The net result should
be less need for 'chaining' under pid hash collisions.
[ the hash scheme is intentionally kept as primitive ]
[ and, therefore, as fast as possible. it employs an ]
[ 'and' approach versus a 'mod' operation since both ]
[ yield similar distribution but the former approach ]
[ was 4 fewer cpu instructions in terms of overhead. ]
[ additionally, for hash collisions, 'chaining' uses ]
[ an array index rather than the usual pointer since ]
[ the HST_t guys may move when they are reallocated. ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch separates the memory allocations into those
used initially from those used in later reallocations.
Thus, we can reduce that iterative realloc() overhead.
Additionally, we'll correct a long standing oops where
multiple history_info structures were created at 'new'
time when only one should have been allocated (jeeze).
[ originally the allocation was strangely based upon ]
[ number of 'items' (???) & later a #define constant ]
Reference(s):
. May, 2016 - subsequent bad history_info logic
commit 9ebadc1438
. Aug, 2015 - original faulty history_info code
commit 7e6a371d8a
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch may be a little misleading in terms of size
since most of the changes just reorder a little logic.
The most significant changes involve two GUEST values.
My original implementation excluded such tics from the
TOTAL calculation and, therefore, the BUSY figure too.
That decision was erroneously based on some code found
in ./kernel/sched/cputime.c which in hindsight applies
only to processes, not those system level cpu figures.
[ another likely oops classified STOLEN tics as IDLE ]
So, this patch attempts to bring those SUM values into
better agreement with the calculations performed for a
root cgroup (see ./kernel/cgroup/rstat.c source file).
[ we differ from those above in that we also include ]
[ the IDLE plus IOWAIT tics in our TOTAL calculation ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
There's a huge toe-stubber awaiting future maintenance
as reflected in that commit below which deals with the
addition of new enumerators to the Item_table. Namely,
whenever the table is grown, one must remember to also
change that existing 'logical_end' enumerator's value.
Well, not anymore! Since that MAXTABLE macro was added
to the procps-private.h header we can now also exploit
it so a 'logical_end' automatically tracks table size.
This change also renders some code associated with the
ITEMTABLE_DEBUG #define unnecessary. So it's gone too.
Reference(s):
. 08/2016, add new enumerators
commit 09e1886c9e
. 08/2020, added MAXTABLE macro
commit c865b06c30
. 08/2020, introduced ITEMTABLE_DEBUG
commit 92d0297e1e
Signed-off-by: Jim Warner <james.warner@comcast.net>
At 'new' time, the major API modules each ensure their
'info' parameter isn't NULL but what it pointed to was
except this single straggler, for some unknown reason.
So, this patch brings him into line with those others.
[ And, without going into the ugly details, this was ]
[ the reason I never experienced an abend originally ]
[ but Craig did. And, though related to stacks mgmt, ]
[ zero initialization was not a factor. Anyway, with ]
[ this patch, everybody would have experienced abend ]
[ under the original (faulty) test_Itemtables logic! ]
Reference(s):
https://www.freelists.org/post/procps/keep-on-patchin,13
Signed-off-by: Jim Warner <james.warner@comcast.net>
Me thinks Craig had the right idea but perhaps not the
most correct solution. As currently structured, all of
the tests now go way too far by checking every 'unref'
and 'new' call when what we're trying for is survival.
In the final analysis, it doesn't matter who issues an
EXIT_FAILURE - that run_tests guy or an early Exit out
of a procps_new() function. They both will produce the
same end result of the desired "FAIL" test diagnostic.
[ and this patch once again allows the slabinfo test ]
Reference(s):
commit 4eeed6dcff
Signed-off-by: Jim Warner <james.warner@comcast.net>
The referenced commit introduced a test program for the API
but it would often fail due to:
The given pointer for _new() not being NULL
The return value for _new not checked, so the subsequent _unref()
would free() random memory
slabinfo checks failing due to permission denied errors.
The first two are fixed, as well as returning a fail to the test
if they don't return correctly, with slabinfo waiting to see if there
is a way of initialising the structure without reading the slabinfo.
References:
commit e616409aa4
Signed-off-by: Craig Small <csmall@dropbear.xyz>
This change sets the stage for exploiting the recently
added ITEMTABLE_DEBUG #define. All tests are performed
in a single module (after trying 6 separate programs).
The chances of each test detecting errors is extremely
remote (at least while I'm maintaining these modules).
However, this single program approach has one flaw and
it relates to the response whenever an error is found.
Each of those six new API modules calls Exit() if they
detect an error. Otherwise, incorrect results would be
produced at the least or an abend encountered at most.
This means that multiple 'make check' invocations will
be needed if more than 1 module actually was in error.
All in all, it is a small price for a large assurance.
Reference(s):
https://www.freelists.org/post/procps/keep-on-patchin,7
Signed-off-by: Jim Warner <james.warner@comcast.net>
The recent work on updating the <meminfo> and <vmstat>
modules with some newly added linux fields reminded me
(again) of a need for some mechanism guaranteeing that
a header file agrees with the source file assumptions.
Sadly, in the past, if a table entry was omitted or if
the table and header are ordered differently, then the
library would silently return the wrong results values
or even potentially experience a SIGSEGV abnormal end.
This patch offers a much needed development assist for
ensuring that Item_table entries are synchronized with
header file enumerators in terms of number plus order.
It's intended solely for our use as libprocps evolves.
Now, by activating ITEMTABLE_DEBUG, either directly or
via ./configure CFLAGS='-DITEMTABLE_DEBUG', the number
and order will be verified. It is envisioned that this
feature will be used at least once prior to a release.
Signed-off-by: Jim Warner <james.warner@comcast.net>
This macro should help the following patch be a little
less prolix. Besides, this private header could/should
do just a little more to help with our newlib efforts.
Signed-off-by: Jim Warner <james.warner@comcast.net>
This cosmetic change just prepares for the later patch
introducing validation of Item_tables and enumerators.
[ and, we'll now have better 'set' function names of ]
[ 'set_diskstats_ENUM' instead of the more redundant ]
[ current 'set_diskstats_DISKSTATS_ENUM' convention. ]
[ now our only exception is the <slabinfo> api where ]
[ a full enumerator identifier with 'SLAB' & 'SLABS' ]
[ prefixes are used, and 'SLABINFO_noop/extra' guys. ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
Since last visited, there have been several new fields
added and one field deleted in the /proc/meminfo file.
[ references shown below represent linux git commits ]
Reference(s):
. 4/2020, added 'ShadowCallStack'
commit 628d06a48f57c36abdc2a024930212e654a501b7
. 9/2019, added 'FileHugePages' & 'FilePmdMapped'
commit 60fbf0ab5da1c360e02b7f7d882bf1c0d8f7e32a
. 9/2019, removed 'Quicklists'
commit 13224794cb0832caa403ad583d8605202cabc6bc
Signed-off-by: Jim Warner <james.warner@comcast.net>
This commit attempts to smooth some wrinkles impacting
any future libprocps user exploitation. The 2 problems
relate exclusively to our XTRA_PROCPS_DEBUG provision.
1. The 'xtra-procps-debug.h' header had an include for
'procps-private.h', which was not an installed header.
So the STRINGIFY macros will now be embedded directly.
2. Each of the new api headers referenced '<proc/...>'
rather than '<procps/...>' for the debugging #include.
So, we must drop that prefix in favor of a quoted file
name so that debugging builds work regardless of where
that 'xtra-procps-debug.h' header happens to be found.
Reference(s):
. Jun, 2020 - changed target install directory
commit d64755ea5f
. Sep, 2018 - #include migrated from procps.h
commit fab37662ef
Signed-off-by: Jim Warner <james.warner@comcast.net>
I almost feel like I should apologize for this kind of
patch ( *almost* ). But, since this comment was unique
to the <pids> api and since it was especially designed
to align with the following comment and since the next
comment already carried the alignment emphasis, I will
refrain from issuing any apology and submit it anyway.
[ plus to prove that i am not totally anal-retentive ]
[ and can exercise some restraint there are two more ]
[ places where i COULD have added alignment emphasis ]
[ see lines 804-806 and lines 1360-1361 for my proof ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch just trades the 'typestr' for the much more
meaningful 'valid range' when a relative enum has been
deemed invalid under that XTRA_PROCPS_DEBUG provision.
[ and we'll also make one comment a bit more generic ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
In that commit referenced below, a promise was made to
revisit an 'escape_str' function in efforts to make it
private to the library. The problem was it's needed by
both ps plus the library which is why it was exported.
So, in an effort to remove it from libprocps.sym, this
patch duplicates all the required code in ps/output.c.
Now, each version can be made private to their caller.
[ along the way we'll use this opportunity to remove ]
[ the 'restrict' qualifiers from function parameters ]
[ while swatting a compiler warning referenced below ]
Reference(s):
. April 2016, most escape functions made private
commit d916d5db86
proc/escape.c: In function `escape_command':
proc/escape.c:182:23: warning: initialization of `const char **' from incompatible pointer type `char **' [-Wincompatible-pointer-types]
182 | const char **lc = (char**)pp->cmdline;
| ^
Signed-off-by: Jim Warner <james.warner@comcast.net>
There was a time when that procps.h file served a more
traditional role. Prior to the commit referenced below
it held just macros plus manifest constants. But, with
that change, such items were replaced with a series of
includes embracing all the library exported functions.
That approach was known to disguise errors which would
have otherwise yielded a compiler warning. And without
such a warning, there was no way to address the error.
So this patch will trade the all inclusive header file
approach for individual includes only where necessary.
Reference(s):
. April 2016, procps.h header file revamped
commit ccb6ae8de1
. Sept 2018, top abandoned use of procps.h
commit a6dfc2382e
Signed-off-by: Jim Warner <james.warner@comcast.net>
The meminfo & vmstat file read buffers are now defined
through symbolic constants next to the symbolic names.
[ recent attention to the /proc/stat buffer size led ]
[ to this (hopefully) better approach to definitions ]
Reference(s):
commit 81f4a6acdf
Signed-off-by: Jim Warner <james.warner@comcast.net>
We won't go as far as the merge request shown below in
increasing buffer size, but we will reduce by half the
total iterations while growing it to achieve one read.
[ and this seems in line with what was recently done ]
[ to help top under a massively parallel environment ]
[ thru 2 abreast ('4') and combined cpus ('!') modes ]
Reference(s:
https://gitlab.com/procps-ng/procps/-/merge_requests/105
Signed-off-by: Jim Warner <james.warner@comcast.net>
Now that we may be getting serious with documentation,
that stat module was revisited with an eye toward user
friendliness. Heck, even this author puzzled over some
of the existing notes and naming conventions employed.
So, this patch will adjust some identifiers and expand
the notes to (hopefully) better serve potential users.
The most significant change was making the STAT_TIC_ID
always valid for numa nodes, even if any are inactive.
Thus the -22222 special STAT_NODE_INVALID constant now
is applied only to STAT_TIC_NUMA_NODE. It will be used
on the cpu summary and reaps with STAT_REAP_CPUS_ONLY.
And it will also mark any numa node that was inactive.
Signed-off-by: Jim Warner <james.warner@comcast.net>
In preparation for a documentation attempt, this patch
renames 'struct reap' to 'struct reaped' in two of our
interfaces. That will bring all the non-pid interfaces
into agreement since the <stat> api already used that.
Signed-off-by: Jim Warner <james.warner@comcast.net>
Thanks to Konstantin for discovering 2 problems in the
issue referenced below. That 15+ year old logic went a
little too far overboard wrestling with a utf8 string.
Henceforth, we will not treat 'x9b' as special. And we
also will handle a 'combining acute accent' correctly.
Reference(s):
https://gitlab.com/procps-ng/procps/-/issues/176
Signed-off-by: Jim Warner <james.warner@comcast.net>
From http://man7.org/linux/man-pages/man5/proc.5.html:
(22) starttime %llu
The time the process started after system boot. In
kernels before Linux 2.6, this value was expressed
in jiffies. Since Linux 2.6, the value is expressed
in clock ticks (divide by sysconf(_SC_CLK_TCK)).
Well, shit! With release 4.0 on March 25th the lxc/lxd
folks have stuck it to us once again. They changed the
cgroup lxc prefix used to identify the container name.
Signed-off-by: Jim Warner <james.warner@comcast.net>
All TIC delta fields are checked for possible negative
results and set to zero when found. This is done so as
to protect against potential anomalies which depend on
kernel version and/or toggling cpus offline or online.
[ it's probably unnecessary with the latest kernels, ]
[ except for iowait. documentation suggests it might ]
[ decrease which would then create a negative delta. ]
The same approach is employed for most of the 'system'
deltas (ctxt, intr & procs_created). However, with two
of the fields (procs_blocked & procs_running) negative
results were allowed. But it now seems such a division
is unwise so this patch will allow all to go negative.
[ rather than force any 'system' delta value to show ]
[ what's logical, we'll now let all reflect reality. ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch just standardizes/normalizes the whitespace
employed within a couple of nearly identical #defines.
Signed-off-by: Jim Warner <james.warner@comcast.net>
Under newlib, the only caller of the readproc routines
is that pids module. And in every case, the address of
some static proc_t structure has always been provided.
As a result, there is no need for the logic supporting
calloc() for a possible NULL pointer which was present
in both of those readproc() and readeither() routines.
Additionally, that pids module takes ownership of most
dynamically acquired 'str' plus 'strv' memory whenever
assigning to a results structure. So, henceforth under
the free_acquired() guy we will only free those string
fields which might exist when not explicitly selected.
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch just eliminates a parameter present for the
simple_readtask() function which is not needed nor has
it ever actually been used. It will make calls to that
function (via taskreader ptr) slightly more efficient.
Signed-off-by: Jim Warner <james.warner@comcast.net>
When some cleanup was performed on the readproc module
in the commit shown below, some residual code involved
with the 'did_fake' flag remained. Since such logic no
longer served any real need, this patch will whack it.
Reference(s):
. cleanup of readproc functions
commit 887bb51016
Signed-off-by: Jim Warner <james.warner@comcast.net>
No libprocps user expects signal values to be returned
as 'long long' quantities. More importantly the <PIDS>
api only returns a 'str' result for signal categories.
So this patch eliminates all the conditional code that
depends on the absence of the #define 'SIGNAL_STRING'.
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch will just correct an oops introduced in the
commit shown below. Thank goodness both 'str' & 'strv'
occupy the same storage location in that result union.
Reference(s):
. standardize 'errno' management
commit 06be33b43e
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch just implements an equivalent to the master
branch 'CPU_ZEROTICS' provision. However, the original
impetus for that earlier implementation was ultimately
attributed to a likely kernel anomaly since corrected.
As a result, in this newlib implementation we take the
opposite approach to the default behavior. There is no
adjustment to TIC_SUM_DELTA values if fewer ticks than
expected are recorded, unless the define is activated.
The commit shown below explains why the 'CPU_ZEROTICS'
define was retained in spite of the fix to the kernel.
Reference(s):
. issue referencing CPU_ZEROTICS
https://gitlab.com/procps-ng/procps/issues/132
. master branch CPU_ZEROTICS summary
commit ee3ed4b45e
. lengthy thread leading to CPU_ZEROTICS
https://www.freelists.org/post/procps/CStates-handling-new-switch
Signed-off-by: Jim Warner <james.warner@comcast.net>
Some parts of our newlib implementation are the result
of functions which have been propagated from module to
module. In particular, those 'cleanup_stacks' routines
are all similar & likely originated in the <pids> api.
In that interface there was a need to free dynamically
acquired memory before the result structure was reused
to satisfy subsequent 'get', 'select' or 'reap' calls.
This, in turn, led to a concept of 'dirty' stacks with
the need to call one of two 'cleanup_stack' functions.
None of the remaining interfaces deal with such memory
yet they each had their own 'cleanup_stack' functions.
Those functions were responsible for resetting each of
the result unions to zero, excluding any 'noop' items.
The bottom line is that for all interfaces, repetitive
calls would require iterating through the stack(s) two
separate times: once to 'cleanup' another to 'assign'.
With this commit we will reduce iterations to just the
'assign' routine. A reset to zero will be accomplished
in the 'extra' item set routine (which is the only one
actually requiring any reset). All other items will be
reinitialized automatically by a new current set value
or upon reallocation when an items compliment changes.
In the <pids> interface, any freeing of dynamic memory
could have been accomplished by adding that 'freefunc'
check to the 'assign' function. However, that requires
an Item_table test with every item. Instead, we'll now
satisfy such needs as the very first step in those set
functions responsible for dynamically acquired memory.
[ the <pids> api retains 2 'cleanup_stack' functions ]
[ to accommodate stack(s) 'reset' & to serve 'unref' ]
Lastly, all the 'itemize_stack' functions were tweaked
by eliminating an unnecessary initialization of result
unions. That objective was already accomplished by the
calloc() in a 'stacks_alloc' function or the remaining
'cleanup_stack' routine found in the <pids> interface.
Signed-off-by: Jim Warner <james.warner@comcast.net>
These changes are an outgrowth of the research/testing
behind the previous commit. There is no commingling of
select/reap stacks in interfaces beyond the <pids> api
since there's no need to support any 'reset' function.
However, those <pids> changes prompted a review of all
interfaces offering that 'stacks_fetch' function, thus
revealing 2 instances of useless logic/wasted efforts.
Signed-off-by: Jim Warner <james.warner@comcast.net>
When the <pids> api was refactored in the commit shown
below, one objective was enabling the simultaneous use
of 'get' & 'select/reap' functions. Unlike other 'get'
functions, this <pids> 'get' acts as an iterator where
successive calls will return successive tasks/threads.
However, that goal wasn't quite met since a stack used
by 'get' was commingled with the 'select/reap' stacks.
Such commingling supported the 'reset' function, again
a provision which was unique to this <pids> interface.
Unfortunately, some poor assumptions in 'stacks_fetch'
produced a SEGV whenever 'reap/select' followed 'get'.
Thus, this patch addresses those issues and guarantees
such commingled stacks (extents) will be accommodated.
Reference(s):
. standardize portions of interface, <PIDS> api
commit 9ebadc1438
Signed-off-by: Jim Warner <james.warner@comcast.net>
The commit referenced below addressed (some) anomalies
surrounding 'strv' pointers. However, there remained a
couple quirks involving a potential NULL return value.
Any NULL values returned from the old library readproc
guys would cause no real harm for newlib. But they did
produce the misleading "[ duplicate ENUM_ID ]" result.
The following all represent potential NULL results and
suggest shortcomings in testing of that earlier patch.
. kernel threads do not have cgroup, cmdline & environ
. even if present environ could require root to access
So, this patch reverts a portion of the earlier commit
and ensures when some vectored string is not available
a traditional dash ('-') is the 'strv' returned value.
[ and we'll also correct one typo in the header file ]
Reference(s):
. eliminated a final potential NULL, <PIDS> api
commit 09503dc597
Signed-off-by: Jim Warner <james.warner@comcast.net>
This commit is intended as a refinement of the patches
mentioned below, where origins/sources of newlib items
were added to the header files for user documentation.
However, if those additions are to be truly effective,
along with kernel documentation (where available), the
following prerequisites must also have been satisfied:
. our identifiers closely align with linux field names
. our derived items are documented or self-documenting
Satisfying those prerequisites prompted this patch and
for these changes, kernel sources were emphasized over
available documentation (shame on me, it should always
have been so). And, while some 'new' fields were found
to be conditional, they were included unconditionally.
These changes appear more extensive than they actually
need be since I have attempted to enforce some spacing
conventions. So, I've summarize the significant things
in the sections that follow. For a proper perspective,
use: 'git diff --ignore-space-change' (good as alias).
___________________________________________ <PIDS> api
This api is unique in that there exists many different
file/directory origins subordinate to /proc/<pid>. And
our item identifiers are sometimes coerced so as to be
able to group related or similar enumerators together.
So, users needed more help relating our identifiers to
an actual documented field. Thus, we will now also add
the field names as with 'stat: delayacct_blkio_ticks'.
Each item ending with a '_C' now consistently includes
both the parent's count/time plus waited for children.
That 'RTPRIO' guy was renamed/relocated as PRIORITY_RT
since its original name is an implementation artifact.
___________________________________________ <STAT> api
The only api change was to correct a typo ('dervied').
_________________________________________ <VMSTAT> api
Even ignoring white space, this interface received the
largest number of changes. Mostly, this was because of
deficiencies in the proc(5) documentation. Recall that
this documentation already sorely lacks any substance.
Usually, just kernel releases are noted, not contents.
When compared to kernel source, that proc(5) contained
many non-existent fields and also omitted many others.
________________________________________ <MEMINFO> api
Sadly, with this api many of the changes were simply a
correction of some earlier 'human error' where several
fields where hashed then tracked but never represented
with an item enumerator in this meminfo.h header file.
_______________________________________ <SLABINFO> api
The 'SLABS' (summary) & 'SLABNODE' items were reversed
since the former are derived from the separate caches.
More significantly, those 'SLABNODE' guys were renamed
to 'SLAB' since they concern individual caches and the
concept of 'nodes' is really an implementation detail.
Also, several enumerators were changed to more closely
agree with official slabinfo(5) documentation referred
to in what we're treating as a base document: proc(5).
Lastly, while those 'SLABS' items are solely a product
of our library and not represented in slabinfo(5), the
names attempt to parallel those found as 'SLAB' items.
______________________________________ <DISKSTATS> api
One enumeration identifier was changed so as to better
reflect its relationship to that actual documentation:
'Documentation/iostats.txt', as referenced in proc(5).
Reference(s):
. 12/2018, item origins added (and commit msg history)
commit 96d59cbf46
. 01/2019, <stat> origins tweaked
commit 201e816b26
Signed-off-by: Jim Warner <james.warner@comcast.net>
Rather than offer three separate patches, they've been
consolidated in this single commit. All are related in
that they surfaced while preparing a subsequent patch.
------------------------------------------------------
library: correct a broken '#if define', <SLABINFO> api
It was introduced (embarrassingly) in the patch below.
Reference(s):
commit 97d078a9af
------------------------------------------------------
library: correct a broken 'GET' macro, <DISKSTATS> api
In the patch referenced below, which purported to make
all the 'GET' macros robust, the 'DISKSTATS_GET' macro
was broken. A necessary parameter wasn't passed to the
subsequently invoked function: procps_diskstats_get().
Reference(s):
commit bef8c7fb70
------------------------------------------------------
library: correct a broken 'sort' func, <DISKSTATS> api
In the commit shown below, an attempt to normalize the
errno handling, the sort function inadvertently lost 1
crucial line of code which produces a consistent SEGV.
Reference(s):
commit 06be33b43e
Signed-off-by: Jim Warner <james.warner@comcast.net>
Since the patch referenced below traded a compile-time
'sizeof' directive for a run-time 'strlen' call, there
is no need to declare lxc patterns as explicit arrays.
We'll also use the actual lxc patterns by omitting the
beginning slashes ('/') for both of those definitions.
And, looking to the future when most/all lxc users are
using the most recent lxc release, we will make things
slightly more efficient by reversing those two pattern
literals so the most recent pattern was checked first.
Of course, such a change only benefits tasks which are
running in a container. For the majority of processes,
both literals will be compared in that 'if' statement,
assuming the 'LXC' field is currently being displayed.
[ plus, a leftover parenthesis pair has been removed ]
Reference(s):
commit 288d759b8b
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch just polishes the 'origin' comments for the
<STAT> header file. In particular those derived/unique
items (the 'SUM' guys) will now be properly explained.
[ in order to employ the 'derived from above' phrase ]
[ with their DELTA versions, all SUM items had to be ]
[ relocated (and some renamed). in turn, that had an ]
[ impact on many portions of the .c source file too. ]
Reference(s):
. summary calculations introduced
commit 2c86c4984a
. origins added to header files
commit 96d59cbf46
Signed-off-by: Jim Warner <james.warner@comcast.net>
This small change is an outgrowth of the research into
the bug represented by that merge request shown below.
With the master branch, a real buglet was subsequently
addressed. In this newlib branch, no bug existed since
the <stat> API relies solely on just cpus reflected in
(and parsed from) the kernel's /proc/stat pseudo file.
[ since that procps_stat_new() priming read about to ]
[ be performed will value info->cpus.total, there is ]
[ no need to separately invoke a procps_cpu_count(). ]
Reference(s):
https://gitlab.com/procps-ng/procps/merge_requests/82
Signed-off-by: Jim Warner <james.warner@comcast.net>
The merge request shown below prompted (thankfully) an
examination of our lxc containers logic in readproc.c.
As it turns out, the lxc folks changed that eyecatcher
used to identify containers within a task cgroup file.
So this patch, with little extra cost, will enable the
libprocps lxc_containers() guy to handle both strings.
[ additionally, I was shocked to find lxc allows the ]
[ eyecatcher to be changed at ./configure time. such ]
[ a provision has always existed. unfortunately, the ]
[ changed value was only available to root, assuming ]
[ one wished to tackle that undocumented liblxc api. ]
Reference(s):
. what prompted lxc support reevaluation
https://gitlab.com/procps-ng/procps/merge_requests/82
. original lxc support introduced
commit 0557504f9c
Signed-off-by: Jim Warner <james.warner@comcast.net>
A lack of documentation seems to be the major obstacle
to releasing this new library. So, in an effort to get
the ball rolling again, this patch adds the origins of
each item as a comment to six of the new header files.
However, before reviewing how such changes may benefit
that documentation objective, it seemed appropriate to
first reflect on newlib's background & current status.
___________________________________________ BACKGROUND
Discussions about and work on a new library began back
in July 2012 but quickly died. After a lull of 2 years
those discussions were resumed in August 2014 but soon
died also (and no code survived the gitorious demise).
With those early discussions, the recommended approach
was to encapsulate all of the libprocps data offerings
in individual functions. When it came to extensibility
it was suggested we should rely on symbols versioning.
Unfortunately that approach would have made for a huge
Application Programming Interface virtually impossible
to master or even document. And, runtime call overhead
would have been substantial for ps and especially top.
So, an alternative design was sought but there were no
new suggestions/contributions via freelists or gitlab.
Thus, in spite of a lack of library design experience,
the procps-ng team (Craig & Jim) set out to develop an
alternative API, more concise and with lower overhead.
Reference(s):
. 07/01/2012, begin library design discussion
https://www.freelists.org/post/procps/Old-library-calls
. 08/12/2014, revival of library design discussion
https://www.freelists.org/post/procps/libprocs-redesign
_____________________________________ DESIGN EVOLUTION
Our newlib branch first appeared on June 14, 2015. And
our current API actually represents the 4th generation
during the past 3 years of evolution. First, there was
a basic 'new', 'get' and 'unref' approach, using enums
to minimize the proliferation of 'get' function calls.
Then, in anticipation of other programs like ps, where
multiple fields times multiple processes would greatly
increase the number of 'get' function calls, a concept
of 'chains' was introduced. This became generation #2.
Such 'chains' proved unnecessarily complex so 'stacks'
replaced them. This was considered the 3rd generation,
but too many implementation details were still exposed
requiring those users to 'alloc', 'read', 'fill', etc.
Finally, a 4th generation emerged representing several
refinements to standardize and minimize those exported
functions, thus hiding all implementation details from
the users. Lastly, handling of 'errno' was normalized.
Reference(s):
. 06/14/2015, revival of new API discussion
https://www.freelists.org/post/procps/The-library-API-again
. 06/24/2015, birth of the newlib branch
https://www.freelists.org/post/procps/new-library
. 06/29/2015, 2nd generation introduced 'chains'
https://www.freelists.org/post/procps/new-library,8
. 07/22/2015, 3rd generation introduced 'stacks'
https://www.freelists.org/post/procps/newlib-stacks-vs-chains
. 06/18/2016, 4th generation refinements begin
https://www.freelists.org/post/procps/newlib-generation-35
. 11/10/2017, 4th generation standardized 'errno'
https://www.freelists.org/post/procps/some-more-master-newlib-stuff
_______________________________________ CURRENT DESIGN
Central to this new design is a simple 'result' struct
reflecting an item plus its value (thanks to a union).
As a user option, these item structures can be grouped
into 'stacks', yielding many results with just 1 call.
Such a 'stack' can be seen as a variable length record
whose content/order is determined solely by the users.
Within that 'result' structure, the union has standard
C language types so there is never a doubt how a value
should be used in a printf statement. Given that linux
requires a least a 32-bit platform the only difference
in capacity surrounds 'long' integers. And, where such
types might be used, the 32-bit maximums are adequate.
The items themselves are simply enumerators defined in
the respective header files. A user can name any items
of interest then the library magically provides result
structure(s). The approach was proven to be extensible
without breaking the ABI (in commit referenced below).
The 6 major APIs each provide for the following calls:
. 'new' ---------> always required as the first call .
. 'ref' -------------------------> strictly optional .
. 'unref' --------> optional, if ill-behaved program .
. 'get' --------------------> retrieve a single item .
. 'select' ----------------> retrieve multiple items .
And the 'get' and 'select' functions provide for delta
results representing the difference between successive
get/select calls (or a 'new' then 'get/select' call).
For the <diskstats>, <pids>, <slabinfo> & <stat> APIs,
where results are unpredictable, a 'reap' function can
return multiple result structures for multiple stacks.
The <pids> API differs from others in that those items
of interest must be provided at 'new' or 'reset' time,
a function unique to this API. And the <pids> 'select'
function requires PIDs or UIDs which are to be fetched
which then operates as a subset of 'reap'. Lastly, the
'get' function is an iterator for successive PIDs/TIDs
returning items previously identified via 'new/reset'.
To provide assistance to users during development, the
special header 'proc/xtra-procps-debug.h' is available
to check type usage against library expectations. That
check is activated by including this header explicitly
or via build using: ./configure '-DXTRA_PROCPS_DEBUG'.
Reference(s):
. 08/05/2016, type validation introduced
https://www.freelists.org/post/procps/newlib-types-validation
commit e3270d463d
. 08/11/2016, extensibility while preserving ABI example
https://www.freelists.org/post/procps/new-meminfo-fields
commit 09e1886c9e
_________________________ INITIAL DOCUMENTATION EFFORT
The initial attempt, referenced below, dealt primarily
with the <pids> interface. Separate man pages for each
exported function were created. Plus there was another
document describing the items, among other miscellany.
Adopting such an approach encounters several problems:
1. In order to use these man pages, users are required
to already know how to use the library. Or alternately
one could randomly search each of them while trying to
ascertain which function call satisfies their need and
what exactly was the proper compliment/order required.
2. While we can explain what all of those <pids> items
represent, that certainly isn't true for all the APIs.
See the gaps in kernel documentation for <meminfo> and
complete lack of documentation with that <vmstat> API.
3. Our documentation effort should take pains to avoid
unnecessary implementation details. Here's an example:
. "The pointer to info will have memory"
. "allocated and a structure created."
Alternatively, the following conveys user requirements
while not offering any internal implementation detail:
. "You must provide the address of a NULL"
. "info structure pointer."
Reference(s):
. 01/04/2017, initial documentation offering
https://www.freelists.org/post/procps/Using-reap-and-get
commit 2598e9f2ce
___________________ RECOMMENDED DOCUMENTATION APPROACH
I recommend that the newlib documentation consist of 3
man pages only. The first would cover the 5 major APIs
and their common functions. The second would deal with
the <pids> API exclusively, explaining how it differs.
Any remaining exported libproc functions which are yet
to be included could be represented in a 3rd document.
For these new documents the following are are assumed:
1. Since we will not be able to document all items, we
shouldn't try to document any items. We should instead
rely on proc(5) or Documentation/filesystems/proc.txt.
2. Program development often involves referencing some
header file(s). So, make that an absolute requirement.
3. With the addition of item origins, represented with
this commit, and considering that 'types' were already
present, the header file might be all some users need.
4. And who knows, when a user of our libproc complains
about gaps in their documentation, it might prompt the
kernel folks to correct those long standing omissions.
To summarize, I suggest that we replace that libproc.3
document with a more general one explaining the basics
of accessing this new library and the common calls for
most of the major interfaces. We can then create a new
document (libproc-pids.3?), which explains differences
in using the <PIDS> application programming interface.
A final document (libproc-misc.3?) covers what's left.
Signed-off-by: Jim Warner <james.warner@comcast.net>
This change is being made in anticipation of adding the
source origin of each item to the <pids.h> header file.
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch will bring the <meminfo> API into line with
that proc(5) document. There were several undocumented
fields that were not noted and these two were omitted:
. 'MmapCopy' was conditional on the #define CONFIG_MMU
. 'Quicklists' depends on the #define CONFIG_QUICKLIST
And we're about to get the following new field in 4.20
which will be represented, at least, in that proc.txt:
. 'KReclaimable' will include SReclaimable plus others
Signed-off-by: Jim Warner <james.warner@comcast.net>
If we ever were to eliminate the procps.h header file,
as discussed in the thread referenced below, then that
would impair the current XTRA_PROCPS_DEBUG provisions.
The only remaining way to verify result types would be
to explicitly include that <proc/xtra-procps-debug.h>.
So, this commit will once again enable the ./configure
provision for defining the -DXTRA_PROCPS_DEBUG option.
Reference(s):
https://www.freelists.org/post/procps/newlib-Qualys-patches,6
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch is the first of three implementing a newlib
branch version of that Jan Rybar master merge request.
With this series we'll ultimately extend 'EXE' support
to both ps and top (plus, everyone else who wants it).
Reference(s):
. master branch merge request
https://gitlab.com/procps-ng/procps/merge_requests/66
Signed-off-by: Jim Warner <james.warner@comcast.net>
Following that patch referenced below, the top SUPGRPS
field would produce a segmentation fault and ps SUPGRP
would often show "(null)". Such problems resulted from
some faulty logic in the status2proc() routine dealing
with 'Groups' (supgid) which served as a source field.
For many processes the original code produced an empty
string which prevented conversion to the expected "-".
Moreover, prior to release 3.3.15 such an empty string
will become 0 after strtol() which pwcache_get_group()
translates to 'root' yielding very misleading results.
So, now we'll check for empty '/proc/#/status/Groups:'
fields & consistently provide a "-" value for callers.
[ we'll also protect against future problems in that ]
[ new qualys logic by always ensuring valid 'supgrp' ]
[ pointers - logic which revealed our original flaw! ]
Reference(s):
. original qualys patch
0071-proc-readproc.c-Harden-supgrps_from_supgids.patch
Signed-off-by: Jim Warner <james.warner@comcast.net>
This refactor was done in response to the Qualys patch
referenced below, which deals with some 'readeither()'
flaws under the master branch. Under our newlib branch
those flaws mostly disappear since the function is now
private. But without a redesign the #define is broken.
When the #define FALSE_THREADS is active, some special
strings showing "[ duplicate ENUM ]" will appear under
each child thread. Note that the real reason for those
appearing isn't being exercised, only their mechanics.
In reality, they only show when a user duplicates such
enums in a results stack & only 1 instance can own it.
Reference(s):
. original qualys patch
0084-proc-readproc.c-Work-around-a-design-flaw-in-readeit.patch
. QUICK_THREADS became FALSE_THREADS
commit c546d9dd44
Signed-off-by: Jim Warner <james.warner@comcast.net>
In the new library 'cmd' is dynamically allocated just
like 'cmdline'. This will align us with the ref below.
Reference(s):
. master branch increase to 64
commit 2cfdbbe897
Signed-off-by: Jim Warner <james.warner@comcast.net>
readeither() caches (in new_p) a pointer to the proc_t of a task-group
leader, but readeither()'s callers can do pretty much anything with the
proc_t structure passed to and/or returned by this function. For
example, they can 1/ free it or 2/ recycle it (by passing it to
readeither() as x).
1/ leads to a use-after-free, and 2/ leads to unexpected behavior when
taskreader()/simple_readtask() is called with new_p equal to x (this is
not a theoretical flaw: 2/ happens in readproctab3() when want_task()
returns false and p is a group leader).
As a workaround, we keep a copy of new_p's first member (tid) in static
storage, and the next times we enter readeither() we check this "canary"
against the tid in new_p: if they differ, we reset new_p to NULL, which
forces the allocation of a new proc_t (the new "leader", or reference).
This always detects 2/ (because free_acquired(x,1) memsets x and hence
new_p); always detects 1/ if freed via free_acquired() and/or freeproc()
(very likely, otherwise memory may be leaked); probably detects 1/ even
if freed directly via free() (because the canary is the first member of
proc_t, likely to be overwritten by free()); but can not detect 1/ if
free() does not write to new_p's chunk at all.
Moreover, accessing new_p->tid to check the canary in case 1/ is itself
a use-after-free, so a better long-term solution should be implemented
at some point (we wanted to avoid intrusive and backward-incompatible
changes in this library function, hence this imperfect workaround).
---------------------------- adapted for newlib branch
. adapted via 'patch' (rejected due to 'xcalloc' ref)
. with loss of both readproctab functions, most no longer true
Signed-off-by: Jim Warner <james.warner@comcast.net>
If QUICK_THREADS is not defined (it is not by default, but most
distributions enable it) and task_dir_missing is true (only on very old
kernels), then readtask() forgets to reset some of the struct proc_t t's
members, which later results in double-free()s in free_acquired().
For now, we simply synchronized the list of members to be reset with the
list of members freed in free_acquired().
---------------------------- adapted for newlib branch
. now 'cmd' is also dynamic
. just synchronized with those freed in free_acquired
. QUICK_THREADS is now FALSE_THREADS, serving different purpose
. entire patch will be effectively reverted with upcoming refactor
Signed-off-by: Jim Warner <james.warner@comcast.net>
Replace xmalloc() with xcalloc().
---------------------------- adapted for newlib branch
. trade xcalloc() for calloc()
. thus we must account for potential ENOMEM
Signed-off-by: Jim Warner <james.warner@comcast.net>
Replace memcpy+strcpy with snprintf.
---------------------------- adapted for newlib branch
. adapted via 'patch' (without rejections)
Signed-off-by: Jim Warner <james.warner@comcast.net>
Check the return value of snprintf(), otherwise dst may point
out-of-bounds when it reaches the end of the dst_buffer (the snprintf()
always returns 1 in that case, even if there is not enough space left),
and vMAX becomes negative and is passed to snprintf() as a size_t.
---------------------------- adapted for newlib branch
. adapted via 'patch (without rejections)
Signed-off-by: Jim Warner <james.warner@comcast.net>
This detects an integer overflow of "strlen + 1", prevents an integer
overflow of "tot + adj + (2 * pSZ)", and avoids calling snprintf with a
string longer than INT_MAX. Truncate rather than fail, since the callers
do not expect a failure of this function.
---------------------------- adapted for newlib branch
. logic is now in pids.c
. former 'vectorize_this_str' is now 'pids_vectorize_this'
Signed-off-by: Jim Warner <james.warner@comcast.net>
1/ Prevent an out-of-bounds write if sz is 0.
2/ Limit sz to INT_MAX, because the return value is an int, not an
unsigned int (and because if INT_MAX is equal to SSIZE_MAX, man 2 read
says "If count is greater than SSIZE_MAX, the result is unspecified.")
3/ Always null-terminate dst (unless sz is 0), because a return value of
0 because of an open() error (for example) is indistinguishable from a
return value of 0 because of an empty file.
4/ Use an unsigned int for i (just like n), not an int.
5/ Check for snprintf() truncation.
---------------------------- adapted for newlib branch
. adapted via 'patch (without rejections)
Signed-off-by: Jim Warner <james.warner@comcast.net>
Note: this is by far the most important and complex patch of the whole
series, please review it carefully; thank you very much!
For this patch, we decided to keep the original function's design and
skeleton, to avoid regressions and behavior changes, while fixing the
various bugs and overflows. And like the "Harden file2str()" patch, this
patch does not fail when about to overflow, but truncates instead: there
is information available about this process, so return it to the caller;
also, we used INT_MAX as a limit, but a lower limit could be used.
The easy changes:
- Replace sprintf() with snprintf() (and check for truncation).
- Replace "if (n == 0 && rbuf == 0)" with "if (n <= 0 && tot <= 0)" and
do break instead of return: it simplifies the code (only one place to
handle errors), and also guarantees that in the while loop either n or
tot is > 0 (or both), even if n is reset to 0 when about to overflow.
- Remove the "if (n < 0)" block in the while loop: it is (and was) dead
code, since we enter the while loop only if n >= 0.
- Rewrite the missing-null-terminator detection: in the original
function, if the size of the file is a multiple of 2047, a null-
terminator is appended even if the file is already null-terminated.
- Replace "if (n <= 0 && !end_of_file)" with "if (n < 0 || tot <= 0)":
originally, it was equivalent to "if (n < 0)", but we added "tot <= 0"
to handle the first break of the while loop, and to guarantee that in
the rest of the function tot is > 0.
- Double-force ("belt and suspenders") the null-termination of rbuf:
this is (and was) essential to the correctness of the function.
- Replace the final "while" loop with a "for" loop that behaves just
like the preceding "for" loop: in the original function, this would
lead to unexpected results (for example, if rbuf is |\0|A|\0|, this
would return the array {"",NULL} but should return {"","A",NULL}; and
if rbuf is |A|\0|B| (should never happen because rbuf should be null-
terminated), this would make room for two pointers in ret, but would
write three pointers to ret).
The hard changes:
- Prevent the integer overflow of tot in the while loop, but unlike
file2str(), file2strvec() cannot let tot grow until it almost reaches
INT_MAX, because it needs more space for the pointers: this is why we
introduced ARG_LEN, which also guarantees that we can add "align" and
a few sizeof(char*)s to tot without overflowing.
- Prevent the integer overflow of "tot + c + align": when INT_MAX is
(almost) reached, we write the maximal safe amount of pointers to ret
(ARG_LEN guarantees that there is always space for *ret = rbuf and the
NULL terminator).
---------------------------- adapted for newlib branch
. there were many formatting differences
. i introduced several myself (especially comments)
. stdlib 'realloc' used, not that home grown xrealloc
. stdlib 'realloc' required extra 'return NULL' statement
Signed-off-by: Jim Warner <james.warner@comcast.net>
1/ Replace sprintf() with snprintf() (and check for truncation).
2/ Prevent an integer overflow of ub->siz. The "tot_read--" is needed to
avoid an off-by-one overflow in "ub->buf[tot_read] = '\0'". It is safe
to decrement tot_read here, because we know that tot_read is equal to
ub->siz (and ub->siz is very large).
We believe that truncation is a better option than failure (implementing
failure instead should be as easy as replacing the "tot_read--" with
"tot_read = 0").
---------------------------- adapted for newlib branch
. no real changes, patch refused due to mem alloc & failure return
Signed-off-by: Jim Warner <james.warner@comcast.net>
