Some parts of our newlib implementation are the result
of functions which have been propagated from module to
module. In particular, those 'cleanup_stacks' routines
are all similar & likely originated in the <pids> api.
In that interface there was a need to free dynamically
acquired memory before the result structure was reused
to satisfy subsequent 'get', 'select' or 'reap' calls.
This, in turn, led to a concept of 'dirty' stacks with
the need to call one of two 'cleanup_stack' functions.
None of the remaining interfaces deal with such memory
yet they each had their own 'cleanup_stack' functions.
Those functions were responsible for resetting each of
the result unions to zero, excluding any 'noop' items.
The bottom line is that for all interfaces, repetitive
calls would require iterating through the stack(s) two
separate times: once to 'cleanup' another to 'assign'.
With this commit we will reduce iterations to just the
'assign' routine. A reset to zero will be accomplished
in the 'extra' item set routine (which is the only one
actually requiring any reset). All other items will be
reinitialized automatically by a new current set value
or upon reallocation when an items compliment changes.
In the <pids> interface, any freeing of dynamic memory
could have been accomplished by adding that 'freefunc'
check to the 'assign' function. However, that requires
an Item_table test with every item. Instead, we'll now
satisfy such needs as the very first step in those set
functions responsible for dynamically acquired memory.
[ the <pids> api retains 2 'cleanup_stack' functions ]
[ to accommodate stack(s) 'reset' & to serve 'unref' ]
Lastly, all the 'itemize_stack' functions were tweaked
by eliminating an unnecessary initialization of result
unions. That objective was already accomplished by the
calloc() in a 'stacks_alloc' function or the remaining
'cleanup_stack' routine found in the <pids> interface.
Signed-off-by: Jim Warner <james.warner@comcast.net>
These changes are an outgrowth of the research/testing
behind the previous commit. There is no commingling of
select/reap stacks in interfaces beyond the <pids> api
since there's no need to support any 'reset' function.
However, those <pids> changes prompted a review of all
interfaces offering that 'stacks_fetch' function, thus
revealing 2 instances of useless logic/wasted efforts.
Signed-off-by: Jim Warner <james.warner@comcast.net>
When the <pids> api was refactored in the commit shown
below, one objective was enabling the simultaneous use
of 'get' & 'select/reap' functions. Unlike other 'get'
functions, this <pids> 'get' acts as an iterator where
successive calls will return successive tasks/threads.
However, that goal wasn't quite met since a stack used
by 'get' was commingled with the 'select/reap' stacks.
Such commingling supported the 'reset' function, again
a provision which was unique to this <pids> interface.
Unfortunately, some poor assumptions in 'stacks_fetch'
produced a SEGV whenever 'reap/select' followed 'get'.
Thus, this patch addresses those issues and guarantees
such commingled stacks (extents) will be accommodated.
Reference(s):
. standardize portions of interface, <PIDS> api
commit 9ebadc1438
Signed-off-by: Jim Warner <james.warner@comcast.net>
The commit referenced below addressed (some) anomalies
surrounding 'strv' pointers. However, there remained a
couple quirks involving a potential NULL return value.
Any NULL values returned from the old library readproc
guys would cause no real harm for newlib. But they did
produce the misleading "[ duplicate ENUM_ID ]" result.
The following all represent potential NULL results and
suggest shortcomings in testing of that earlier patch.
. kernel threads do not have cgroup, cmdline & environ
. even if present environ could require root to access
So, this patch reverts a portion of the earlier commit
and ensures when some vectored string is not available
a traditional dash ('-') is the 'strv' returned value.
[ and we'll also correct one typo in the header file ]
Reference(s):
. eliminated a final potential NULL, <PIDS> api
commit 09503dc597
Signed-off-by: Jim Warner <james.warner@comcast.net>
This commit is intended as a refinement of the patches
mentioned below, where origins/sources of newlib items
were added to the header files for user documentation.
However, if those additions are to be truly effective,
along with kernel documentation (where available), the
following prerequisites must also have been satisfied:
. our identifiers closely align with linux field names
. our derived items are documented or self-documenting
Satisfying those prerequisites prompted this patch and
for these changes, kernel sources were emphasized over
available documentation (shame on me, it should always
have been so). And, while some 'new' fields were found
to be conditional, they were included unconditionally.
These changes appear more extensive than they actually
need be since I have attempted to enforce some spacing
conventions. So, I've summarize the significant things
in the sections that follow. For a proper perspective,
use: 'git diff --ignore-space-change' (good as alias).
___________________________________________ <PIDS> api
This api is unique in that there exists many different
file/directory origins subordinate to /proc/<pid>. And
our item identifiers are sometimes coerced so as to be
able to group related or similar enumerators together.
So, users needed more help relating our identifiers to
an actual documented field. Thus, we will now also add
the field names as with 'stat: delayacct_blkio_ticks'.
Each item ending with a '_C' now consistently includes
both the parent's count/time plus waited for children.
That 'RTPRIO' guy was renamed/relocated as PRIORITY_RT
since its original name is an implementation artifact.
___________________________________________ <STAT> api
The only api change was to correct a typo ('dervied').
_________________________________________ <VMSTAT> api
Even ignoring white space, this interface received the
largest number of changes. Mostly, this was because of
deficiencies in the proc(5) documentation. Recall that
this documentation already sorely lacks any substance.
Usually, just kernel releases are noted, not contents.
When compared to kernel source, that proc(5) contained
many non-existent fields and also omitted many others.
________________________________________ <MEMINFO> api
Sadly, with this api many of the changes were simply a
correction of some earlier 'human error' where several
fields where hashed then tracked but never represented
with an item enumerator in this meminfo.h header file.
_______________________________________ <SLABINFO> api
The 'SLABS' (summary) & 'SLABNODE' items were reversed
since the former are derived from the separate caches.
More significantly, those 'SLABNODE' guys were renamed
to 'SLAB' since they concern individual caches and the
concept of 'nodes' is really an implementation detail.
Also, several enumerators were changed to more closely
agree with official slabinfo(5) documentation referred
to in what we're treating as a base document: proc(5).
Lastly, while those 'SLABS' items are solely a product
of our library and not represented in slabinfo(5), the
names attempt to parallel those found as 'SLAB' items.
______________________________________ <DISKSTATS> api
One enumeration identifier was changed so as to better
reflect its relationship to that actual documentation:
'Documentation/iostats.txt', as referenced in proc(5).
Reference(s):
. 12/2018, item origins added (and commit msg history)
commit 96d59cbf46
. 01/2019, <stat> origins tweaked
commit 201e816b26
Signed-off-by: Jim Warner <james.warner@comcast.net>
Rather than offer three separate patches, they've been
consolidated in this single commit. All are related in
that they surfaced while preparing a subsequent patch.
------------------------------------------------------
library: correct a broken '#if define', <SLABINFO> api
It was introduced (embarrassingly) in the patch below.
Reference(s):
commit 97d078a9af
------------------------------------------------------
library: correct a broken 'GET' macro, <DISKSTATS> api
In the patch referenced below, which purported to make
all the 'GET' macros robust, the 'DISKSTATS_GET' macro
was broken. A necessary parameter wasn't passed to the
subsequently invoked function: procps_diskstats_get().
Reference(s):
commit bef8c7fb70
------------------------------------------------------
library: correct a broken 'sort' func, <DISKSTATS> api
In the commit shown below, an attempt to normalize the
errno handling, the sort function inadvertently lost 1
crucial line of code which produces a consistent SEGV.
Reference(s):
commit 06be33b43e
Signed-off-by: Jim Warner <james.warner@comcast.net>
Since the patch referenced below traded a compile-time
'sizeof' directive for a run-time 'strlen' call, there
is no need to declare lxc patterns as explicit arrays.
We'll also use the actual lxc patterns by omitting the
beginning slashes ('/') for both of those definitions.
And, looking to the future when most/all lxc users are
using the most recent lxc release, we will make things
slightly more efficient by reversing those two pattern
literals so the most recent pattern was checked first.
Of course, such a change only benefits tasks which are
running in a container. For the majority of processes,
both literals will be compared in that 'if' statement,
assuming the 'LXC' field is currently being displayed.
[ plus, a leftover parenthesis pair has been removed ]
Reference(s):
commit 288d759b8b
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch just polishes the 'origin' comments for the
<STAT> header file. In particular those derived/unique
items (the 'SUM' guys) will now be properly explained.
[ in order to employ the 'derived from above' phrase ]
[ with their DELTA versions, all SUM items had to be ]
[ relocated (and some renamed). in turn, that had an ]
[ impact on many portions of the .c source file too. ]
Reference(s):
. summary calculations introduced
commit 2c86c4984a
. origins added to header files
commit 96d59cbf46
Signed-off-by: Jim Warner <james.warner@comcast.net>
This small change is an outgrowth of the research into
the bug represented by that merge request shown below.
With the master branch, a real buglet was subsequently
addressed. In this newlib branch, no bug existed since
the <stat> API relies solely on just cpus reflected in
(and parsed from) the kernel's /proc/stat pseudo file.
[ since that procps_stat_new() priming read about to ]
[ be performed will value info->cpus.total, there is ]
[ no need to separately invoke a procps_cpu_count(). ]
Reference(s):
https://gitlab.com/procps-ng/procps/merge_requests/82
Signed-off-by: Jim Warner <james.warner@comcast.net>
The merge request shown below prompted (thankfully) an
examination of our lxc containers logic in readproc.c.
As it turns out, the lxc folks changed that eyecatcher
used to identify containers within a task cgroup file.
So this patch, with little extra cost, will enable the
libprocps lxc_containers() guy to handle both strings.
[ additionally, I was shocked to find lxc allows the ]
[ eyecatcher to be changed at ./configure time. such ]
[ a provision has always existed. unfortunately, the ]
[ changed value was only available to root, assuming ]
[ one wished to tackle that undocumented liblxc api. ]
Reference(s):
. what prompted lxc support reevaluation
https://gitlab.com/procps-ng/procps/merge_requests/82
. original lxc support introduced
commit 0557504f9c
Signed-off-by: Jim Warner <james.warner@comcast.net>
A lack of documentation seems to be the major obstacle
to releasing this new library. So, in an effort to get
the ball rolling again, this patch adds the origins of
each item as a comment to six of the new header files.
However, before reviewing how such changes may benefit
that documentation objective, it seemed appropriate to
first reflect on newlib's background & current status.
___________________________________________ BACKGROUND
Discussions about and work on a new library began back
in July 2012 but quickly died. After a lull of 2 years
those discussions were resumed in August 2014 but soon
died also (and no code survived the gitorious demise).
With those early discussions, the recommended approach
was to encapsulate all of the libprocps data offerings
in individual functions. When it came to extensibility
it was suggested we should rely on symbols versioning.
Unfortunately that approach would have made for a huge
Application Programming Interface virtually impossible
to master or even document. And, runtime call overhead
would have been substantial for ps and especially top.
So, an alternative design was sought but there were no
new suggestions/contributions via freelists or gitlab.
Thus, in spite of a lack of library design experience,
the procps-ng team (Craig & Jim) set out to develop an
alternative API, more concise and with lower overhead.
Reference(s):
. 07/01/2012, begin library design discussion
https://www.freelists.org/post/procps/Old-library-calls
. 08/12/2014, revival of library design discussion
https://www.freelists.org/post/procps/libprocs-redesign
_____________________________________ DESIGN EVOLUTION
Our newlib branch first appeared on June 14, 2015. And
our current API actually represents the 4th generation
during the past 3 years of evolution. First, there was
a basic 'new', 'get' and 'unref' approach, using enums
to minimize the proliferation of 'get' function calls.
Then, in anticipation of other programs like ps, where
multiple fields times multiple processes would greatly
increase the number of 'get' function calls, a concept
of 'chains' was introduced. This became generation #2.
Such 'chains' proved unnecessarily complex so 'stacks'
replaced them. This was considered the 3rd generation,
but too many implementation details were still exposed
requiring those users to 'alloc', 'read', 'fill', etc.
Finally, a 4th generation emerged representing several
refinements to standardize and minimize those exported
functions, thus hiding all implementation details from
the users. Lastly, handling of 'errno' was normalized.
Reference(s):
. 06/14/2015, revival of new API discussion
https://www.freelists.org/post/procps/The-library-API-again
. 06/24/2015, birth of the newlib branch
https://www.freelists.org/post/procps/new-library
. 06/29/2015, 2nd generation introduced 'chains'
https://www.freelists.org/post/procps/new-library,8
. 07/22/2015, 3rd generation introduced 'stacks'
https://www.freelists.org/post/procps/newlib-stacks-vs-chains
. 06/18/2016, 4th generation refinements begin
https://www.freelists.org/post/procps/newlib-generation-35
. 11/10/2017, 4th generation standardized 'errno'
https://www.freelists.org/post/procps/some-more-master-newlib-stuff
_______________________________________ CURRENT DESIGN
Central to this new design is a simple 'result' struct
reflecting an item plus its value (thanks to a union).
As a user option, these item structures can be grouped
into 'stacks', yielding many results with just 1 call.
Such a 'stack' can be seen as a variable length record
whose content/order is determined solely by the users.
Within that 'result' structure, the union has standard
C language types so there is never a doubt how a value
should be used in a printf statement. Given that linux
requires a least a 32-bit platform the only difference
in capacity surrounds 'long' integers. And, where such
types might be used, the 32-bit maximums are adequate.
The items themselves are simply enumerators defined in
the respective header files. A user can name any items
of interest then the library magically provides result
structure(s). The approach was proven to be extensible
without breaking the ABI (in commit referenced below).
The 6 major APIs each provide for the following calls:
. 'new' ---------> always required as the first call .
. 'ref' -------------------------> strictly optional .
. 'unref' --------> optional, if ill-behaved program .
. 'get' --------------------> retrieve a single item .
. 'select' ----------------> retrieve multiple items .
And the 'get' and 'select' functions provide for delta
results representing the difference between successive
get/select calls (or a 'new' then 'get/select' call).
For the <diskstats>, <pids>, <slabinfo> & <stat> APIs,
where results are unpredictable, a 'reap' function can
return multiple result structures for multiple stacks.
The <pids> API differs from others in that those items
of interest must be provided at 'new' or 'reset' time,
a function unique to this API. And the <pids> 'select'
function requires PIDs or UIDs which are to be fetched
which then operates as a subset of 'reap'. Lastly, the
'get' function is an iterator for successive PIDs/TIDs
returning items previously identified via 'new/reset'.
To provide assistance to users during development, the
special header 'proc/xtra-procps-debug.h' is available
to check type usage against library expectations. That
check is activated by including this header explicitly
or via build using: ./configure '-DXTRA_PROCPS_DEBUG'.
Reference(s):
. 08/05/2016, type validation introduced
https://www.freelists.org/post/procps/newlib-types-validation
commit e3270d463d
. 08/11/2016, extensibility while preserving ABI example
https://www.freelists.org/post/procps/new-meminfo-fields
commit 09e1886c9e
_________________________ INITIAL DOCUMENTATION EFFORT
The initial attempt, referenced below, dealt primarily
with the <pids> interface. Separate man pages for each
exported function were created. Plus there was another
document describing the items, among other miscellany.
Adopting such an approach encounters several problems:
1. In order to use these man pages, users are required
to already know how to use the library. Or alternately
one could randomly search each of them while trying to
ascertain which function call satisfies their need and
what exactly was the proper compliment/order required.
2. While we can explain what all of those <pids> items
represent, that certainly isn't true for all the APIs.
See the gaps in kernel documentation for <meminfo> and
complete lack of documentation with that <vmstat> API.
3. Our documentation effort should take pains to avoid
unnecessary implementation details. Here's an example:
. "The pointer to info will have memory"
. "allocated and a structure created."
Alternatively, the following conveys user requirements
while not offering any internal implementation detail:
. "You must provide the address of a NULL"
. "info structure pointer."
Reference(s):
. 01/04/2017, initial documentation offering
https://www.freelists.org/post/procps/Using-reap-and-get
commit 2598e9f2ce
___________________ RECOMMENDED DOCUMENTATION APPROACH
I recommend that the newlib documentation consist of 3
man pages only. The first would cover the 5 major APIs
and their common functions. The second would deal with
the <pids> API exclusively, explaining how it differs.
Any remaining exported libproc functions which are yet
to be included could be represented in a 3rd document.
For these new documents the following are are assumed:
1. Since we will not be able to document all items, we
shouldn't try to document any items. We should instead
rely on proc(5) or Documentation/filesystems/proc.txt.
2. Program development often involves referencing some
header file(s). So, make that an absolute requirement.
3. With the addition of item origins, represented with
this commit, and considering that 'types' were already
present, the header file might be all some users need.
4. And who knows, when a user of our libproc complains
about gaps in their documentation, it might prompt the
kernel folks to correct those long standing omissions.
To summarize, I suggest that we replace that libproc.3
document with a more general one explaining the basics
of accessing this new library and the common calls for
most of the major interfaces. We can then create a new
document (libproc-pids.3?), which explains differences
in using the <PIDS> application programming interface.
A final document (libproc-misc.3?) covers what's left.
Signed-off-by: Jim Warner <james.warner@comcast.net>
This change is being made in anticipation of adding the
source origin of each item to the <pids.h> header file.
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch will bring the <meminfo> API into line with
that proc(5) document. There were several undocumented
fields that were not noted and these two were omitted:
. 'MmapCopy' was conditional on the #define CONFIG_MMU
. 'Quicklists' depends on the #define CONFIG_QUICKLIST
And we're about to get the following new field in 4.20
which will be represented, at least, in that proc.txt:
. 'KReclaimable' will include SReclaimable plus others
Signed-off-by: Jim Warner <james.warner@comcast.net>
If we ever were to eliminate the procps.h header file,
as discussed in the thread referenced below, then that
would impair the current XTRA_PROCPS_DEBUG provisions.
The only remaining way to verify result types would be
to explicitly include that <proc/xtra-procps-debug.h>.
So, this commit will once again enable the ./configure
provision for defining the -DXTRA_PROCPS_DEBUG option.
Reference(s):
https://www.freelists.org/post/procps/newlib-Qualys-patches,6
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch is the first of three implementing a newlib
branch version of that Jan Rybar master merge request.
With this series we'll ultimately extend 'EXE' support
to both ps and top (plus, everyone else who wants it).
Reference(s):
. master branch merge request
https://gitlab.com/procps-ng/procps/merge_requests/66
Signed-off-by: Jim Warner <james.warner@comcast.net>
Following that patch referenced below, the top SUPGRPS
field would produce a segmentation fault and ps SUPGRP
would often show "(null)". Such problems resulted from
some faulty logic in the status2proc() routine dealing
with 'Groups' (supgid) which served as a source field.
For many processes the original code produced an empty
string which prevented conversion to the expected "-".
Moreover, prior to release 3.3.15 such an empty string
will become 0 after strtol() which pwcache_get_group()
translates to 'root' yielding very misleading results.
So, now we'll check for empty '/proc/#/status/Groups:'
fields & consistently provide a "-" value for callers.
[ we'll also protect against future problems in that ]
[ new qualys logic by always ensuring valid 'supgrp' ]
[ pointers - logic which revealed our original flaw! ]
Reference(s):
. original qualys patch
0071-proc-readproc.c-Harden-supgrps_from_supgids.patch
Signed-off-by: Jim Warner <james.warner@comcast.net>
This refactor was done in response to the Qualys patch
referenced below, which deals with some 'readeither()'
flaws under the master branch. Under our newlib branch
those flaws mostly disappear since the function is now
private. But without a redesign the #define is broken.
When the #define FALSE_THREADS is active, some special
strings showing "[ duplicate ENUM ]" will appear under
each child thread. Note that the real reason for those
appearing isn't being exercised, only their mechanics.
In reality, they only show when a user duplicates such
enums in a results stack & only 1 instance can own it.
Reference(s):
. original qualys patch
0084-proc-readproc.c-Work-around-a-design-flaw-in-readeit.patch
. QUICK_THREADS became FALSE_THREADS
commit c546d9dd44
Signed-off-by: Jim Warner <james.warner@comcast.net>
In the new library 'cmd' is dynamically allocated just
like 'cmdline'. This will align us with the ref below.
Reference(s):
. master branch increase to 64
commit 2cfdbbe897
Signed-off-by: Jim Warner <james.warner@comcast.net>
readeither() caches (in new_p) a pointer to the proc_t of a task-group
leader, but readeither()'s callers can do pretty much anything with the
proc_t structure passed to and/or returned by this function. For
example, they can 1/ free it or 2/ recycle it (by passing it to
readeither() as x).
1/ leads to a use-after-free, and 2/ leads to unexpected behavior when
taskreader()/simple_readtask() is called with new_p equal to x (this is
not a theoretical flaw: 2/ happens in readproctab3() when want_task()
returns false and p is a group leader).
As a workaround, we keep a copy of new_p's first member (tid) in static
storage, and the next times we enter readeither() we check this "canary"
against the tid in new_p: if they differ, we reset new_p to NULL, which
forces the allocation of a new proc_t (the new "leader", or reference).
This always detects 2/ (because free_acquired(x,1) memsets x and hence
new_p); always detects 1/ if freed via free_acquired() and/or freeproc()
(very likely, otherwise memory may be leaked); probably detects 1/ even
if freed directly via free() (because the canary is the first member of
proc_t, likely to be overwritten by free()); but can not detect 1/ if
free() does not write to new_p's chunk at all.
Moreover, accessing new_p->tid to check the canary in case 1/ is itself
a use-after-free, so a better long-term solution should be implemented
at some point (we wanted to avoid intrusive and backward-incompatible
changes in this library function, hence this imperfect workaround).
---------------------------- adapted for newlib branch
. adapted via 'patch' (rejected due to 'xcalloc' ref)
. with loss of both readproctab functions, most no longer true
Signed-off-by: Jim Warner <james.warner@comcast.net>
If QUICK_THREADS is not defined (it is not by default, but most
distributions enable it) and task_dir_missing is true (only on very old
kernels), then readtask() forgets to reset some of the struct proc_t t's
members, which later results in double-free()s in free_acquired().
For now, we simply synchronized the list of members to be reset with the
list of members freed in free_acquired().
---------------------------- adapted for newlib branch
. now 'cmd' is also dynamic
. just synchronized with those freed in free_acquired
. QUICK_THREADS is now FALSE_THREADS, serving different purpose
. entire patch will be effectively reverted with upcoming refactor
Signed-off-by: Jim Warner <james.warner@comcast.net>
Replace xmalloc() with xcalloc().
---------------------------- adapted for newlib branch
. trade xcalloc() for calloc()
. thus we must account for potential ENOMEM
Signed-off-by: Jim Warner <james.warner@comcast.net>
Replace memcpy+strcpy with snprintf.
---------------------------- adapted for newlib branch
. adapted via 'patch' (without rejections)
Signed-off-by: Jim Warner <james.warner@comcast.net>
Check the return value of snprintf(), otherwise dst may point
out-of-bounds when it reaches the end of the dst_buffer (the snprintf()
always returns 1 in that case, even if there is not enough space left),
and vMAX becomes negative and is passed to snprintf() as a size_t.
---------------------------- adapted for newlib branch
. adapted via 'patch (without rejections)
Signed-off-by: Jim Warner <james.warner@comcast.net>
This detects an integer overflow of "strlen + 1", prevents an integer
overflow of "tot + adj + (2 * pSZ)", and avoids calling snprintf with a
string longer than INT_MAX. Truncate rather than fail, since the callers
do not expect a failure of this function.
---------------------------- adapted for newlib branch
. logic is now in pids.c
. former 'vectorize_this_str' is now 'pids_vectorize_this'
Signed-off-by: Jim Warner <james.warner@comcast.net>
1/ Prevent an out-of-bounds write if sz is 0.
2/ Limit sz to INT_MAX, because the return value is an int, not an
unsigned int (and because if INT_MAX is equal to SSIZE_MAX, man 2 read
says "If count is greater than SSIZE_MAX, the result is unspecified.")
3/ Always null-terminate dst (unless sz is 0), because a return value of
0 because of an open() error (for example) is indistinguishable from a
return value of 0 because of an empty file.
4/ Use an unsigned int for i (just like n), not an int.
5/ Check for snprintf() truncation.
---------------------------- adapted for newlib branch
. adapted via 'patch (without rejections)
Signed-off-by: Jim Warner <james.warner@comcast.net>
Note: this is by far the most important and complex patch of the whole
series, please review it carefully; thank you very much!
For this patch, we decided to keep the original function's design and
skeleton, to avoid regressions and behavior changes, while fixing the
various bugs and overflows. And like the "Harden file2str()" patch, this
patch does not fail when about to overflow, but truncates instead: there
is information available about this process, so return it to the caller;
also, we used INT_MAX as a limit, but a lower limit could be used.
The easy changes:
- Replace sprintf() with snprintf() (and check for truncation).
- Replace "if (n == 0 && rbuf == 0)" with "if (n <= 0 && tot <= 0)" and
do break instead of return: it simplifies the code (only one place to
handle errors), and also guarantees that in the while loop either n or
tot is > 0 (or both), even if n is reset to 0 when about to overflow.
- Remove the "if (n < 0)" block in the while loop: it is (and was) dead
code, since we enter the while loop only if n >= 0.
- Rewrite the missing-null-terminator detection: in the original
function, if the size of the file is a multiple of 2047, a null-
terminator is appended even if the file is already null-terminated.
- Replace "if (n <= 0 && !end_of_file)" with "if (n < 0 || tot <= 0)":
originally, it was equivalent to "if (n < 0)", but we added "tot <= 0"
to handle the first break of the while loop, and to guarantee that in
the rest of the function tot is > 0.
- Double-force ("belt and suspenders") the null-termination of rbuf:
this is (and was) essential to the correctness of the function.
- Replace the final "while" loop with a "for" loop that behaves just
like the preceding "for" loop: in the original function, this would
lead to unexpected results (for example, if rbuf is |\0|A|\0|, this
would return the array {"",NULL} but should return {"","A",NULL}; and
if rbuf is |A|\0|B| (should never happen because rbuf should be null-
terminated), this would make room for two pointers in ret, but would
write three pointers to ret).
The hard changes:
- Prevent the integer overflow of tot in the while loop, but unlike
file2str(), file2strvec() cannot let tot grow until it almost reaches
INT_MAX, because it needs more space for the pointers: this is why we
introduced ARG_LEN, which also guarantees that we can add "align" and
a few sizeof(char*)s to tot without overflowing.
- Prevent the integer overflow of "tot + c + align": when INT_MAX is
(almost) reached, we write the maximal safe amount of pointers to ret
(ARG_LEN guarantees that there is always space for *ret = rbuf and the
NULL terminator).
---------------------------- adapted for newlib branch
. there were many formatting differences
. i introduced several myself (especially comments)
. stdlib 'realloc' used, not that home grown xrealloc
. stdlib 'realloc' required extra 'return NULL' statement
Signed-off-by: Jim Warner <james.warner@comcast.net>
1/ Replace sprintf() with snprintf() (and check for truncation).
2/ Prevent an integer overflow of ub->siz. The "tot_read--" is needed to
avoid an off-by-one overflow in "ub->buf[tot_read] = '\0'". It is safe
to decrement tot_read here, because we know that tot_read is equal to
ub->siz (and ub->siz is very large).
We believe that truncation is a better option than failure (implementing
failure instead should be as easy as replacing the "tot_read--" with
"tot_read = 0").
---------------------------- adapted for newlib branch
. no real changes, patch refused due to mem alloc & failure return
Signed-off-by: Jim Warner <james.warner@comcast.net>
1/ Use a "size_t num" instead of an "unsigned num" (also, do not store
the return value of sscanf() into num, it was unused anyway).
2/ Check the return value of strchr() and strrchr().
3/ Never jump over the terminating null byte with "S = tmp + 2".
---------------------------- adapted for newlib branch
. newlib doesn't use that 'unlikely' crap
. the cmd field is now also dynamic (like cmdline)
. thus we must account for potential ENOMEM
Signed-off-by: Jim Warner <james.warner@comcast.net>
1/ Prevent an integer overflow of t.
2/ Avoid an infinite loop if s contains characters other than comma,
spaces, +, -, and digits.
3/ Handle all possible return values of snprintf().
---------------------------- adapted for newlib branch
. we can't use xrealloc(), so we use realloc() instead
. and must account for a mem failure via a return of 1
Signed-off-by: Jim Warner <james.warner@comcast.net>
1/ Do not read past the terminating null byte when hashing the name.
2/ S[x] is used as an index, but S is "char *S" (signed) and hence may
index the array out-of-bounds. Bit-mask S[x] with 127 (the array has 128
entries).
3/ Use a size_t for j, not an int (strlen() returns a size_t).
Notes:
- These are (mostly) theoretical problems, because the contents of
/proc/PID/status are (mostly) trusted.
- The "name" member of the status_table_struct has 8 bytes, and
"RssShmem" occupies exactly 8 bytes, which means that "name" is not
null-terminated. This is fine right now, because status2proc() uses
memcmp(), not strcmp(), but it is worth mentioning.
---------------------------- adapted for newlib branch
. newlib doesn't use that 'unlikely' crap
. newlib also had a '#ifdef FALSE_THREADS'
Signed-off-by: Jim Warner <james.warner@comcast.net>
This function is unused (SIGNAL_STRING is defined by default, and if it
is not, procps does not compile -- for example, there is no "outbuf" in
help_pr_sig()) but fix it anyway. There are two bugs:
- it accepts non-hexadecimal characters (anything >= 0x30);
- "(c - (c>0x57) ? 0x57 : 0x30)" is always equal to 0x57.
---------------------------- adapted for newlib branch
. newlib doesn't use that 'unlikely' crap
Signed-off-by: Jim Warner <james.warner@comcast.net>
This solves several problems:
1/ outbuf[1] was written to, but not outbuf[0], which was left
uninitialized (well, SECURE_ESCAPE_ARGS() already fixes this, but do it
explicitly as well); we know it is safe to write one byte to outbuf,
because SECURE_ESCAPE_ARGS() guarantees it.
2/ If bytes was 1, the write to outbuf[1] was an off-by-one overflow.
3/ Do not call escape_str() with a 0 bufsize if bytes == overhead.
4/ Prevent various buffer overflows if bytes <= overhead.
Simply rearrange the old comparisons. The new comparisons are safe,
because we know from previous checks that:
1/ wlen > 0
2/ my_cells < *maxcells (also: my_cells >= 0 and *maxcells > 0)
3/ len > 1
4/ my_bytes+1 < bufsize (also: my_bytes >= 0 and bufsize > 0)
This should never happen, because wcwidth() is called only if iswprint()
returns nonzero. But belt-and-suspenders, and make it visually clear
(very important for the next patch).
The SECURE_ESCAPE_ARGS() macro solves several potential problems
(although we found no problematic calls to the escape*() functions in
procps's code-base, but had to thoroughly review every call; and this is
library code):
1/ off-by-one overflows if the size of the destination buffer is 0;
2/ buffer overflows if this size (or "maxroom") is negative;
3/ integer overflows (for example, "*maxcells+1");
4/ always null-terminate the destination buffer (unless its size is 0).
---------------------------- adapted for newlib branch
. the escape.c now has just a single exported function
. thus SECURE_ESCAPE_ARGS() is needed in only 2 places
. unlike that original patch, macro is executed 1 time
( not like 'escape_command' calling 'escape_strlist' )
( which might then call 'escape_str' multiple times! )
Signed-off-by: Jim Warner <james.warner@comcast.net>
In the human_readable case; otherwise the strcat() that follows may
append bytes to the previous contents of buf.
Also, slightly enlarge buf, as it was a bit too tight.
Could also replace all sprintf()s with snprintf()s, but all the calls
here output a limited number of characters, so they should be safe.
---------------------------- adapted for newlib branch
. the source file is now proc/uptime.c
. function is now named 'procps_uptime_sprint()'
. new human readable function 'procps_uptime_sprint_short()'
. both were already initialized, so just raised size of 2 buffers
Signed-off-by: Jim Warner <james.warner@comcast.net>
In proc/slab.c, functions parse_slabinfo20() and parse_slabinfo11(),
sscanf() might overflow curr->name, because "String input conversions
store a terminating null byte ('\0') to mark the end of the input; the
maximum field width does not include this terminator."
Add one byte to name[] for this terminator.
---------------------------- adapted for newlib branch
. file is now proc/slabinfo.c (not .h)
. manifest constant renamed SLABINFO_NAME_LEN
. older parse_slabinfo11() function no longer present
Signed-off-by: Jim Warner <james.warner@comcast.net>
When 'newlib' was introduced, in the commit referenced
below, the use of that glibc '__BEGIN_DECLS' macro was
standardized. However, as issue #88 revealed, this may
result in a fatal build error with other environments.
So, this patch just trades that macro for the standard
'#ifdef __cplusplus' conventions (thus avoiding use of
all those '#include <features.h>' directives as well).
Reference(s):
. newlib introduced
commit a410e236ab
. procps-ng-3.3.13 issue
https://gitlab.com/procps-ng/procps/issues/88
. some additional discussion
https://www.freelists.org/post/procps/PATCH-Replace-glibcspecific-macros-in-procnumah,1
. musl wiki (see: sys/cdefs.h error messages)
https://wiki.musl-libc.org/faq.html
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch simply eliminates that glibc specific macro
from all header files which contain no public callable
functions. After all, if user code can't link to them,
then protection from C++ name mangling is unnecessary.
[ we also remove any related '#include <features.h>' ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
Now that the procio logic was removed from the library
we must move the header file, lest we break make dist.
In the process, we will relocate that source file too.
[ we'll take a slightly different approach than that ]
[ used under the master branch by exploiting those 2 ]
[ non-library directories 'include' and 'lib', while ]
[ avoiding any sysctl hard coded function prototype. ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
The procio functions that were in the library have been
moved into sysctl. sysctl is not linked to libprocps in
newlib and none of the other procps binaries would need
to read/write large data to the procfs.
References:
be6b048a41
to be able to read and write large buffers below /proc.
The buffers and file offsets are handled dynamically
on the required buffer size at read, that is lseek(2)
is used to determine this size. Large buffers at
write are split at a delimeter into pieces and also
lseek(2) is used to write each of them.
Signed-off-by: Werner Fink <werner@suse.de>
With the documentation update in the commit referenced
below, we should also account for such threads as they
will already be represented in the task/thread totals.
[ and do it in a way that might avoid future changes ]
Reference(s):
commit 91df65b9e7
Signed-off-by: Jim Warner <james.warner@comcast.net>
This removes the following error by stating the task ID can only be 10
characters wide, as it is an integer.
proc/readproc.c: In function ‘simple_nexttid’:
proc/readproc.c:1185:46: warning: ‘%s’ directive output may be truncated writing up to 255 bytes into a region of size between 41 and 51 [-Wformat-truncation=]
snprintf(path, PROCPATHLEN, "/proc/%d/task/%s", p->tgid, ent->d_name);
^~
proc/readproc.c:1185:3: note: ‘snprintf’ output between 14 and 279 bytes into a destination of size 64
snprintf(path, PROCPATHLEN, "/proc/%d/task/%s", p->tgid, ent->d_name);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This commit removes some obsolete parameter validation
code which was needed back when certain functions were
public, called directly by users (1st/2nd generation).
Now that they're static they can be safely eliminated.
Signed-off-by: Jim Warner <james.warner@comcast.net>
With older library logic having been modified to avoid
using those potentially deadly alloc.h routines, while
improving 'errno' handling, we're ready to standardize
and enhance newlib's approach to any potential errors.
In so doing, we'll establish the following objectives:
. . . . . . . . . . . . . functions returning an 'int'
. an error will be indicated by a negative number that
is always the inverse of some well known errno.h value
. . . . . . . . . . . functions returning an 'address'
. any error will be indicated by a NULL return pointer
with the actual reason found in the formal errno value
And, when errno is manipulated directly we will strive
to do so whenever possible within those routines which
have been declared with PROCPS_EXPORT. In other words,
in the user callable functions defined in source last.
[ But, that won't always be possible. In particular, ]
[ all the 'read_failed' functions will sometimes set ]
[ 'errno' so that they can serve callers returning a ]
[ NULL or an int without duplicating a lot of logic. ]
[ Also, that includes one subordinate function which ]
[ was called by 'read_failed' in the <slabinfo> API. ]
------------------------------------------------------
Along the way, several additional miscellaneous issues
were addressed. They're listed here now for posterity.
. the '-1' return value passed outside the library was
eliminated since it would erroneously equate to -EPERM
. the stacks_fetch functions in <diskstats> and <stat>
weren't checked for their possible minus return values
. hash create was not checked in <meminfo> or <vmstat>
. fixed 'new' function faulty parm check in <slabinfo>
Signed-off-by: Jim Warner <james.warner@comcast.net>
While that old master branch library may utilize those
memory allocation functions found in the alloc module,
it was inappropriate for this newlib branch to subject
callers to a stderr message followed by an early exit.
Of course, the old libprocps offered a message handler
override provision (xalloc_err_handler) but that, too,
would seem to be inappropriate for our modern library.
[ remember the battles fought with that damn libnuma ]
So, this commit will tweak those old inherited sources
setting the stage for standardized return values/errno
settings in connection with a memory allocation error.
------------------------------------------------------
Along the way, we'll address the following miscellany:
. Completely eliminate usage of anything from alloc.h.
This, of course, entails our own error checking of the
alternative allocation calls from stdlib.h & string.h.
. Eliminate use of the strdup function where possible,
as with 'procps_uptime' and 'procps_loadavg' routines.
. Whack some obsolete code (getslabinfo) in sysinfo.c.
Signed-off-by: Jim Warner <james.warner@comcast.net>
. ensure whitespace exists between the code & comments
[ changing txt slightly keeps right margin alignment ]
. strive for more consistency with some comment styles
[ don't use C '/*' style where C++ '//' style exists ]
. removed the instance of double space in 1 assignment
[ still striving for consistency in whitespace usage ]
. fixed comment relating to number of 'derived fields'
[ the <meminfo> api recently added one new such enum ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch just rearranges 1 item in the stacks_extent
struct to make it equivalent to all the other modules.
Signed-off-by: Jim Warner <james.warner@comcast.net>
As an oversight, delta values for SWAP amounts weren't
included in the <meminfo> API. Since any runtime costs
of including them only amount to slightly more storage
this commit will simply correct the earlier oversight.
Signed-off-by: Jim Warner <james.warner@comcast.net>
When supplying the -p command to uptime, it does not display any
sections where the value is less than 1; however, after a reboot, this
causes the command to just output "up". Showing 0 minutes when the
system has been up for less than a minute makes it clear a reboot just
occurred.
References:
commit 325d68b7c3
Signed-off-by: Craig Small <csmall@enc.com.au>
Awhile back, that former QUICK_THREADS #define evolved
into the development (only) FALSE_THREADS which can be
used to ensure a 'duplicate ENUM' convention is output
when certain string fields can't be easily duplicated.
Unfortunately, that original implementation was marred
with zeros being displayed for /proc/$$/meminfo fields
in all the child threads for a multi-threaded process.
So this commit corrects that zero memory field buglet.
Reference(s):
. QUICK_THREADS becomes FALSE_THREADS
commit c546d9dd44
Signed-off-by: Jim Warner <james.warner@comcast.net>
The <meminfo> module attempted to duplicate the former
sysinfo memory calculations wherein 'SReclaimable' was
added to 'Cached' for the 'kb_main_cached' equivalent.
But, this original approach was wrong for two reasons.
1. The addition occurred too late to impact the 'USED'
calculation which could then cause an underflow in the
top memory display if 'SReclaimable' was heavily used.
2. In changing the actual /proc/meminfo 'Cached' value
it meant that users could not rely on that proc(5) man
page when interpreting the MEMINFO_MEM_CACHED results.
So this commit adds a new enumerator for the inclusive
cached amount plus repositions the calculation so that
a MEMINFO_MEM_USED result will exclude 'SReclaimable'.
Signed-off-by: Jim Warner <james.warner@comcast.net>
In response to that suggestion referenced below, these
changes allow display of task/thread level NUMA nodes.
Currently, only the 'top' program offers any NUMA type
support and it is limited to the Summary Area display.
With this commit both the 'top' and 'ps' programs will
be able to display NUMA nodes associated with threads.
Reference(s):
https://gitlab.com/procps-ng/procps/issues/58
Signed-off-by: Jim Warner <james.warner@comcast.net>
Reference(s):
proc/readproc.c: In function 'statm2proc'
proc/readproc.c:600:9: warning: variable 'num' set but not used [-Wunused-but-set-variable]
proc/stat.c: In function 'stat_derive_unique':
proc/stat.c:429:1: warning: no return statement in function returning non-void [-Wreturn-type]
ps/parser.c: In function 'arg_type':
ps/parser.c:1098:3: warning: this 'if' clause does not guard... [-Wmisleading-indentation]
ps/parser.c:1099:34: note: ...this statement, but the latter is misleadingly indented as if it is guarded by the 'if'
lib/signals.c: In function 'strtosig':
lib/signals.c:243:9: warning: this 'if' clause does not guard... [-Wmisleading-indentation]
lib/signals.c:245:13: note: ...this statement, but the latter is misleadingly indented as if it is guarded by the 'if'
slabtop.c: In function 'print_summary':
slabtop.c:223:29: warning: unused variable 'stats' [-Wunused-variable]
watch.c: In function 'process_ansi':
watch.c:232:5: warning: this 'if' clause does not guard... [-Wmisleading-indentation]
watch.c:235:2: note: ...this statement, but the latter is misleadingly indented as if it is guarded by the 'if'
Signed-off-by: Jim Warner <james.warner@comcast.net>
When those standardized 'derived' TIC enumerators were
introduced, a problem with potential DELTA distortions
was also introduced when toggling cpus offline/online.
It has always been true that the 1st (summary) line in
/proc/stat will experience a decrease in total tics if
a new cpu is brought online. Such decreases are mostly
due to reductions in 'idle' and 'iowait' tics. Exactly
why such a counterintuitive phenomenon should occur is
a mystery, but this has been acknowledged in proc.txt.
A separate potential distortion arises with individual
cpus. And, here it extends to both bringing processors
online plus taking them offline too. When that happens
the order of the cpus array tracking is upset, placing
the 'new' values in some other processor's array slot.
But even if we were to occupy the same slot, the issue
regarding reductions in 'idle' & 'iowait' still apply.
In all cases, when a DELTA field was found to be minus
it was forced to zero via the 'TICsetH' macro. However
the 'derived' calculations are subject to new forms of
distortion with their own DELTA values. For example we
could find DELTA_SUM_USER + DELTA_SUM_SYSTEM exceeding
DELTA_SUM_TOTAL, an illogical/inappropriate condition.
So this commit moves former protections for individual
cpus to the stat_derive_unique() function and modifies
it to also extend protections to the 'derived' values.
In the process we now protect the cpu 'summary' counts
which were unfortunately previously overlooked (oops).
Reference(s):
. 'derived' types introduced
commit 2c86c4984a
Since its introduction, our evolved /proc/stat API has
relied on a static buffer of 8192 bytes. This approach
is probably Ok for other /proc files but it would only
accommodate around 100 processors. If such a threshold
were exceeded then this interface could never succeed.
Now days 100 processors doesn't seem at all excessive.
So this commit trades that static buffer for a dynamic
self-tuning one. And since so much former top CPU code
was already rolled into this module, we just stole the
already proven top dynamic buffer management code too.
[ this also meant switching low level unbuffered I/O ]
[ calls to standard library buffered I/O calls. that ]
[ is exactly what <slabinfo> and <diskstats> employ. ]
Reference(s):
. 1st gen readstat introduction
commit a410e236ab
Signed-off-by: Jim Warner <james.warner@comcast.net>
With the addition of those new derived SUM values, any
CPUs taken offline or brought online would distort the
historical (delta) results. So this patch just forces
a history reset when such transitions are encountered.
Reference(s):
. derived SUM provisions introduced
commit 2c86c4984a
Signed-off-by: Jim Warner <james.warner@comcast.net>
For each of those interfaces employing a priming read,
all the other 'read' functions begin with the module's
name except this guy which began with 'read_slabinfo'.
Now, they'll all begin with their module name then end
the same with a '_read_failed' boolean hinting suffix.
Signed-off-by: Jim Warner <james.warner@comcast.net>
Upon reflection, at the point where the 'priming read'
was introduced, any possibility of history distortions
was also eliminated. This was true because all of the
'old' (zeroed) data will have been replaced with 'new'
data whenever a user finally calls get, select & reap.
Thus, any DELTA values will automatically reflect that
interval between 'new' and subsequent retrieval calls.
[ diskstats didn't actually employ a 1st time switch ]
[ like the others so we have changed a comment only. ]
[ but that module will retain something similar used ]
[ inside node_update whenever a new node is created. ]
Reference(s):
. priming read added to slabinfo
commit 5d5a52a380
. priming read added to diskstats
commit ecd64f4445
. priming read added to meminfo, stat, vmstat
commit 1a2b62c779
Signed-off-by: Jim Warner <james.warner@comcast.net>
This commit arose out of the discussion (and research)
surrounding the issue cited below. It is an attempt to
consolidate and standardize the calculation of jiffies
categories (e.g. 'idle', 'busy', etc.) once & for all.
Also included is the enum STAT_TIC_NUM_CONTRIBUTORS in
case anyone, in the future, decides to calculate usage
based upon elapsed time * Hz (like top does in process
level %CPU stats). In such an event, a total number of
CPUs or NUMA Nodes would be needed for proper scaling.
Reference(s):
https://gitlab.com/procps-ng/procps/issues/48
Signed-off-by: Jim Warner <james.warner@comcast.net>
Unlike the ps kludge under the master branch to ensure
that namespaces appear the same under both 32 & 64-bit
models, this newlib branch already used a proper type.
However source data still carried the original type as
'signed long' versus that more proper 'unsigned long'.
So, this patch makes sources & destinations identical.
Reference(s):
. master branch ps kludge
commit c41c614b0c
Signed-off-by: Jim Warner <james.warner@comcast.net>
This commit will resolve the RedHat Bugzilla #1322111.
[ import from identical commit against master branch ]
[ but without trailing whitespace, thank you so much ]
Imported by: Jim Warner <james.warner@comcast.net>
In each module employing a priming read at 'new' time,
should that read fail, a call to 'unref' will be made.
However, there is a hidden dependency that these calls
must never occur before the context 'refcount' was set
due to the way an 'unref' conditional was constructed.
So this commit just ensures that 'unref' will function
as expected, even if called with a 'refcount' of zero.
Signed-off-by: Jim Warner <james.warner@comcast.net>
Removing the Item_table 'stat' oldflags for WCHAN_ADDR
was wrong since that 'stat' field is not a constant 0.
Rather, it could assume these 3 values: -1, 0, and +1.
I have not been able to pin down a '-1' result, but it
probably means some sort of permission error (-EPERM).
The '1' or '0' values were supposed to distinguish the
tasks that were or were not blocked (whether there was
a wchan address). However, in practice there is little
correlation between those values and availability of a
kernel symbol in /proc/$$/wchan (perhaps due to race).
Anyway, the real point is that a 'stat' wchan does not
now intentionally contain an address. Thus, outputting
'ffffff', '-' or '1' in programs like ps is senseless.
So this patch just eliminates PIDS_WCHAN_ADDR from our
item enumerators leaving only the PIDS_WCHAN_NAME guy.
Now the new library can't be blamed for bad addresses!
Reference(s):
. removed Item_table 'oldflags'
commit c4aa6c0ab4
. linux removal of wchan addresses
commit b2f73922d119686323f14fbbe46587f863852328
Signed-off-by: Jim Warner <james.warner@comcast.net>
It seems inappropriate to blindly include fields known
to always be zero in our brand new library. Therefore,
this patch removes support for three such enumerators.
[ that stat 'it_real_value' (PIDS_ALARM) field could ]
[ have been made obsolete before a linux 2.6 release ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
This commit just contains some tweaks to comments plus
a few adjustments to whitespace for alignment purposes
and a normalization of the header inclusion #define's.
[ plus a spelling error in one header file was fixed ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
When this module was upgraded to 3rd generation in the
patch referenced below, numa node support was migrated
from the top program into newlib. The 'guest_nice' and
'guest' tics were overlooked as top did not need them.
So, this commit corrects that oversight and achieves a
proper symmetry between the cpu & numa jiffies counts.
Reference(s):
. 3rd gen redesign, numa support imported
commit abc71a46ad
Signed-off-by: Jim Warner <james.warner@comcast.net>
A priming read at 'new' time in that <slabinfo> module
was important so that permission problems are detected
early. Plus, it also had the potential of making delta
values valid when 'get' or 'select' were first called.
It is for that latter reason that such a read was also
incorporated in the <diskstats> module 'new' function.
No other module, however, employed such priming reads.
This patch just brings those potential benefits to all
of our other newlib modules with the exception of that
<pids> guy. That module is, of necessity, sufficiently
different from those others to justify such exclusion.
Not only are there precious few DELTA enums in <pids>,
but the costs of a priming read would be much greater.
[ otherwise, these newly added priming reads have no ]
[ measurable negative impact on performance/timings. ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
The <slabinfo> header provides 3 groups of enumerators
with prefixes of SLABINFO, SLABS & SLABNODE. The first
is strictly user oriented & isn't supported internally
by any structure. The other two, however, have structs
associated with 'em but, unfortunately, 1 is misnamed.
The 'struct slabs_node' is associated with 'nodes' and
supports the enumerators with the SLABNODE prefix. But
the 'struct slabs_hist' was associated with 'hist' yet
supports those enumerators with just the SLABS prefix.
We do not care very much what some structure is called
but we do care about an identifier used manipulate it.
This patch will trade the 'hist' identifier associated
with 'struct slabs_hist' for a more congruous 'slabs'.
[ it's awful when the author can't remember what the ]
[ true meaning of an identifier is after creating it ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
If, in fact, numa nodes are dynamic (like that current
total of on-line cpus) the existing logic was lacking.
It included an early return before checking the total.
So, this commit ensures that the nodes total is always
set or updated consistently in only a single function.
There's no need to set it at the time 'new' is called.
[ and since under our existing code this nodes total ]
[ could never possibly have been zero, the erroneous ]
[ test (with the early return) has now been whacked! ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
Still unhappy with a minor memory leak associated with
libnuma, I experimented with omitting the dlclose that
was issued at module's end. For some reason which will
remain a mystery, the valgrind leak then went bye-bye.
So this patch just omits one use of dlclose and relies
on whatever kernel magic is at work to free the memory
when each process ends. We kept, however, the original
code (now commented-out) to serve as a future caution.
There remains one potential (but unlikely) dlclose use
near the original dlopen. But there will be no leak as
that 'numa_node_of_cpu' will not yet have been called.
This seems to be the culprit that triggers such leaks.
None of this libnuma shit would likely have come close
to hitting our fan had the numa developers provided us
with 'new' and 'unref' functions like our newlib does.
Signed-off-by: Jim Warner <james.warner@comcast.net>
Recent profiling and timings have resulted in improved
newlib performance. This patch completes that process.
It just normalizes naming conventions employed for all
allocations involving reaped stacks & history support.
The modules offering a 'reap' function will also offer
the now standardized corresponding STACKS_INCR define.
The modules which provide dynamic history support will
now have a separate #define called NEWOLD_INCR used in
allocations/reallocations. And, while values currently
are set equal to that STACKS_INCR value, in the future
some reason for divorcing those two may be discovered.
----------------------------- for future reference ---
In those modules which contain the STACKS_INCR #define
it is tempting to specify a large value so as to avoid
repeated calls to malloc/realloc. However, in doing so
an extra runtime price will be paid in 'cleanup_stack'
calls with any iterative programs like top or slabtop.
So, with the current values a balance has been sought.
Signed-off-by: Jim Warner <james.warner@comcast.net>
Profiling revealed a large amount of time spent in the
'escape_str_utf8' function (escape.c) with both of our
NLS branches (newlib and master). That same result was
not seen under an ancient top-3.2.8 program & library.
Well, the 3.2.8 result was ultimately explained by the
absence of a 'setlocale', necessary under NLS support.
Thus, when that ancient library tested for locale, all
it got was 'ANSI_...' & assumed 'UTF-8' wasn't active.
But after a hack to that ancient code to place it on a
par with newlib/master, I still found cost differences
that led me to revisit an old change referenced below.
It turns out that 'iswprint' costs far more than would
a call of 'isprint', even with the extra support code.
So this commit just reverts that five year old change.
Reference(s):
commit 7b0fc19e9d
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch contains the following miscellaneous stuff:
. The pids_stacks_fetch() routine might call for newly
allocated stacks to be itemized. However, that job was
already tended to by the pids_stacks_alloc() function.
So, this patch just eliminates a redundant invocation.
------------------------------------------------------
. The concept of 'dirty_stacks' has not kept pace with
the evolving stacks implementation. Originally, stacks
were considered dirty only if free() of dynamic memory
was needed before refreshing any single result struct.
Later, with the introduction of the 'extra' item and a
promise to reset it to zero, 'dirty' was much broader.
So, this patch just treats the dirty flg as others do.
------------------------------------------------------
. Lastly, a word or three about performance & timings.
Tuning efforts concentrated on the <pids> API and top.
And unless an oldlib equivalent to the preceding patch
is applied (favoring stat vs. status), newlib top will
often outperform the oldlib version (obviously wrong).
So assuming /proc/stat is preferred in both libraries,
generally speaking, a cpu and elapsed time increase of
1-5% was found for this new stacks oriented interface.
Of course, there's also an increased memory footprint.
There are some occasions, however, when the newlib top
is at a substantial disadvantage. For example if WCHAN
or TTY is displayed, such items will be present in all
newlib reaped stacks (i.e. every process). But old top
would only incur such overhead with displayable tasks.
Thus, oldlib top could outperform newlib by up to 25%,
for example, if only fields requiring NO library flags
were displayed. However, such a scenario is not likely
since only GID, UID, PID, TGID & WCHAN would be shown.
In the usual case, that overhead associated with WCHAN
and/or TTY is overshadowed by other top runtime costs.
All in all a pleasing outcome I deem quite acceptable.
------------------------------------------------------
Signed-off-by: Jim Warner <james.warner@comcast.net>
Long ago, in a galaxy far away (when top was in charge
of library FILL flgs) /proc/status was to be preferred
over /proc/stat if a field could be satisfied by both.
This was done to avoid costly 64-bit math emulation in
a 32-bit application due to 'unsigned long long' data.
Well it's time to acknowledge the prevalence of 64-bit
platforms. And in such an environment the cost picture
has shifted significantly. It now costs 14 times (wow)
as much to access /proc/status compared to /proc/stat.
In other words, even with '%llu' variables, a sscanf()
call in stat2proc() beats the pants off that home brew
gperf based hashing employed by the status2proc() guy.
In fact, status2proc incurs higher costs than found in
the most expensive aspect of top's forest view option.
Here's a gprof extract to illustrate the costs. It was
produced with an rcfile requiring fields from both the
/proc/stat & /proc/status pseudo files (among others).
There were 5000 iterations in each of 4 separate gprof
runs subsequently merged into 1 gmon.sum for analysis.
% self self
time seconds calls us/call name
----- ------- ------- ------- -----------
28.65 4.10 4689423 0.87 status2proc
26.14 3.74 40000 93.50 forest_adds
...
01.96 0.28 4689427 0.06 stat2proc
[ since forest_adds is recursive, the calls value is ]
[ the non-recursive #, its 'call graph' shows totals ]
Anyway, now that such cost is known this patch becomes
what is euphemistically known as the usual no-brainer.
[ jeeze, was it really this long between profilings? ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch will begin some refinements associated with
gprof. Initially, functions names have been changed to
help in identifying potential bottlenecks. This effort
also included the obscure set, free and sort routines.
Plus the following additional modifications were made:
. the stacks_alloc prologue was generalized plus added
to a couple of modules where it had not yet propagated
. a couple of the '// end ...' comments were corrected
. some functions have been formally tagged as 'inline'
Signed-off-by: Jim Warner <james.warner@comcast.net>
The immediately prior commit demonstrated how our APIs
might be expanded in at some point in the future while
maintaining binary compatibility in previous programs.
However, since we've yet to release the 1st version of
our new library, there's no need to violate alphabetic
ordering just yet. So, this patch restores that order.
Signed-off-by: Jim Warner <james.warner@comcast.net>
With the 4.8 kernel, 2 new fields will be added to the
meminfo pseudo file. This commit, soon to be replaced,
is intended as an example of how such changes might be
incorporated plus still maintain binary compatibility.
This actually goes further than is strictly necessary,
by retaining meminfo_item ordering for 'set' functions
and the creation of hash table entries. However, there
is only 1 true requirement, that of Item_table entries
which must always agree exactly with item enumerators.
All of the other changes could be done alphabetically.
Ok, so what happens when an old program encounters the
new expanded meminfo items? Well, if it was thoroughly
tested against an old library, it won't even see those
new fields. On the other hand, if it somehow exceeds a
previous MEMINFO_logical_end, then it will just get an
extra result structure or two, with no real harm done.
[ this patch is being replace by the very next patch ]
[ so that our iniitial newlib release can maintain a ]
[ strict alphabetic ordering in all areas initially! ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
When the numa stuff was imported from the top program,
that #include for dlopen() was not made conditional as
it should have been. Well, here it is being corrected.
Signed-off-by: Jim Warner <james.warner@comcast.net>
Rather than return a 0 result for all VAL type errors,
return what would have been without validation active.
This will enable a program like pgrep to still print a
result even though it used some incorrect type member.
With this commit, our VAL macro validations logic will
behave in exactly the same way as the GET validations.
While warning messages may be issued, except for a bad
enumerator, values will always be returned to callers.
Signed-off-by: Jim Warner <james.warner@comcast.net>
One ought not to assume that random memory access will
always succeed or, when it does, that an obviously bad
item enumerator will always be found at that location.
Thus, this patch corrects some really poor assumptions
associated with the 'xtra_procps_debug.h' header file.
[ and it does so in somewhat contorted ways so as to ]
[ avoid several darn gcc -Wnonnull warning messages! ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
When users call the native 'get' functions they have a
responsibility to check that the result struct address
was indeed returned. But when using those 'GET' macros
there was no protection for possible NULL dereference.
So this patch will add some protection for a potential
failure of an underlying 'get' function. And should it
occur then those 'GET' macros will just return a zero.
Plus, we'll also mirror that behavior in the debugging
header should the XTRA_PROCPS_DEBUG #define be active.
And, we might as well add a warning when invalid items
are passed to 'GET' macros, just like we do for 'VAL'.
[ lastly, we added the missing opening parens/braces ]
[ to 2 'GET' macros in that xtra-procps-debug.h file ]
[ which went unnoticed until the qa folks caught up. ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
During development, we now have a means for validating
that a type referenced in application code matches the
actual type set by the library. The new feature can be
activated through either of the following two methods:
1) ./configure CFLAGS='-DXTRA_PROCPS_DEBUG' (all pgms)
2) an #include <proc/xtra-procps-debug.h> (single pgm)
[ in the future, one could add a formal configure.ac ]
[ provision. but for now a manual approach is safer. ]
Lastly, for any module which provides a sort function,
the handling for both 'noop' & 'extra' enumerators was
made consistent. Now, 'noop' is not sorted and 'extra'
will be sorted as that module's widest supported type.
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch will set the stage for validating the types
referenced in the result union. For now, the parameter
representing that 'info' structure will remain unused.
[ and while we're at it, let us correct a faulty GET ]
[ macro in the diskstats header. that puppy missed a ]
[ parm which ain't so good if that guy is ever used! ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
This commit attempts to minimize the variety of types
currently used. Plus, the following were also changed:
. the MEM fields were switched to parallel the VM guys
. PIDS_MEM_VIRT -> PIDS_MEM_VIRT_PGS
. PIDS_MEM_VIRT_KIB -> PIDS_MEM_VIRT
. made NICE 's_int' so that it then parallels PRIORITY
. change RTPRIO & SCHED_CLASS from 'ul_int' to 's_int'
. removed Item_table 'oldflags' for an obsoleted field
. PIDS_WCHAN_ADDR
. added calculations like TICS_ALL_C for the following
. PIDS_TICS_USER_C
. PIDS_TICS_SYSTEM_C
. these three new 'TICS' fields have been incorporated
. PIDS_TICS_BLKIO - jiffies spent in block i/o
. PIDS_TICS_GUEST - jiffies spent as a guest
. PIDS_TICS_GUEST_C - as above, includes dead children
. that PIDS_TICS_DELTA was renamed PIDS_TICS_ALL_DELTA
( so it did not hide between TICS_BLKIO & TICS_GUEST )
( and to make clearer what's included: utime + stime )
. eliminated 'sl_int' entirely from that result struct
[ often, the <pids> module changes necessitated that ]
[ readproc header and source files had to change too ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
With this patch, all of the modules which offer a reap
function (pids, diskstats, slabinfo and stat too) will
now also provide for sorting whatever had been reaped.
It was easy to overlook a sort function for our <STAT>
guy given the paucity of CPUs on your typical personal
desktop or laptop. However, out in the world one might
find boxes with hundreds of CPUs plus many NUMA nodes.
Hey, who are we to disallow sorts on something another
person might see as useful under the above conditions?
And, there's always something to be said for symmetry.
[ of course, several minor tweaks were also included ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
. a more appropriate error return code was substituted
. a safer form of comparison is utilized in two places
. a STAT_VAL macro replaced by more proper MEMINFO_VAL
. several of the silly 'xerrx' formats had been missed
. a few whitespace changes have also been incorporated
Signed-off-by: Jim Warner <james.warner@comcast.net>
With this patch we've completed a progression toward a
standard approach to naming conventions which follows:
* Only functions will begin with that 'procps_' prefix
. ........................................... examples
. procps_vmstat_get ()
. procps_diskstats_select ()
- ----------------------------------------------------
* Exposed structures begin with the module/header name
. ........................................... examples
. struct pids_info
. struct stat_reaped
- ----------------------------------------------------
* Item enumerators begin like structs, but capitalized
. ........................................... examples
. VMSTAT_COMPACT_FAIL
. MEMINFO_DELTA_ACTIVE
[ slabinfo varies slightly due to some item variants ]
. SLABINFO_extra
. SLABS_SIZE_ACTIVE
. SLABNODE_OBJS_PER_SLAB
[ could cure with a prefix of SLABINFO, but too long ]
- ----------------------------------------------------
* Other enumerators work exactly like item enumerators
. ........................................... examples
. PIDS_SORT_ASCEND
. STAT_REAP_CPUS_AND_NODES
- ----------------------------------------------------
* Macros and constants begin just like the enumerators
. ........................................... examples
. #define SLABINFO_GET
. #define DISKSTATS_TYPE_DISK
- ----------------------------------------------------
Signed-off-by: Jim Warner <james.warner@comcast.net>
Many of our item enumerator identifiers are very long,
especially in that <VMSTAT> module. Additionally, they
all contain the exact same universal 'PROCPS_' prefix.
The origins for this are likely found in the desire to
avoid name clashes with other potential include files.
But with procps-ng newlib, we've probably gone way too
far. Did 'PROCPS_PIDS_TICS_SYSTEM' actually offer more
protection against clash than 'PIDS_TICS_SYSTEM' does?
I don't think so. Besides, no matter how big that name
becomes, one can never guarantee they'll never be some
clash. And, conversely, extremely short names will not
always create conflict. Of course, in either case when
some clash occurs, one can always #undef that problem.
Thus, this commit will eliminate that 'PROCPS_' prefix
making all of those enum identifiers a little shorter.
And, we'll still be well above some ridiculously short
(criminally short) names found in some common headers:
- - - - - - - - - - <term.h>
- 'tab', 'TTY', etc
- - - - - - - - - - - - - - - - <search.h>
- 'ENTER', ENTRY', 'FIND', etc
------------------------------------------------------
Finally, with this as a last of the wholesale changes,
we will have established the naming conventions below:
. only functions will begin with that 'procps_' prefix
. exposed structures begin with the module/header name
. item enumerators begin like structs, but capitalized
. other enumerators work exactly like item enumerators
. macros and constants begin just like the enumerators
------------------------------------------------------
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch attempts to standardize the naming of those
most important (declared not defined) context structs.
The present practice represents a hodge podge of names
only some of which reflect the source /proc file name.
And 2 of those file names embed a literal 'info' which
is likely the origin of that required parm identifier.
Now we'll append a universal '_info' to such structure
names, while including the names of those /proc pseudo
files where possible. In any case, that context struct
will *always* begin with the actual module/header file
name. And only the following two sound a little weird!
---------> 'meminfo_info' + 'slabinfo_info' <---------
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch will bring this interface up to our 3rd gen
standards. The following summarizes the major changes.
* New delta provisions have been added to most fields.
There are, of course, some fields for which a delta is
inappropriate. They include the identifying items such
as name, type, major and minor. Plus the io_inprogress
field which already acts, in effect, as a delta value.
* To provide delta support, dev_node historical values
have become persistent. By the same token, the library
must provide for future removal of disks/partitions. A
timestamp is used to detect 'stale' data which will be
deleted so as not to satisfy some get, select or reap.
* Such persistent support is provided by a linked list
which, by default, grows from the bottom down so as to
maintain compatibility with the /proc/diskstats order.
Initially, I was tempted to use the GNU tsearch (tree)
provisions until I discovered the overhead of building
that tree plus costs of a subsequent 'twalk'. Besides,
walking such a tree means retrieval order would differ
from an order required/expected by the vmstat program.
* The '/sys/block' directory is no longer scanned with
every refresh cycle. Rather, it's only accessed when a
node is first encountered. Then, that node's 'type' is
persistent for its lifetime like several other fields.
* A sort provision was included, at virtually no cost,
even though such a provision was not currently needed.
Signed-off-by: Jim Warner <james.warner@comcast.net>
Where possible, libprocps files convey the name of the
actual source pseudo file under the '/proc' directory.
This brings diskstats into line with such an approach.
Signed-off-by: Jim Warner <james.warner@comcast.net>
[ plus we also play catch up on some earlier changes ]
[ that impacted skill.c, after using --enable-skill! ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
A collection of miscellaneous code and comment tweaks.
[ such changes will stop when desk checking ends too ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
That #define QUICK_THREADS was impossible to implement
under the new <pids> interface. Plus it was also found
to distort some thread information (referenced below).
So, it's always been inactive under the newlib branch.
However, it will (with small changes) still serve some
useful purpose in our library. Now, when the redefined
FALSE_THREADS is active, those special strings showing
"[ duplicate ENUM ]" will appear for any child thread.
Note: the real reason for such strings appearing isn't
being exercised, only their mechanics. In actual usage
they are substituted when a user duplicates such items
in a results stack & only the 1st instance can own it.
With this patch, we are simply fooling the <pids> code
into thinking an item was duplicated via a NULL value.
Reference(s):
. from master branch
commit 25a6ecdbfb
Signed-off-by: Jim Warner <james.warner@comcast.net>
Yes, all of these changes are strictly cosmetic. It is
likely symptomatic of some deep-seated character flaw.
[ or, it might be because of a certain pride in this ]
[ new library and the desire to make it even better! ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
Ever since their introduction, plus continuing through
several evolutions, both the meminfo and vmstat 'read'
functions employed a 'do while' loop for /proc access.
However, that loop construct was wrong since identical
tests were already done (twice!) within each loop body
itself, then accompanied by its own 'break' statement.
So, we will now transform them both into forever loops
which will help us to emphasize such break statements.
[ plus, let's return an error should nothing be read ]
[ lastly, eliminate 1 erroneous PROCPS_EXPORT prefix ]
Reference(s):
. original meminfo introduction
commit a20e88e4e7
. original vmstat introduction
commit a410e236ab
Signed-off-by: Jim Warner <james.warner@comcast.net>
The procps library attempts to work out the tty of a process
through several methods. For things like /dev/tty123 or
/dev/foo it works fine.
For tty devices that put the minor number in a directory
of the major name this fails. So then we have to fallback
to stating things like the processes STDERR and try again.
Considering a lot of processes sit on ttys such as
/dev/pts/3 this is a lot of wasted time. At the point of
entering driver_name we know "/dev/pts" and we know "3"
we just didn't join them up the right way as this is old
code.
This change now looks for /dev/pts/3 as well. It does it
after looking for /dev/pts3 so the behaviour is the same.
References:
https://bugs.debian.org/770215
Signed-off-by: Craig Small <csmall@enc.com.au>
This commit brings all of those 'fetch' type functions
(supporting some form of 'reap') into closer alignment
with one another. The biggest impact is to be found in
the <stat> module, which now provides for the separate
copy of stack pointers which will be exposed to users.
The reason such a copy was not employed initially with
<stat>, unlike those for <pids> and <slabinfo>, is due
to the fact that such stacks were never sortable. Thus
the original raw consolidated extent pointers wouldn't
have been disturbed. But that meant no NULL delimiter.
So with this commit, all reap/fetch operations now use
pointer copies when returning results to callers. And,
all such arrays are now NULL delimited meaning callers
can choose their own access fencepost: totals or NULL.
Signed-off-by: Jim Warner <james.warner@comcast.net>
This commit will consolidate the three separate copies
of the item enumerators currently supporting that reap
function (summary, cpus & nodes) into one shared copy.
That select function will continue to maintain its own
dedicated items copy, albeit via a new item structure.
[ and while we're at it, we'll move the 'select' guy ]
[ to its proper alphabetical place, after our 'reap' ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
Unlike other cached string fields (such as USER name),
that lxcname field was defined as 'const char *'. This
required a custom 'set' function using a cast in order
to avoid a compiler warning regarding that assignment.
So this commit brings lxcname into line with all those
other cached string fields. And while we're at it, the
reason some string fields have no 'freefunc' will also
be explained with some new comments in the Item_table.
Signed-off-by: Jim Warner <james.warner@comcast.net>
As those 3rd generation newlib APIs evolved so too did
the extents_free_all() function. Most versions of this
function required the callers to first verify that the
extents anchor wasn't empty, which was poor etiquette.
This simple function should have been much more robust
and forgiving. With this commit, it fnally becomes so.
Signed-off-by: Jim Warner <james.warner@comcast.net>
With all our 3rd generation interfaces, we're now well
positioned to preserve binary compatibility should new
fields be added to any public structure (assuming that
the 'result' union already contains its largest type).
This remains true even for the <pids> interface, which
unlike the others, has one structure embedded within a
separate struct rather than declaring a pointer to it.
The counts struct was positioned after the stacks ptrs
array so as to preserve that ABI if ever new ints were
added. Logically, however, the counts (especially that
total) should precede the stacks array if we wished to
properly place a horse (total) before a cart (stacks).
So to enable relocating those counts we will no longer
embed that structure, but provide a pointer to it. And
this will make accessing syntax feel more natural too.
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch represents the partial revert of the commit
referenced below. And, so that yours truly (the author
for goodness sake) doesn't shoot his tootsies again in
the future, a cautionary programmer comment was added.
Reference(s):
commit dea4033418
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch represents the following minor adjustments:
. old PROCTAB used with select & reap more descriptive
. reformatted PROCPS_PIDS_VAL macro like other 3rd gen
Signed-off-by: Jim Warner <james.warner@comcast.net>
The following commit message is shared with 4 patches.
------------------------------------------------------
Under the newlib interface most of our 'get' functions
represent a bit of a compromise in that the actual raw
values are coerced into one (probably ok) return type.
That approach creates the possibility of truncation at
best, and wouldn't serve future needs should something
other than numeric data be added to the 'get' results.
This commit trades the current compromise for a return
value guaranteed to satisfy all future needs, namely a
pointer to a particular api's specific results struct.
The impact on existing programs is minimal, especially
when using a new supplied macro. Otherwise, native 'C'
syntax could be used, but may feel somewhat unnatural.
[ as an aside, this new approach allows us to delete ]
[ all 'getsfunc' table entries & the supporting code ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
The following commit message is shared with 4 patches.
------------------------------------------------------
Under the newlib interface most of our 'get' functions
represent a bit of a compromise in that the actual raw
values are coerced into one (probably ok) return type.
That approach creates the possibility of truncation at
best, and wouldn't serve future needs should something
other than numeric data be added to the 'get' results.
This commit trades the current compromise for a return
value guaranteed to satisfy all future needs, namely a
pointer to a particular api's specific results struct.
The impact on existing programs is minimal, especially
when using a new supplied macro. Otherwise, native 'C'
syntax could be used, but may feel somewhat unnatural.
[ as an aside, this new approach allows us to delete ]
[ all 'getsfunc' table entries & the supporting code ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
The following commit message is shared with 4 patches.
------------------------------------------------------
Under the newlib interface most of our 'get' functions
represent a bit of a compromise in that the actual raw
values are coerced into one (probably ok) return type.
That approach creates the possibility of truncation at
best, and wouldn't serve future needs should something
other than numeric data be added to the 'get' results.
This commit trades the current compromise for a return
value guaranteed to satisfy all future needs, namely a
pointer to a particular api's specific results struct.
The impact on existing programs is minimal, especially
when using a new supplied macro. Otherwise, native 'C'
syntax could be used, but may feel somewhat unnatural.
[ as an aside, this new approach allows us to delete ]
[ all 'getsfunc' table entries & the supporting code ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
The following commit message is shared with 4 patches.
------------------------------------------------------
Under the newlib interface most of our 'get' functions
represent a bit of a compromise in that the actual raw
values are coerced into one (probably ok) return type.
That approach creates the possibility of truncation at
best, and wouldn't serve future needs should something
other than numeric data be added to the 'get' results.
This commit trades the current compromise for a return
value guaranteed to satisfy all future needs, namely a
pointer to a particular api's specific results struct.
The impact on existing programs is minimal, especially
when using a new supplied macro. Otherwise, native 'C'
syntax could be used, but may feel somewhat unnatural.
[ as an aside, this new approach allows us to delete ]
[ all 'getsfunc' table entries & the supporting code ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
All our numa logic was well encapsulated under #ifndef
NUMA_DISABLE (as ./configure --disable-numa), with the
exception of 1 case label. The reason it didn't create
any problems is because the sole consumer (top) itself
responded to the presence of #define NUMA_DISABLE too.
But if that option wasn't used, an oops still awaited.
When experimenting with a 32-bit library plus a 64-bit
kernel, a bug was revealed should the numa library not
have been found. Our 'reap' function must tolerate the
potential of a 0 return from that stacks_fetch_tics().
Now, if numa is desired but not possible, we'll be ok.
[ and, we'll build numa history from fresher sources ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
On MacOS the system already has user_from_uid and group_from_gid.
These are renamed pwcache_get_user and pwcache_get_group.
They were also exported but did not appear in the symbol file
or used by any of the procps binaries. They are no longer exported.
References:
https://gitlab.com/procps-ng/procps/issues/34
Signed-off-by: Craig Small <csmall@enc.com.au>
When those items were made dynamic at 'new' time, some
other functions, previously assured of their presence,
failed to verify a 'reset' had acually been requested.
This commit just corrects that oversight and avoids an
attempt to 'assign_results' when no items are present.
Reference(s):
. when items/numitems became optional
commit 9ebadc1438
Signed-off-by: Jim Warner <james.warner@comcast.net>
Following is a summary of significant changes (if any)
to each of these now upgraded 3rd gen library modules.
<meminfo> ............................................
. eliminated duplicate decl of 'struct procps_meminfo'
. standardized/normalized results struct union members
. added 'std' & 'var' dividers in .c file, like <pids>
. how did i miss relocating all these friggin' #undefs
. cleanup 'get' return logic (remove a redundant 'if')
<pids> ...............................................
. repositioned the procps_pidsinfo structure in header
. removed the extra trailing comma from enum pids_item
. standardized/normalized results struct union members
<slabinfo> ...........................................
. corrected comment typo (jeeze, in an 'aligned' para)
. standardized/normalized results struct union members
. added 'std' & 'var' dividers in .c file, like <pids>
. removed an obsolete #undef from procps_slabinfo_sort
. cleanup 'get' return logic (remove a redundant 'if')
<stat> ...............................................
. how did i miss relocating all these friggin' #undefs
. corrected an initialization fencepost used with numa <=== see Craig, here's a bug fix
. removed the extra trailing comma from enum stat_item
. standardized/normalized results struct union members
. added 'std' & 'var' dividers in .c file, like <pids>
. strengthen those parm checks in procps_stat_get func
. cleanup 'get' return logic (remove a redundant 'if')
<vmstat> .............................................
. standardized/normalized results struct union members
. added 'std' & 'var' dividers in .c file, like <pids>
. cleanup 'get' return logic (remove a redundant 'if')
[ virtually all of these tweaks reflect the author's ]
[ continuing pursuit of an unreasonable goal -- that ]
[ of a 'perfect' (plus 'pretty') C language program! ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
After reviewing the hsearch code in glibc, performance
will almost certainly benefit from abandoning a strcmp
approach in favor of hashing, just like that <vmstat>.
[ As an aside, now having struggled toward that goal ]
[ of opaqueness & making our API as user friendly as ]
[ possible, haven't we earned the rights to evaluate ]
[ other implementations? For example, GNU's hsearch? ]
[ We expose none of our 'info' struct details to the ]
[ users, but GNU exposes their 'hsearch_data' thingy ]
[ right there in <search.h>. But worse, they require ]
[ the user to zero it out before 1st use. Jeeze, you ]
[ mean that a function called hcreate_r could not do ]
[ its own memset? Aw, come on GNU! What's with that? ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
With the dust now settling on all those 3rd generation
upgrades, this patch tries to provide some consistency
among the separate modules involved. Someday we should
consider a 4th generation where all redundant code has
been removed and isolated in a new shared source file.
Following is a summary of significant changes (if any)
to each of these now upgraded 3rd gen library modules.
<meminfo> ............................................
. strictly formatting/comment changes, code unaffected
<pids> ...............................................
. replaced a local mkSTR macro with existing STRINGIFY
. added fetch narrative explaining duplicate addresses
<slabinfo> ...........................................
. rearranged some free logic for procps_slabinfo_unref
. added fetch narrative explaining duplicate addresses
<stat> ...............................................
. added #define ENFORCE_LOGICAL, just as in <slabinfo>
. replaced a local mkSTR macro with existing STRINGIFY
. alphabetized the function declarations in the header
<vmstat> .............................................
. made one coverity concession with read_vmstat_failed
[ several of these changes may reflect this author's ]
[ continuing pursuit of an unreasonable goal -- that ]
[ of a 'perfect' (plus 'pretty') C language program! ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
The 'procps.h' header includes all other headers which
contain exported (public) functions, while this header
has some (private) macros required for implementation.
If we mix the two, by including procps.h in this file,
we'll lose all hope of the compiler catching the kinds
of errors corrected in an immediately preceding patch.
[ we must be diligent, always constantly striving to ]
[ protect jimmy from ill effects due to copy & paste ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
Needless to say, when one copies and then pastes, it's
always better when those results are refined somewhat.
The qsort callback declarations used the wrong results
type and 2 of their 3 input parameters were wrong too.
So this patch will fix that & generalize declarations.
[ plus add comment guidance about noop & extra enums ]
[ & provide for 'noop' sorting, as is done in <PIDS> ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
Though all those callback's parameters are ignored and
qsort treats them as pointers to void, it's wrong when
3rd parm is 'enum pids_item', not 'struct sort_parms'.
So we will fix it in a way that lessens the likelihood
of another such a mistake when some new type is added.
Signed-off-by: Jim Warner <james.warner@comcast.net>
This commit simply tries to keep naming plus formating
conventions on a par with the continuing climb up that
learning curve. These changes were suggested following
<slabinfo> sources upgrade from 2nd to 3rd generation.
Signed-off-by: Jim Warner <james.warner@comcast.net>
After normalizing the <SLABINFO> module, much of which
was based on this module, a structure which is crucial
to concurrent select/reap support is renamed the same.
Such a need for concurrent support was revealed during
slabtop testing. That program requests (select) global
data after calling for individual node data (reap) but
before those stacks have been accessed. Therefore, one
set of 'items'/'extents' could not possibly be shared.
So now we will know the former struct fetch_support as
ext_support. The former name was potentially confusing
since the struct was used by both 'select' and 'reap'.
However, only 'reap' ever called the 'fetch' function.
[ this <STAT> already accommodated concurrent usage. ]
[ in fact it contains 3 separate sets of items/exts. ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
Oops, the PROCPS_VMSTAT_logical_end enum reflected the
source state before all those DELTA enum's were added.
[ plus add a couple of overlooked #undef directives, ]
[ and whack an unneeded stacks_alloc assignment too! ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
This patch was prompted through work on the <slabinfo>
upgrade from 2nd gen to 3rd. And while this assignment
caused no real harm, it most certainly was misleading.
[ plus add a couple of overlooked #undef directives! ]
Signed-off-by: Jim Warner <james.warner@comcast.net>
Before this major redesign, the slabs interface likely
was our messiest 2nd generation attempt at opaqueness.
Beyond the standard 'new', 'ref' & 'unref', there were
a total of 12 exported functions. Now, there are four.
The 1st step was to remove several of those functions.
These were quick to go since they were not used (yet):
. procps_slabnode_count
. procps_slabnode_getname
. procps_slabnode_getstack
Then, the following were internalized so users needn't
be burdened with implementation details in the future:
. procps_slabinfo_read (renamed: read_slabinfo_failed)
. procps_slabnode_stacks_alloc (renamed: stacks_alloc)
Still others evolved into the minimal interface we had
strived for in the other upgraded 3rd generation APIs:
. procps_slabnode_get -----------> procps_slabinfo_get
. separate stack_alloc/fill --> procps_slabinfo_select
. separate stacks_alloc/fill ---> procps_slabinfo_reap
. procps_slabnode_stacks_sort --> procps_slabinfo_sort
------------------------------------------------------
Beyond those reductions, the major modifications were:
. This API tries to be as forgiving as possible and as
such won't throw errors when a caller request makes no
sense. For example, if a 'get' or 'select' requested a
SLABNODE item (with no current means to id that node),
results will be zero. By the same token, should 'reap'
include a global SLABS item (meaning those values will
be duplicated in *every* node stack) it'll be allowed.
. If the above behavior is undesired, a new #define of
ENFORCE_LOGICAL can be used to restrict certain items.
. Permission problems will now be caught at 'new' time
thanks to a priming 'read' call. That read also serves
to make DELTA values potentially useful at 1st access.
. Separate slab/slabnode enumerators were consolidated
into one, simplifying validation & the results struct.
. Several internal parameter checks were relaxed since
they were already checked by the caller. Besides if we
cannot trust our own code we might as well hang it up.
. That sort provision was made more efficient and will
offer the ascending choice, in addition to descending.
------------------------------------------------------
Lastly, some additional thoughts regarding the future:
. It would not be difficult to expand 'select' to also
accept a nodeid, or to clone it as 'select_node'. And,
should the same be extended to 'get', a results struct
could be returned instead of signed long accommodating
the extra data type(s) like a node name (string data).
. The 'get' function is not currently affected by that
define ENFORCE_LOGICAL. However, at some future point
perhaps -EINVAL would be more appropriate than a zero.
Signed-off-by: Jim Warner <james.warner@comcast.net>
Where possible, libprocps files convey the name of the
actual source pseudo file under the '/proc' directory.
This patch brings slab into line with such a standard.
Signed-off-by: Jim Warner <james.warner@comcast.net>
The original test did not NULL the unallocated info pointer. This
was incorrectly calling the procps_pids_new() function.
Signed-off-by: Craig Small <csmall@dropbear.xyz>
This interface represented a 2nd generation attempt at
the opaque newlib approach. In other words, it did not
involve the 1st generation 'chains'. Instead, 'stacks'
were employed. But the interface wasn't user friendly.
Users were required to create their own stacks, before
calling 'getstack' to retrieve multiple results with a
single call. Even worse, sometimes 'read' was required
before calling 'get' when working with single results.
So this commit represents the 3rd generation approach.
We eliminate the burden of 'read' and creating stacks.
Rather, beyond those standard 'new', 'ref' and 'unref'
functions, we'll offer just 'get' (single result) plus
a 'select' function (for multiple results in 1 stack).
And along the way, this commit vastly expands the data
extracted from /proc/vmstat. All values that currently
exist (and their delta equivalents) are now available.
Deltas were included for everything because there's no
real runtime costs beyond using a little extra memory.
The only problem is a lack of documentation for all of
those fields, as is reflected in the references below.
Oh well, maybe someday someone will dig through kernel
sources & finally plug that rather large document gap.
[ as an aside, rather than using a 'strcmp' approach ]
[ when parsing the /proc/vmstat file, as is found in ]
[ the <meminfo> module, we exploit those hash search ]
[ provisions that are found in the <search.h> header ]
Reference(s):
http://www.spinics.net/lists/linux-man/msg09096.htmlhttp://www.linuxinsight.com/proc_vmstat.html
Signed-off-by: Jim Warner <james.warner@comcast.net>
With an eye to the future, rather than managing a long
switch/case construct in the procps_stat_get function,
we'll adopt the approach used in the <meminfo> module.
By making 'get' processing table driven too, just like
'select' is already, that may ease future maintenance.
Along the way a rather large boo-boo was fixed dealing
with that /proc/stat 'procs_running' field which crept
in with the commit referenced below. It wasn't spelled
correctly and thus was never captured as PROC_RUNNING.
Reference(s):
commit abc71a46ad
Signed-off-by: Jim Warner <james.warner@comcast.net>
When this interface was normalized/standardized, under
the commit referenced below, the parameters were never
validated in the 'get' function. Let's plug that hole.
Reference(s):
commit 407f1b71de
Signed-off-by: Jim Warner <james.warner@comcast.net>
Update the pids tests for tweaked API. Unsure why the toomany
test doesn't work, may point to a problem in the change as it used
to trigger, or the library has flexibility.
In that reference below a specific systemd problem was
fixed in the commit shown. However lurking deep within
the <pids> interface was yet one final case where NULL
could be returned, involving 'strv' and the following:
. a user requested both a single string vector (always
returned as a normal string) & the vectorized version,
as with PROCPS_PIDS_CMDLINE and PROCPS_PIDS_CMDLINE_V.
. a user simply duplicated some vectorized enum items.
The root of that NULL problem is the fact those single
string vectors shared the same proc_t field with their
true vectorized version. So while multiple occurrences
for most strings could be satisfied with strdup versus
the normal ownership usurpation, those true vectorized
fields could not be quite so easily copied/duplicated.
Thus newlib chose to return a NULL result.strv pointer
under either of the above scenarios (which perhaps was
just a user boo-boo in the first place). In any event,
the NULL was a potential for true string vectors only.
Now, since newlib is the sole caller into the readproc
module, separate fields have been created for what are
just normal strings (never vectorized) and those which
remain the true vectorized versions. And, former flags
which only worked if combined, now act as stand alone.
Thus, both PROCPS_PIDS_CMDLINE & PROCPS_PIDS_CMDLINE_V
can be used simultaneously (as they should have been).
Also with this patch, items which a user duplicates in
the stack (beyond the first such item) will return the
the string "[ duplicate ENUM_ID ]". This practice will
apply to both single strings and true vectorized ones.
In addition to informing users of their error, it will
also mean potential NULLs need now never be a concern.
Reference(s);
http://www.freelists.org/post/procps/systemd-binary-vs-library
commit 0580a7b4c6
Signed-off-by: Jim Warner <james.warner@comcast.net>
