Skip to content

Add top-like monitoring tool#333

Draft
ben-grande wants to merge 21 commits into
QubesOS:mainfrom
ben-grande:qvm-top
Draft

Add top-like monitoring tool#333
ben-grande wants to merge 21 commits into
QubesOS:mainfrom
ben-grande:qvm-top

Conversation

@ben-grande

@ben-grande ben-grande commented Jan 28, 2025

Copy link
Copy Markdown
Contributor

Add top-like monitoring tool

Inspiration and code from qui-domains and visuals from xentop.

  • Colors entries according to state
  • Footer with clock to show last time the screen was refreshed
  • Allows scrolling domain list when it is longer than the screen height,
    with a less-type hint indicating how many lines to scroll
  • To scroll, readline style (Emacs) navigation is supported, with some
    Vi keys also supported
  • Inexpensive calls using "admin.vm.Stats"

Requires: QubesOS/qubes-core-admin#827

This PR is based on top of:

@marmarek

Copy link
Copy Markdown
Member

is a blocking operation as well as proc.stdout.read(2)

Do you mean read() hangs for you? Maybe the service waits for something before starting to output any data (doesn't look like it based on the other PR...).

@ben-grande

ben-grande commented Jan 28, 2025 via email

Copy link
Copy Markdown
Contributor Author

@marmarek

Copy link
Copy Markdown
Member

Based on discussion in the other PR, this should be converted to use stat events (and events extended to include necessary info). Converting to draft until then.

@marmarek
marmarek marked this pull request as draft June 10, 2026 00:49
@ben-grande
ben-grande force-pushed the qvm-top branch 2 times, most recently from 75a9ee5 to bc049c9 Compare June 10, 2026 19:25
@ben-grande

Copy link
Copy Markdown
Contributor Author

LLM helped on draw_table(). It's almost stable. It works nicely on R4.2, but not on R4.3. I don't know yet why builds are failing on manpages.

@codecov

codecov Bot commented Jun 11, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 1.25733% with 1178 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.30%. Comparing base (0c7b921) to head (f1ca237).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
qubesadmin/tools/qvm_top.py 0.00% 1171 Missing ⚠️
qubesadmin/app.py 54.54% 5 Missing ⚠️
qubesadmin/tools/__init__.py 83.33% 1 Missing ⚠️
qubesadmin/vm/__init__.py 66.66% 1 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (0c7b921) and HEAD (f1ca237). Click for more details.

HEAD has 1 upload less than BASE
Flag BASE (0c7b921) HEAD (f1ca237)
2 1
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #333      +/-   ##
==========================================
- Coverage   76.79%   68.30%   -8.49%     
==========================================
  Files          53       54       +1     
  Lines        9402    10586    +1184     
==========================================
+ Hits         7220     7231      +11     
- Misses       2182     3355    +1173     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ben-grande

Copy link
Copy Markdown
Contributor Author

One thing I don't plan to fix is the memory reporting being wrong, as only qmemman has the knowledge about memory of each qube, and getting information from it is not currently nice with it's current API (synchonous), and I don't want to delay reporting statistics because qmemman is busy for seconds at a time.

@ben-grande

Copy link
Copy Markdown
Contributor Author

Maybe it's not necessary, reading past conversations, something that I didn't understand one year ago (qmemman, admin.vm.Stats), I think I will settle for meminfo: QubesOS/qubes-core-agent-linux#550 (comment).

@ben-grande

Copy link
Copy Markdown
Contributor Author
qvm-top

xentop has cpu_time for each vcpu, but that may be too much. Xentop has actually a lot of things that can be show about virtual blocks also, but I don't like that much info. Customizing delay is not important to us, the screen refreshed in a loop and the delay is pretty insignificant and is not causing issues to navigate up and down. One think I like is that xentop has sort order, but I can't really see which column is currently being sorted, it's supposed to be bold. Anyway, I like how qvm-top is right now and don't plan to improve further (unless I think about this again, tomorrow...).

@ben-grande

ben-grande commented Jun 11, 2026

Copy link
Copy Markdown
Contributor Author

And one more bug, for dom0, it should use cpu_usage_raw instead of cpu_usage. It seems that the value gets much closer to what xentop reports about Domain-0.

@ben-grande

Copy link
Copy Markdown
Contributor Author

And yes, vm-stats from my core-admin PR is reporting stubdomains, but as the client VMCollection doesn't know about stubdomains, it's never considered. Maybe I could workaround this limitation.

@ben-grande
ben-grande force-pushed the qvm-top branch 4 times, most recently from 4ab3c3a to 8027157 Compare June 12, 2026 01:46
@ben-grande

Copy link
Copy Markdown
Contributor Author
new-qvm-top
  • stubdom reporting, can't show memory usage for it, but can show CPU stats
  • NA is better than lying how much memory the qube is using
  • I don't know why Xen adds one mebibyte to maxmem_kb (domain_getinfo) but not on static-max xenstore key.

@marmarek

Copy link
Copy Markdown
Member

I don't know why Xen adds one mebibyte to maxmem_kb (domain_getinfo) but not on static-max xenstore key.

I'm not sure either, but I guess domain_getinfo includes some internal overhead in Xen (EPT? maybe something else too?), but static-max is just what OS can really see.

Anyway, I don't like exposing stubdomains in a high level tool, it should be counted as part of the target qube; if anything, maybe some separate column for some internal overhead, but not separate row.
If somebody really want to look under the hood, they have xentop and other low level tools.

When checking libvirt event callbacks, VIR_DOMAIN_EVENT_STOPPED is the
only event called when shutting down or destroying/killing a domain, so
the state transitions from Running->Halted, therefore, never sending
"Halting" or "Dying".

I checked the clients, and only qubes-manager uses "Dying", but it has a
"Halting" equivalent, which all translate to the same visuals as
"Transient".
Variables set on the module level so it can be reused by other clients.
@ben-grande

Copy link
Copy Markdown
Contributor Author

PR rebased on top of:

Besides the name "Transient" means impermanent, it could as well take a
long time in case the qrexec agent on the qube broke. Also it doesn't
help anything knowing that via the power state, it should be broadcasted
in another manner. It also doesn't tell the clients why it is in a
transient state, while "Starting" and "Halting" really does tell us what
is happening.

For: QubesOS/qubes-issues#10964
For: QubesOS/qubes-issues#10966
Cut in half the to get derived qubes.
Inspiration and code from qui-domains and visuals from xentop.

- Color data, when possible, qube names are colored according to label,
  it works okay on XTerm (except orang) but colors render perfectly on
  Xfce4-terminal. The black label is just printed as the default
  foreground color, because using standout on it would be unnecessary
  highlight.
- Clock to show last time the screen was refreshed
- Allows scrolling domain list when it is longer than the screen height,
  with a less-style type hint indicating pages
- To scroll, readline style (Emacs) navigation is supported, with some
  Vi keys also supported. Mouse navigation is also supported.
- Inexpensive calls using "admin.vm.Stats"
- Can show only columns you specify
- Shows all non-halted domains by default, but can show only specified
  ones also
- Screen refresh is delayed until needed, small sleep to avoid user
  perceiving delays when scrolling
- Can sort per column and click again to reverse the sort, with a little
  indicator at the end of the column
- Can filter list of domains after the program is already running
- Can act based on common actions available from all selected domains,
  such as kill if qubes are running, start if qubes are halted
Strings are trusted, they are just label and name. But some values such
as those reported by the qube via xenstore keys, are not so trusted and
can be large integers. Instead of printing dubious data, print an
ellipsis to symbolize it wouldn't fit. It might be a programming error,
might be a malicious information, fall on the safe side.
- Swap column introduced to know when the qube should have more memory,
  swaping is never nice.
- Columns are ordered in "total", "standard", "internal".
- Do not consider percentage intensity when irrelevant
The "min_width" is used to avoid the columns moving too much.
This is not perfect, if the list change and the qube is not in the
visible section anymore, it will go back to the first row.
It is still shown when "show_halted" is enabled, just to view the qubes
in the system, but obviously, it can't display any statistic or run any
action on it.
@ben-grande
ben-grande force-pushed the qvm-top branch 4 times, most recently from 96844be to 7e6632a Compare July 16, 2026 16:14
@ben-grande
ben-grande marked this pull request as draft July 20, 2026 11:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants