"used memory" metric (not same as htop, etc)

[IoannisTM]

Hello,

Describe the bug

btop shows wrong memory used metric.
It worked fine but I think this happened after the upgrade to MacOS 12.3, although I cannot be sure 100% this is the cause.

To Reproduce

just run btop. not additional steps required

Screenshots
In Activity monitor, asitop and htop the used memory show to be ~17GB
btop shows ~30GB. It looks like it also calculates the cached files? I am not sure.
screenshot below

Info (please complete the following information):

• btop++ version: 1.2.5
• Binary: installed via brew
• Architecture: Apple silicon M1 Pro
• Platform: MacOS 12.3
• Terminal used: iTerm2

Additional context

contents of ~/.config/btop/btop.log
log is filled with similar errors like the below

2022/03/18 (01:25:29) | ERROR: Invalid RGB decimal value: ""#3d4070""
2022/03/18 (01:25:29) | ERROR: Invalid RGB decimal value: ""#073642""
2022/03/18 (01:25:29) | ERROR: Invalid RGB decimal value: ""#edb400""
2022/03/18 (01:25:29) | ERROR: Invalid RGB decimal value: ""#adc700""
2022/03/18 (01:25:29) | ERROR: Invalid RGB decimal value: ""#586e75""
2022/03/18 (01:25:29) | ERROR: Invalid RGB decimal value: """"
2022/03/18 (01:25:29) | ERROR: Invalid RGB decimal value: ""#a3a8f7""
2022/03/18 (01:25:29) | ERROR: Invalid RGB decimal value: """"
2022/03/18 (01:25:29) | ERROR: Invalid RGB decimal value: ""#6c71c4""
2022/03/18 (01:25:29) | ERROR: Invalid RGB decimal value: ""#268bd2""
2022/03/18 (01:25:29) | ERROR: Invalid RGB decimal value: ""#d6a200""
2022/03/18 (01:25:29) | ERROR: Invalid RGB decimal value: ""#268bd2""
2022/03/18 (01:25:29) | ERROR: Invalid RGB decimal value: ""#705500""

[jakenvac]

Came here to report the same issue. Glad it's not just me.

[usmcamp0811]

I just noticed the same issue on my server; I installed bpytop and it seems to report the correct memorry. Dunno what logs if any I could provide to help debug.

[jmakov]

Same here with btop 1.2.13 on Manjaro 23.0.4 showing 13.1GB whereas htop 1.56GB.

[imwints]

This has already been discussed here

I'm not convinced the calculation should change without knowing the reasoning for how it's calculated in free, htop etc.
To me it's makes sense that only non available memory is marked as "used". Maybe also free should be subtracted, but was under the assumption that free was included in available.

[...] According to this https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=34e431b0ae398fc54ea69ff85ec700722c9da773 MemAvailable is an estimate of how much memory a process could allocate without swapping. I am sure it includes MemFree.

MemUsed is a really just a question of definition and expectation and likely for this reason not defined in /proc/meminfo at all.

If you are very strict with your definitiion then you would have to say
used = total - free
but this would iritate lots of users.

The free implementation predates MemAvailable and therefore the developers had to come up with a definition of used.
free chose the definition
used = total - free - cached - buffers - SReclaimable

and btop chose
used = total - available

I can live with either of the definitions but i agree that the current implementation best shows to a user how much main memory is used and cannot be reclaimed easily.

The issue is just the expectation other programs like free, htop and top cause when they will be inevitable compared against btop. [...]

I agree that this is confusing since it doesn't align with the definition of free memory that other programs tent to have. Might be worth to look at and implement an option to toggle this behavior.

[aristocratos]

I see no issue having alternative ways of calculating the mem usage.
Beside the current and the htop way of calculating, might as well also add the used = total - free variant since that would be the calculation actually best fitting the definition for used memory.

Btw, leaving comments saying only "+1", "me too" or variants thereof, really serves no purpose.
If you are not providing any new information beside that you have the same issue, just leave a thumbs up or similar on the original post to not spam the thread with posts that just makes it messier to browse if a discussion gets started.

@leaf-ts

btop shows wrong memory used metric.

I'm curious which one is right? Since none of the programs show the same "used memory" in your screenshot.

[jmakov]

So guess it depends what for ppl use this info. My case is if having e.g. 16GB of RAM installed, running OS etc. programs that take N GB of RAM already, can I run another prog that I know will need 4GB of memory. Currently I can't answer this question with btop. Not sure what the use case of including buffers&caches in "used" is (am genuinely interested to know for what cases this info makes any sense) . htop colors e.g. buffers differently. Perhaps that would be an option to avoid trying to answer what the "real" number is - everybody can choose to look at diff color :)

[aristocratos]

@jmakov

can I run another prog that I know will need 4GB of memory. Currently I can't answer this question with btop.

Sure you can, look at the "available" memory, since you know... you wanna know how much memory you have available.

Not sure what the use case of including buffers&caches in "used" is (am genuinely interested to know for what cases this info makes any sense) . htop colors e.g. buffers differently. Perhaps that would be an option to avoid trying to answer what the "real" number is - everybody can choose to look at diff color :)

Me neither, and as answered above the calculation isn't based on cache and buffers.

The current calculation for used memory is "total memory - available memory" and thus shows how much memory is in "active" use and can not immediately be freed.

And you can already see a separate stat for how much memory is cached, so not sure why there would by any confusion there?

[jmakov]

You're right, I'm sorry, don't know how I missed diff stats for memory already.

Sure you can, look at the "available" memory, since you know... you wanna know how much memory you have available.

So forgive me if this is sth obvious to you but it's not at all to me. Looking at the OP's pic, according to btop, he can't launch a prog that will take 4GB of memory without touching swap. But according to htop he can.
So if I'd have 100 users on server1 and would look at btop, it would say (again, taking the OP's example), that server1 can't handle another user who will use 4GB (available=1.96GB from the OP's pic) and we'd invest in another server to handle users 101, 102 etc. But actually server1 could handle user 101 (according to e.g. htop) without swapping.

[aristocratos]

@jmakov

The available memory metric is directly reported from the kernel and described as:

An estimate of how much memory is available for starting new applications, without swapping.

From https://man7.org/linux/man-pages/man5/proc.5.html

But my mistake, this is only true for Linux.

In OP's case on MacOS there is no "available" metric provided, so there is probably room for improvement in how the "used" (and also "available") memory is calculated.

I will have to look it over the calculations for MacOs, maybe @joske have insights in to what the memory calculations for MacOs (and the BSD's) was based on.

[jmakov]

That solves my confusion, thank you very much for taking the time to ELI5 :) . Hope it can serve as future ref to others.

[joske]

It's absolutely not clear to me what these fields are supposed to mean. What is the difference between 'free' and 'available'? All OSes have different ways of accounting virtual memory, and expose a lot of details, but not clear how to map these to btop's 4 numbers.

For macOS, the memory metrics come from:

mem.stats.at("free") = p.free_count * Shared::pageSize;
mem.stats.at("cached") = p.external_page_count * Shared::pageSize;
mem.stats.at("used") = (p.active_count + p.inactive_count + p.wire_count) * Shared::pageSize;
mem.stats.at("available") = Shared::totalMem - mem.stats.at("used");

Looking at https://opensource.apple.com/source/xnu/xnu-3789.51.2/osfmk/mach/vm_statistics.h.auto.html, it seems the cached should use p.swapouts and not p.external_page_count.

[joske]

But when you compare the 'cached' field with Activity Monitor, it seems to match the value of external_page_count and not swapouts 🤔

[aristocratos]

Should be fixed for MaxOS in latest commit.

The issue seemed to be that we were counting inactive memory for MacOS, which we don't for the BSD's.

Now the used metric seems to be the same as in htop.

[ivan-volnov]

What about BSD? You closed issue #728 as a duplicate of this. But there is no fix for BSD.

[aristocratos]

@ivan-volnov
BSD uses the same calculation as MacOs (the way MacOs calculates after the fix).

My bad by making the assumption that would be right for BSD too, will have to investigate.