Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

replace safe_set with set #147

Merged
merged 6 commits into from
Aug 3, 2022
Merged

replace safe_set with set #147

merged 6 commits into from
Aug 3, 2022

Conversation

kingluo
Copy link
Contributor

@kingluo kingluo commented Jul 28, 2022

When the number of metrics increase, especially due to label value changes or add/remove labels on reload, it's likely to use up the shdict, when mostly we want to remove the old items in shdict making use of the lru cache mechanism of shdict. The old items are supposed to be ignored.

In fact, even we insist safe_set, the inc we used in other places of codes may also purge old items. So why not make them consistent?

@knyar
Copy link
Owner

knyar commented Jul 29, 2022

Thank you for proposing this.

In general, I believe that the library should maintain data integrity and never lose metric values. If shdict gets full and lru eviction kicks in, it should be treated as an error: nginx_metric_errors_total should be incremented, and error log messages should say that reported metric values might be incomplete.

safe_set is used as an attempt to maintain data integrity, but I think you are absolutely right that eviction can also happen when a previously nonexistent dictionary key is incremented using :incr. The fact that we do not detect evictions in such cases is a bug.

What I think would be a good way forward is to proceed with replacing safe_set with set, but also catch the forcible return value every time we write to a shdict (using set or incr) and report errors when it becomes true (which means that the dict is full).

@kingluo
Copy link
Contributor Author

kingluo commented Jul 29, 2022

I agree. Let me have a try.

@kingluo
Copy link
Contributor Author

kingluo commented Aug 2, 2022

@knyar Please review my commits and approve the github workflow, thanks.

@knyar knyar changed the base branch from master to eviction August 3, 2022 06:45
@knyar
Copy link
Owner

knyar commented Aug 3, 2022

Thank you! I am going to merge this into a separate branch to make a few follow-up adjustments before merging this to master.

@knyar knyar merged commit f587efa into knyar:eviction Aug 3, 2022
knyar pushed a commit that referenced this pull request Aug 5, 2022
hanshuebner added a commit to Kong/kong that referenced this pull request May 30, 2024
- replace safe_set with set

  cherry pick of knyar/nginx-lua-prometheus@5b6a209
  See knyar/nginx-lua-prometheus#147

- improve LRU eviction error message

- improve histogram consistency

  cherry-pick of knyar/nginx-lua-prometheus@0d790d0
hanshuebner added a commit to Kong/kong that referenced this pull request May 30, 2024
- replace safe_set with set

  cherry pick of knyar/nginx-lua-prometheus@5b6a209
  See knyar/nginx-lua-prometheus#147

- improve LRU eviction error message

- improve histogram consistency

  cherry-pick of knyar/nginx-lua-prometheus@0d790d0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants