Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Track temp dirs and remove when staled. #30

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

mashhurs
Copy link
Contributor

@mashhurs mashhurs commented Jun 9, 2023

Description

This PR mainly focuses on two changes:

  1. Tracks every created temp files make sure they are removed.
    Introduced temp_files container (FileRepository locks for thread safety) to keep and track all rotated files. When file is removed from track list?
  • when recently created file becomes stale, means no event written to the file

When file and tempdir are physically deleted?

  • when uploaded to S3 - both temp dir and file are deleted.
  • when becomes stale: no event written to the file during 15 mins - both temp dir and file are deleted.
  • !NEW: deleting file (file.delete!) didn't succeed, means third party process (especially on Windows, security scanners) is keeping open the IO and Logstash removes the file. Physically, for this case, file will be deleted but temp dir stays. - both temp dir is deleted.

asdas

+def remove_staled_files
+  with_lock do |factory|
+    factory.temp_files = factory.temp_files.delete_if do |temp_file|
+     is_staled = temp_file.size == 0 && (Time.now - temp_file.ctime > @stale_time)
+      is_temp_dir_empty = false
+     begin
+        # checking Dir emptiness and remove file
+        # reading file and checking size doesn't make sense as it will not possible after temp_file.size == 0 filter
+        temp_file.delete! if is_staled
+        is_temp_dir_empty = Dir.empty?(::File.join(temp_file.temp_path, temp_file.prefix)) unless is_staled
+        temp_file.delete! if is_temp_dir_empty
+      rescue => e
+        @logger.error("An error occurred while sweeping temp dir.", :exception => e.class, :message => e.message, :path => temp_file.path, :backtrace => e.backtrace)
+      end
+      is_staled || is_temp_dir_empty
+    end
+    # mark as deleted once we finish tracking all temp files created
+    @is_deleted = factory.temp_files.length == 0
+  end
+end
  1. Moves the periodic stale sweeper to higher level, same level as periodic file rotator to make KISS. Also, confusing codes are cleaned such as properly namings (ex: periodic file rotator, periodic file sweeper), correcting misleading comments/log-descriptions.

@mashhurs mashhurs self-assigned this Sep 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant