Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 refactor timer coros awspr multi worker #30

Open
wants to merge 36 commits into
base: 2.31.12-all-cherrypicks
Choose a base branch
from

Conversation

PettitWesley
Copy link
Owner


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
This commit combines many changes designed to
improve S3 stability:
- code clean up and refactoring for readability
- re-implement preserve_data_ordering code to
  fix several bugs and simplify logic.
- all uploads now happen outside of cb_flush
- uploads performed with new scheduled timer
  jobs with coroutines
- S3 always uses async IO, with mutex lock
  to protect it from concurrency issues
- remove trailing slash in store_dir to
  prevent double // in final path

Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
…t async timers on the sched

Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
@PettitWesley
Copy link
Owner Author

I still don't know why the build is failing. I'm gonna try rebasing these changes on top of master and debug there instead.

[ 40%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_config.c.o
In file included from /home/ec2-user/fluent-bit/include/fluent-bit/flb_io.h:27:0,
                 from /home/ec2-user/fluent-bit/include/fluent-bit/flb_upstream.h:29,
                 from /home/ec2-user/fluent-bit/include/fluent-bit/flb_output_thread.h:24,
                 from /home/ec2-user/fluent-bit/src/flb_async_timer.c:22:
/home/ec2-user/fluent-bit/include/fluent-bit/flb_output.h: In function ‘flb_output_flush_create’:
/home/ec2-user/fluent-bit/include/fluent-bit/flb_output.h:591:35: error: dereferencing pointer to incomplete type ‘struct flb_out_thread_instance’
         pthread_mutex_lock(&th_ins->flush_mutex);
                                   ^~
[ 40%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_socket.c.o
make[2]: *** [src/CMakeFiles/fluent-bit-static.dir/flb_async_timer.c.o] Error 1
make[2]: *** Waiting for unfinished jobs....
[ 40%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_network.c.o
make[1]: *** [src/CMakeFiles/fluent-bit-static.dir/all] Error 2
make: *** [all] Error 2

Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant