Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added information about Lambda Interruption #199

Merged
merged 6 commits into from
Oct 20, 2016

Conversation

halilduygulu
Copy link

Added information about Lambda Interruption which happened on my project this week. The information comes from aws support ticket and lambda engineering team according to support staff.

Copy link
Collaborator

@ThanosBaskous ThanosBaskous left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the submission, @halilduygulu!

Do you have a linkable source source for this (even AWS support comment forums-level)?

@@ -1055,6 +1055,7 @@ Lambda
- 🔸Lambda is a new technology. As of mid 2016, only a few companies are using it for large-scale production applications.
- 🔸Managing lots of Lambda functions is a workflow challenge, and tooling to manage Lambda deployments is still immature.
- 🔸AWS’ official workflow around managing function [versioning and aliases](https://docs.aws.amazon.com/lambda/latest/dg/versioning-aliases.html) is painful.
- ❗Currently (October,2016) Lambda functions can sometimes stop working for 2-3 minutes for failure recovery purposes according to a support ticket answer from Lambda dev team. They are working to prevent this in the future.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: "(as of October, 2016)"

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

@@ -1055,6 +1055,7 @@ Lambda
- 🔸Lambda is a new technology. As of mid 2016, only a few companies are using it for large-scale production applications.
- 🔸Managing lots of Lambda functions is a workflow challenge, and tooling to manage Lambda deployments is still immature.
- 🔸AWS’ official workflow around managing function [versioning and aliases](https://docs.aws.amazon.com/lambda/latest/dg/versioning-aliases.html) is painful.
- ❗Currently (October,2016) Lambda functions can sometimes stop working for 2-3 minutes for failure recovery purposes according to a support ticket answer from Lambda dev team. They are working to prevent this in the future.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not 100% clear on what "for failure recovery purposes" means - is it 2-3 minutes while recovering from failure?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As i sad it was a support ticket response. I don't have link to it because it is in our account, but i can copy paste the exact response here.

I've received an update from the engineering team.
They replied that customer may see their stream stop processing for three minute or less very infrequently due to certain failure recovery modes in our system. Their stream processing should then resume and recover from the delay.
They are working improvements to our system that should eliminate these sorts of pauses, we don't have a solid ETA on when that will be available.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also it happened to my 2 functions 4 times in total last week.

Copy link

@mklopets mklopets Oct 30, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a huge difference between "Lambda functions can stop working" and "streams can stop processing" though. This should belong in the Kinesis section then, imo.

Copy link
Collaborator

@ThanosBaskous ThanosBaskous left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an excellent addition, thank you @halilduygulu - a few requests related to the undocumented nature of the behavior.

@@ -1055,6 +1055,7 @@ Lambda
- 🔸Lambda is a new technology. As of mid 2016, only a few companies are using it for large-scale production applications.
- 🔸Managing lots of Lambda functions is a workflow challenge, and tooling to manage Lambda deployments is still immature.
- 🔸AWS’ official workflow around managing function [versioning and aliases](https://docs.aws.amazon.com/lambda/latest/dg/versioning-aliases.html) is painful.
- ❗Currently (as of October, 2016) Lambda functions can sometimes stop working for 2-3 minutes for failure recovery purposes according to a support ticket answer from Lambda dev team. They are working to prevent this in the future.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this isn't documented officially, would you mind doing the following?

  1. Adding a "undocumented" scroll icon after the serious gotcha red exclamation point? (📜)
  2. Convert "October, 2016" to be a link pointing to your support ticket text? ( https://github.com/open-guides/og-aws/pull/199/files/c99bddb4ee2437587f1e188d47be2bb1da01f81d#r83529126 )

@halilduygulu
Copy link
Author

halilduygulu commented Oct 17, 2016

@ThanosBaskous I made changes, could you check again. Thanks for reviews.

Copy link
Collaborator

@ThanosBaskous ThanosBaskous left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fixes, @halilduygulu! I have one more request to change something that was in the original review - sorry that I didn't catch it the first time around.

@@ -1055,6 +1055,7 @@ Lambda
- 🔸Lambda is a new technology. As of mid 2016, only a few companies are using it for large-scale production applications.
- 🔸Managing lots of Lambda functions is a workflow challenge, and tooling to manage Lambda deployments is still immature.
- 🔸AWS’ official workflow around managing function [versioning and aliases](https://docs.aws.amazon.com/lambda/latest/dg/versioning-aliases.html) is painful.
- ❗📜 Currently [as of October, 2016](https://github.com/open-guides/og-aws/pull/199/files/c99bddb4ee2437587f1e188d47be2bb1da01f81d#r83529126) Lambda functions can sometimes stop working for 2-3 minutes for failure recovery purposes according to a support ticket answer from Lambda dev team. They are working to prevent this in the future.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please replace "dev" with "development.

Copy link
Author

@halilduygulu halilduygulu Oct 18, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done and thanks for accepting.

@ThanosBaskous
Copy link
Collaborator

Thanks @halilduygulu! Since this doesn't have a topic, assigning to @jlevy for secondary review.

@ThanosBaskous ThanosBaskous assigned jlevy and unassigned halilduygulu Oct 18, 2016
@jlevy jlevy merged commit cc1c153 into open-guides:master Oct 20, 2016
@jlevy
Copy link
Collaborator

jlevy commented Oct 20, 2016

👍 Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants