Merge pull request opensearch-project#344 from okhasawn/linkCheckWork…

…flow

Add Link Checking Workflow

.github/workflows/linkCheck.yml

@@ -0,0 +1,28 @@
+name: Link Checker
+on:
+  push:
+    branches:
+      - "*"
+  pull_request:
+    branches:
+      - "*"
+
+jobs:
+  linkchecker:
+
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v2
+      - name: lychee Link Checker
+        id: lychee
+        uses: lycheeverse/lychee-action@v1.5.0
+        with:
+          args: --verbose --accept=200,403,429  "**/*.html" "**/*.md" "**/*.txt" "**/*.json"
+            --exclude "file:///github/workspace/*"
+            --exclude "http://localhost*"
+            --exclude "https://localhost*"
+            --exclude-mail
+          fail: true
+        env:
+          GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}

SECURITY.md

@@ -14,7 +14,7 @@ When configuring the Traffic Replayer, you can provide a set of credentials that
 
 In this case, an attacker could send a series of large and potentially malformed requests to the source cluster via the capture proxy. These messages would be relayed to the Kafka cluster, and if they were PUT/POST/UPDATE requests, block sending the request to the source cluster until the message was finished. If the attacker is able to use this strategy to tie up the proxy and/or Kafka cluster, all other incoming mutating requests to the source cluster would be blocked.
 
-We have partially mitigated this by preventing the proxy from blocking for more than a fixed period of time (10 seconds by default, configurable [here](./TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/Main.java#L182)), however the flow of messages to Kafka could still be disrupted.
+We have partially mitigated this by preventing the proxy from blocking for more than a fixed period of time (10 seconds by default, configurable in this file [here](./TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java), however the flow of messages to Kafka could still be disrupted.
 
 If you are concerned about this scenario, we recommend fully mitigating it by putting a load-shedder in front of the proxy.
 

TrafficCapture/trafficReplayer/README.md

@@ -11,10 +11,9 @@ encoded [TrafficStream](../captureProtobufs/src/main/proto/TrafficCaptureStream.
 Currently, these TrafficStream objects are ingested via stdin and are reconstructed into entire traffic channels. This
 involves some buffering for those connections whose contents are divided into a number of TrafficStream objects.
 Read and write observations are extracted from TrafficStream objects into source requests and source responses.
-The [TrafficCaptureToHttpTransactionAccumulator](src/main/java/org/opensearch/migrations/replay/TrafficCaptureToHttpTransactionAccumulator.java)
+The [CapturedTrafficToHttpTransactionAccumulator](src/main/java/org/opensearch/migrations/replay/CapturedTrafficToHttpTransactionAccumulator.java)
 takes full requests (as defined by the data, not necessarily by the HTTP format) and sends them to
-an [IPacketToHttpHandler](
-src/main/java/org/opensearch/migrations/replay/datahandlers/IPacketToHttpHandler.java). The packet handler is
+an [IPacketConsumer](src/main/java/org/opensearch/migrations/replay/datahandlers/IPacketConsumer.java). The packet handler is
 responsible for doing
 any transformation of the request and sending it to the target server. It is also responsible for aggregating the HTTP
 response from the server and returning that as a CompletableFuture via finalizeRequest().
@@ -25,9 +24,9 @@ other pertinent information is sent to stdout.
 ## The Netty Request Transformation Pipeline
 
 There are two implementations of
-IPacketToHttpHandler, [NettyPacketToHttpHandler](../trafficReplayer/src/main/java/org/opensearch/migrations/replay/datahandlers/NettyPacketToHttpHandler.java),
+IPacketToHttpHandler, [NettyPacketToHttpConsumer](../trafficReplayer/src/main/java/org/opensearch/migrations/replay/datahandlers/NettyPacketToHttpConsumer.java),
 which will send packets to the target server
-and [HttpJsonTransformer](../trafficReplayer/src/main/java/org/opensearch/migrations/replay/datahandlers/http/HttpJsonTransformer.java)
+and [HttpJsonTransformingConsumer](./src/main/java/org/opensearch/migrations/replay/datahandlers/http/HttpJsonTransformingConsumer.java)
 that is capable of transforming the message as per directives passed to the JsonTransformer.
 
 Examples of transformations that the HttpJsonTransfomer needs to run include mapping the host header to match the new
@@ -40,9 +39,9 @@ over the constructed JSON document.
 Since payloads can be arbitrarily complex (compression, chunking), and may not be subject to any changes via the
 transformation rules, the HttpJsonTransformer creates the channel pipeline *only* to parse the HTTP headers. The
 transformation is run on this partial, in-construction
-[HttpJsonMessageWithFaultablePayload](../trafficReplayer/src/main/java/org/opensearch/migrations/replay/datahandlers/http/HttpJsonMessageWithFaultablePayload.java)
+[HttpJsonMessageWithFaultablePayload](./src/main/java/org/opensearch/migrations/replay/datahandlers/http/HttpJsonMessageWithFaultingPayload.java)
 message. When the transformation (or any other code), attempts to access the payload contents, it will throw a
-[PayloadNotLoadedException](TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/datahandlers/PayloadNotLoadedException.java)
+[PayloadNotLoadedException](./src/main/java/org/opensearch/migrations/replay/datahandlers/PayloadNotLoadedException.java)
 exception. That exception triggers the HttpJsonTransformer to add channel handlers to the pipeline to parse the HTTP
 content stream into JSON, transform it, and to repackage it as the HTTP headers indicate, observing the content-encoding
 (gzip, etc) and transfer-encoding (chunked)/content-length values. Fixed length streams will be used by default, with
@@ -78,7 +77,7 @@ target URI.
 ## Transformations
 
 Transformations are performed via a simple class defined by
-[JsonTransformer](../trafficReplayer/src/main/java/org/opensearch/migrations/transform/JsonTransformer.java). Currently,
+[IJsonTransformer](../trafficReplayer/src/main/java/org/opensearch/migrations/transform/IJsonTransformer.java). Currently,
 this class uses [JOLT](https://github.com/bazaarvoice/jolt) to perform transforms that are composed of modular
 operations that are defined in the [resources](../trafficReplayer/src/main/resources/jolt/operations) associated with
 the package. Future work will include adding more JSON transformations and other potential JSON transformation tools

deployment/README.md

@@ -1,8 +1,6 @@
 ### Deployment
 This directory is aimed at housing deployment/distribution methods for various migration related images and infrastructure. It is not specific to any given platform and should be expanded to more platforms as needed. 
 
-It is worth noting that there is not a hard divide between these subdirectories and deployments such as [opensearch-service-migration](./cdk/opensearch-service-migration) will use Dockerfiles in the [docker](./docker) directory for some of its container deployments.
-
 
 ### Deploying Migration solution to AWS
 

deployment/copilot/README.md

@@ -193,7 +193,11 @@ copilot svc exec -a migration-copilot -e dev -n migration-console -c "bash"
 
 ### Addons
 
-Addons are a Copilot concept for adding additional AWS resources outside the core ECS resources that it sets up. An example of this can be seen in the [traffic-replayer](traffic-replayer/addons/taskRole.yml) service which has an `addons` directory and yaml file which adds an IAM ManagedPolicy to the task role that Copilot creates for the service. This added policy is to allow communication with MSK.
+Addons are a Copilot concept for adding additional AWS resources outside the core ECS resources that it sets up. 
+
+An example of this can be seen in the `traffic-replayer/addons/taskRole.yml`  service which has an `addons` directory and yaml file.
+
+That yaml file adds an IAM ManagedPolicy to the task role that Copilot creates for the service. This added policy is to allow communication with MSK. (Note that `taskRole.yml` will only exist after building.)
 
 Official documentation on Addons can be found [here](https://aws.github.io/copilot-cli/docs/developing/addons/workload/).
 

experimental/knowledge_base/README.md

@@ -2,6 +2,6 @@
 
 The Knowledge Base is the collection of expectations we have about the behavior of clusters of various versions and across upgrades.
 
-It can be used in various ways, but was designed to be a component of the Upgrade Testing Framework. There is more extensive documentation of how it works in [experimental/upgrades/README.md](../experimental/upgrades/README.md).
+It can be used in various ways, but was designed to be a component of the Upgrade Testing Framework. There is more extensive documentation of how it works in [experimental/upgrades/README.md](../upgrades/README.md).
 
 Further discussion on adding additional expectations to this knowledge base, can be found in the PR [here](https://github.com/opensearch-project/opensearch-migrations/pull/68)