Changing op_type in ingest pipeline in case of _bulk operation

[lockwooddev]

Problem
I tried using an ingest pipeline to write to a data stream, but the client writing to the pipeline is using the bulk API without create as op_type. This would require a code change to the writing client.

Proposed solution
It would be nice to have roughly the following processor in my ingest pipeline:

        {
            "set": {
                "field": "op_type",
                "value": "create"
            }
        }

Alternative
Alternatively I could not use a data stream or change the client (which I don't want to do in this case)

I hope the case I present is clear. I would love some feedback on this strange case I have.
Thanks in advance!

[mvillafuertem]

Hi @lockwooddev @ryanbogan , any update about this? it seems that I have this problem

org.opensearch.dataprepper.plugins.sink.opensearch.BulkRetryStrategy - operation = Index, status = 400, error = only write ops with an op_type of create are allowed in data streams

[gaobinlong]

Sounds like a valid use case, now we supports modifying part of the metadata fields:

OpenSearch/server/src/main/java/org/opensearch/ingest/IngestDocument.java

Line 865 in 087355f

IF_PRIMARY_TERM("_if_primary_term");

, we can also support modifying the op_type parameter during the execution of ingest pipeline, will dive deep into this.