Add http.[request|response].mime_type

[webmat]

Discussion points

• Should we make a concession that a response mime_type can be set based on the Content-Type header?
• Should this field use wildcard instead of keyword?
• Since populating this requires analysis of the body content, I think it may not be applicable to many sources, so I'm suggesting level extended

Closes #749

[webmat]

@ebeahan @neu5ron In #749 I considered making a concession and perhaps allowing http.response.mime_type to be populated based on Content-Type, when the source of the data is trusted. By this I meant only for web server logs, not a network IDS. An operator can likely trust their own servers. Whereas from a NIDS point of view, the responses may be untrusted.

Although at this time, this PR does not allow for that. After further thought, I think that concession would have little utility. And when ECS adds support for HTTP headers, that concession would now have zero utility anymore.

Do you agree with keeping the mime_type strictly based on body content?

[webmat]

@simitt Would love to have your thoughts on this, or someone else from APM. I think it's straightforward. Does APM capture Mime Type already? After a quick check on a demo app, it looks like it doesn't.

[ebeahan]

Should we make a concession that a response mime_type can be set based on the Content-Type header?

If we don't make the concession, *.mime_type field would be populated when the HTTP request/response is examined to determine the effective MIME type vs the Content-Type field? For example, mime sniffing in a browser? Ok - I think I understand better now 😄 .

Now that I have better understanding, I don't think we should make the concession since the support for HTTP headers would address Content-Type as you mentioned.

Should this field use wildcard instead of keyword?

I lean towards keyword. I could see leading wildcard queries being common here. However, since it should be also be using a curated set of values set of values, it should be a lower cardinality field and may perform better as keyword (reference: keyword vs. wildcard comparison table from the wildcard intro blog).

Since populating this requires analysis of the body content, I think it may not be applicable to many sources, so I'm suggesting level extended

👍

[webmat]

examined to determine the effective MIME type

Yes exactly. And if they differ, it's actually meant to be something to look into. It's a feature not a bug :-)

I also agree with sticking to keyword.

[simitt]

@webmat APM is not collecting a mime_type from the body; it would be stored as part of a request or response header if contained and not configured otherwise.