Getting ECS to 1.0

[webmat]

Getting ECS to Beta1

Here's an attempt at summing the work that should be completed, in order to get ECS to Beta1.

Why this plan?

• We need to get agreement on what should be in 1.0, and what is kept for later
• We need to have this plan in writing, so we can set expectations on timeline and amount of effort remaining
• With everyone slowly ramping up to conform to ECS, we need to make it clear what's still planned to change
• We need to have as much of this ready as early as possible, so that people working to implement ECS are working towards the right thing :-)

How?

Part of the strategy is dividing the fields into ECS Core (which should be more stable) and ECS Extended (which may change a little more in the future), and making sure at least Core is very stable for 1.0. We'll do our best to stabilize ECS Extended as well, on a best effort basis.

I may have included too much in this plan for 1.0, challenging each issue's inclusion for 1.0 (especially if they're ECS Extended) is welcome ;-)

This issue will contain lots of checkmark tasks. All of these should eventually become a concrete GitHub issue. But this is not the case yet: the idea is to get my view of this plan out there, discuss and firm up the high level plan ASAP. Then we can firm up what needs to happen where, create subsequent tasks or PRs, to get to 1.0.

We should discuss/challenge this plan here, and update the body of the overarching plan (this issue) accordingly.

I think it will be helpful to label all issues that that go into this plan with a label such as ga-1.0.

Finally, some of these issues are still written in extreme shorthand (e.g. 1 or 2 words), that may only make sense for the folks closest to the ECS discussions. Feel free to ask for clarifications :-)

Without further ado, here's the plan how I see it at the moment.

Multiple issues

• This issue is for work considered blockers to declare GA
• Beta1 related but not blocker tasks #161 is for work related to GA but non-blocker tasks
• Post GA tasks #160 is for post-GA tasks

ECS plan for Beta1

• Perform the actual release (tag, announcements, etc)

ECS Core/Extended/Other (93)

• Document which field is Core & Extended (ECS Core and ECS Extended #93)
  • Mention the 3rd level as well, right now named "Other". There's discussions around having "Application", "Service", "Protocol"
  • Assign each field to Core or Extended in the yml
  • What do we do with "Other"? Leave unassigned in the yml?
• Update the script generating the doc to indicate Core vs Extended fields (Add column 'level' to display which field is core/extended #149)

Top Level Objects

The "Top Level Objects" section is meant to outline which objects will stay at the top level and which ones are slated to be 1) nested; 2) become a "reuseable" object (nestable in different places); or 3) are new objects.

The work that needs to happen in these TLOs (changes on the fields themselves) is listed in the other sections below.

Core

• agent
• cloud
• container
• destination
• device
• event
• error
• file
• geo
  • reuseable object
• http
  • take out temporarily (Temporarily remove the http object... #171)
• host
• kubernetes (removed in Remove two top level objects: kubernetes and tls. #132)
• log
• network
• organization
• os
  • reuseable object
• process
• service
• source
• tls (removed in Remove two top level objects: kubernetes and tls. #132)
• url
• user
  • reuseable object
• user_agent
  • Move to use cases (Temporarily remove the user_agent field set... #172)

Structure

• Compose vs Reuse
  • close discussion (Compose ECS objects vs reuse objects #71, Making of all objects reusable #72, Document how / whether objects should be reused in different places in ECS #94)
  • Document which objects are reuseable, how and where (Document how / whether objects should be reused in different places in ECS #94)
  • reuseable:
    • geo (closes Doing geoip on more than one address... #9)
    • os
    • user
    • threat
  • Remove nested os in some tables where it was initially duplicated manually a while ago. Now that it's a reusable object, it's expected in these places, but the duplication of the fields under user_agent.os and host.os is superfluous (Remove duplicate definitions of os from host.os and user_agent.os... #168)

Datatypes

• New convention for text indexing
  • (Almost) always keyword at the top level (Switch most fields to keyword as the type #137)
  • Analyzed will be the nested field if needed. Users can add analyzed nested field without breaking compatibility with ECS. (Introduce the new convention for multi-fields text indexing to the README. #140)
  • Two exceptions to this rule, to avoid further breaking changes. The following fields will be text only indexing: message, error.message.
  • Rewrite the doc that talks about multi-fields to reflect the new approach. (Introduce the new convention for multi-fields text indexing to the README. #140)
  • We will need to close issues / undo all other multi-fields. See Move to the ElasticSearch convention of using ".keyword" to denote nested fields indexed with type "keyword"  #103, Pore over all existing text fields to identify more that should be multi-fields #104, Rename all nested fields mentioned in #103 to .keyword #118, Make 4 more text fields multi-field. #121

Naming

• hostname (Question about template usage for ecs #31, host.name vs hostname, host.ip vs ip inconsistencies #62, pr Rename hostname to host.name fields to be consistent #74)
• url.original (pr Rename url.href to url.original #109)

Documentation

• Clarify use of hostname, subdomain, domain in source/destination (Clarify use of hostname, subdomain, domain in source/destination #84, hostname PR Remove src/dst.hostname, rename url.hostname to url.domain. #175)
  • Note: subdomain-related fields have been taken out of beta 1
• Adjust the README intro for the release (Adjust the readme intro for the Beta1 release #167, PR Prepare the readme intro and changelog for 1.0.0-beta1 #174)

Network discussions

• network protocol layers (pr Revises network.* to cover more use cases #81)
  • protocol.name and .id (pr Adds network.protocol.name and network.protocol.id  #79)

ECS

• Rename event.version to ecs.version (pr Rename event.version to ecs.version. #169)

[webmat]

I just updated the issue based on today's ECS meeting.

[ruflin]

Hi @webmat thanks for putting this together. Here my notes when I read the above issue:

• Document Core, Extended, Other: As mentioned before for simplicity reason, there is no other. Introducing Application / Service later can be easily done and is not a breaking change to fields.
• Schema management: Most important for 1.0 is having all the fields fixed. Creating multiple templates, fields.yml can still follow later. Most initial user of ECS I assume will have their own templates.
• Fields: Lets finish our internal spreadsheet discussion around what field should be where and then we can open a few PR's to adjust it. Not sure if it makes sense to list them all here
• Non ECS Fields: Would be nice to come to a conclusion here but not required for 1.0. Same for sharing the schema.
• Datatypes: long vs int does not really matter.
• Naming: This is the issue we really need to resolve
• Upcoming fields: Solved by definining just custom as the only reserved option
• Docs: Most parts can follow after GA
• Adding fields: Every single field we plan to add, can be added after GA. Adding fields is not breaking and we can do a 1.0.1 or 1.1 soon afterwards.
• Changes in Elastic Stack: These changes can all happen after GA but work should start now.

In summary: For me the highest priority has getting agreement on all the fields we have right now in the Github repository and update the fields.yml accordingly. Everything around it like docs, generation of template etc. is nice to have. We could also call the release with all fields fixed 0.9 to make it more clear more automation and docs are following. From my perspective the only breaking changes that can happen to ECS are when we change names of the fields.

[webmat]

@MikePaquette @ruflin

I've added a new H1 section to put the things related to GA, but not blockers to declaring the schema itself GA.

• The "automation" and "Related projects" H2 sub-sections were already at the end of my list, so they are already in that new section for later.
• I've also created another "documentation" section for the things that aren't that pressing to document either.
• I've moved a few odd tasks that were elsewhere but about "automation" to that new section for after GA

Another thing of note, I've removed the following task: "Review current fields and consider making some longs into ints (e.g. port)"

[webmat]

I've just moved 15 tasks out of 1.0 in this plan. They're now at the beginning of the "Post 1.0" section. Let me know if there's any disagreement with this. Some of it we've discussed internally recently, and some of it is simply non-critical issues I had optimistically included in the 1.0 plan.

[ruflin]

Can we move everything which is post GA and for other projects into a different issue? So this issue is purely about going GA?

[webmat]

@ruflin I had already been thinking about splitting this in 3, actually. This issue's purpose would become to track work for internal availability (IA), create a new one for some more work in getting this ready for GA, and finally one that would be post-ga (but related to GA, close after).

WDYT?

[ruflin]

SGTM

[webmat]

I've removed a bunch of mentions of "GA" and 1.0 (not all), in favour of "Beta1". Beta1 is what we're shipping as of Stack 7.0.

@ruflin I've moved a few things to later:

Moved to #161:

• Reflect object reuse in template.json
• Reflect object reuse in schema.csv
• Fix script issue with multi_fields, introduced when adding for Core/Extended

Moved to #160:

• Introduce related.* field set (Propose new top-level prefix 'related' #67)
• Discuss nesting http (and eventually all network protocol field sets) under network?

And added new tasks:

• Adjust the readme for the release (Adjust the readme intro for the Beta1 release #167)
• Remove nested os in some tables where it was initially duplicated

[webmat]

We forgot to close this :-D