Revamp post filtering system #18058

ClearlyClaire · 2022-04-20T13:46:01Z

New filter management UI

It's now possible to regroup filter keywords in categories.

New filtered post UI

Instead of just displaying “Filtered”, the filter category name is displayed, and an opportunity is given to show the filtered post:

test.mp4

API changes and implementation guidelines

Note that everything described here is subject to change before the feature actually gets released.

Backward compatibility

This PR provides limited backward compatibility with the /api/v1/filters API:

GET /api/v1/filters is supported and will return all keywords, including when multiple keywords are in a single category. Existing pre-update keywords should retain their identifier, though there is no guarantee this will be the case.
GET /api/v1/filters/:id is supported and will return a keyword, even if that keyword shares a category with other keywords. If that keyword was created pre-update, it should retain its identifier, though there is no guarantee this will be the case.
POST /api/v1/filters is supported, and will create a one-keyword filter with a title matching the keyword
PUT /api/v1/filters/:id is partially supported, and will error out if it attempts to change expires_in, irreversible or context for a filter shared by multiple keywords.

New API

New entities

Filter: has attributes id, title, context, expires_at, filter_action (currently either hide or warn), and optionally keywords (list of FilterKeyword entities).
This entity will likely be extended in the future with attributes for other rule types (e.g. conversations).
FilterKeyword: has attributes id, keyword and whole_word
FilterResult: has attributes filter (a Filter without a keywords attribute) as well as keyword_matches.
This entity will likely be extended in the future with attributes for other rule types (e.g. conversation_matches).

Changes to other entities

Status entities now optionally have a filtered attribute which is a list of FilterResult entities.

New API endpoints

GET /api/v2/filters to list filters (including keywords)
POST /api/v2/filters to create a new filter
keywords_attributes can also be passed to create keywords in one request
GET /api/v2/filters/:id to read a particular filter
PUT /api/v2/filters/:id to update a filter
keywords_attributes can also be passed to edit, delete or add keywords in one request
DELETE /api/v2/filters/:id to delete a particular filter
GET /api/v2/filters/:id/keywords to list keywords for a filter
POST /api/v2/filters/:filter_id/keywords/:id to add a new keyword to a filter
GET /api/v1/filters/keywords/:id to read a particular keyword
PUT /api/v1/filters/keywords/:id to edit a particular keyword
DELETE /api/v1/filters/keywords/:id to delete a particular keyword

Implementation guidelines

Determining matched filters

Since the server provides a filtered attribute on Status entities, that can be used to check which filters apply without implementations needing to implement the rule matching themselves.

However, client implementations may still want to perform rule matching client-side, as this would allow retroactively apply filter changes without re-fetching posts from the server. When doing so, they should take care of not ignoring filtered entries for which there are other attributes than keyword_matches, so as to handle future extensions of the filtering system.

Applying actions on matched rules

Matched filters need to be filtered based on context (home, notifications, public, thread or profile) and expiration date.

When at least one active matched filter has hide for filter_action, the post should not be shown at all. Otherwise, if at least one active matched filter has warn for filter_action, the post should be hidden with a warning, and the user should be able to reveal the post after being informed of which filters matched (identified by title rather than the exact matched keywords).

For extension purposes, unknown values for filter_action should be treated as warn.

TODO

This project was funded through the NGI0 Discovery Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG Communications Networks, Content and Technology under grant agreement No 825322.

ClearlyClaire · 2022-04-21T16:47:50Z

It's not possible yet to add multiple keywords to a category (missing API, and the /settings page need more work), but here is how it gets displayed:

ariasuni · 2022-04-21T17:04:05Z

I like it a lot, but it looks a bit too large in my opinion. I have currently 33 filters, which could probably be changed into 10 groups, and I don’t want to scroll too much.

I feel like keywords should be between title and «metadata» (places where filter is applied, expiration time). Also, the longest metadata line in English will overflow; maybe metadata should be placed on its own line below title/keywords/action buttons so that metadata line never overflows on desktop (and separated into two different lines on mobile?).

ClearlyClaire · 2022-04-21T17:31:22Z

I like it a lot, but it looks a bit too large in my opinion. I have currently 33 filters, which could probably be changed into 10 groups, and I don’t want to scroll too much.

I honestly have the same concern :/

I feel like keywords should be between title and «metadata» (places where filter is applied, expiration time).

I am not sure about that, I want to eventually have other kinds of filters, e.g. filtering specific conversations (see the following mockup):

Also, the longest metadata line in English will overflow; maybe metadata should be placed on its own line below title/keywords/action buttons so that metadata line never overflows on desktop (and separated into two different lines on mobile?).

Also a good point.

ClearlyClaire · 2022-04-21T19:23:16Z

@ariasuni reworked a bit that design by moving things around and reducing some margins, I think it addresses most of your concerns

ariasuni · 2022-04-22T11:09:09Z

It much better in my opinion! Keyword list and conversation count could be side-to-side, though.

jayrope · 2022-08-11T14:05:13Z

Bug: When creating a new filter i get an error message upon trying to save this including a set keyword. The error does not appear, when not setting a keyword.
Unclear: Is the filter active, even when no keyword is set? It seems so, but that wouldn't make any sense, would it? Generally the keyword is meant to trigger the filter, right?
GUI: I think the new design wastes space and find the former one line display much more usable to get a quick overview on many filters.

However, thank you much for all you do.

ClearlyClaire · 2022-08-11T15:43:22Z

Bug: When creating a new filter i get an error message upon trying to save this including a set keyword. The error does not appear, when not setting a keyword.

What error message do you get? This is likely #18742

Unclear: Is the filter active, even when no keyword is set? It seems so, but that wouldn't make any sense, would it? Generally the keyword is meant to trigger the filter, right?

When no keyword is set, the filter does nothing. There are other filter types planned, but maybe the UI should be tweaked to make it clearer that the filter won't do anything without a keyword.

GUI: I think the new design wastes space and find the former one line display much more usable to get a quick overview on many filters.

I'm split on this one. I can understand how the one-line display may be more usable if you have many different filters, but I think this one is more usable if your keywords are logically grouped, which I think fits the general case more.

jayrope · 2022-08-11T16:12:11Z

Thanks for getting into this, ClearlyClaire.

What error message do you get? This is likely #18742

It is similar, but i get the server error already upon creating a new filter when adding a keyword at all (v. 3.5.3 on mastodon.online that is).

When no keyword is set, the filter does nothing. There are other filter types planned, but maybe the UI should be tweaked to make it clearer that the filter won't do anything without a keyword.

I suppose i cannot currently create any new and working filters then, as setting a keyword produces that error. Uhm.

GUI: ...

I'm split on this one. I can understand how the one-line display may be more usable if you have many different filters, but I think this one is more usable if your keywords are logically grouped, which I think fits the general case more.

Why not give the user a choice of display? That way you'll satisfy both power filterers as well as people, who use only a few. I am filtering a lot. It is one of my most used tools to regulate the flood into a lean experience. The current display turns out to be a major scroll monster for me :).

Thank you again for anything you do.

ClearlyClaire · 2022-08-11T16:57:06Z

Thanks for getting into this, ClearlyClaire.

What error message do you get? This is likely #18742

It is similar, but i get the server error already upon creating a new filter when adding a keyword at all (v. 3.5.3 on mastodon.online that is).

Yeah, it's likely the same thing then. It can occur whenever adding a keyword, whether there are keywords in the filter or not.

When no keyword is set, the filter does nothing. There are other filter types planned, but maybe the UI should be tweaked to make it clearer that the filter won't do anything without a keyword.

I suppose i cannot currently create any new and working filters then, as setting a keyword produces that error. Uhm.

Yes, unfortunately, this issue may prevent you from creating any new filter… I think you can retry over and over again and eventually it should work… but you may need to retry a lot of times :/

GUI: ...

I'm split on this one. I can understand how the one-line display may be more usable if you have many different filters, but I think this one is more usable if your keywords are logically grouped, which I think fits the general case more.

Why not give the user a choice of display? That way you'll satisfy both power filterers as well as people, who use only a few. I am filtering a lot. It is one of my most used tools to regulate the flood into a lean experience. The current display turns out to be a major scroll monster for me :).

Because it's not just the display that changed, but the whole model, and the old display would not quite work with the new model…

linux-lukas · 2022-10-31T22:23:39Z

When Revamp the filter system, has it been considered that filter rules created in versions below v4.0 can be migrated to the new scheme from v4.0 onwards without loss as users? This means that the user does not have to create the filters again.

ariasuni · 2022-11-01T14:10:08Z

I have the new filter system and my old filters are still here.

linux-lukas · 2022-11-01T14:13:50Z

I have the new filter system and my old filters are still here.

Okay, that sounds good, are they then "uncategorised" or how are the old filters ordered?

ClearlyClaire · 2022-11-02T08:53:48Z

Okay, that sounds good, are they then "uncategorised" or how are the old filters ordered?

They are all added as their own category.

stom79 · 2022-11-18T07:59:29Z

Hi @ClearlyClaire
I try to implement that feature with the API. Could you tell more about payload for POST/PUT endpoints?
For instance with /api/v2/filters

I tried this:

    @FormUrlEncoded
    @POST("filters")
    Call<Filter> addFilter(
            @Header("Authorization") String token,
            @Field("title") String title,
            @Field("expires_at") Date expires_at,
            @Field("filter_action") String filter_action,
            @Field("context[]") List<String> context,
            @Field("keywords_attributes[]") List<Filter.KeywordsAttributes> keywordsAttributes
    );

    public static class KeywordsAttributes implements Serializable {
        @SerializedName("id")
        public String id;
        @SerializedName("keyword")
        public String keyword;
        @SerializedName("whole_word")
        public boolean whole_word;
        @SerializedName("_destroy")
        public boolean _destroy;
    }

But Keywords are not added

If I use the PUT call, only context is changed (title, filter_action, keywords) are not updated.

ClearlyClaire · 2022-11-18T09:26:23Z

That is weird, those should work… or at least I can't see anything immediately wrong with it. POST should not include ids for keyword attributes though.

Are you sure you are hitting /api/v2/filters and not /api/v1/filters still?

trwnh · 2022-11-18T09:44:02Z

@stom79 unsure if related, but expires_at should be expires_in

i can't replicate any issues with a raw curl POST http://mastodon.local/api/v2/filters?title=test&context[]=home&keywords_attributes[][keyword]=word though...

stom79 · 2022-11-18T11:30:40Z

yes I use /api/v2 with /filters, I can make it work by using

  @Field("keywords_attributes[][id]") List<String> keywordId,
  @Field("keywords_attributes[][keyword]") List<String> keywords,
  @Field("keywords_attributes[][whole_word]") List<Boolean> wholeWords

instead of

 @Field("keywords_attributes[]") List<Filter.KeywordsAttributes> keywordsAttributes

But whole_word is always set to true. I will check why.

PUT to /api/v2/filters should also update keywords if the id is passed? Currently that doesn't add new ones but it doesn't change their value.

---------- All is working fine. Thank you.

* Add model for custom filter keywords * Use CustomFilterKeyword internally Does not change the API * Fix /filters/edit and /filters/new * Add migration tests * Remove whole_word column from custom_filters (covered by custom_filter_keywords) * Redesign /filters Instead of a list, present a card that displays more information and handles multiple keywords per filter. * Redesign /filters/new and /filters/edit to add and remove keywords This adds a new gem dependency: cocoon, as well as a npm dependency: cocoon-js-vanilla. Those are used to easily populate and remove form fields from the user interface when manipulating multiple keyword filters at once. * Add /api/v2/filters to edit filter with multiple keywords Entities: - `Filter`: `id`, `title`, `filter_action` (either `hide` or `warn`), `context` `keywords` - `FilterKeyword`: `id`, `keyword`, `whole_word` API endpoits: - `GET /api/v2/filters` to list filters (including keywords) - `POST /api/v2/filters` to create a new filter `keywords_attributes` can also be passed to create keywords in one request - `GET /api/v2/filters/:id` to read a particular filter - `PUT /api/v2/filters/:id` to update a new filter `keywords_attributes` can also be passed to edit, delete or add keywords in one request - `DELETE /api/v2/filters/:id` to delete a particular filter - `GET /api/v2/filters/:id/keywords` to list keywords for a filter - `POST /api/v2/filters/:filter_id/keywords/:id` to add a new keyword to a filter - `GET /api/v2/filter_keywords/:id` to read a particular keyword - `PUT /api/v2/filter_keywords/:id` to edit a particular keyword - `DELETE /api/v2/filter_keywords/:id` to delete a particular keyword * Change from `irreversible` boolean to `action` enum * Remove irrelevent `irreversible_must_be_within_context` check * Fix /filters/new and /filters/edit with update for filter_action * Fix Rubocop/Codeclimate complaining about task names * Refactor FeedManager#phrase_filtered? This moves regexp building and filter caching to the `CustomFilter` class. This does not change the functional behavior yet, but this changes how the cache is built, doing per-custom_filter regexps so that filters can be matched independently, while still offering caching. * Perform server-side filtering and output result in REST API * Fix numerous filters_changed events being sent when editing multiple keywords at once * Add some tests * Use the new API in the WebUI - use client-side logic for filters we have fetched rules for. This is so that filter changes can be retroactively applied without reloading the UI. - use server-side logic for filters we haven't fetched rules for yet (e.g. network error, or initial timeline loading) * Minor optimizations and refactoring * Perform server-side filtering on the streaming server * Change the wording of filter action labels * Fix issues pointed out by linter * Change design of “Show anyway” link in accordence to review comments * Drop “irreversible” filtering behavior * Move /api/v2/filter_keywords to /api/v1/filters/keywords * Rename `filter_results` attribute to `filtered` * Rename REST::LegacyFilterSerializer to REST::V1::FilterSerializer * Fix systemChannelId value in streaming server * Simplify code by removing client-side filtering code The simplifcation comes at a cost though: filters aren't retroactively applied anymore.

Tak · 2023-01-16T18:27:45Z

@stom79 I'm seeing a similar issue here, also using retrofit, can you clarify what ended up working for you?
If I PUT to /api/v2/filters/{id} with @Field("keywords_attributes[][id]") List<String> keywordId etc., it creates new keywords with new ids (yes, I'm passing the old ids) (and gives 422 if I pass whole_word at all)
If I use a list of objects or maps instead, the call succeeds but the keywords are ignored.

stom79 · 2023-01-16T18:42:59Z

@Tak Our endpoints are here for filters

https://codeberg.org/tom79/Fedilab/src/branch/main/app/src/main/java/app/fedilab/android/client/endpoints/MastodonFiltersService.java

And our activity that manages filters

https://codeberg.org/tom79/Fedilab/src/branch/main/app/src/main/java/app/fedilab/android/activities/FilterActivity.java

We created objects for retrofit

https://codeberg.org/tom79/Fedilab/src/branch/main/app/src/main/java/app/fedilab/android/client/entities/api/Filter.java#L74

And
https://codeberg.org/tom79/Fedilab/src/branch/main/app/src/main/java/app/fedilab/android/client/entities/api/Filter.java#L89

We use them directly with retrofit for POST/PUT calls
https://codeberg.org/tom79/Fedilab/src/branch/main/app/src/main/java/app/fedilab/android/client/endpoints/MastodonFiltersService.java#L81

Currently, I am not with my computer, but I will check that more carefully tomorrow.

@Tak I just checked. The only way we made it work was to use @Body Filter.FilterParams filter where Filter.FilterParams is our params sent to the api. It represents the filter and its attached keywords (two classes).

https://codeberg.org/tom79/Fedilab/src/branch/main/app/src/main/java/app/fedilab/android/client/entities/api/Filter.java#L74

POST/PUT calls: https://codeberg.org/tom79/Fedilab/src/branch/main/app/src/main/java/app/fedilab/android/client/endpoints/MastodonFiltersService.java#L48-L63

In the activity we loop through existing keywords to attach them to the filter params (when editing):

https://codeberg.org/tom79/Fedilab/src/branch/main/app/src/main/java/app/fedilab/android/activities/FilterActivity.java#L115-L121

And new ones are added on the flow:

https://codeberg.org/tom79/Fedilab/src/branch/main/app/src/main/java/app/fedilab/android/activities/FilterActivity.java#L164-L169

* Add model for custom filter keywords * Use CustomFilterKeyword internally Does not change the API * Fix /filters/edit and /filters/new * Add migration tests * Remove whole_word column from custom_filters (covered by custom_filter_keywords) * Redesign /filters Instead of a list, present a card that displays more information and handles multiple keywords per filter. * Redesign /filters/new and /filters/edit to add and remove keywords This adds a new gem dependency: cocoon, as well as a npm dependency: cocoon-js-vanilla. Those are used to easily populate and remove form fields from the user interface when manipulating multiple keyword filters at once. * Add /api/v2/filters to edit filter with multiple keywords Entities: - `Filter`: `id`, `title`, `filter_action` (either `hide` or `warn`), `context` `keywords` - `FilterKeyword`: `id`, `keyword`, `whole_word` API endpoits: - `GET /api/v2/filters` to list filters (including keywords) - `POST /api/v2/filters` to create a new filter `keywords_attributes` can also be passed to create keywords in one request - `GET /api/v2/filters/:id` to read a particular filter - `PUT /api/v2/filters/:id` to update a new filter `keywords_attributes` can also be passed to edit, delete or add keywords in one request - `DELETE /api/v2/filters/:id` to delete a particular filter - `GET /api/v2/filters/:id/keywords` to list keywords for a filter - `POST /api/v2/filters/:filter_id/keywords/:id` to add a new keyword to a filter - `GET /api/v2/filter_keywords/:id` to read a particular keyword - `PUT /api/v2/filter_keywords/:id` to edit a particular keyword - `DELETE /api/v2/filter_keywords/:id` to delete a particular keyword * Change from `irreversible` boolean to `action` enum * Remove irrelevent `irreversible_must_be_within_context` check * Fix /filters/new and /filters/edit with update for filter_action * Fix Rubocop/Codeclimate complaining about task names * Refactor FeedManager#phrase_filtered? This moves regexp building and filter caching to the `CustomFilter` class. This does not change the functional behavior yet, but this changes how the cache is built, doing per-custom_filter regexps so that filters can be matched independently, while still offering caching. * Perform server-side filtering and output result in REST API * Fix numerous filters_changed events being sent when editing multiple keywords at once * Add some tests * Use the new API in the WebUI - use client-side logic for filters we have fetched rules for. This is so that filter changes can be retroactively applied without reloading the UI. - use server-side logic for filters we haven't fetched rules for yet (e.g. network error, or initial timeline loading) * Minor optimizations and refactoring * Perform server-side filtering on the streaming server * Change the wording of filter action labels * Fix issues pointed out by linter * Change design of “Show anyway” link in accordence to review comments * Drop “irreversible” filtering behavior * Move /api/v2/filter_keywords to /api/v1/filters/keywords * Rename `filter_results` attribute to `filtered` * Rename REST::LegacyFilterSerializer to REST::V1::FilterSerializer * Fix systemChannelId value in streaming server * Simplify code by removing client-side filtering code The simplifcation comes at a cost though: filters aren't retroactively applied anymore.

ClearlyClaire force-pushed the features/revamped-filters branch 7 times, most recently from 1fc65f9 to 4f71f67 Compare April 21, 2022 16:34

ClearlyClaire force-pushed the features/revamped-filters branch 8 times, most recently from cbc16f0 to 0794b17 Compare April 22, 2022 10:52

ClearlyClaire force-pushed the features/revamped-filters branch 7 times, most recently from 245fcf2 to fc648fb Compare April 27, 2022 20:48

ClearlyClaire added the NLnet Funded by NLnet, see https://nlnet.nl/project/Mastodon/ label May 2, 2022

ClearlyClaire force-pushed the features/revamped-filters branch from fc648fb to 84bed86 Compare May 12, 2022 09:14

ClearlyClaire force-pushed the features/revamped-filters branch from 84bed86 to fb0d34e Compare June 1, 2022 06:37

ClearlyClaire mentioned this pull request Jul 31, 2022

Change mutes to no longer hide statuses merely mentioning muted account #17631

Closed

Cassolotl mentioned this pull request Sep 17, 2022

Searches containing filtered terms should say so. "You have this search term filtered. Click here to edit your filter list." #13247

Closed

ClearlyClaire mentioned this pull request Oct 30, 2022

We need a proper blacklist/whitelist feature for CWs and posts #19532

Open

ClearlyClaire mentioned this pull request Nov 7, 2022

Feature request: keyword triggered auto-CWs #7947

Closed

omentic mentioned this pull request Dec 19, 2022

Display filtering reason without clicking through glitch-soc/mastodon#2020

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revamp post filtering system #18058

Revamp post filtering system #18058

ClearlyClaire commented Apr 20, 2022 •

edited

Loading

ClearlyClaire commented Apr 21, 2022

ariasuni commented Apr 21, 2022

ClearlyClaire commented Apr 21, 2022

ClearlyClaire commented Apr 21, 2022

ariasuni commented Apr 22, 2022

jayrope commented Aug 11, 2022

ClearlyClaire commented Aug 11, 2022

jayrope commented Aug 11, 2022

ClearlyClaire commented Aug 11, 2022

linux-lukas commented Oct 31, 2022

ariasuni commented Nov 1, 2022

linux-lukas commented Nov 1, 2022

ClearlyClaire commented Nov 2, 2022

stom79 commented Nov 18, 2022

ClearlyClaire commented Nov 18, 2022

trwnh commented Nov 18, 2022

stom79 commented Nov 18, 2022 •

edited

Loading

Tak commented Jan 16, 2023

stom79 commented Jan 16, 2023 •

edited

Loading

Revamp post filtering system #18058

Revamp post filtering system #18058

Conversation

ClearlyClaire commented Apr 20, 2022 • edited Loading

New filter management UI

New filtered post UI

API changes and implementation guidelines

Backward compatibility

New API

New entities

Changes to other entities

New API endpoints

Implementation guidelines

Determining matched filters

Applying actions on matched rules

TODO

ClearlyClaire commented Apr 21, 2022

ariasuni commented Apr 21, 2022

ClearlyClaire commented Apr 21, 2022

ClearlyClaire commented Apr 21, 2022

ariasuni commented Apr 22, 2022

jayrope commented Aug 11, 2022

ClearlyClaire commented Aug 11, 2022

jayrope commented Aug 11, 2022

ClearlyClaire commented Aug 11, 2022

linux-lukas commented Oct 31, 2022

ariasuni commented Nov 1, 2022

linux-lukas commented Nov 1, 2022

ClearlyClaire commented Nov 2, 2022

stom79 commented Nov 18, 2022

ClearlyClaire commented Nov 18, 2022

trwnh commented Nov 18, 2022

stom79 commented Nov 18, 2022 • edited Loading

Tak commented Jan 16, 2023

stom79 commented Jan 16, 2023 • edited Loading

ClearlyClaire commented Apr 20, 2022 •

edited

Loading

stom79 commented Nov 18, 2022 •

edited

Loading

stom79 commented Jan 16, 2023 •

edited

Loading