Support Privilege Grants to Roles and Groups in Materialization #626

soksamnanglim · 2023-10-04T05:58:54Z

resolves #415
docs: dbt-labs/docs.getdbt.com#4193

Problem

Prior to this PR, users were unable to configure grants to roles and groups via grant config on resource.
A valid workaround existed. Users could prefix their permissions with "group" or "role", which introduced concerns:

user error.
get_show_grant did not query for group or role grants on the table. Expected behavior: grants configured via the workaround would be automatically revoked. Actual behavior: workaround grants remain.

Solution

Users may now extend grant configurations to roles and groups like this:

models:
    - name: specific_model
      config:
        grants:
            select: 
                user: ['chidi', 'tahani', 'eleanor']
                role: ['human', 'developer']
                group: ['good']
            insert:
                user: ['michael']
                role: ['architect']
                group: ['bad']

To not break existing dbt projects, dbt users granting privileges to only users may continue configuring grants the following way:

models:
    - name: specific_model
      config:
        grants:
            select: ['user_1']

Upon running, dbt will continue to automatically grant and revoke differences in privileges to users, groups, and roles.

Checklist

I have read the contributing guide and understand what's expected of me
I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

soksamnanglim · 2023-10-04T05:59:56Z

Ran all of the integration tests, however the test suite for materialized views fails for me even with no proposed changes. Would love some help on that! Kept getting a TypeError: NoneType is not subscriptable

Also had to refactor many of the integration tests from dbt-core as many methods/macros were overriden to support the new config, would love your input in that area!

tests/functional/adapter/test_grants.py

.changes/unreleased/Features-20231004-103938.yaml

colin-rogers-dbt · 2023-10-04T19:58:13Z

Need to update CI workflow to include new env vars: https://github.com/dbt-labs/dbt-redshift/blob/main/.github/workflows/integration.yml#L185

soksamnanglim · 2023-10-04T20:18:49Z

Hmmm... integration tests for grants are still failing after I updated the CI workflow.

colin-rogers-dbt · 2023-10-04T23:14:09Z

Hmmm... integration tests for grants are still failing after I updated the CI workflow.

Failing test:

____________ TestIncrementalGrantsRedshift.test_incremental_grants _____________
[gw2] linux -- Python 3.8.18 /home/runner/work/dbt-redshift/dbt-redshift/.tox/integration-redshift/bin/python

self = <test_grants.TestIncrementalGrantsRedshift object at 0x7f5ec33e4d90>
project = <dbt.tests.fixtures.project.TestProjInfo object at 0x7f5eb8a58e80>
get_test_users = ['dbt_test_user_1', 'dbt_test_user_2', 'dbt_test_user_3']
get_test_groups = [], get_test_roles = []

    def test_incremental_grants(self, project, get_test_users, get_test_groups, get_test_roles):
        # for debugging
        print("incremental test")
    
        # we want the test to fail, not silently skip
        test_users = get_test_users
        test_groups = get_test_groups
        test_roles = get_test_roles
        select_privilege_name = self.privilege_grantee_name_overrides()["select"]
        assert len(test_users) == 3
>       assert len(test_groups) == 3
E       assert 0 == 3
E        +  where 0 = len([])

soksamnanglim · 2023-10-05T16:48:41Z

Failing test:

It's the same error that came up also before updating CI workflow. Perhaps, dbt_test_group_i and dbt_test_role_i need to be created in dbt's test cluster?

tests/functional/adapter/conftest.py

colin-rogers-dbt · 2023-10-09T21:26:49Z

tests/functional/adapter/grants/test_model_grants.py

+            "schema.yml": updated_schema,
+        }
+
+    def test_view_table_grants(self, project, get_test_users, get_test_groups, get_test_roles):


Running locally I see issues with the expected/actual grants (it looks like the actual grants includes a superset of all users). Kind of hard to debug since there are so many test cases in this test. Can we break this up into separate test cases?

Hi Colin, let me take a look.

The tests that are failing successfully pass in my local environment. Is there a way to set up my local environment to mirror dbt's local + remote testing environment? I follow the contributing guide, but some test results conflict between local and remote environment.

Remotely, it seems the fixtures are not manifesting even with the fix. https://github.com/dbt-labs/dbt-redshift/actions/runs/6461848915/job/17542365369?pr=626#step:7:275

get_test_users = ['dbt_test_user_1', 'dbt_test_user_2', 'dbt_test_user_3'] get_test_groups = [], get_test_roles = []

Let me try refactoring base_grants to see if that fixes the issue.

@soksamnanglim took another look at the tests and it looks like they're being invoked twice in some places. Simplified by making the "base" classes standard test classes since they shouldn't need to be inherited

also confirmed that the roles/groups are setup in our test DB via:
select * from SVV_ROLES; and select * from pg_group;

After running revoke SELECT on "database"."schema"."my_model" from dbt_test_user_1, group dbt_test_group_1, role dbt_test_role_1;, test_model_grants successfully passes for me.

It seems that the privileges were not revoked properly the first time, but seems like they are automatically revoking fine now.

I'd like to figure out whether the code for automatic revoking/granting has a bug in it or my testing set up for models is incorrect—Can we try manually revoking those privileges from the dbt testing cluster and running the grant test suite multiple times to ensure its behaving deterministically in the dbt environment?

soksamnanglim · 2023-12-05T00:34:07Z

Any updates here?

tests/functional/adapter/conftest.py

Co-authored-by: Colin Rogers <111200756+colin-rogers-dbt@users.noreply.github.com>

dbeatty10 · 2024-07-10T21:35:43Z

Looking at the results from the most recent CI tests, there are 3 tests that are not passing:

FAILED tests/functional/adapter/grants/test_incremental_grants.py::TestIncrementalGrantsRedshift::test_incremental_grants - AssertionError
FAILED tests/functional/adapter/grants/test_model_table_grants.py::TestModelGrantsTableRedshift::test_table_grants - AssertionError
FAILED tests/functional/adapter/grants/test_snapshot_grants.py::TestSnapshotGrantsRedshift::test_snapshot_grants - AssertionError

The failed assertion for each looks identical to me:

TestIncrementalGrantsRedshift

tests/functional/adapter/grants/test_incremental_grants.py::TestIncrementalGrantsRedshift::test_incremental_grants:

actual_grants = {'select': {'group': ['dbt_test_group_1'], 'user': ['dbt_test_user_1']}}
expected_grants = {'select': {'group': ['dbt_test_group_1'], 👉 'role': ['dbt_test_role_1'] 👈, 'user': ['dbt_test_user_1']}}

TestModelGrantsTableRedshift

tests/functional/adapter/grants/test_model_table_grants.py::TestModelGrantsTableRedshift::test_table_grants:

actual_grants = {'select': {'group': ['dbt_test_group_1'], 'user': ['dbt_test_user_1']}}
expected_grants = {'select': {'group': ['dbt_test_group_1'], 👉 'role': ['dbt_test_role_1'] 👈 , 'user': ['dbt_test_user_1']}}

TestSnapshotGrantsRedshift

tests/functional/adapter/grants/test_snapshot_grants.py::TestSnapshotGrantsRedshift::test_snapshot_grants:

actual_grants = {'select': {'group': ['dbt_test_group_1'], 'user': ['dbt_test_user_1']}}
expected_grants = {'select': {'group': ['dbt_test_group_1'], 👉 'role': ['dbt_test_role_1'] 👈, 'user': ['dbt_test_user_1']}}

dbeatty10 · 2024-07-10T22:36:57Z

I checked that the CI environment has values for all the expected environment variables:

22:15:11  DBT_TEST_USER_1: dbt_test_user_1
22:15:11  DBT_TEST_USER_2: dbt_test_user_2
22:15:11  DBT_TEST_USER_3: dbt_test_user_3
22:15:11  DBT_TEST_GROUP_1: dbt_test_group_1
22:15:11  DBT_TEST_GROUP_2: dbt_test_group_2
22:15:11  DBT_TEST_GROUP_3: dbt_test_group_3
22:15:11  DBT_TEST_ROLE_1: dbt_test_role_1
22:15:11  DBT_TEST_ROLE_2: dbt_test_role_2
22:15:11  DBT_TEST_ROLE_3: dbt_test_role_3

added roles and groups to env-setup.sh

I'm not seeing any changes to env-setup.sh, so I'm guessing they weren't pushed on accident.

Before trying to make changes to env-setup.sh, I'm going to take a look at the roles listed within select * from svv_roles r as seen by the CI environment.

dbeatty10 · 2024-07-10T23:10:13Z

It looks to me like the logic within tests/functional/adapter/conftest.py that is meant to create groups and roles if they don't exist is not working, because the following query has 0 rows in the CI environment:

select * from svv_roles r

So to unblock the CI tests, I manually executed in the following within Redshift:

CREATE ROLE dbt_test_role_1;
CREATE ROLE dbt_test_role_2;
CREATE ROLE dbt_test_role_3;

jtcohen6 · 2024-07-11T14:33:45Z

dbt/include/redshift/macros/adapters/apply_grants.sql

+    'role' as grantee_type,
+    r.role_name as grantee,
+    p.privilege_type
+from svv_roles r


From pairing with @dbeatty10 - in order to query svv_table_info, this requires:

that dbt's user has role <somerole>

grant access system table to role <somerole>;

References:

https://docs.aws.amazon.com/redshift/latest/dg/r_SVV_ROLES.html

https://www.redshiftresearchproject.org/white_papers/downloads/users_groups_roles_and_privileges.html

dbeatty10 · 2024-07-16T14:38:42Z

dbt/include/redshift/macros/adapters/apply_grants.sql

+
+{% endmacro %}
+
+{% macro redshift__apply_grants(relation, grant_config, should_revoke=True) %}


Overriding default__apply_grants.

dbeatty10 · 2024-07-16T14:40:18Z

dbt/adapters/redshift/impl.py

+        return grants_dict
+
+    @available
+    def diff_of_two_nested_dicts(self, dict_a, dict_b):


Instead of diff_of_two_dicts.

dbeatty10 · 2024-07-16T14:41:22Z

dbt/adapters/redshift/impl.py

+
+    # grant-related methods
+    @available
+    def standardize_grants_dict(self, grants_table):


Overriding standardize_grants_dict.

dbeatty10 · 2024-07-16T14:44:35Z

dbt/include/redshift/macros/adapters/apply_grants.sql

-{% macro redshift__get_show_grant_sql(relation) %}
+{# ------- DCL STATEMENT TEMPLATES ------- #}
+
+{%- macro redshift__get_grant_sql(relation, privilege, grantee_dict) -%}


Overriding default__get_grant_sql.

dbeatty10 · 2024-07-16T14:45:15Z

dbt/include/redshift/macros/adapters/apply_grants.sql

+    {%- endfor -%}
+{%- endmacro -%}
+
+{%- macro redshift__get_revoke_sql(relation, privilege, revokee_dict) -%}


Overriding default__get_revoke_sql.

dbeatty10 · 2024-07-16T14:56:47Z

dbt/include/redshift/macros/adapters/apply_grants.sql

+    and tp.table_schema=replace(split_part('{{ relation }}', '.', 2), '"', '')
+    and tp.table_name=replace(split_part('{{ relation }}', '.', 3), '"', '')


We should use Relation class attributes (like here and here) rather than splitting on dot separators.

Suggested change

and tp.table_schema=replace(split_part('{{ relation }}', '.', 2), '"', '')

and tp.table_name=replace(split_part('{{ relation }}', '.', 3), '"', '')

and tp.table_schema = '{{ relation.schema }}'

and tp.table_name = '{{ relation.identifier }}'

dbeatty10 · 2024-07-16T15:00:07Z

dbt/include/redshift/macros/adapters/apply_grants.sql

+    and rp.namespace_name=replace(split_part('{{ relation }}', '.', 2), '"', '')
+    and rp.relation_name=replace(split_part('{{ relation }}', '.', 3), '"', '')


We should use Relation class attributes (like here and here) rather than splitting on dot separators.

Suggested change

and rp.namespace_name=replace(split_part('{{ relation }}', '.', 2), '"', '')

and rp.relation_name=replace(split_part('{{ relation }}', '.', 3), '"', '')

and rp.namespace_name = '{{ relation.schema }}'

and rp.relation_name = '{{ relation.identifier }}'

pgvillena · 2024-08-20T21:55:51Z

Any updates here?

Support Privilege Grants to Roles and Groups in Materialization

a1ec09c

cla-bot bot added the cla:yes label Oct 4, 2023

soksamnanglim marked this pull request as ready for review October 4, 2023 16:40

soksamnanglim requested a review from a team as a code owner October 4, 2023 16:40

soksamnanglim requested a review from mikealfare October 4, 2023 16:40

Support Privilege Grants to Roles and Groups in Materialization

a0ca37d

colin-rogers-dbt added the ok to test label Oct 4, 2023

colin-rogers-dbt reviewed Oct 4, 2023

View reviewed changes

tests/functional/adapter/test_grants.py Outdated Show resolved Hide resolved

colin-rogers-dbt closed this Oct 4, 2023

colin-rogers-dbt reopened this Oct 4, 2023

colin-rogers-dbt reviewed Oct 4, 2023

View reviewed changes

.changes/unreleased/Features-20231004-103938.yaml Outdated Show resolved Hide resolved

soksamnanglim added 2 commits October 4, 2023 13:00

rectified code with pre-commit

ca046c9

Update CI workflow to include new env vars

2bb40b8

dbeatty10 mentioned this pull request Oct 5, 2023

Accept users, groups, or roles in grants config for dbt-redshift dbt-labs/docs.getdbt.com#4193

Open

Add fixture to create grants/roles for test purposes

f464d44

colin-rogers-dbt reviewed Oct 9, 2023

View reviewed changes

tests/functional/adapter/conftest.py Show resolved Hide resolved

colin-rogers-dbt reviewed Oct 9, 2023

View reviewed changes

colin-rogers-dbt and others added 8 commits October 11, 2023 15:44

Merge branch 'main' into extend_grant_privilege

7b89a16

Fixed ImportError related to misnaming BaseModelGrantsRedshift

72a2410

Remove debugging comment

0f6442e

Change away from inheritance of BaseGrants

bfc08f6

Merge branch 'main' into extend_grant_privilege

53cc135

clean up duplicate tests

b546431

Merge branch 'dbt-labs:main' into extend_grant_privilege

87c491d

Merge branch 'main' into extend_grant_privilege

574193c

Merge branch 'main' into extend_grant_privilege

66782c4

mikealfare mentioned this pull request Jan 3, 2024

[ADAP-1086] [Bug] Grants are failing during incremental runs when data shares are present on serverless #699

Closed

2 tasks

dataders added the user docs [docs.getdbt.com] Needs better documentation label May 2, 2024

dataders mentioned this pull request May 2, 2024

add example to redshift grants section dbt-labs/docs.getdbt.com#5156

Closed

colin-rogers-dbt and others added 2 commits May 6, 2024 12:43

Merge branch 'main' into extend_grant_privilege

96fc734

Merge branch 'main' into extend_grant_privilege

4a28c99

dbeatty10 reviewed Jul 9, 2024

View reviewed changes

tests/functional/adapter/conftest.py Outdated Show resolved Hide resolved

Update package due to decoupling of dbt-adapters from dbt-core

1f2f9e0

mirnawong1 mentioned this pull request Jul 10, 2024

Update grants.md for Redshift roles and groups dbt-labs/docs.getdbt.com#5739

Closed

3 tasks

Update body of changelog entry

53956b4

Co-authored-by: Colin Rogers <111200756+colin-rogers-dbt@users.noreply.github.com>

dbeatty10 added 2 commits July 10, 2024 15:43

Update whitespace

5901cf2

Try printing the environment variables configured in the CI environment

f37d91d

dbeatty10 added 2 commits July 10, 2024 16:38

Examine roles in the Redshift database

220c810

Remove debugging logic

c56c61d

dbeatty10 added 4 commits July 10, 2024 17:39

Debug output for setting up groups and roles for CI

00231be

Force creation of groups and roles

30043a7

Fix typo

12d339d

Debugging database roles

3aeff0d

jtcohen6 reviewed Jul 11, 2024

View reviewed changes

dbeatty10 added 4 commits July 11, 2024 09:02

Re-run CI tests

c8e06f5

Remove debugging

eb07cfb

Add groups and roles for various release workflows in GitHub Actions

655cb53

Database groups and roles for testing locally

5247bab

dbeatty10 reviewed Jul 16, 2024

View reviewed changes

colin-rogers-dbt added the community This PR is from a community member label Jul 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Privilege Grants to Roles and Groups in Materialization #626

Support Privilege Grants to Roles and Groups in Materialization #626

soksamnanglim commented Oct 4, 2023 •

edited

Loading

soksamnanglim commented Oct 4, 2023

colin-rogers-dbt commented Oct 4, 2023

soksamnanglim commented Oct 4, 2023

colin-rogers-dbt commented Oct 4, 2023

soksamnanglim commented Oct 5, 2023

colin-rogers-dbt Oct 9, 2023

soksamnanglim Oct 11, 2023 •

edited

Loading

soksamnanglim Oct 11, 2023 •

edited

Loading

colin-rogers-dbt Oct 13, 2023

colin-rogers-dbt Oct 13, 2023

soksamnanglim Oct 24, 2023 •

edited

Loading

soksamnanglim commented Dec 5, 2023

dbeatty10 commented Jul 10, 2024

dbeatty10 commented Jul 10, 2024

dbeatty10 commented Jul 10, 2024

jtcohen6 Jul 11, 2024

dbeatty10 Jul 16, 2024

dbeatty10 Jul 16, 2024

dbeatty10 Jul 16, 2024

dbeatty10 Jul 16, 2024

dbeatty10 Jul 16, 2024

dbeatty10 Jul 16, 2024

dbeatty10 Jul 16, 2024

pgvillena commented Aug 20, 2024


		{% endmacro %}

		{% macro redshift__apply_grants(relation, grant_config, should_revoke=True) %}

		and tp.table_schema=replace(split_part('{{ relation }}', '.', 2), '"', '')
		and tp.table_name=replace(split_part('{{ relation }}', '.', 3), '"', '')

		and rp.namespace_name=replace(split_part('{{ relation }}', '.', 2), '"', '')
		and rp.relation_name=replace(split_part('{{ relation }}', '.', 3), '"', '')

Support Privilege Grants to Roles and Groups in Materialization #626

Are you sure you want to change the base?

Support Privilege Grants to Roles and Groups in Materialization #626

Conversation

soksamnanglim commented Oct 4, 2023 • edited Loading

Problem

Solution

Checklist

soksamnanglim commented Oct 4, 2023

colin-rogers-dbt commented Oct 4, 2023

soksamnanglim commented Oct 4, 2023

colin-rogers-dbt commented Oct 4, 2023

soksamnanglim commented Oct 5, 2023

Choose a reason for hiding this comment

soksamnanglim Oct 11, 2023 • edited Loading

Choose a reason for hiding this comment

soksamnanglim Oct 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

soksamnanglim Oct 24, 2023 • edited Loading

Choose a reason for hiding this comment

soksamnanglim commented Dec 5, 2023

dbeatty10 commented Jul 10, 2024

TestIncrementalGrantsRedshift

TestModelGrantsTableRedshift

TestSnapshotGrantsRedshift

dbeatty10 commented Jul 10, 2024

dbeatty10 commented Jul 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pgvillena commented Aug 20, 2024

soksamnanglim commented Oct 4, 2023 •

edited

Loading

soksamnanglim Oct 11, 2023 •

edited

Loading

soksamnanglim Oct 11, 2023 •

edited

Loading

soksamnanglim Oct 24, 2023 •

edited

Loading