Add Rails/UniqueValidationWithoutIndex cop #197

pocke · 2020-02-08T08:30:30Z

Target problem

The target is the ActiveRecord's uniqueness validation.

When you define uniqueness validation in a model, you also should add a unique index.
It has two reasons.

First, duplicated records may occur even if ActiveRecord's validation is defined.
The Rails Guide mentions this problem.

This helper validates that the attribute's value is unique right before the object gets saved. It does not create a uniqueness constraint in the database, so it may happen that two different database connections create two records with the same value for a column that you intend to be unique. To avoid that, you must create a unique index on that column in your database.
https://guides.rubyonrails.org/active_record_validations.html#uniqueness

Second, it will cause slow queries.
The validation executes a SELECT statement with the target column when inserting/updating a record. If the column does not have an index and the table is large, the query will be heavy.

Solution

Add a Rails/UniqueValidationWithoutIndex cop to detect the problem.

And, I implemented a db/schema.rb loader for the cop.
So now we can implement cops with DB schema information! I think we can more powerful cops for ActiveRecord with the loader.

Note

This pull request adds active_support dependency. But it actually does not increase dependencies in most cases because rubocop-rails is used in Rails projects.

Limitation

This cop ignores validates_uniqueness_of method in the current implementation.
Of course, it's better if the cop supports validates_uniqueness_of. But personally I think it is not important. Because Rails/Validation cop marks it "old-style", so I guess not many users use the method, especially rubocop-rails users.

But if someone sends a pull request to support the method, it will be acceptable. I just omitted it now to keep the first PR simple.

Before submitting the PR make sure the following are checked:

Wrote good commit messages.
Commit message starts with [Fix #issue-number] (if the related issue exists).
Feature branch is up-to-date with master (if not - rebase it).
Squashed related commits together.
Added tests.
Added an entry to the Changelog if the new code introduces user-observable changes. See changelog entry format.
The PR relates to only one subject with a clear title
and description in grammatically correct, complete sentences.
Run bundle exec rake default. It executes all tests and RuboCop for itself, and generates the documentation.

pocke · 2020-02-08T09:45:27Z

lib/rubocop/rails/schema_loader.rb

+  module Rails
+    # It loads db/schema.rb and return Schema object.
+    # Cops refers database schema information with this module.
+    module SchemaLoader


This is the schema loader.

I implemented it as a mixin at first, but it was a bad idea. So now it is a non-mixin module.

If it is a mixin, the @schema cache only works for a single cop and single file. So it needs to parse db/schema.rb many times.
But if the non-mixin module, just once parsing is enough.

pocke · 2020-02-08T09:47:36Z

config/default.yml

@@ -496,6 +496,13 @@ Rails/UniqBeforePluck:
    - aggressive
  AutoCorrect: false

+Rails/UniqueValidationWithoutIndex:
+  Description: 'Uniqueness validation should be with a unique index.'
+  Enabled: true


Should be the status pending, or not? I'm not sure, so if it should be pending, please tell me 🙏

RuboCop core has not yet released cop with pending. Here, let's try with true first.

pocke · 2020-02-08T10:18:32Z

Oops, I found a false positive with validation for a relation.

# If the table has a unique index for `user_id` column, the following validation is not offensive.
# But the cop registers an offense to the validation.
belongs_to :user
validates :user, uniqueness: true

I'll fix it soon.

pocke · 2020-02-08T11:09:54Z

Oops, I found a false positive with validation for a relation.

Fixed and force-pushed.

koic · 2020-02-09T11:33:53Z

lib/rubocop/cop/mixin/active_record_helper.rb

+
+module RuboCop
+  module Cop
+    # A mixin to extend cops for ActiveRecord features


The proper notation is Active Record in this case. Can you update it in this PR?

The proper names of Rails components have a space in between the words, like "Active Support". ActiveRecord is a Ruby module, whereas Active Record is an ORM.

https://github.com/rails/rails/blob/v6.0.2.1/guides/source/api_documentation_guidelines.md#wording

koic · 2020-02-09T11:34:09Z

lib/rubocop/cop/mixin/active_record_helper.rb

+    module ActiveRecordHelper
+      extend NodePattern::Macros
+
+      def_node_search :find_set_table_name, <<-PATTERN


Can you use squiggly heredoc in this PR?

Oops, I guess the cause is my old-style snippets. Thank you notice that!
https://github.com/pocke/dotfiles/blob/c5def33338c0a9f4b32086986c0fc4fd6250f4ff/snippets/ruby.snip#L73-L83

koic · 2020-02-09T11:34:52Z

lib/rubocop/cop/rails/unique_validation_without_index.rb

+      class UniqueValidationWithoutIndex < Cop
+        include ActiveRecordHelper
+
+        MSG = 'Uniqueness validation should be with a unique index'


Can you add a period to the end of the message?

koic · 2020-02-09T11:36:45Z

rubocop-rails.gemspec

@@ -4,6 +4,7 @@ $LOAD_PATH.unshift File.expand_path('lib', __dir__)
 require 'rubocop/rails/version'
 require 'English'

+# rubocop:disable Metrics/BlockLength


Can you specify in the exclusion list of this cop at .rubocop.yml?
https://github.com/rubocop-hq/rubocop-rails/blob/v2.4.2/.rubocop.yml#L80-L84

koic · 2020-02-09T11:37:14Z

rubocop-rails.gemspec

@@ -33,6 +34,8 @@ Gem::Specification.new do |s|

  # Rack::Utils::SYMBOL_TO_STATUS_CODE, which is used by HttpStatus cop, was
  # introduced in rack 1.1
+  s.add_runtime_dependency 'activesupport'


This dependency should be added above the comment for Rack :-)

I did not confirm the result of sorting the dependencies by rubocop -a. 🙈

pocke · 2020-02-09T13:37:46Z

I confirmed the comments and fixed them, then force-pushed. Thank you for reviewing!

andyw8 · 2020-02-09T16:37:27Z

What about if someone has set config.active_record.schema_format = :sql? There will no db/schema.rb

andyw8 · 2020-02-09T16:40:04Z

lib/rubocop/cop/rails/unique_validation_without_index.rb

+module RuboCop
+  module Cop
+    module Rails
+      # When you define uniqueness validation in Active Record model,


Missing 'a' before 'define'?

andyw8 · 2020-02-09T16:41:29Z

lib/rubocop/cop/rails/unique_validation_without_index.rb

+  module Cop
+    module Rails
+      # When you define uniqueness validation in Active Record model,
+      # you also should add a unique index for the column. It has two reasons.


'It has two reasons' reads a little awkward – maybe change to 'There are two reasons' or 'There are two reasons for this'?

andyw8 · 2020-02-09T16:46:16Z

lib/rubocop/rails/schema_loader.rb

+
+      # It parses `db/schema.rb` and return it.
+      # It returns `nil` if it can't find `db/schema.rb`.
+      # So a cop that uses the loader should handle `nil` properly.


What do you think about having it raise an exception if schema.rb is missing?

Personally I think missing schema.rb should be handled, it means it shouldn't display a stack trace. Because it is not a bug of rubocop-rails.
But I think it is a good idea to notify missing shcema.rb to the user. So, how about a warning instead of an exception?
For example: "db/schema.rb is missing. Disable Rails/UniqueValidationWithoutIndex cop if you don't use db/schema.rb."

I don't think there is a need to raise warnings or errors. OTOH, documenting the dependency on db/schema.rb will help users.

I do not have a strong opinion for it because I think both of them have pros and cons.
The warning can tell users that the cop does not work. It may be a bug, or using schema.sql instead.
The current behavior conceals it. but it will be easy to use rubocop-rails, because no-configuration is necessary even if the cop doesn't work.

By the way, I updated the documentation to mention existence of db/schema.rb.
https://github.com/rubocop-hq/rubocop-rails/pull/197/files#diff-af4cb6c7e18c12f0a26b2ef45466f731R15

koic · 2020-02-10T03:28:30Z

spec/rubocop/cop/rails/unique_validation_without_index_spec.rb

+
+  context 'without db/schema.rb' do
+    it 'does nothing' do
+      expect_no_offenses(<<-RUBY)


There are still some <<- :-)

pocke · 2020-02-14T15:07:57Z

What about if someone has set config.active_record.schema_format = :sql? There will no db/schema.rb

Currently the cop does nothing if db/schema.rb is missing.
I guess we can support the sql format, but I guess it is more difficult than parsing db/schema.rb.
Pull request for sql format is welcome, but I do not have any plan for it.

koic · 2020-02-15T07:17:20Z

Yeah. I think the cop's responsibility should not handle the existence of db/schema.rb. Each cop responsibility should be simple. So, it should be essentially SRP: Single Responsibility Principle.

pocke · 2020-02-17T14:58:04Z

I applied the reviewed comments, Thanks!
And I commented on the comment thread. #197 (comment)

@koic I like the both of warning and no-warning ideas, so if you'd like no-warning, could you merge this pull request?

koic · 2020-02-17T15:04:31Z

Yeah, I'm going to merge this one! Thanks!

rubocop/rubocop-rails#197 introduced a new cop to rubocop-rails that checks for unique validation on an index if it's present in the model. This caused a number of failures when upgrading to 2.5.0 that this fixes.

mathieujobin · 2020-04-17T06:17:24Z

I love this, thank you

mathieujobin · 2020-04-17T06:27:07Z

I confirm it does this supports validation with a scope and using relation names instead of id columns!!!

validates :book, uniqueness: { scope: [:shelf] }

pocke changed the title ~~[WIP] Rails/unique validation without index~~ [WIP] Add Rails/UniqueValidationWithoutIndex cop Feb 8, 2020

pocke force-pushed the Rails/UniqueValidationWithoutIndex branch 2 times, most recently from ac6cf91 to d23f303 Compare February 8, 2020 09:40

pocke commented Feb 8, 2020

View reviewed changes

pocke changed the title ~~[WIP] Add Rails/UniqueValidationWithoutIndex cop~~ Add Rails/UniqueValidationWithoutIndex cop Feb 8, 2020

pocke marked this pull request as ready for review February 8, 2020 09:48

pocke force-pushed the Rails/UniqueValidationWithoutIndex branch from d23f303 to 585b2d9 Compare February 8, 2020 11:09

koic reviewed Feb 9, 2020

View reviewed changes

pocke force-pushed the Rails/UniqueValidationWithoutIndex branch from 585b2d9 to b3797d5 Compare February 9, 2020 13:36

andyw8 reviewed Feb 9, 2020

View reviewed changes

koic reviewed Feb 10, 2020

View reviewed changes

Add Rails/UniqueValidationWithoutIndex cop

9c9eac6

pocke force-pushed the Rails/UniqueValidationWithoutIndex branch from b3797d5 to 9c9eac6 Compare February 17, 2020 14:38

koic merged commit 2a217d0 into rubocop:master Feb 17, 2020

pocke deleted the Rails/UniqueValidationWithoutIndex branch February 17, 2020 15:08

pocke mentioned this pull request Feb 18, 2020

The SchemaLoader raises an error on GitLab's schema.rb #200

Closed

koic mentioned this pull request Apr 13, 2020

Rails/UniqueValidationWithoutIndex should not fail when using scope #231

Open

This was referenced May 20, 2020

Add missing unique indexes - part 1 forem/forem#7977

Merged

Add missing unique indexes - part 2 forem/forem#8002

Merged

AndrewHYi mentioned this pull request Aug 25, 2020

BUG: An error occurred while Rails/UniqueValidationWithoutIndex cop was inspecting <file> (uniqueness: in validation) #318

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Rails/UniqueValidationWithoutIndex cop #197

Add Rails/UniqueValidationWithoutIndex cop #197

pocke commented Feb 8, 2020 •

edited

Loading

pocke Feb 8, 2020

pocke Feb 8, 2020 •

edited

Loading

koic Feb 9, 2020

pocke commented Feb 8, 2020

pocke commented Feb 8, 2020

koic Feb 9, 2020

koic Feb 9, 2020

pocke Feb 9, 2020

koic Feb 9, 2020

koic Feb 9, 2020

koic Feb 9, 2020

pocke Feb 9, 2020

pocke commented Feb 9, 2020

andyw8 commented Feb 9, 2020

andyw8 Feb 9, 2020

andyw8 Feb 9, 2020

andyw8 Feb 9, 2020

pocke Feb 14, 2020

koic Feb 15, 2020

pocke Feb 17, 2020

koic Feb 10, 2020

pocke commented Feb 14, 2020

koic commented Feb 15, 2020 •

edited

Loading

pocke commented Feb 17, 2020

koic commented Feb 17, 2020

mathieujobin commented Apr 17, 2020

mathieujobin commented Apr 17, 2020 •

edited

Loading

Add Rails/UniqueValidationWithoutIndex cop #197

Add Rails/UniqueValidationWithoutIndex cop #197

Conversation

pocke commented Feb 8, 2020 • edited Loading

Target problem

Solution

Note

Limitation

Choose a reason for hiding this comment

pocke Feb 8, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pocke commented Feb 8, 2020

pocke commented Feb 8, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pocke commented Feb 9, 2020

andyw8 commented Feb 9, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pocke commented Feb 14, 2020

koic commented Feb 15, 2020 • edited Loading

pocke commented Feb 17, 2020

koic commented Feb 17, 2020

mathieujobin commented Apr 17, 2020

mathieujobin commented Apr 17, 2020 • edited Loading

pocke commented Feb 8, 2020 •

edited

Loading

pocke Feb 8, 2020 •

edited

Loading

koic commented Feb 15, 2020 •

edited

Loading

mathieujobin commented Apr 17, 2020 •

edited

Loading