{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":734629358,"defaultBranch":"master","name":"delta","ownerLogin":"tlm365","currentUserCanPush":false,"isFork":true,"isEmpty":false,"createdAt":"2023-12-22T07:20:58.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/49281946?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1724233298.0","currentOid":""},"activityList":{"items":[{"before":"aabfc4fdc871b278907f124fb8b0b0e1a1c21e46","after":"ebed6e893727e89a0379c0dbbdb5f3aeca7c7ac4","ref":"refs/heads/cleanup-file-util","pushedAt":"2024-08-22T12:44:53.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"Migrate javafmtCheckSettings to project/Checkstyle\n\nSigned-off-by: Tai Le Manh <manhtai.lmt@gmail.com>","shortMessageHtmlLink":"Migrate javafmtCheckSettings to project/Checkstyle"}},{"before":"6ffae97e87774ec503967bd4a0da0c3d60998568","after":"aabfc4fdc871b278907f124fb8b0b0e1a1c21e46","ref":"refs/heads/cleanup-file-util","pushedAt":"2024-08-22T11:40:45.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"Migrate javafmtCheckSettings to project/Checkstyle\n\nSigned-off-by: Tai Le Manh <manhtai.lmt@gmail.com>","shortMessageHtmlLink":"Migrate javafmtCheckSettings to project/Checkstyle"}},{"before":"50d9235a729330c991ee87fcd853fe8477539b40","after":"6ffae97e87774ec503967bd4a0da0c3d60998568","ref":"refs/heads/cleanup-file-util","pushedAt":"2024-08-22T11:19:36.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"Migrate javafmtCheckSettings to project/Checkstyle\n\nSigned-off-by: Tai Le Manh <manhtai.lmt@gmail.com>","shortMessageHtmlLink":"Migrate javafmtCheckSettings to project/Checkstyle"}},{"before":"69f9c908e55e678976e05e7ff837e62e551155b6","after":"50d9235a729330c991ee87fcd853fe8477539b40","ref":"refs/heads/cleanup-file-util","pushedAt":"2024-08-21T09:55:19.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"[Kernel] Cleanup the FileNames.java utility methods\n\nSigned-off-by: Tai Le Manh <manhtai.lmt@gmail.com>","shortMessageHtmlLink":"[Kernel] Cleanup the FileNames.java utility methods"}},{"before":null,"after":"69f9c908e55e678976e05e7ff837e62e551155b6","ref":"refs/heads/cleanup-file-util","pushedAt":"2024-08-21T09:41:38.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"[Kernel] Cleanup the FileNames.java utility methods\n\nSigned-off-by: Tai Le Manh <manhtai.lmt@gmail.com>","shortMessageHtmlLink":"[Kernel] Cleanup the FileNames.java utility methods"}},{"before":"67c618302066b2e1b2977234e635dfba485deede","after":null,"ref":"refs/heads/remove-redundant-test","pushedAt":"2024-08-21T07:28:16.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"}},{"before":"6dd57b72794a7472509a54ea98a7c9faeb6e92ac","after":null,"ref":"refs/heads/checkstyle","pushedAt":"2024-08-21T07:28:15.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"}},{"before":"cb6e386864fbd606cde81b8cffbecaf2eff868c9","after":"01bf60743b77c47147843e9083129320490f1629","ref":"refs/heads/master","pushedAt":"2024-08-21T07:25:25.000Z","pushType":"push","commitsCount":63,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"[TEST-ONLY + Refactor] Fix Vacuum test code to make use of artificial clock everywhere (#3572)\n\n<!--\r\nThanks for sending a pull request!  Here are some tips for you:\r\n1. If this is your first time, please read our contributor guidelines:\r\nhttps://github.com/delta-io/delta/blob/master/CONTRIBUTING.md\r\n2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]\r\nYour PR title ...'.\r\n  3. Be sure to keep the PR description updated to reflect all changes.\r\n  4. Please write your PR title to summarize what this PR proposes.\r\n5. If possible, provide a concise example to reproduce the issue for a\r\nfaster review.\r\n6. If applicable, include the corresponding issue number in the PR title\r\nand link it in the body.\r\n-->\r\n\r\n#### Which Delta project/connector is this regarding?\r\n<!--\r\nPlease add the component selected below to the beginning of the pull\r\nrequest title\r\nFor example: [Spark] Title of my pull request\r\n-->\r\n\r\n- [X] Spark\r\n- [ ] Standalone\r\n- [ ] Flink\r\n- [ ] Kernel\r\n- [ ] Other (fill in here)\r\n\r\n## Description\r\nFix Vacuum test code to make use of artificial clock everywhere\r\n\r\n## How was this patch tested?\r\nExisting tests\r\n\r\n## Does this PR introduce _any_ user-facing changes?\r\n\r\nNo","shortMessageHtmlLink":"[TEST-ONLY + Refactor] Fix Vacuum test code to make use of artificial…"}},{"before":"890889a3b841f8157c833f813728b49d7276c73b","after":"cb6e386864fbd606cde81b8cffbecaf2eff868c9","ref":"refs/heads/master","pushedAt":"2024-08-06T02:44:32.000Z","pushType":"push","commitsCount":14,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"[Spark] Add commit version and logical records in Delta DML metrics (#3458)\n\n## Description\r\n\r\nExtend `delta.dml.{merge, update, delete}.stats` metrics with the\r\nfollowing fields:\r\n- `commitVersion` The commit version of the DML version. This allows\r\nassociating DML metrics with commit metrics and distinguishing DML\r\noperations that did not commit.\r\n- `numLogicalRecordsAdded` and `numLogicalRecordsRemoved`: The number of\r\nlogical records in AddFile and RemoveFile actions to be committed. These\r\nmetrics can be compared to the row-level metrics emitted by the DML\r\noperations.\r\n\r\nFinally, this commit adds the `isWriteCommand` field in DELETE metrics\r\nto distinguish DELETE operations that are performed in the context of\r\nWRITE commands that selectively overwrite data.\r\n\r\n\r\n## How was this patch tested?\r\nLog-only changes. Existing tests.","shortMessageHtmlLink":"[Spark] Add commit version and logical records in Delta DML metrics (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2441860332\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3458\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3458/hovercard\" href=\"https://github.com/delta-io/delta/pull/3458\">d…</a>"}},{"before":"7320d6bc9ea301d13247756c8d070d85bce6c646","after":"890889a3b841f8157c833f813728b49d7276c73b","ref":"refs/heads/master","pushedAt":"2024-08-01T04:58:08.000Z","pushType":"push","commitsCount":115,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"[Spark] Fix the inconsistencies in min/max Delta Log stats for special characters (#3430)\n\n## Description\r\n\r\nWhen truncating maxValue strings longer than 32 characters for\r\nstatistics, it's crucial to ensure the final truncated string is\r\nlexicographically greater than or equal to the input string in UTF-8\r\nencoded bytes.\r\n\r\nPreviously, we used the Unicode replacement character as the tieBreaker,\r\ncomparing it directly against one byte of the next character at a time.\r\nThis approach was insufficient because the tieBreaker could incorrectly\r\nwin against the non-first bytes of other characters (e.g., � < 🌼 but � >\r\nthe second byte of 🌼). We now compare one UTF-8 character (i.e. upto 2\r\nScala UTF-16 characters depending on surrogates) at a time to address\r\nthis issue.\r\n\r\nWe also start using U+10FFFD i.e. character with highest Unicode code\r\npoint as the tie-breaker now.\r\n\r\n## How was this patch tested?\r\nUTs","shortMessageHtmlLink":"[Spark] Fix the inconsistencies in min/max Delta Log stats for specia…"}},{"before":"3af433517bf5a42b1774cb63a8bd1d262e7d933d","after":"7320d6bc9ea301d13247756c8d070d85bce6c646","ref":"refs/heads/master","pushedAt":"2024-06-11T17:09:07.000Z","pushType":"push","commitsCount":59,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"[Spark] Log duration of DeltaSource operations (#2846)\n\n#### Which Delta project/connector is this regarding?\r\n\r\n- [x] Spark\r\n- [ ] Standalone\r\n- [ ] Flink\r\n- [ ] Kernel\r\n- [ ] Other (fill in here)\r\n\r\n## Description\r\n\r\nThis PR adds logging the main DeltaSource operations to allow us to\r\nmeasure the duration of these operations.\r\n\r\n## How was this patch tested?\r\n\r\nExisting tests to make sure nothing breaks.\r\n\r\n## Does this PR introduce _any_ user-facing changes?\r\n\r\nNo","shortMessageHtmlLink":"[Spark] Log duration of DeltaSource operations (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2225152402\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/2846\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/2846/hovercard\" href=\"https://github.com/delta-io/delta/pull/2846\">delta-io#2846</a>)"}},{"before":"2ef9a7df233a5464ddadb9ff12579fba5858d7a8","after":"6dd57b72794a7472509a54ea98a7c9faeb6e92ac","ref":"refs/heads/checkstyle","pushedAt":"2024-05-22T16:51:31.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"vkorukanti","name":"Venki Korukanti","path":"/vkorukanti","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1719945?s=80&v=4"},"commit":{"message":"minor changes","shortMessageHtmlLink":"minor changes"}},{"before":"8d68d07093d9b2fda7fec72eadf1f8a4fac3a89b","after":"2ef9a7df233a5464ddadb9ff12579fba5858d7a8","ref":"refs/heads/checkstyle","pushedAt":"2024-05-22T16:50:04.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"vkorukanti","name":"Venki Korukanti","path":"/vkorukanti","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1719945?s=80&v=4"},"commit":{"message":"minor cleanup","shortMessageHtmlLink":"minor cleanup"}},{"before":"2237bb4251377eba83016d3360d17fcb8dd81c9f","after":"8d68d07093d9b2fda7fec72eadf1f8a4fac3a89b","ref":"refs/heads/checkstyle","pushedAt":"2024-05-22T16:20:02.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"[INFRA] Improve the java checkstyle checks to log the errors to the sbt console\n\nSigned-off-by: Tai Le Manh <manhtai.lmt@gmail.com>","shortMessageHtmlLink":"[INFRA] Improve the java checkstyle checks to log the errors to the s…"}},{"before":"943960b0184c1082fdcd543a106fd21bfaf68040","after":"2237bb4251377eba83016d3360d17fcb8dd81c9f","ref":"refs/heads/checkstyle","pushedAt":"2024-05-18T16:18:11.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"[INFRA] Improve the java checkstyle checks to log the errors to the sbt console\n\nSigned-off-by: Tai Le Manh <manhtai.lmt@gmail.com>","shortMessageHtmlLink":"[INFRA] Improve the java checkstyle checks to log the errors to the s…"}},{"before":null,"after":"943960b0184c1082fdcd543a106fd21bfaf68040","ref":"refs/heads/checkstyle","pushedAt":"2024-05-18T15:37:52.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"[INFRA] Improve the java checkstyle checks to log the errors to the sbt console\n\nSigned-off-by: Tai Le Manh <manhtai.lmt@gmail.com>","shortMessageHtmlLink":"[INFRA] Improve the java checkstyle checks to log the errors to the s…"}},{"before":"8a8e757eba08abe83a4ddbb328442a9c6125ce03","after":"3af433517bf5a42b1774cb63a8bd1d262e7d933d","ref":"refs/heads/master","pushedAt":"2024-05-18T07:57:46.000Z","pushType":"push","commitsCount":3,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"[Spark] Pass sparkSession to commitOwnerBuilder (#3112)\n\n<!--\r\nThanks for sending a pull request!  Here are some tips for you:\r\n1. If this is your first time, please read our contributor guidelines:\r\nhttps://github.com/delta-io/delta/blob/master/CONTRIBUTING.md\r\n2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]\r\nYour PR title ...'.\r\n  3. Be sure to keep the PR description updated to reflect all changes.\r\n  4. Please write your PR title to summarize what this PR proposes.\r\n5. If possible, provide a concise example to reproduce the issue for a\r\nfaster review.\r\n6. If applicable, include the corresponding issue number in the PR title\r\nand link it in the body.\r\n-->\r\n\r\n#### Which Delta project/connector is this regarding?\r\n<!--\r\nPlease add the component selected below to the beginning of the pull\r\nrequest title\r\nFor example: [Spark] Title of my pull request\r\n-->\r\n\r\n- [X] Spark\r\n- [ ] Standalone\r\n- [ ] Flink\r\n- [ ] Kernel\r\n- [ ] Other (fill in here)\r\n\r\n## Description\r\n\r\n<!--\r\n- Describe what this PR changes.\r\n- Describe why we need the change.\r\n \r\nIf this PR resolves an issue be sure to include \"Resolves #XXX\" to\r\ncorrectly link and close the issue upon merge.\r\n-->\r\nUpdates CommitOwnerBuilder.build so that it can take in a sparkSession\r\nobject. This allows it to read CommitOwner-related dynamic confs from\r\nthe sparkSession while building it.\r\n\r\n\r\n## Does this PR introduce _any_ user-facing changes?\r\n\r\n<!--\r\nIf yes, please clarify the previous behavior and the change this PR\r\nproposes - provide the console output, description and/or an example to\r\nshow the behavior difference if possible.\r\nIf possible, please also clarify if this is a user-facing change\r\ncompared to the released Delta Lake versions or within the unreleased\r\nbranches such as master.\r\nIf no, write 'No'.\r\n-->\r\nNo","shortMessageHtmlLink":"[Spark] Pass sparkSession to commitOwnerBuilder (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2301716762\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3112\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3112/hovercard\" href=\"https://github.com/delta-io/delta/pull/3112\">delta-io#3112</a>)"}},{"before":"4ee7f4d669558b3d19785152298c899f70d7b055","after":"8a8e757eba08abe83a4ddbb328442a9c6125ce03","ref":"refs/heads/master","pushedAt":"2024-05-17T09:46:46.000Z","pushType":"push","commitsCount":10,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"[SPARK] Promote Variant Type Table Feature to Preview (#3091)\n\n## Description\r\npromotes variant type table feature to preview by removing `-dev` suffix\r\nand appending `-preview` suffix. We will support `-preview` forever.\r\n\r\n## How was this patch tested?\r\nexisting tests","shortMessageHtmlLink":"[SPARK] Promote Variant Type Table Feature to Preview (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2294654532\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3091\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3091/hovercard\" href=\"https://github.com/delta-io/delta/pull/3091\">delta-io#3091</a>)"}},{"before":"f6ebe24a559bd435ec241475b51b96602d26a6c0","after":"4ee7f4d669558b3d19785152298c899f70d7b055","ref":"refs/heads/master","pushedAt":"2024-05-13T17:35:51.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"[Spark] Protocol version downgrade in the presence of table features (#2841)\n\n## Description\r\nThis PR adds support for protocol versions downgrade when table features\r\nexist in the protocol. The downgraded protocol versions should be the\r\nminimum required to support all available table features. For example,\r\n`Protocol(3, 7, DeletionVectors, RowTracking)` can be downgraded to\r\n`Protocol(1, 7, RowTracking)` after removing the DV feature.\r\n\r\n## How was this patch tested?\r\nAdded new UTs in DeltaProtocolVersionSuite. Furthermore, existing UTs\r\ncover a significant part of the functionality. These these are the\r\nfollowing:\r\n\r\n- Downgrade protocol version on table created with (3, 7).\r\n- Downgrade protocol version on table created with (1, 7).\r\n- Protocol version downgrade on a table with table features and added\r\nlegacy feature.\r\n- Protocol version is not downgraded when writer features exist.\r\n- Protocol version is not downgraded when reader+writer features exist.\r\n- Protocol version is not downgraded when multiple reader+writer\r\nfeatures exist.\r\n\r\n\r\n## Does this PR introduce _any_ user-facing changes?\r\nYes. Dropping a table feature from a table with multiple features may\r\nnow result to a Protocol versions downgrade. For example, `Protocol(3,\r\n7, DeletionVectors, RowTracking)` can now be downgraded to `Protocol(1,\r\n7, RowTracking)`.","shortMessageHtmlLink":"[Spark] Protocol version downgrade in the presence of table features (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2222226034\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/2841\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/2841/hovercard\" href=\"https://github.com/delta-io/delta/pull/2841\">d…</a>"}},{"before":"cb070925b8ef5313582f679b207d47a660a78697","after":"f6ebe24a559bd435ec241475b51b96602d26a6c0","ref":"refs/heads/master","pushedAt":"2024-05-13T14:06:16.000Z","pushType":"push","commitsCount":211,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"[Kernel][Writes] Write `timestamp` as `INT64` type to Parquet data files (#3084)\n\n## Description\r\nWrite the `timestamp` as `INT64` physical format in Parquet. Currently,\r\nit is written as `INT96` which is a very old method of writing timestamp\r\nand deprecated a long time ago. Also, collect statistics, for\r\n`timestamp` type columns.\r\n\r\n## How was this patch tested?\r\nUpdate the existing tests.","shortMessageHtmlLink":"[Kernel][Writes] Write <code>timestamp</code> as <code>INT64</code> type to Parquet data fi…"}},{"before":"c5ff2365390a10c87da04ffec437d8d218d98490","after":"cb070925b8ef5313582f679b207d47a660a78697","ref":"refs/heads/master","pushedAt":"2024-03-22T03:23:52.000Z","pushType":"push","commitsCount":8,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"[Build] Add `storage` module dependency to Kernel examples\n\n## Description\r\nPrep for #2770 PR","shortMessageHtmlLink":"[Build] Add <code>storage</code> module dependency to Kernel examples"}},{"before":null,"after":"67c618302066b2e1b2977234e635dfba485deede","ref":"refs/heads/remove-redundant-test","pushedAt":"2024-03-20T16:06:37.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"Remove redundant imports and test suite\n\nSigned-off-by: Tai Le Manh <manhtai.lmt@gmail.com>","shortMessageHtmlLink":"Remove redundant imports and test suite"}},{"before":"72fad38a3b003122ccdd88144cfc4f4110d9e154","after":"c5ff2365390a10c87da04ffec437d8d218d98490","ref":"refs/heads/master","pushedAt":"2024-03-20T15:43:59.000Z","pushType":"push","commitsCount":6,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"[Spark][Test-Only] Add Row Tracking Clone tests with empty source table (#2742)\n\n## Description\r\nAdd Materialized Row Tracking columns CLONE tests, making sure we have\r\nenough Row Tracking test coverage necessary to gear us towards [enabling\r\nRow Tracking outside of\r\ntesting](https://github.com/delta-io/delta/pull/2059).\r\n\r\n## How was this patch tested?\r\nExisting + added UTs.\r\n\r\n## Does this PR introduce _any_ user-facing changes?\r\nNo.","shortMessageHtmlLink":"[Spark][Test-Only] Add Row Tracking Clone tests with empty source tab…"}},{"before":"d0477bba4bd796967ace5f989f3285f1107baada","after":"72fad38a3b003122ccdd88144cfc4f4110d9e154","ref":"refs/heads/master","pushedAt":"2024-03-19T08:57:19.000Z","pushType":"push","commitsCount":13,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"[Spark] Column mapping removal: support tables with deletion vectors, column constraints and generated columns. (#2753)\n\n## Description\r\nAdd additional tests for tables with deletion vectors, generated columns\r\nand column constraints for column mapping removal.\r\n\r\n## How was this patch tested?\r\nNew unit tests\r\n\r\n## Does this PR introduce _any_ user-facing changes?\r\nNo","shortMessageHtmlLink":"[Spark] Column mapping removal: support tables with deletion vectors,…"}},{"before":"e9081d6df58fdb3889e40a0c765060cab97a2c45","after":"d0477bba4bd796967ace5f989f3285f1107baada","ref":"refs/heads/master","pushedAt":"2024-03-12T14:39:21.000Z","pushType":"push","commitsCount":8,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"[Kernel][LogReplay] Make a single read request for all checkpoint files\n\n## Description\r\nCurrently, the `kernel-api` reads one file (either checkpoint or commit\r\nfile) at a time. Once the file is fully read, then the next file is read\r\nrequest is issued. This makes reading large checkpoints split over\r\nmultiple files slower. Instead `kernel-api` could issue read requests\r\nfor all checkpoint files at once (in case of multi-part checkpoints)\r\nusing the `ParquetHandler.readParquetFiles` and let the implementations\r\nof the `ParquetHandler` prefetch or using multiple threads to read the\r\ncheckpoint parts concurrently.\r\n\r\nThis PR makes the change to `kernel-api` to issue one read request for\r\nall checkpoint files that need to be read for state reconstructions.\r\n\r\nResolves #2668\r\nResolves #1965\r\n\r\n## How was this patch tested?\r\nExisting tests and a benchmark with a test only parallel parquet reader.\r\nHere are the sample benchmark results with the test only parallel\r\nParquet reader. `Score` tells the average time to construct the Delta\r\ntable state. `parallelReaderCount` indicates the number of parallel\r\nParquet reading threads used.\r\n\r\n```\r\nBenchmark                                     (parallelReaderCount)  Mode  Cnt     Score     Error  Units\r\nBenchmarkParallelCheckpointReading.benchmark                      0  avgt    5  1565.520 ±  20.551  ms/op\r\nBenchmarkParallelCheckpointReading.benchmark                      1  avgt    5  1064.850 ±  19.699  ms/op\r\nBenchmarkParallelCheckpointReading.benchmark                      2  avgt    5   785.918 ± 176.285  ms/op\r\nBenchmarkParallelCheckpointReading.benchmark                      4  avgt    5   729.487 ±  51.470  ms/op\r\nBenchmarkParallelCheckpointReading.benchmark                     10  avgt    5   693.757 ±  41.252  ms/op\r\nBenchmarkParallelCheckpointReading.benchmark                     20  avgt    5   702.656 ±  19.145  ms/op\r\n```","shortMessageHtmlLink":"[Kernel][LogReplay] Make a single read request for all checkpoint files"}},{"before":"2fddb8b0b87b08ec57b13e86fcd2feec83ea10f1","after":"e9081d6df58fdb3889e40a0c765060cab97a2c45","ref":"refs/heads/master","pushedAt":"2024-03-08T15:13:33.000Z","pushType":"push","commitsCount":6,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"[CI] Disable running unidoc from the spark CI tests for now (#2732)\n\n#### Which Delta project/connector is this regarding?\r\n<!--\r\nPlease add the component selected below to the beginning of the pull\r\nrequest title\r\nFor example: [Spark] Title of my pull request\r\n-->\r\n\r\n- [ ] Spark\r\n- [ ] Standalone\r\n- [ ] Flink\r\n- [ ] Kernel\r\n- [X] Other (fill in here)\r\n\r\n## Description\r\n\r\nThis removes compiling unidoc from the spark CI tests since we've seen a\r\nflaky failure on #2727.\r\n\r\nWe will follow up and create a separate job just for unidoc (which makes\r\nmore sense anyways as it should run for changes in any of the projects.)\r\n\r\n## How was this patch tested?\r\n\r\nCI runs.","shortMessageHtmlLink":"[CI] Disable running unidoc from the spark CI tests for now (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2175015802\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/2732\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/2732/hovercard\" href=\"https://github.com/delta-io/delta/pull/2732\">delta-io…</a>"}},{"before":"aff2f564aba5462dd7c2ee27c9faa051841cb3ad","after":null,"ref":"refs/heads/2638","pushedAt":"2024-03-07T08:38:58.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"}},{"before":"5cc01848eb787f673d2f22b87d353eef236fa2e8","after":"2fddb8b0b87b08ec57b13e86fcd2feec83ea10f1","ref":"refs/heads/master","pushedAt":"2024-03-07T05:51:17.000Z","pushType":"push","commitsCount":5,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"Add tests to check the behaviour of CLONE with row IDs.\n\nCloses https://github.com/delta-io/delta/pull/2678\n\nGitOrigin-RevId: 9473242559e6957b3ce56925cfe1da256972ba1b","shortMessageHtmlLink":"Add tests to check the behaviour of CLONE with row IDs."}},{"before":"9b3fa0a1a05e51b38cec083afb41226beb399b0f","after":"5cc01848eb787f673d2f22b87d353eef236fa2e8","ref":"refs/heads/master","pushedAt":"2024-03-06T16:13:27.000Z","pushType":"push","commitsCount":8,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"Add RemoveColumnMappingSuite\n\nAdd missing RemoveColumnMappingSuite.\n\nCloses delta-io/delta#2697\n\nGitOrigin-RevId: 992c940b42748701bd5f080ee2219d8aa09c9570","shortMessageHtmlLink":"Add RemoveColumnMappingSuite"}},{"before":"a688dee173851380320695338266e6b08d28a54b","after":"aff2f564aba5462dd7c2ee27c9faa051841cb3ad","ref":"refs/heads/2638","pushedAt":"2024-03-04T18:28:17.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"tlm365","name":"Tai Le Manh","path":"/tlm365","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/49281946?s=80&v=4"},"commit":{"message":"[Kernel][Java to Scala test conversion] Convert and refactor TestParquetBatchReader\n\nSigned-off-by: Tai Le Manh <manhtai.lmt@gmail.com>","shortMessageHtmlLink":"[Kernel][Java to Scala test conversion] Convert and refactor TestParq…"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"Y3Vyc29yOnYyOpK7MjAyNC0wOC0yMlQxMjo0NDo1My4wMDAwMDBazwAAAAShSCO4","startCursor":"Y3Vyc29yOnYyOpK7MjAyNC0wOC0yMlQxMjo0NDo1My4wMDAwMDBazwAAAAShSCO4","endCursor":"Y3Vyc29yOnYyOpK7MjAyNC0wMy0wNFQxODoyODoxNy4wMDAwMDBazwAAAAQMD4mp"}},"title":"Activity · tlm365/delta"}