deps: upgrade zlib to 1.2.11 #10980

sam-github · 2017-01-24T17:57:39Z

Update zlib.

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
commit message follows commit guidelines

Affected core subsystem(s)

deps

sam-github · 2017-01-24T18:07:59Z

No idea why the bot thinks this shouldn't land on 7.x, it cherry-picked clean for me onto 7.x staging.

mscdex · 2017-01-24T18:11:25Z

Yeah I think something has been wrong with the bot for awhile now, I've had to manually un-label that countless times.

sam-github · 2017-01-24T18:24:09Z

I thought the bot just didn't take into account that a PR would land as long as earlier PRs landed first - but this change depends on nothing, so it looks like its quite broken.

bnoordhuis · 2017-01-24T23:54:56Z

The fips buildbot failed to download https://openssl.org/source/openssl-fips-2.0.9.tar.gz.

@nodejs/build Can we cache it somewhere? It's a point of failure and not very good netizenship to download it with every CI run.

bnoordhuis

Rubber-stamp LGTM. You might be able to shrink the diff if you dos2unix the files first.

sam-github · 2017-01-25T17:53:51Z

Thanks, @bnoordhuis, done.

I suppose zlib to be pretty stable, are the clean test results sufficient to land this, or is there anything else that can be done to increase confidence?

bnoordhuis · 2017-01-25T18:30:24Z

A green CI should be good enough.

sam-github · 2017-01-25T20:42:32Z

@nodejs/ctc some more thumbs up please, and I'll land tomorrow.

/cc @MylesBorins

shigeki · 2017-01-26T14:15:29Z

deps/zlib/zlib.h

+     deflateGetDictionary returns Z_OK on success, or Z_STREAM_ERROR if the
+   stream state is inconsistent.
+*/
+


Does this correspond to semver-minor? Looking at other diffs in zlib.h, there seems API/ABI compatible fortunately.

You believe that it is ABI compatible? Do we have a way of testing this? @MylesBorins you mentioned an "ABI roaster"?

You believe that it is ABI compatible?

As far as, I look at this but I do not have a confidence.

Do we have a way of testing this? @MylesBorins you mentioned an "ABI roaster"?

I'm not sure. It is nice to have a tools to check it.

shigeki · 2017-01-26T14:21:04Z

I found that I had missed to add -DHAVE_HIDDEN in zlib.gyp when gcc is used in the last upgrade . https://github.com/nodejs/node/blob/master/deps/zlib/configure#L735. Do we need this?

sam-github · 2017-01-27T19:58:48Z

re -DHAVE_HIDDEN, would adding this to zlib.gyp make the ABI different? It looks like it would.

shigeki · 2017-01-31T11:02:28Z

HAVE_HIDDEN is defined by default in zlib when gcc is used in order to hide internal functions. According to follow the upstream preferences, I think it would be better to add it.

sam-github · 2017-01-31T15:47:03Z

I agree it would be better to define HAVE_HIDDEN, but I think changing the visibility of symbols is unrelated to this PR to update the source, and since symbol visibility effects the API, it would be semver-major.

Can I open a seperate PR for HAVE_HIDDEN?

shigeki · 2017-02-01T09:13:53Z

Can I open a seperate PR for HAVE_HIDDEN?

Sure, let's move it.

sam-github · 2017-02-01T18:08:10Z

HAVE_HIDDEN: #11082

sam-github · 2017-02-01T18:11:12Z

abi check: https://ci.nodejs.org/job/citgm-abi-smoker/28/

jasnell · 2017-02-02T20:25:51Z

#11082 has landed

jasnell · 2017-02-17T01:09:43Z

What's the status on this one? :-)

This is a special LTS to fix a number of regressions that were found on the 6.10.x release line. This includes: * a fix for memory leak in the crypto module that was introduced in 6.10.1 * a fix for a regression introduced to the windows repl in 6.10.0 * a backported fix for V8 to stop a segfault that could occur when using spread syntax It also includes an upgrade to zlib 1.2.11 to fix a numberof low severity CVEs that were present in zlib 1.2.8. http://seclists.org/oss-sec/2016/q4/602 Notable changes * crypto: - fix memory leak if certificate is revoked (Tom Atkinson) #12089 * deps: - upgrade zlib to 1.2.11 (Sam Roberts) #10980 - backport V8 fixes for spread syntax regression causing segfaults (Michaël Zasso) #12037 * repl: - Revert commit that broke REPL display on Windows (Myles Borins) #12123

This is a maintenance release to fix a memory leak that was introduced in 4.8.1. It also includes an upgrade to zlib 1.2.11 to fix a number of low severity CVEs that were present in zlib 1.2.8. http://seclists.org/oss-sec/2016/q4/602 Notable changes: * crypto: - fix memory leak if certificate is revoked (Tom Atkinson) #12089 * deps: - upgrade zlib to 1.2.11 (Sam Roberts) #10980

This is a special LTS to fix a number of regressions that were found on the 6.10.x release line. This includes: * a fix for memory leak in the crypto module that was introduced in 6.10.1 * a fix for a regression introduced to the windows repl in 6.10.0 * a backported fix for V8 to stop a segfault that could occur when using spread syntax It also includes an upgrade to zlib 1.2.11 to fix a numberof low severity CVEs that were present in zlib 1.2.8. http://seclists.org/oss-sec/2016/q4/602 Notable changes * crypto: - fix memory leak if certificate is revoked (Tom Atkinson) #12089 * deps: - upgrade zlib to 1.2.11 (Sam Roberts) #10980 - backport V8 fixes for spread syntax regression causing segfaults (Michaël Zasso) #12037 * repl: - Revert commit that broke REPL display on Windows (Myles Borins) #12123

This is a special LTS to fix a number of regressions that were found on the 6.10.x release line. This includes: * a fix for memory leak in the crypto module that was introduced in 6.10.1 * a fix for a regression introduced to the windows repl in 6.10.0 * a backported fix for V8 to stop a segfault that could occur when using spread syntax It also includes an upgrade to zlib 1.2.11 to fix a numberof low severity CVEs that were present in zlib 1.2.8. http://seclists.org/oss-sec/2016/q4/602 Notable changes * crypto: - fix memory leak if certificate is revoked (Tom Atkinson) nodejs#12089 * deps: - upgrade zlib to 1.2.11 (Sam Roberts) nodejs#10980 - backport V8 fixes for spread syntax regression causing segfaults (Michaël Zasso) nodejs#12037 * repl: - Revert commit that broke REPL display on Windows (Myles Borins) nodejs#12123

This is a maintenance release to fix a memory leak that was introduced in 4.8.1. It also includes an upgrade to zlib 1.2.11 to fix a number of low severity CVEs that were present in zlib 1.2.8. http://seclists.org/oss-sec/2016/q4/602 Notable changes: * crypto: - fix memory leak if certificate is revoked (Tom Atkinson) nodejs#12089 * deps: - upgrade zlib to 1.2.11 (Sam Roberts) nodejs#10980

rmg · 2017-04-05T23:06:11Z

deps/zlib/zutil.h

-#ifdef __50SERIES /* Prime/PRIMOS */
-#  define OS_CODE  0x0f
+#ifdef __APPLE__
+#  define OS_CODE 19


This might be the source of #12244. This would slightly change the gzip header that is produced by zlib.gzip() on macOS. From looking at the code it seems that macOS would have previously fallen through to the default of 0x03 (Unix).

is this something we would want to float a patch for on LTS?

I am on the fence. Its clearly an observable change, @flippmoke saw it, but its not clear that we are that committed to binary identicality of output encoding. Then again, it seems a low-risk change to float a patch on the LTS branches to make them absolutely identical in output. @flippmoke, do you have an opinion on this? Do you feel that your test suites were depending on unstable encoding details, or do you feel that this is a regression?

@sam-github I would say we were surprised to see a in encoding change due to a patch release. However, I can understand how this is not a big deal for the vast majority of use cases for users. It is possibly confusing as well for OSX users that might use zlib elsewhere that they will not get the same binary results. However, I feel this is more an issue with zlib issuing this in a patch release rather then node doing this in a patch release.

Also just a note I am somewhat confused by the MAC OS header above it as well and when _ _ APPLE _ _applies compared to it. https://github.com/nodejs/node/pull/10980/files/dfa8abe1b500b86ffb3951736c209e57d83ed3ed#diff-a12b3cb4ae22eb7956dc70bda62773a3L121

@flippmoke I believe that is a relic of the pre-xcode days when MetroWerks/CodeWarrior was the compiler of choice on Mac. Yay legacy!

This is a maintenance release to fix a memory leak that was introduced in 4.8.1. It also includes an upgrade to zlib 1.2.11 to fix a number of low severity CVEs that were present in zlib 1.2.8. http://seclists.org/oss-sec/2016/q4/602 Notable changes: * crypto: - fix memory leak if certificate is revoked (Tom Atkinson) nodejs/node#12089 * deps: - upgrade zlib to 1.2.11 (Sam Roberts) nodejs/node#10980 Signed-off-by: Ilkka Myller <ilkka.myller@nodefield.com>

This is a special LTS to fix a number of regressions that were found on the 6.10.x release line. This includes: * a fix for memory leak in the crypto module that was introduced in 6.10.1 * a fix for a regression introduced to the windows repl in 6.10.0 * a backported fix for V8 to stop a segfault that could occur when using spread syntax It also includes an upgrade to zlib 1.2.11 to fix a numberof low severity CVEs that were present in zlib 1.2.8. http://seclists.org/oss-sec/2016/q4/602 Notable changes * crypto: - fix memory leak if certificate is revoked (Tom Atkinson) nodejs/node#12089 * deps: - upgrade zlib to 1.2.11 (Sam Roberts) nodejs/node#10980 - backport V8 fixes for spread syntax regression causing segfaults (Michaël Zasso) nodejs/node#12037 * repl: - Revert commit that broke REPL display on Windows (Myles Borins) nodejs/node#12123 Signed-off-by: Ilkka Myller <ilkka.myller@nodefield.com>

This is a maintenance release to fix a memory leak that was introduced in 4.8.1. It also includes an upgrade to zlib 1.2.11 to fix a number of low severity CVEs that were present in zlib 1.2.8. http://seclists.org/oss-sec/2016/q4/602 Notable changes: * crypto: - fix memory leak if certificate is revoked (Tom Atkinson) nodejs/node#12089 * deps: - upgrade zlib to 1.2.11 (Sam Roberts) nodejs/node#10980 Signed-off-by: Ilkka Myller <ilkka.myller@nodefield.com>

This is a special LTS to fix a number of regressions that were found on the 6.10.x release line. This includes: * a fix for memory leak in the crypto module that was introduced in 6.10.1 * a fix for a regression introduced to the windows repl in 6.10.0 * a backported fix for V8 to stop a segfault that could occur when using spread syntax It also includes an upgrade to zlib 1.2.11 to fix a numberof low severity CVEs that were present in zlib 1.2.8. http://seclists.org/oss-sec/2016/q4/602 Notable changes * crypto: - fix memory leak if certificate is revoked (Tom Atkinson) nodejs/node#12089 * deps: - upgrade zlib to 1.2.11 (Sam Roberts) nodejs/node#10980 - backport V8 fixes for spread syntax regression causing segfaults (Michaël Zasso) nodejs/node#12037 * repl: - Revert commit that broke REPL display on Windows (Myles Borins) nodejs/node#12123 Signed-off-by: Ilkka Myller <ilkka.myller@nodefield.com>

puzrin · 2017-07-20T01:30:23Z

Hi, i suspect regression. I have pako, zlib port to js, and it uses node's zlib to verify data. After 6.10.2 tests fails with deflate { level: 0 }

See https://travis-ci.org/nodeca/pako/jobs/255485506.

To reproduce - clone paco's repo, and run tests in desired version of node. 6.10.1 works, all next - not.

Can anyone comment?

sam-github · 2017-07-21T00:11:19Z

The binary output format changed, particularly for OS X, is there any chance you have assertions based on the zipped binary data? I cloned pako and did an npm test on Linux and it passed with node 8.1.4, are you testing on OS X? That would be a smoking gun, see #12244 and #12404

puzrin · 2017-07-21T01:09:56Z

is there any chance you have assertions based on the zipped binary data

That's exactly what i do. I use pako & node's zlib with all possible options and check that results are binary equal https://github.com/nodeca/pako/blob/master/test/deflate.js

Also note, that in my case it fails with { level: 0 } - when no compression applied and input data pass through almost as is (but with header + crc, except raw deflate)

I cloned pako and did an npm test on Linux and it passed with node 8.1.4

Sorry i've just disabled broken tests. If you enable those back, tests will fail in all nodes with zlib 1.2.11.

I use Ubuntu 14.04lts

About os type in header (pako writes "unix") - that make sense for gzip only, not for deflate.

sam-github · 2017-07-21T05:24:34Z

I'll run again just to verify, but if you think this is a bug, can you provide a standalone repro?

But, if you are doing things like the following, you might want to consider whether it is a valid test:

p = plain text
pako = pako.compress(p)
node = node.compress(p)
assert(pako == node)

I don't think above is. There are usually multiple legal compressed forms, what the algorithms guarantee is: orig = uncompress(compress(orig)), they don't guarantee the compressed form.

p = plain text
pako = pako.compress(p)
node = node.compress(p)
assert(pako.uncompress(node) == p)
assert(node.uncompress(pako) == p)

Again, I haven't read your test suite, so above is just a note about things that we've seen a few people have problems with in their test suites (not applications) after the zlib update.

puzrin · 2017-07-21T06:21:49Z

I'll run again just to verify, but if you think this is a bug, can you provide a standalone repro?

I do not know, is it a bug or not. But pako is not "yet another deflate implementation", it's a real port of zlib 1.2.8. And i suspect, zlib did not changed binary output (i don't count 1 byte of OS code in header). So, prior to dig deep, i asked, if anybody know about similar issues.

I'll triage in 1-2 weeks and tell you result.

puzrin · 2017-07-21T19:28:18Z

@sam-github https://github.com/puzrin/node_zlib1211_l0

It seems, zlib really changed binary representation of deflate "stored" mode. That's very strange but i don't consider it as bug.

sam-github · 2017-07-21T20:58:30Z

@puzrin Thanks for looking into this. I don't regard it as a bug either.

nodejs-github-bot added zlib Issues and PRs related to the zlib subsystem. dont-land-on-v7.x labels Jan 24, 2017

sam-github added lts-watch-v4.x and removed dont-land-on-v7.x labels Jan 24, 2017

mscdex mentioned this pull request Jan 24, 2017

bot incorrectly adding 'dont-land-on-v7.x' nodejs/github-bot#116

Closed

bnoordhuis approved these changes Jan 24, 2017

View reviewed changes

sam-github force-pushed the update-zlib branch from c269e41 to 9fd1f63 Compare January 25, 2017 16:33

nodejs-github-bot added the dont-land-on-v7.x label Jan 25, 2017

sam-github removed the dont-land-on-v7.x label Jan 25, 2017

gibfahn mentioned this pull request Jan 25, 2017

Cache CI downloads nodejs/build#599

Closed

shigeki reviewed Jan 26, 2017

View reviewed changes

shigeki approved these changes Feb 1, 2017

View reviewed changes

bricss mentioned this pull request Feb 14, 2017

Proposal - 7.6.0 (Current) #11185

Merged

MylesBorins mentioned this pull request Mar 29, 2017

V6.10.2 proposal #12128

Merged

MylesBorins mentioned this pull request Mar 29, 2017

v4.8.2 proposal #12129

Merged

flippmoke mentioned this pull request Apr 5, 2017

zlib different on OSX and Linux in newest releases #12244

Closed

rmg reviewed Apr 5, 2017

View reviewed changes

MylesBorins added land-on-v4.x and removed lts-watch-v4.x labels Apr 13, 2017

puzrin mentioned this pull request Jul 20, 2017

Fix broken tests { level: 0 } nodeca/pako#107

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

deps: upgrade zlib to 1.2.11 #10980

deps: upgrade zlib to 1.2.11 #10980

sam-github commented Jan 24, 2017

sam-github commented Jan 24, 2017

mscdex commented Jan 24, 2017

sam-github commented Jan 24, 2017

bnoordhuis commented Jan 24, 2017

bnoordhuis left a comment

sam-github commented Jan 25, 2017

bnoordhuis commented Jan 25, 2017

sam-github commented Jan 25, 2017

shigeki Jan 26, 2017

sam-github Jan 27, 2017

shigeki Jan 31, 2017

shigeki commented Jan 26, 2017

sam-github commented Jan 27, 2017

shigeki commented Jan 31, 2017

sam-github commented Jan 31, 2017

shigeki commented Feb 1, 2017

sam-github commented Feb 1, 2017

sam-github commented Feb 1, 2017

jasnell commented Feb 2, 2017

jasnell commented Feb 17, 2017

rmg Apr 5, 2017

MylesBorins Apr 6, 2017

sam-github Apr 6, 2017

flippmoke Apr 6, 2017

rmg Apr 11, 2017

puzrin commented Jul 20, 2017 •

edited

Loading

sam-github commented Jul 21, 2017

puzrin commented Jul 21, 2017 •

edited

Loading

sam-github commented Jul 21, 2017

puzrin commented Jul 21, 2017 •

edited

Loading

puzrin commented Jul 21, 2017 •

edited

Loading

sam-github commented Jul 21, 2017

deps: upgrade zlib to 1.2.11 #10980

deps: upgrade zlib to 1.2.11 #10980

Conversation

sam-github commented Jan 24, 2017

Checklist

Affected core subsystem(s)

sam-github commented Jan 24, 2017

mscdex commented Jan 24, 2017

sam-github commented Jan 24, 2017

bnoordhuis commented Jan 24, 2017

bnoordhuis left a comment

Choose a reason for hiding this comment

sam-github commented Jan 25, 2017

bnoordhuis commented Jan 25, 2017

sam-github commented Jan 25, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shigeki commented Jan 26, 2017

sam-github commented Jan 27, 2017

shigeki commented Jan 31, 2017

sam-github commented Jan 31, 2017

shigeki commented Feb 1, 2017

sam-github commented Feb 1, 2017

sam-github commented Feb 1, 2017

jasnell commented Feb 2, 2017

jasnell commented Feb 17, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

puzrin commented Jul 20, 2017 • edited Loading

sam-github commented Jul 21, 2017

puzrin commented Jul 21, 2017 • edited Loading

sam-github commented Jul 21, 2017

puzrin commented Jul 21, 2017 • edited Loading

puzrin commented Jul 21, 2017 • edited Loading

sam-github commented Jul 21, 2017

puzrin commented Jul 20, 2017 •

edited

Loading

puzrin commented Jul 21, 2017 •

edited

Loading

puzrin commented Jul 21, 2017 •

edited

Loading

puzrin commented Jul 21, 2017 •

edited

Loading