Enable MacOS build on Travis CI #116

yongtang · 2019-02-24T02:51:25Z

Enabled macOS build for all ops (including kafka/kinesis/ignite/video)
Enable macOS tests for all ops except video.
Updated the script to launch ignite process, kafka/zookeeper process, kinesis localstack emulator on macOS (not through container as Travis does not have docker available on macOS).
Changed port number from 42300 to 10800 for ignite, as on macOS there is no docker available so no port mapping.
Updated librdkafka to 0.11.6 (from 0.11.5)
Changed the required tensorflow package to >=1.13.0,<1.14.0. Also, there is only one place to specify tensorflow package version and that is in setup.py. No multiple places like before to avoid inconsistency.
Additional refactor of .travis.yaml to consolidate macOS and linux build. Note .travis/python.release.sh is still the command to run to build the release binary like before.

Signed-off-by: Yong Tang yong.tang.github@outlook.com

yongtang · 2019-02-24T16:08:35Z

There are still some work to do and clean up, but I think we are very close to get macOS support.

Right now the following ops has been enabled:

arrow
hadoop
image
libsvm
lmdb
parquet

Not done yet:

ignite
kafka
kinesis
video

For ignite, kafka, kinesis, we will need to setup servers or emulators (with or without docker) for testing on Travis CI.
For video, on macOS there is no standard install like ubuntu for FFmpeg. We have to stick with one or two versions of ffmpeg install (cannot support all).

Will do some cleanup but overall I think we may have a nightly build of macOS support pretty soon.

@damienpontifex as you might be interested.

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Encountered a scneario where all tests are skipped Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

yongtang · 2019-02-26T22:24:43Z

/cc @terrytangyuan @dmitrievanthony @BryanCutler to take a look. We have enabled the build all ops for macOS, and the tests of most of the ops for macOS. Tests for kinesis/kafka/video/ignite are not done yet.

The PR also adds some enhancement and additional comments about each configuration in Travis CI:

Developer Build: The builds are done in Ubuntu 16.04/18.04 to make sure anyone could clone the source code and build locally.
Install Build: The whl binaries are generated with Ubuntu 14.04 (to match tensorflow), and tested with Ubuntu 16.04/18.04. This is to make sure that user could use pip to install the releases and run locally (with their ubuntu 16.04/18.04 machines)
Preview Build: Nightly releases for TF 2.0 preview.
Nightly Build: Nightly releases based on TF 1.13.1.

Please take a look

BryanCutler

Do the developer and install build jobs both build binaries and run tests, then only the build job makes wheels? Is it possible they could be combined and just test the pip install as a last step?

BryanCutler · 2019-02-26T22:58:23Z

.travis/python.release.sh

+# docker run -i -t --rm -v $PWD:/v -w /v --net=host buildpack-deps:14.04 /v/.travis/python.release.sh
+
+export TENSORFLOW_INSTALL="$(python setup.py --package-version)"
+export PYTHON_VERSION="python python3 python3.5 python3.6"


Could python2.7 python3.4 be explicitly versioned too?

@BryanCutler The build of whl that is pushed to PyPi.org, has to run on Ubuntu 14.04 as otherwise the pip install tensorflow-io will not work with `pip install tensorflow'.

If we build whl in Ubuntu 18.04, then some users that is using an old version of Ubuntu (16.04), will find out that they could run tensorflow but not tensorflow-io. This will not be a good experience for them.

So we stick with Ubuntu 14.04 for building whl. And the generated binary will be tested on Ubuntu 16.04/18.04.

On the other hand, some developers may not be interested in building whl themselves. They only want to build the repo and run locally, maybe debug something. They likely will not use Ubuntu 14.04 (too old). That is why we also test building the source with Ubuntu 16.04/18.04.

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

BryanCutler · 2019-02-26T23:59:59Z

Ahh, ok makes sense thanks for the explanation. I didn't see that there were three versions of Ubuntu, I thought it was all 16.04/18.04. I see why we need to build on 14.04 then, sound fine to me.

…

On Tue, Feb 26, 2019, 3:49 PM Yong Tang ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In .travis/python.release.sh <#116 (comment)>: > -rm -rf patchelf-0.9* -curl -sOL https://bootstrap.pypa.io/get-pip.py -python3.6 get-pip.py -python3.5 get-pip.py -python3 get-pip.py -python get-pip.py -rm -rf get-pip.py -python3 -m pip install -q auditwheel==1.5.0 -python3 -m pip install -q wheel==0.31.1 - -if [[ ! -z ${TENSORFLOW_INSTALL} ]]; then - python -m pip install -q ${TENSORFLOW_INSTALL} +# docker run -i -t --rm -v $PWD:/v -w /v --net=host buildpack-deps:14.04 /v/.travis/python.release.sh + +export TENSORFLOW_INSTALL="$(python setup.py --package-version)" +export PYTHON_VERSION="python python3 python3.5 python3.6" @BryanCutler <https://github.com/BryanCutler> The build of whl that is pushed to PyPi.org, has to run on Ubuntu 14.04 as otherwise the pip install tensorflow-io will not work with `pip install tensorflow'. If we build whl in Ubuntu 18.04, then some users that is using an old version of Ubuntu (16.04), will find out that they could run tensorflow but not tensorflow-io. This will not be a good experience for them. So we stick with Ubuntu 14.04 for building whl. And the generated binary will be tested on Ubuntu 16.04/18.04. On the other hand, some developers may not be interested in building whl themselves. They only want to build the repo and run locally, maybe debug something. They likely will not use Ubuntu 14.04 (too old). That is why we also test building the source with Ubuntu 16.04/18.04. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#116 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEUwdammCQxgizmI5GOHqB8cGoE7XLNmks5vRcfwgaJpZM4bOTKN> .

yongtang · 2019-02-27T00:07:53Z

@BryanCutler The Ubuntu 14.04 will be out of support (EOL) 04/2019, so likely TensorFlow will start building on Ubuntu 16.04. Then we could drop building on 14.04.

There are also discussions of moving the build of TensorFlow on CentOS 6, for the purpose of manylinux2010. Don't know if this will be the direction or not yet. Though we could always build tenworflow-io with the same environment as tensorflow to make sure it works for any scenario.

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Hopefully to fix kafka trap 6 issue on macOS (happens on 1.13.0rc2) Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

yongtang · 2019-02-27T10:06:27Z

Looks like we are all good now and all tests passes on macOS (except for video/ffmpeg). Here is the list of changes in the PR:

Enabled macOS build for all ops (including kafka/kinesis/ignite/video)
Enable macOS tests for all ops except video.
Updated the script to launch ignite process, kafka/zookeeper process, kinesis localstack emulator on macOS (not through container as Travis does not have docker available on macOS).
Changed port number from 42300 to 10800 for ignite, as on macOS there is no docker available so no port mapping.
Updated librdkafka to 0.11.6 (from 0.11.5)
Changed the required tensorflow package to >=1.13.0,<1.14.0. Also, there is only one place to specify tensorflow package version and that is in setup.py. No multiple places like before to avoid inconsistency.
Additional refactor of .travis.yaml to consolidate macOS and linux build. Note .travis/python.release.sh is still the command to run to build the release binary like before.

yongtang · 2019-02-27T10:07:02Z

/cc @BryanCutler @dmitrievanthony @terrytangyuan @damienpontifex

BryanCutler

Nice work, thanks for doing this! Just one minor question otherwise, LGTM

BryanCutler · 2019-02-27T18:08:08Z

tests/test_ignite/start_ignite.sh

 -v ${SCRIPT_PATH}:/data apacheignite/ignite:${IGNITE_VERSION} /data/bin/start-plain.sh

 # Start Apache Ignite with IGFS.
 docker run -itd --name ignite-igfs -p 10500:10500 \
 -v ${SCRIPT_PATH}:/data apacheignite/ignite:${IGNITE_VERSION} /data/bin/start-igfs.sh

 # Wait Apache Ignite to be started
-sleep 10
+#sleep 10


is waiting no longer needed or done somewhere else?

Thanks @BryanCutler. Just removed this one as Ignite is only going to be used after build is complete (takes about 10-15min) so I think no need to wait explicitly.

terrytangyuan · 2019-02-27T19:27:10Z

Great job @yongtang!

damienpontifex · 2019-02-27T20:46:39Z

Brilliant @yongtang!
I'm looking forward to trying this all out

yongtang · 2019-02-27T20:50:32Z

@damienpontifex the build is based on macOS 10.12+xcode 9 (on Travis CI). Let me know if you encounter any issues.

Enable MacOS build on Travis CI

yongtang force-pushed the travis branch 18 times, most recently from 6a939a7 to 6b740dd Compare February 24, 2019 14:59

yongtang mentioned this pull request Feb 25, 2019

Fix tiff lib and remove stub ffmpeg libraries for macOS support #119

Merged

yongtang added 3 commits February 26, 2019 04:52

Enable MacOS build in Travis CI

d53da89

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Enable R test for whl installation

85cc4db

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Add mising R dependency, and improve the test condition.

5b13be7

Encountered a scneario where all tests are skipped Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

yongtang force-pushed the travis branch from 506f61e to 5b13be7 Compare February 26, 2019 15:59

yongtang changed the title ~~[WIP] Enable MacOS build on Travis CI~~ Enable MacOS build on Travis CI Feb 26, 2019

yongtang force-pushed the travis branch 2 times, most recently from 73cc59b to 2fb3818 Compare February 26, 2019 20:36

Enable Mac build for all ops (tests are skipped for now)

3e77165

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

yongtang force-pushed the travis branch from 2fb3818 to 3e77165 Compare February 26, 2019 20:52

BryanCutler reviewed Feb 26, 2019

View reviewed changes

Update to use python version explicitly

852a9a0

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

yongtang added 4 commits February 27, 2019 03:37

Enable localstack/Kinesis, Kafka, Ignite and tests

d690475

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Use after_success so that exit will not trigger nightly upload

fe231b2

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Change required package to >=1.13.0,<1.14.0

c215710

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Change librdkafka to v0.11.6

3aeb75b

Hopefully to fix kafka trap 6 issue on macOS (happens on 1.13.0rc2) Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

yongtang mentioned this pull request Feb 27, 2019

[WIP] Use buildpack-deps as the base image to speed up Travis CI #112

Closed

BryanCutler approved these changes Feb 27, 2019

View reviewed changes

yongtang merged commit 937acfc into tensorflow:master Feb 27, 2019

yongtang deleted the travis branch February 27, 2019 20:47

yongtang mentioned this pull request Mar 2, 2019

Build tensorflow-io package on MacOS #4

Closed

i-ony pushed a commit to i-ony/io that referenced this pull request Feb 8, 2021

Merge pull request tensorflow#116 from yongtang/travis

5b934cf

Enable MacOS build on Travis CI

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable MacOS build on Travis CI #116

Enable MacOS build on Travis CI #116

yongtang commented Feb 24, 2019 •

edited

Loading

yongtang commented Feb 24, 2019

yongtang commented Feb 26, 2019

BryanCutler left a comment

BryanCutler Feb 26, 2019

yongtang Feb 26, 2019

BryanCutler commented Feb 26, 2019 via email

yongtang commented Feb 27, 2019

yongtang commented Feb 27, 2019

yongtang commented Feb 27, 2019

BryanCutler left a comment

BryanCutler Feb 27, 2019

yongtang Feb 27, 2019

terrytangyuan commented Feb 27, 2019

damienpontifex commented Feb 27, 2019

yongtang commented Feb 27, 2019

Enable MacOS build on Travis CI #116

Enable MacOS build on Travis CI #116

Conversation

yongtang commented Feb 24, 2019 • edited Loading

yongtang commented Feb 24, 2019

yongtang commented Feb 26, 2019

BryanCutler left a comment

Choose a reason for hiding this comment

BryanCutler Feb 26, 2019

Choose a reason for hiding this comment

yongtang Feb 26, 2019

Choose a reason for hiding this comment

BryanCutler commented Feb 26, 2019 via email

yongtang commented Feb 27, 2019

yongtang commented Feb 27, 2019

yongtang commented Feb 27, 2019

BryanCutler left a comment

Choose a reason for hiding this comment

BryanCutler Feb 27, 2019

Choose a reason for hiding this comment

yongtang Feb 27, 2019

Choose a reason for hiding this comment

terrytangyuan commented Feb 27, 2019

damienpontifex commented Feb 27, 2019

yongtang commented Feb 27, 2019

yongtang commented Feb 24, 2019 •

edited

Loading