Add FFmpeg support to open video files #30

yongtang · 2018-12-16T17:45:13Z

This fix adds a preliminary video file support (VideoDataset) by linking and using FFmpeg's libraries.

The video.VideoDataset takes input of filename list, and generates outputs of RGB24 images (uint8) in (height, width, 3).

The reason to link against FFmpeg's libraries is because FFmpeg has a LGPL 2.1+ license.

On Ubuntu 18.04, FFmpeg libraries consists of:

libavformat.so.57
libavcodec.so.57
libavutil.so.55
libswscale.so.4

Because FFmpeg only guarantee's major version conpatibility, linking to a lower version (e.g., libavformat.so.54 on Ubuntu 14.04) may not work.

To support multiple versions of Ubuntu, multiple copies of header file combined with linking to different versions of .so might be needed.

It needs to be fully tested though and only limited major version should be supported.

~~At the moment, only ffmpeg libraries in Ubuntu 18.04 are used.~~
UPDATE:
At the moment, Ubuntu 14.04, 16.04, and 18.04 are all supported. We build 3 .so files and distributed at the same time:

_video_ops_ffmpeg_3.4.so (Ubuntu 18.04)
_video_ops_ffmpeg_2.8.so (Ubuntu 16.04)
_video_ops_libav_9.20.so (Ubuntu 14.04)

Additional platform could be added if there is enough need from the community.

To install required libraries on Ubuntu 18.04:

apt-get -y -qq install libavformat57 libavcodec57 libavutil55 libswscale4

To build the pip package:

bazel build -s --verbose_failures build_pip_pkg

NOTE: This PR fixed #11

Signed-off-by: Yong Tang [email protected]

googlebot · 2018-12-17T01:04:25Z

So there's good news and bad news.

👍 The good news is that everyone that needs to sign a CLA (the pull request submitter and all commit authors) have done so. Everything is all good there.

😕 The bad news is that it appears that one or more commits were authored or co-authored by someone other than the pull request submitter. We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that here in the pull request.

Note to project maintainer: This is a terminal state, meaning the cla/google commit status will not change from this state. It's up to you to confirm consent of all the commit author(s), set the cla label to yes (if enabled on your project), and then merge this pull request when appropriate.

googlebot · 2018-12-17T01:11:53Z

CLAs look good, thanks!

terrytangyuan · 2018-12-17T03:37:30Z

tensorflow_io/video/python/kernel_tests/video_test.py

+# License for the specific language governing permissions and limitations under
+# the License.
+# ==============================================================================
+"""Tests for SequenceFileDataset."""


Update this to VideoDataset instead of SequenceFileDataset? Same for other places

Thanks @terrytangyuan, the PR has been updated.

yongtang · 2018-12-18T21:32:57Z

I think this PR is almost ready. I am working on adding Ubuntu 16.04 and 14.04 support (on top of 18.04 support that is already in place). Will update the PR shortly to completion.

yongtang · 2018-12-18T23:46:41Z

I think this PR is ready. We support Ubuntu 14.04, 16.04, and 18.04 at the moment, with one distributed package in pip. Tests also pass on those platforms.

martinwicke · 2018-12-19T00:32:11Z

This is a thing of beauty. I especially appreciate the multi-version tooling.

Since these are LGPL, we have to make sure to only release dynamically linked versions. Can we make sure (with a test/scripting) that this is always going ot be the case?

yongtang · 2018-12-19T19:37:25Z

@martinwicke The library only uses header files from ffmpeg to compile and build. In Bazel there is a check-license option though I don't know how reliable it is. I think some additional tooling might be possible (maybe a module in bazel) to enforce only header files are included for LGPL types of libraries .

I will take a look at the license tooling issue.

martinwicke · 2018-12-19T19:39:08Z

I think as long as we make sure all releases are built with dynamic linking, we should be ok.

This fix adds a preliminary video file support (`VideoDataset`) by linking and using FFmpeg's libraries. The `video.VideoDataset` takes input of filename list, and generates outputs of RGB24 images (`uint8`) in `(height, width, 3)`. The reason to link against FFmpeg's libraries is because FFmpeg has a LGPL 2.1+ license. On Ubuntu 18.04, FFmpeg libraries consists of: ``` libavformat.so.57 libavcodec.so.57 libavutil.so.55 libswscale.so.4 ``` Because FFmpeg only guarantee's major version conpatibility, linking to a lower version (e.g., libavformat.so.54 on Ubuntu 14.04) may not work. To support multiple versions of Ubuntu, multiple copies of header file combined with linking to different versions of .so might be needed. It needs to be fully tested though and only limited major version should be supported. At the moment, only ffmpeg libraries in Ubuntu 18.04 are used. To install required libraries on Ubuntu 18.04: ``` apt-get -y -qq install libavformat57 libavcodec57 libavutil55 libswscale4 ``` To build the pip package: ``` bazel build -s --verbose_failures build_pip_pkg ``` Signed-off-by: Yong Tang <[email protected]>

Signed-off-by: Yong Tang <[email protected]>

So that build could always succeed. In runtime system .so files will be used. Signed-off-by: Yong Tang <[email protected]>

…ave Ubuntu 18.04 support directly. Signed-off-by: Yong Tang <[email protected]>

Encountered quite some issues with ``` Use --sandbox_debug to see verbose messages from the sandbox /usr/bin/ld.gold: error: cannot find libavformat.so.57 /usr/bin/ld.gold: error: cannot find libavcodec.so.57 /usr/bin/ld.gold: error: cannot find libavutil.so.55 /usr/bin/ld.gold: error: cannot find libswscale.so.4 collect2: error: ld returned 1 exit status Target //tensorflow_io/video:video_py_test failed to build ``` Believe it is related to (or similiar): https://stackoverflow.com/questions/52386530/linker-fails-in-sandbox-when-running-through-bazel-but-works-when-sandboxed-comm?rq=1 Signed-off-by: Yong Tang <[email protected]>

(By default Travis CI uses Ruby) Signed-off-by: Yong Tang <[email protected]>

…ainer Signed-off-by: Yong Tang <[email protected]>

Signed-off-by: Yong Tang <[email protected]>

…nore error Signed-off-by: Yong Tang <[email protected]>

Signed-off-by: Yong Tang <[email protected]>

…brary and use `cc_import_library` rule to prevent including GPL code. Signed-off-by: Yong Tang <[email protected]>

Signed-off-by: Yong Tang <[email protected]>

yongtang · 2019-01-08T23:03:52Z

@terrytangyuan @martinwicke The PR has been updated.

I added a customized implementation of bazel rule def (cc_import_library) which controls the dependency so that library will only be linked to, and no source files (like GPL) are included or compiled.

It is pretty much similar to cc_import in Bazel's standard rule, except that cc_import does not support .so with soname. I created an issue in bazel repo: bazelbuild/bazel#7059

I think this FFmpeg PR is ready, please take a look and let me know if there are any issues.

terrytangyuan

LGTM!

yongtang · 2019-01-13T17:49:32Z

Thanks. Will merge this PR in, and have a follow up PR with an example.

* feat: reading from bigtable (#2) Implements reading from bigtable in a synchronous manner. * feat: RowRange and RowSet API. * feat: parallel read (#4) In this pr we make the read methods accept a row_set reading only rows specified by the user. We also add a parallel read, that leverages the sample_row_keys method to split work among workers. * feat: version filters (#6) This PR adds support for Bigtable version filters. * feat: support for other data types (#5) * fix: linter fixes (#8) * feat docs (#9) * fix: building on windows (#12) * fix: refactor bigtable package to api folder (#14) moved bigtable to tfensorflow_io.python.api * fix: tests hanging (#30) changed path to bigtable emulator and cbt in tests moved arguments' initializations to the body of the function in bigtable_ops.py fixed interleaveFromRange of column filters when using only one column * fix: temporarily disable macos tests (#32) * disable tests on macos Co-authored-by: Kajetan Boroszko <[email protected]> Co-authored-by: Kajetan Boroszko <[email protected]>

yongtang mentioned this pull request Dec 16, 2018

Support video format for tensorflow-io #11

Closed

yongtang force-pushed the ffmpeg branch 2 times, most recently from 7c8fc9c to 66df035 Compare December 17, 2018 01:04

yongtang force-pushed the ffmpeg branch from 66df035 to aabf043 Compare December 17, 2018 01:11

terrytangyuan reviewed Dec 17, 2018

View reviewed changes

yongtang force-pushed the ffmpeg branch 11 times, most recently from 272d247 to abadf38 Compare December 18, 2018 14:14

yongtang changed the title ~~[WIP] Add FFmpeg support to open video files~~ Add FFmpeg support to open video files Dec 18, 2018

yongtang force-pushed the ffmpeg branch 2 times, most recently from a6c0ae3 to a928008 Compare December 18, 2018 23:10

yongtang mentioned this pull request Dec 18, 2018

tf.contrib.ffmpeg: No mention of alternatives after deprecation tensorflow/tensorflow#24424

Closed

yongtang force-pushed the ffmpeg branch from 8567891 to e7ee4de Compare January 5, 2019 04:52

yongtang added 2 commits January 8, 2019 18:58

Add python wrapper for video.VideoDataset

9a500e5

Signed-off-by: Yong Tang <[email protected]>

yongtang added 16 commits January 8, 2019 18:58

Add test case for video.VideoDataset

fdf7499

Signed-off-by: Yong Tang <[email protected]>

Add BUILD file for tensorflow_io.video

cd22417

Signed-off-by: Yong Tang <[email protected]>

Add ffmpeg n3.4.4 dependency

6dcdaa5

Signed-off-by: Yong Tang <[email protected]>

Update BUILD file to include video.VideoDataset

15b2c43

Signed-off-by: Yong Tang <[email protected]>

Disable -//tensorflow_io/video:video_py_test test on Ubuntu 14.04

1df47e2

Signed-off-by: Yong Tang <[email protected]>

Update ffmpeg to use stub .so files

738b8ea

So that build could always succeed. In runtime system .so files will be used. Signed-off-by: Yong Tang <[email protected]>

Add Travis CI for Ubuntu 18.04, use docker run as Travis does not h…

d218d35

…ave Ubuntu 18.04 support directly. Signed-off-by: Yong Tang <[email protected]>

Use python as base vm image for Travis CI Ubuntu 18.04

29bb7dc

(By default Travis CI uses Ruby) Signed-off-by: Yong Tang <[email protected]>

Only Enable Video on Travis CI for Ubuntu 18.04, as it is inside cont…

f26e4ee

…ainer Signed-off-by: Yong Tang <[email protected]>

Add Ubuntu 16.04 support, by build separate so with ffmpeg 2.8

a113c69

Signed-off-by: Yong Tang <[email protected]>

Fix additional issue in ffmpeg 3.4 where cached packet should just ig…

bdb85d3

…nore error Signed-off-by: Yong Tang <[email protected]>

Add Ubuntu 14.04 support, with libav 9.20 built in.

f3884ae

Signed-off-by: Yong Tang <[email protected]>

Add libav dependency and enable video test on Ubuntu 14.04

ccc8a20

Signed-off-by: Yong Tang <[email protected]>

Enable Ubuntu 16.04 for video.VideoDataset

a5cbf32

Signed-off-by: Yong Tang <[email protected]>

Defined a bazel rule of cc_import_library to only accepts system li…

10a402d

…brary and use `cc_import_library` rule to prevent including GPL code. Signed-off-by: Yong Tang <[email protected]>

yongtang force-pushed the ffmpeg branch from e7ee4de to 10a402d Compare January 8, 2019 19:09

Reduce verbose logging output by bazel

4da4dd2

Signed-off-by: Yong Tang <[email protected]>

yongtang mentioned this pull request Jan 12, 2019

Support audio format for tensorflow-io #49

Closed

terrytangyuan approved these changes Jan 12, 2019

View reviewed changes

yongtang mentioned this pull request Jan 12, 2019

Support live video for tensorflow-io #51

Open

yongtang merged commit 5fac290 into tensorflow:master Jan 13, 2019

yongtang deleted the ffmpeg branch January 13, 2019 17:49

wookayin mentioned this pull request Mar 11, 2019

Support decoding video from byte stream for tensorflow-io #144

Closed

missaugustina mentioned this pull request May 8, 2019

Create docs/tutorial/ consistent with other TF projects #227

Closed

smellslikeml mentioned this pull request Apr 2, 2021

FFmpeg scene change detection eto-ai/rikai#156

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add FFmpeg support to open video files #30

Add FFmpeg support to open video files #30

yongtang commented Dec 16, 2018 •

edited

Loading

googlebot commented Dec 17, 2018

googlebot commented Dec 17, 2018

terrytangyuan Dec 17, 2018

yongtang Dec 17, 2018

yongtang commented Dec 18, 2018

yongtang commented Dec 18, 2018

martinwicke commented Dec 19, 2018

yongtang commented Dec 19, 2018

martinwicke commented Dec 19, 2018

yongtang commented Jan 8, 2019

terrytangyuan left a comment •

edited

Loading

yongtang commented Jan 13, 2019

Add FFmpeg support to open video files #30

Add FFmpeg support to open video files #30

Conversation

yongtang commented Dec 16, 2018 • edited Loading

googlebot commented Dec 17, 2018

googlebot commented Dec 17, 2018

terrytangyuan Dec 17, 2018

Choose a reason for hiding this comment

yongtang Dec 17, 2018

Choose a reason for hiding this comment

yongtang commented Dec 18, 2018

yongtang commented Dec 18, 2018

martinwicke commented Dec 19, 2018

yongtang commented Dec 19, 2018

martinwicke commented Dec 19, 2018

yongtang commented Jan 8, 2019

terrytangyuan left a comment • edited Loading

Choose a reason for hiding this comment

yongtang commented Jan 13, 2019

yongtang commented Dec 16, 2018 •

edited

Loading

terrytangyuan left a comment •

edited

Loading