Upgrade intervention required
This release adds a feature to ban channels, replacing the earlier (hastily
added) configuration-based channel-blacklist
. If you have any entries in
channel-blacklist
in your config.yaml
, you will need to migrate them to the
new bans table by using a command after upgrading (the ACP web interface hasn't
been updated for this feature):
./bin/admin.js ban-channel <channel-name> <external-reason> <internal-reason>
The external reason will be displayed when users attempt to join the banned
channel, while the internal reason is only displayed when using the
show-channel-ban
command.
You can later use unban-channel
to remove a ban. The owner of the banned
channel can still delete it, but the banned state will persist, so the channel
cannot be re-registered later.
This release integrates Xaekai's added support for Bandcamp, BitChute, Odysee, and Nicovideo playback support into the main repository. The updated support for custom fonts and audio tracks in custom media manifests is also included, but does not work out of the box -- it requires a separate channel script; this may be addressed in the future.
CyTube has been upgraded to socket.io v4 (from v2).
Breaking change: Newer versions of socket.io require CORS to validate the
origin initiating the socket connection. CyTube allows the origins specified in
the io.domain
and https.domain
configuration keys by default, which should
work for many use cases, however, if you host your website on a different domain
than the socket connection, you will need to configure the allowed origins (see
config.template.yaml under io.cors
).
CyTube enables the allowEIO3
configuration in socket.io by default, which
means that existing clients and bots using socket.io-client v2 should continue
to work.
The legacy metrics recorder (counters.log
file) has been removed. For over 4
years now, CyTube has integrated with Prometheus,
which provides a superior way to monitor the application. Copy
conf/example/prometheus.toml
to conf/prometheus.toml
and edit it to
configure CyTube's Prometheus support.
Due to changes in Soundcloud's authorization scheme, support has been dropped from core due to requiring each server owner to register an API key (which is currently impossible as they have not accepted new API key registrations for years).
If you happen to already have an API key registered, or if Soundcloud reopens registration at some point in the future, feel free to reach out to me for patches to reintroduce support for it.
Some of CyTube's dependencies depends on features in newer versions of node.js. Accordingly, node 10 is no longer supported. Administrators are recommended to use node 12 (the active LTS), or node 14 (the current version).
Twitch has updated their embed player, which adds new requirements for embedding Twitch:
- The origin website must be served over HTTPS
- The origin website must be served over the default port (i.e., the hostname cannot include a port; https://example.com:8443 won't work)
Additionally, third-party cookies must be enabled for whatever internal subdomains Twitch is using.
CyTube now sets the parameters expected by Twitch, and displays an error message if it detects (1) or (2) above are not met.
Old versions of CyTube defaulted to storing channel state in flatfiles located
in the chandump
directory. The default was changed a while ago, and the
flatfile storage mechanism has now been removed.
Admins who have not already migrated their installation to the "database" channel storage type can do so by following these instructions:
- Run
git checkout e3a9915b454b32e49d3871c94c839899f809520a
to temporarily switch to temporarily revert to the previous version of the code that supports the "file" channel storage type - Run
npm run build-server
to build the old version - Run
node lib/channel-storage/migrator.js |& tee migration.log
to migrate channel state from files to the database - Inspect the output of the migration tool for errors
- Set
channel-storage
/type
to"database"
inconfig.yaml
and start the server. Load a channel to verify the migration worked as expected - Upgrade back to the latest version with
git checkout 3.0
andnpm run build-server
- Remove the
channel-storage
block fromconfig.yaml
and remove thechandump
directory since it is no longer needed (you may wish to archive it somewhere in case you later discover the migration didn't work as expected).
If you encounter any errors during the process, please file an issue on GitHub
and attach the output of the migration tool (which if you use the above commands
will be written to migration.log
).
In accordance with node v8 LTS becoming end-of-life on 2019-12-31, CyTube no longer supports v8.
Please upgrade to v10 or v12 (active LTS); refer to https://nodejs.org/en/about/releases/ for the node.js support timelines.
Users can now self-service request their account to be deleted, and it will be
automatically purged after 7 days. In order to send a notification email to
the user about the request, copy the email
configuration
to conf/email.toml
(the same file used for password reset emails).
The sanitize-html
dependency has made a change that results in "
no longer
being replaced by "
when not inside an HTML attribute value. This
potentially breaks any chat filters matching quotes as "
(on my
particular instance, this seems to be quite rare). These filters will need to
be updated in order to continue matching quotes.
Support for node.js 6.x has been dropped, in order to bump the babel preset to generate more efficient code (8.x supports async-await and other ES6+ features natively and is the current node.js LTS).
If you are unable to upgrade to node.js 8.x, you can revert the changes to package.json in this commit, however, be warned that I no longer test on 6.x.
In order to support node.js 10, the bcrypt
dependency has been upgraded to
version 2. bcrypt
version 2 defaults to the $2b$
algorithm, whereas version
1 defaults to the $2a$
algorithm. Existing password hashes will continue to
be readable, however hashes created with version 2 will not be readable by
version 1. See https://github.com/kelektiv/node.bcrypt.js for details.
In addition, the optional dependency on v8-profiler
has been removed, since
this is not compatible with newer versions of v8.
In accordance with the node.js release schedule, node.js 4.x, 5.x, 7.x, and 9.x are end-of-life and are no longer maintained upstream. Accordingly, these versions are no longer supported by CyTube.
Please upgrade to 8.x (LTS) or 10.x (current). 6.x is still supported, but is in the "maintenance" phase upstream, and should be phased out.
Build changes: When the babel
dependency was first added to transpile ES6
code to ES5, an interactive prompt was added to the postinstall
script before
transpilation, in case the user had made local modifications to the files in
lib
which previously would have been detected as a git conflict when pulling.
It has now been sufficiently long that this is no longer needed, so I've removed
it. As always, users wishing to make local modifications (or forks) should edit
the code in src/
and run npm run build-server
to regenerate lib/
.
This commit also removes the bundled www/js/player.js
file in favor of having
postinstall
generate it from the sources in player/
.
As of December 2017, Vid.me is no longer in service. Accordingly, Vid.me support in CyTube has been deprecated.
The Google Drive userscript has been updated once again. Violentmonkey is now explicitly supported. Google login redirects are caught and handled. See directly below on how to regenerate the user script again.
The Google Drive userscript has been updated due to breaking changes in Greasemonkey 4.0. Remember to generate the script by running:
$ npm run generate-userscript "Your Site Name" http://your-site.example.com/r/*
The latest commit introduces a referrer check in the account page handlers. This is added as a short-term mitigation for a recent report that account management functions (such as deleting channels) can be executed without the user's consent if placed in channel JS.
Longer term options are being considered, such as moving account management to a separate subdomain to take advantage of cross-origin checks in browsers, and requiring the user to re-enter their password to demonstrate intent. As always, I recommend admins take extreme caution when accepting channel JS.
Breaking change: the nodemailer
dependency has been upgraded to version
4.x. I also took this opportunity to make some modifications to the email
configuration and move it out of config.yaml
to conf/email.toml
.
To upgrade:
- Run
npm upgrade
(orrm -rf node_modules; npm install
) - Copy
conf/example/email.toml
toconf/email.toml
- Edit
conf/email.toml
to your liking - Remove the
mail:
block fromconfig.yaml
This feature only supports sending via SMTP for now. If there is demand for other transports, feel free to open an issue or submit a pull request.
The /useragreement
default page has been removed. Server administrators can
substitute their own terms of service page by editing templates/footer.pug
This commit removes an old kludge that redirected users to HTTPS (when enabled)
specifically for the account authorization pages (e.g., /login
). The code for
doing this was to work around limitations that no longer exist, and does not
represent current security best practices.
The recommended solution to ensure that users are logged in securely (assuming
you've configured support for HTTPS) is to use
Strict-Transport-Security
to direct browsers to access the HTTPS version of the website at all times. You
can enable this by configuring a reverse proxy (e.g. nginx) in front of CyTube
to intercept HTTP traffic and redirect it to HTTPS, and add the
Strict-Transport-Security
header when returning the response from CyTube.
Support for the old version of Vimeo's OAuth API (the vimeo-oauth
configuration block) has been dropped. It's unlikely anyone was using this,
since you haven't been able to register new API keys for it in years (it was
superseded by a newer OAuth API, which CyTube does not support), and in fact I
lost my credentials for this API and no longer have a way to test it.
Vimeo videos can still be added -- the metadata will be queried from the anonymous API which has been the default since the beginning.
The stats
database table and associated ACP subpage have been removed in favor
of integration with Prometheus. You can enable
Prometheus reporting by copying conf/example/prometheus.toml
to
conf/prometheus.toml
and editing it to your liking. I recommend integrating
Prometheus with Grafana for dashboarding needs.
The particular metrics that were saved in the stats
table are reported by the
following Prometheus metrics:
- Channel count:
cytube_channels_num_active
gauge. - User count:
cytube_sockets_num_connected
gauge (labeled by socket.io transport). - CPU/Memory: default metrics emitted by the
prom-client
module.
More Prometheus metrics will be added in the future to make CyTube easier to monitor :)
The latest commit upgrades socket.io
to version 2.0, a major version change
from 1.4. This release improves performance by switching to uws
for the
websocket transport, and fixes several bugs; you can read about it
here.
For browser clients, the upgrade should basically just work with no
intervention. For node.js clients, all that is needed is to upgrade
socket.io-client
to 2.0. For other clients, work required may vary depending
on whether the implementation has compatibility problems with 2.0.
The latest commit drops support for node.js versions below 6 (the current LTS). This is to allow the babel preset to avoid generating inefficient code to polyfill ES2015+ features that are now implemented in the node.js core.
New versions of node.js can be downloaded from the node.js website, if they are not already available in your distribution's package manager.
Polls are now more strictly validated, including the number of options. The
default limit is 50 options, which you can configure via poll.max-options
.
Commit f8183bea1b37154d79db741ac2845adf282e7514 modifes the schema of the
users
table to include a new column (name_dedupe
) which has a UNIQUE
constraint. This column is populated with a modified version of the user's name
to prevent the registration of usernames which are bitwise distinct but visually
similar. 'l', 'L', and '1' are all mapped to '1'; 'o', 'O', and '0' are all
mapped to '0'; '_' and '-' are mapped to '_'. On first startup after
upgrading, the new column will be added and populated.
This replaces the earlier solution which was put in place to mitigate PR#489 but was overly-restrictive since it wildcarded these characters against any character, not just characters in the same group.
The dependency on sanitize-html
, which previously pointed to a fork, has now
been switched back to the upstream module. XSS filtering has been turned off
for the chat filter replacement itself (since this provides no additional
security), and is now only run on the final chat message after filtering.
Certain chat filters and MOTDs which relied on syntactically incorrect HTML,
such as unclosed tags, may have different behavior now, since sanitize-html
fixes these.
After upgrading the dependency on yamljs
, you may see this error if you didn't
notice and correct a typo in the config.yaml template:
Error loading config file config.yaml:
{ [Error: Unexpected characters near ",".]
message: 'Unexpected characters near ",".',
parsedLine: 88,
snippet: 'title: \'CyTube\',' }
The fix is to edit config.yaml and remove the trailing comma for the title:
property under html-template
. If there are other syntax errors that the old
version didn't detect, you will need to correct those as well.
Longer term, I am looking to move away from using yamljs
to parse
configuration because it's a little buggy and the current configuration system
is confusing.
Google Drive changed the URL schema for retrieving video metadata, which broke CyTube's Google Drive support, even with the userscript. I have updated the userscript source with the new URL, so server administrators will have to regenerate the userscript for their site and users will be prompted to install the newer version.
Additionally, fixing Drive lookups required an update to the mediaquery
module, so you will have to do an npm install
to pull that fix in.
A few weeks ago, the previous Google Drive player stopped working. This is nothing new; Google Drive has consistently broken a few times a year ever since support for it was added. However, it's becoming increasingly difficult and complicated to provide good support for Google Drive, so I've made the decision to phase out the native player and require a userscript for it, in order to bypass CORS and allow each browser to request the video stream itself.
See the updated documentation for details on how to enable this for your users.
A new dependency has been added on cytube-common
, a module that will hold
common code shared between the current version of CyTube and the upcoming work
around splitting it into multiple services. You will need to be sure to run
npm install
after pulling in this change to pull in the new dependency.
This release updates socket.io to version 1.4.0. The updates to socket.io
include a few security-related fixes, so please be sure to run npm install
to ensure the updated version is installed before restarting your CyTube server.
- https://nodesecurity.io/advisories/67
- https://github.com/socketio/engine.io/commit/391ce0dc8b88a6609d88db83ea064040a05ab803
In order to support future clustering support, the legacy /sioconfig
endpoint is being deprecated. Instead, you should make a request to
/socketconfig/<channel name>.json
. See the
documentation for more information.
-
The channel data storage system has been refactored a bit. For compatibility, the default remains to store JSON objects for each channel in the
chandump
folder, however there is now also the option of storing channel data in the database. You can take advantage of this by settingchannel-storage: type: 'database'
in yourconfig.yaml
.- In order to migrate existing channel data from the
chandump
files to the database, runnode lib/channel-storage/migrate.js
.
- In order to migrate existing channel data from the
-
The database storage method uses foreign keys to associate the channel data with the corresponding row in the
channels
table. This requires that the tables be stored using the InnoDB engine rather than MyISAM. If your CyTube tables defaulted to MyISAM, you can fix them by runningALTER TABLE `channels` ENGINE = InnoDB;
- CyTube is now transpiled with babel to allow the use of ES6/ES2015
features. All source files have been moved from
lib
tosrc
. - Running
npm install
ornpm run postinstall
will prompt you to build fromsrc
tolib
. - Running
npm run build-server
will run the build script without any prompts. - After updating with
git pull
, you should runnpm install
ornpm run build-server
in order to rebuild after the changes.
- CyTube now supports subtitles for Google Drive videos. In order to take
advantage of this, you must upgrade mediaquery by running
npm install cytube/mediaquery
. Subtitles are cached in the google-drive-subtitles folder.
- CyTube and CyTube/mediaquery have both been updated to use
calzoneman/status-message-polyfill to polyfill res.statusMessage on older
versions of node (e.g., v0.10). After pulling, run
npm install
to update this dependency. This fixes an issue where HTTP status messages from mediaquery were reported asundefined
, and removes the need for manually looking up status messages inlib/ffmpeg.js
.
-
As part of the video player rewrite, Google Drive and Google+ metadata lookups are now offloaded to CyTube/mediaquery. After pulling the new changes, run
npm install
ornpm update
to update the mediaquery dependency. -
www/js/player.js
is now built from the CoffeeScript source files in theplayer/
directory. Instead of modifying it directly, modify the relevant player implementations inplayer/
and runnpm run build-player
(ornode build-player.js
) to generatewww/js/player.js
. -
Also as part of the video player rewrite, the schema for custom embeds changed so any custom embeds stored in the
channel_libraries
table need to be updated. The automatic upgrade script will convert any custom embeds that are parseable (i.e., not truncated by the width of theid
field using the old format) and will delete the rest (you may see a lot of WARNING: unable to convert xxx messages-- this is normal). Custom embeds in channel playlists in the chandumps will be converted when the channel is loaded.