6.3.0

Version 6.3.0 Released 2023-01-26

(Nearly a full year since the previous release! The diff is 19,773 lines long!)

User support and user interface improvements:

• The S3 DataProvider class has been extended to fully support the
  browse_path feature, allowing files to be registered at arbitrary
  depth within the object namespace tree of the S3 bucket.
• The launch button in the file manager was improved so that external
  sites can directly link to a prepared pair of dataset/tool. Used
  by the CONP project, mostly.
• We added the standard 'This site use a cookie' banner.
• The globally visible list of available tools and datasets was
  cleaned up and re-arranged in two tabs, each with two tables for
  the public and restricted thingies.
• Added a new userfile model in the base distribution: ZipArchive.
• Special API hooks for the LORIS projects were adjusted; although
  they can also be used by non LORIS actors, they're pretty specific.

Admin and codebase improvements:

• A new type of user called an 'AutomatedUser' has been added; it
  is basically the same as a NormalUser, but the type can help admins
  identify accounts that are meant to be accessed by automated
  systems (API calls etc).
• A DataUsage model was added to track and count how often files
  are being downloaded, copied, used in processing, or viewed. Admins
  can selectively enable this on a project by project basis, and
  counts are aggregated on a user + month-by-month basis.
• A DiskQuota model was added, it allows administrator to impose
  limits on the number of files and their total sizes for any user
  on any data provider.
• The communication channels between the portal and the bourreau
  are now completely performed by setting up UNIX-domain sockets
  on the Bourreau side. No longer do we open a network port on
  localhost! The connections are established by proper -L and -R
  SSH options, which now support such sockets.
• A new rake task help developers and admins manage their CBRAIN
  instances (cbrain:models:broken:*)
• The boot mechanism for Bourreau was rewritten as plain bash shell
  wrappers to allow a faster startup than then old Ruby bootstrapping
  code. The Bourreau is still in Ruby, of course, but prepping it
  up no longer requires a costly initial Ruby setup script.
• Speaking of the boot system, Bourreaux servers now launch a
  separate watchdog process (also a bash script) that will ping the
  Bourreau every 20 minutes and force it to shutdown if the DB
  connection (or any SSH tunnel) is shut down unexpectedly. This
  keep the PID file from staying around for no reason.
• We removed from the GitHub-hosted codebase the hardcoded cookie
  secret keys; these were never really a security issue (given in
  'production' mode the admin was supposed to create them), but for
  convenience now any true production or development CBRAIN system
  will generate their own secret key deterministically (yet in a
  non-guessable way).
• Admins can force specific users to not only link their account
  to a GlobusAuth provider, it can also be a specific provider chosen
  by the admin. And once the linkage is done, the password method
  is permanently disabled for such users.
• We cleaned up a bunch of system attributes that are no longer
  used (like port numbers for DB and ActiveResource connections to
  Bourreaux, which are now always tunnelled through SSH)
• Admins users have access to the new 'last' command in the console,
  and the 'p' (ping) command in the ibc interface.
• Support for Apptainer as the new Singularity engine.
• Bourreaux can be configured to log to an external file some
  information about each job submitted (user, jobid, name, user
  Globus name, etc).
• When configuring a ToolConfig, the admin no longer has to explicitely
  duplicate the environment variables that the tool needs depending
  on whether the tools runs in Singularity/Apptainer or not. Before,
  the admins had to set both XYZ=a and SINGULARITYENV_XYZ=a, now
  it's done automatically.
• Admins can now visualize directly in the interface the Boutiques
  descriptor associated with a particular ToolConfig, for a tool
  configured with the new integrator.
• We cleaned up (removed) most of the controller actions that were
  required by the Canadian agency that initially funded CBRAIN (CANARIE).
  These were being monitored by them but they discontinued their
  side.
• Admins can create notification messages that will show up in the
  dashboard of all users (e.g. notice for downtime etc)

Boutiques improvements:

• The (relatively) new Boutiques integrator has been extended with lots
  of modules to let integrators customize the behavior of their tools:

  • BoutiquesAllowedExitCodes
  • BoutiquesFileNameMatcher
  • BoutiquesForcedOutputBrowsePath
  • BoutiquesInputCacheCleaner
  • BoutiquesInputSubdirMaker
  • BoutiquesOutputCacheCleaner
  • BoutiquesOutputFilenameRenamer

• Launching task arrays with a CbrainFileList now allows the user
  to provide extra parameters specific to each row in the file list.
  To do so, the file list should be an ExtendedCbrainFileList and
  the last column should contain a serialized JSON structure that
  can merge to the Boutiques parameters of the task.

• A new tool BoutiquesDescriptorMaker is provided as part of the
  base distribution. It allows a developer to test 'live' what a
  Boutiques descriptor would look like in CBRAIN.

6.2.0

Version 6.2.0 Released 2022-01-28

(After eleven months, the git diff output is over 12,000 lines long!)

Major features:

• A new Boutiques integrator was implemented. This new framework
  is modular and customizable, and much easier to maintain.
  from the developer's perspective. Some technical documentation
  is in this presentation.
• The new Boutiques integrator comes with four new modules that sysadmins
  can configure within each Boutiques descriptor:
  • BoutiquesFileNameMatcher
  • BoutiquesFileTypeVerifier
  • BoutiquesOutputFileTypeSetter
  • BoutiquesPostProcessingCleaner
• Users can link their account to a Globus identity and
  authenticate with Globus from that point on.
• A new DataProvider types has been added: SingBindMountDataProvider.
  It access files in squashfs or ext3 overlays using the singularity
  bindmount feature.
• The NeuroHub interface now shows the user's own private project.
• A new 'stream' action has been added to the userfile model,
  allowing seemless path-based access to file contents.
• Sysadmins have access to a set of new rake tasks to extract old record
  from the resource_usage table, and get them saved into YAML files.
  Replacement monthly summary records can then be reinserted in the database
  so that the usage tracking for all resources stay accurate.

Other enhancements:

• New NOC status and statistics pages: users over time, tools usage
  over time, tasks CPU time over time.
• The main login page has a public list of configured tools and datasets.
• The data structure that maintains statistics about the portal
  requests has been simplified.
• S3 data providers can be onfigued with distinct regions and endpoints.
• A new class of users, AutomatedUser, is available for automatic systems.
• Admins can configure overlays for Singularity-launched tools
  by provding the overlays as userfiles.
• A new tool, SimpleFileExtractor, is provided with the distribution.

6.1.0

CBRAIN/NeuroHub Release Notes

Version 6.1.0 Released 2021-02-26

This release contains several new features and bug fixes.

• The Continuous Integration system has been switched from
  Travis CI to GitHub actions. The control script for the
  running the test suite is under .github/workflows/cbrain_ci.yaml
• The remote script that starts a Bourreau has been switched
  from a ruby script to a simpler bash script; the bash script
  is used in the most common situation of launching a Bourreau,
  and the old ruby script is still invoked in other rarer modes
  of operation.
• Browsing data providers now support 'local subpaths'. This
  is enabled only on some select DataProvider subclasses.
  This mechanism is meant for read-only data providers only,
  and can only be accessed by administrators. Warning: registering
  files that are subsets of other registered files has
  un-predictable consequences.
• A new specialized controller for LORIS hooks has been added.
  This is used by automated operations from LORIS systems.
• NeuroHub users can send direct messages to each other. This
  feature is not accessible on the CBRAIN side yet, but users
  can switch from one interface to the other.
• NeuroHub users can destroy their old project.
• Different internal mailer configurations allow two sets of
  email messages to be set to users depending on whether they
  interact with the CBRAIN side or the NeuroHub side (for
  registration, password recovery etc).
• CBRAIN administrators can now explicitely indicate that
  a tool is known not to modify its input files, allowing
  users to launch such tools on files on which they only
  have read access. In the past, the interface would refuse
  to launch the tools because it didn't know if that would
  modify the files.
• CBRAIN admins can configure tasks that are containerized
  in Singularity to mount specific file overlays in the container
  (for example, for fixed datasets).
• CBRAIN admins can specify special Singularity run-time options
  for tasks that are containerized with Singularity.
• The task launching system will autodetect if a CBRAIN file
  is located on a data provider that stores files in Singularity
  overlays and mount the overlays automatically if the task
  is also run with Singularity.
• NeuroHub users can generate new API tokens just like on the
  CBRAIN side.
• Internal version tracking is performed by git describe
  instead of the old ruby code that basically did the same thing.
• The Boutiques integrator supports a new special custom
  option allowing a task to save back its input automatically.

CBRAIN 6.0.0

CBRAIN/NeuroHub Release Notes

Version 6.0.0 Released 2020-08-19

More than 8 months have passed since the previous release, 5.3.0!

This release introduces the new GUI interface called NeuroHub.
The original CBRAIN interface is still all there; NeuroHub
is an additional set of pages providing a new look and feel, and
new capabilities. NeuroHub is still rather restricted in what
it can do, but users can easily switch back and forth between
the two interfaces at the push of a button (located at the top
and left on each interface).

New CBRAIN features: (some of these apply to NeuroHub too)

• Projects can be tagged as 'un-assignable", so that a member
  of the project cannot assign userfiles or tasks or other resources
  to it (except for the project's creator). This is mostly
  useful for projects that represent fixed datasets that are
  meant to be used but not extended further.
• CBRAIN (and NeuroHub) now creates a single pair of SSH keys
  (one private, one public) for each user that the system can
  use to access external resources. The private key is never
  made visible. Right now this is mostly useful for the new
  UserkeyFlatDirSshDataProvider class in NeuroHub.
• These keys (full pair) can be pushed to a Bourreau so that
  the CBRAIN code running there can connect as the user. Users
  have control over which bourreau to push their key pair to.
• CBRAIN can generate a new API token and show it to the user
  in their 'my account' page.
• When a user accepts an invitation to join a project, the
  person who made the invitation is notified.
• When an admin reviews a signup request, they can tell
  if the request was made using the CBRAIN or the NeuroHub
  signup page.
• Users can see how many active sessions they have (including
  API sessions), and from what IP they connected.
• Admins can set a user's default data provider when creating
  their account.

NeuroHub exclusive features:

• If a user enters their ORCID ID in their profile, they
  can log in using the ORCID authentication platform.
• Projects are limited to "Work" projects; the other special
  projects that CBRAIN supplies by default are hidden.
• Projects can have special members designated as 'editors'.
• Project editors have much the same powers as the project's creator.
• A project's creator can assign a license to it; members will
  have to agree to the license before they can access it.
• Projects can be configured as 'public'. Things assigned to public
  projects can be accessed by any user.
• User can create their own Data Providers. This is limited right
  now to DPs of type 'UserkeyFlatDirSshDataProvider'. The provider
  side will be accessed using the personal SSH key of the user
  who created the DP. Mostly useful for power users who want to
  access remote filesystems.
• Specialized API hooks for LORIS integrations. Right now, only
  one such hook is implemented, file_list_maker.

Other:

• Much code refactoring (e.g. FileInfo class, BrowseProviderFileCaching,
  SshDataProviderBase...).
• Selection boxes are shown with the chosen.jquery.js framework.
• DP consistency checkers improved.

CBRAIN 5.3.0

Version 5.3.0 Released 2019-12-10

New features:

• From a Task's information page, a user can now access a "Publish to Zenodo"
  page. The task's output and runtime information will be published to
  Zenodo. Caveat: pushing data is done synchronously, which blocks the
  browser (TODO, as pushing as a forked process messes up the libcurl library)
• The Network Operation Center now provides weekly, monthly and yearly reports
• Userfile resource usage tracking: whenever files are added, deleted or change size,
  persistent records are made about the change
• Task resource usage tracking: whenever tasks reach a final state, persistent records
  are made. These include the status of the tasks, but also their accumulated CPU
  and wall times
• Administrators can configure 'epilogue' sections of the tool configs, to match
  the existing prologue sections. This allow the admin to surround the running
  script with e.g. sg newgroup bash <<TOKEN (in the prologue) and TOKEN (in
  the epilogue), making sure the script runs with a particular effective GID.
• The QC panels for userfiles have a new layout; also users can toggle between
  one panel or two
• The S3FlatDataProvider class now allows confiuring a S3 provider that starts
  with a prefixed path (e.g. "/a/b/c", and all the files are under that)
• Users now have an 'affiliation' field, with a controled set of values for it
• Userfile custom viewers can return an informative error message informing the
  framework why they aren't available for a particular file
• New built-in viewers for singularity image files, json files, xml files
• The Diagnostics tool has options for generating busy loops (system and/or user)

Bug fixes:

• Many many small ones.

5.2.0

Version 5.2.0 Released 2019-09-13

We added a Code Of Conduct file to the GitHub repo.

New features:

• The CARMIN API has been implemented
  pretty much completely, except for some limitations (users need to find externally
  the ID of CARMIN files in order to prepare the arguments for CBRAIN tasks)
• A new DataProvider that can connect to SquashFS files through singularity
• The Travis CI control scripts have been improved and one can now selectively
  skip some test stages (e.g. just perform the Ruby curl tests...)
• Added a Boutiques descriptor as a demonstration for developers (for the command 'du')

Bug fixes:

• Several fixes for Bourreau workers
• Improvements to API (in particular, downloads of binary data)
• Big fix when changing a user's type
• Added latest Boutiques schema
• Also see comments on this commit

5.1.2

Version 5.1.2 Released 2019-06-07

The release includes new features and bug fixes.

New features:

• Boutiques descriptors can request their shell to
  be something other than 'bash'
• Boutiques tasks that work on a multiple input files
  now properly record provenance of their outputs
• Custom filters now support multi-select lists for
  many attributes
• API requests will be refused if they suddenly come from
  a different origin
• DataProviders have two new fields to support Datalad servers
• The swagger spec defines page and per_page query parameters
• Tasks that no longer have a workdir are deleted by the portal
• API users can group together new tasks under the same batch_id
• The server list will show red indicators if a Bourreau is
  running in an environment different from the portal (e.g.
  production vs development)
• The admin console now has a 'last' command just like in UNIX

Bugs fixed:

• We use Process.setproctitle() instead of writing to $0
• The custom 'confirm' dialog was changed to a standard Rails one
• Project list show the creators properly
• Project buttons allow deletion
• Roaming users will see their IP address update properly
• Extracting files from FileCollections now guess their types
  and will not proceed if the DP is not writable
• The system won't attempt to send emails if it's not configured
  for email anyway
• Very rare duplications of SyncStatus objects, caused by race
  conditions, are detected and fixed at boot time
• Miscellaneous other small fixes

5.1.1

This is mostly a bugfix release. There is one major new
feature: the Boutiques integrator now has hooks to
allow a superclass to override its subclasses behavior (!),
which is useful when coding special integrators. The
current use case is the BidsAppHandler class in the
cbrain-plugins-neuro package.

Bugs fixed:

• Spawned subprocesses woudl no longer log exceptions and just
  disappear; handler code was still using the old Mysql::Error
  instead of Mysql2::Error
• Bourreaux now set the env variable OBJC_DISABLE_INITIALIZE_FORK_SAFETY=yes
  this is needed on latest maxOS versions
• User messages are properly appended to 'read' messages
• Tool forms are adjusted slightly
• The boot process now lists each task descriptor has it is being integrated
• Some better docs

5.1.0

Version 5.1.0 Released 2018-11-16

It is the age of wisdom, it is the age of foolishness.

General changes:

• The old S3DataProvider code has been revamped
  to use the new Amazon SDK
• A new S3FlatDataProvider was added; it can browse the
  objects in a bucket and register them as files
• Parallelized tasks are better are recovery
• Restarting PostProcessing on tasks now works again
• The show_table helpers can pass around the FormBuilder form handler
• Several show_tables can be linked into a single form
• Links to the ExceptionLog objects now work
• Plugins and Boutiques tasks provide proper revision info
• The Tool show page use the show table helpers
• The ToolConfig show page is used to create and edit them
• Better error messages when a Boutiques descriptor fails to integrate
• New rspec test set for the ParamsErrors class
• Improved Travis CI integration scripts

Some changes are related to the console environment:

• New helpers: online, offline, tv
• New generic scopes: utoday, ctodat, uweek, cweek

Several changes are related to the API:

• The Swagger API spec has been updated; it is still a Swagger 2.0 spec
• The controller code has all been adjusted to match it
• A curl-based testing framework was added to test API calls
• A Ruby-based testing framework was added too
• Both testing frameworks use the same set of 'req' files for testing
• A Ruby gem, CbrainClient, was created out of the swagger spec
• The gem is at https://github.com/aces/cbrain-client-gem

5.0.2

Several bug fixes (boring!) and enhancements (yeah!)

• Singularity support fixes (build instead of pull, exec instead of run)
• jQuery fixes
• API is now single-token based (Bearer token)
• API improvements (limits, simpler filters)
• Switch project clears the persistent list
• Containers mount local DPs
• Plugins can provide arbitrary ruby code in their lib/
• A task's work directory can be saved for inspection
• Launching tasks checks accessibility of inputs
• Lots of tiny internal bugs fixes