-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
REPO: Initial transfer from private repo to open source
- Loading branch information
Craig Ulmer
committed
Apr 23, 2018
1 parent
a1d4fad
commit b660e7f
Showing
884 changed files
with
173,546 additions
and
16 deletions.
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
MIT License | ||
|
||
Copyright 2018 National Technology & Engineering Solutions of Sandia, | ||
LLC (NTESS). Under the terms of Contract DE-NA0003525 with NTESS, the U.S. | ||
Government retains certain rights in this software. | ||
|
||
Permission is hereby granted, free of charge, to any person obtaining a copy | ||
of this software and associated documentation files (the "Software"), to deal | ||
in the Software without restriction, including without limitation the rights | ||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
copies of the Software, and to permit persons to whom the Software is | ||
furnished to do so, subject to the following conditions: | ||
|
||
The above copyright notice and this permission notice shall be included in all | ||
copies or substantial portions of the Software. | ||
|
||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | ||
SOFTWARE. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,69 @@ | ||
Release Information | ||
=================== | ||
|
||
This file provides information about different releases of the | ||
faodel tools. Releases are named alphabetically and have a | ||
4-digit ID associated with them that designates the year and month of | ||
the release. | ||
|
||
Cachet (1.1803.1) | ||
----------------- | ||
- Summary: Flattened repos, simplify builds, and do external release | ||
- Release Improvements: | ||
- Quicker build system | ||
- Better stability with OpBox core | ||
- Significant User Visible Changes: | ||
- Renamed gutties to common. Gutties namespace is now faodel. | ||
- Switched default OpBox core to threaded | ||
- Known Issues | ||
- Similar to 0.1801.1 | ||
- Kelpie is still a preview and may have stability issues | ||
- NNTI MPI transport may have stability issues | ||
|
||
Belfry (0.1801.1) | ||
----------------- | ||
- Summary: Rename to FAODEL, expanded Opbox and Kelpie features | ||
- Release Improvements: | ||
- Renamed repo and internals from data-warehouse to FAODEL | ||
- Simplified release process | ||
- Updated LDO API | ||
- Added Opbox threaded core | ||
- Unified network config tags at the Opbox level | ||
- New Kelpie core ("standard") has networking capabilities | ||
- Kelpie has a preliminary IOM interface for posix io | ||
- Significant User Visible Changes: | ||
- The name external config file environment variable changed from | ||
CONFIG to FAODEL_CONFIG | ||
- The Opbox network configurations tags are now the same for either | ||
NNTI or libfabric | ||
- Known Issues | ||
- Mutrino builds are still challenging | ||
- OpBox/Dirman does not allow new connections after one app shuts down | ||
- IOM API expected to need changes for other backends | ||
- Kelpie row operations not implemented | ||
- Libfabric wrapper does not implement Atomics and lacks maturity | ||
- Sparse documentation | ||
|
||
Amigo (0.1707.2) | ||
---------------- | ||
- Summary: Bug fix release, for friendly users | ||
- Release Improvements: | ||
- Improved stability of NNTI at large scale | ||
- Fixed include dir bug in DataWarehouseConfig.cmake | ||
- Fixed "always on" debug message in Kelpie | ||
- Known Issues | ||
- Same as 1707.1 | ||
|
||
Amigo (0.1707.1) | ||
---------------- | ||
- Summary: First packaged release, for friendly users | ||
- Release Improvements: | ||
- Stable versions of SBL, Gutties, WebHook, NNTI, Lunasa, and Opbox | ||
- Experimental version of Kelpie (nonet) | ||
- Switched to Graith CMake modules | ||
- Initial doxygen and readme documentation | ||
- Known Issues | ||
- Mutrino builds are still challenging | ||
- Kelpie only supports nonet core (for local testing) | ||
- OpBox/Dirman does not allow new connections after one app shuts down | ||
- Sparse documentation (especially examples) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,16 +1,86 @@ | ||
# FAODEL | ||
|
||
Flexible, Asynchronous, Object Data-Exchange Libraries: HPC libraries for moving data between applications | ||
|
||
This repo is a placeholder for the external release of FAODEL from Sandia | ||
National Laboratories. We are currently going through our legal process | ||
to convert the internal version of this software into an **open source** | ||
release. We expect this to be completed by mid **March 2018**. | ||
Until then, please look at the faodel-pubs repo for papers and presentations | ||
related to this work. | ||
|
||
- Update: 3/12 Cleared first round of reviews | ||
- Update: 3/15 Finished copyright paperwork and submitted | ||
- Update: 3/20 Cleared another review | ||
- Update: 3/27 Paperwork submitted to DOE | ||
- Update: 4/05 DOE granted permission to assert copyright, software deemed to be Publicly Available | ||
FAODEL Overview | ||
=============== | ||
FAODEL (Flexible, Asynchronous, Object Data-Exchange Libraries) is a | ||
collection of software libraries that are used to implement | ||
different data management services on high-performance computing (HPC) | ||
platforms. This project is part of the Advanced Technology Development | ||
and Mitigation (ATDM) effort for NNSA's ASC program at Sandia | ||
National Laboratories. | ||
|
||
Components | ||
---------- | ||
FAODEL is composed of multiple libraries: | ||
|
||
- [Kelpie](src/kelpie/README_Kelpie.md): Kelpie is a distributed memory | ||
service that enables applications to migrate different data objects | ||
between compute nodes in a platform. It utilizes out-of-band RDMA | ||
communication to enable different MPI jobs to interact with each | ||
other. | ||
- [OpBox](src/opbox/README_OpBox.md): OpBox is a communication engine | ||
responsible for orchestrating complex communication patterns in a | ||
distributed system. Rather than use traditional remote-procedure | ||
call (RPC) techniques, communication is facilitated through state | ||
machines called Ops. Ops allow distributed protocols to run | ||
asynchronously, without explicit maintenance by user services. | ||
- [Lunasa](src/lunasa/README_Lunasa.md): Lunasa is a memory management unit | ||
for data that may be transmitted on the network using RDMAs. In | ||
order to reduce network registration overheads, Lunasa allocates | ||
sizable amounts of registered memory and then suballocates it to | ||
applications through tcmalloc. User allocations are described by | ||
Lunasa Data Objects (LDOs), which provide reference counting and | ||
object description in the stack. | ||
- [NNTI](src/nnti/README_NNTI.md): NNTI is a low-level, RDMA portability | ||
layer for high-performance networks. It provides application with | ||
the ability to send messages and coordinate RDMA transfers via | ||
registered memory. | ||
- [WebHook](src/webhook/README_WebHook.md): WebHook is a network service for | ||
FAODEL nodes that enables users and applications to query | ||
and change the state of a node via an HTTP connection. | ||
- [Common](src/common/README_Common.md): Common is a collection of data types | ||
and software functions that are used throughout FAODEL. | ||
- [SBL](src/sbl/README_SBL.md): The Simplified Boost Logging (SBL) library | ||
provides a way to map log information in FAODEL components to | ||
Boost's logging library. | ||
|
||
Additional Information | ||
====================== | ||
This release includes files to help guide users. The files are: | ||
|
||
- [INSTALL](INSTALL.md): Details about how to configure, build, | ||
install and run the software provided in this release. This document | ||
is a good starting point, as the build process can be challenging on | ||
different platforms. | ||
- [LICENSE](LICENSE.md): The FAODEL code uses the MIT license. | ||
- [NEWS](NEWS.md): The news file provides a history of major changes provided | ||
with each release of this software. Developers should review this document | ||
when switching to a new release. | ||
|
||
Contributors | ||
============ | ||
The following developers contributed code to the FAODEL: | ||
|
||
- Nathan Fabian | ||
- Todd Kordenbrock | ||
- Scott Levy | ||
- Shyamali Mukherjee | ||
- Gary Templet | ||
- Craig Ulmer | ||
- Patrick Widener | ||
|
||
The following helped contribute ideas and provided feedback for the project: | ||
|
||
- Margaret Lawson | ||
- Jay Lofstead | ||
- Ron Oldfield | ||
- Jeremy Wilke | ||
|
||
This release includes third-party software that contains its own licensing | ||
and copyright info: | ||
- gperftools (in tpl/gperftools) | ||
- Boost ASIO examples (in src/webhook/server) | ||
|
||
Copyright | ||
========= | ||
Copyright 2018 National Technology & Engineering Solutions of Sandia, LLC | ||
(NTESS). Under the terms of Contract DE-NA0003525 with NTESS, the U.S. | ||
Government retains certain rights in this software. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,60 @@ | ||
|
||
message( STATUS "Faodel configuring from ${CMAKE_CURRENT_LIST_FILE}" ) | ||
|
||
# CMake built-ins that we'll use | ||
include( FindPkgConfig ) | ||
include( FindPackageHandleStandardArgs ) | ||
include( CMakePackageConfigHelpers ) | ||
|
||
include( "${CMAKE_CURRENT_LIST_DIR}/FaodelTargets.cmake" ) | ||
|
||
@PACKAGE_INIT@ | ||
|
||
# Compute paths | ||
|
||
set_and_check( Faodel_INCLUDE_DIR "@PACKAGE_INCLUDE_INSTALL_DIR@" ) | ||
set_and_check( Faodel_LIBRARY_DIR "@PACKAGE_LIBRARY_INSTALL_DIR@" ) | ||
set_and_check( Faodel_BINARY_DIR "@PACKAGE_BINARY_INSTALL_DIR@" ) | ||
|
||
set(Faodel_CXX_COMPILER @CMAKE_CXX_COMPILER@ ) | ||
set(Faodel_C_COMPILER @CMAKE_C_COMPILER@ ) | ||
set(Faodel_CXX_FLAGS "@CMAKE_CXX_FLAGS@" ) | ||
|
||
set( Faodel_BUILT_WITH_MPI @USE_MPI@ ) | ||
set( Faodel_BUILT_CMAKE_SYSTEM_NAME @CMAKE_SYSTEM_NAME@ ) | ||
set( Faodel_BUILT_NETWORK_LIBRARY @NETWORK_LIBRARY@ ) | ||
|
||
|
||
# | ||
# Pre-requisites for Faodel | ||
# | ||
# | ||
|
||
# Don't consider this a success until the end | ||
set( Faodel_FOUND 0 ) | ||
|
||
# | ||
# Require C++11 from whatever compiler is in use | ||
# | ||
set( CMAKE_CXX_STANDARD 11 ) | ||
set( CMAKE_CXX_STANDARD_REQUIRED YES ) | ||
set( CMAKE_CXX_EXTENSIONS OFF ) | ||
|
||
find_package( Threads ) | ||
|
||
message( STATUS "[Faodel] Locating required third-party libraries" ) | ||
|
||
include( ${CMAKE_CURRENT_LIST_DIR}/FaodelTPLs.cmake ) | ||
|
||
if( NOT USE_MPI AND Faodel_BUILT_WITH_MPI ) | ||
# The user has indicated that they don't want an MPI_BUILD (USE_MPI not set) but they've | ||
# found a FAODEL install that was built with MPI. This will almost certainly result in | ||
# missing symbols at link time (if not a compile failure because mpi.h can't be found). | ||
# So let's warn about this. | ||
message( WARNING | ||
"[Faodel] This build isn't configuring to use MPI (USE_MPI not set), but Faodel was built with MPI. This is almost certainly not what you want." ) | ||
endif() | ||
|
||
message( STATUS "[Faodel] configured, available targets in ${CMAKE_CURRENT_LIST_DIR}/FaodelTargets.cmake" ) | ||
|
||
set(Faodel_FOUND 1) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
set(PACKAGE_VERSION "@PROJECT_VERSION@") | ||
|
||
# Check whether the requested PACKAGE_FIND_VERSION is compatible | ||
if("${PACKAGE_VERSION}" VERSION_LESS "${PACKAGE_FIND_VERSION}") | ||
set(PACKAGE_VERSION_COMPATIBLE FALSE) | ||
else() | ||
set(PACKAGE_VERSION_COMPATIBLE TRUE) | ||
if ("${PACKAGE_VERSION}" VERSION_EQUAL | ||
"${PACKAGE_FIND_VERSION}") | ||
set(PACKAGE_VERSION_EXACT TRUE) | ||
endif() | ||
endif() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
set (CTEST_PROJECT_NAME "${PROJECT_NAME}") |
Oops, something went wrong.