Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[cdac] cdac-build-tool #100650

Merged
merged 67 commits into from
Apr 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
8518e9f
checkpoint
lambdageek Mar 28, 2024
12aef6d
add old tooling notes
lambdageek Mar 28, 2024
1eff4d0
add an emitter
lambdageek Apr 1, 2024
22e217b
checkpoint read in the directory header
lambdageek Apr 1, 2024
dd9279d
checkpoint plausible-looking offsets
lambdageek Apr 2, 2024
492af29
checkpoint: field stuff is working
lambdageek Apr 3, 2024
459d124
checkpoint json
lambdageek Apr 3, 2024
28f8256
contract parsing
lambdageek Apr 3, 2024
418739d
refactor
lambdageek Apr 3, 2024
1c1524d
indirect pointer value support
lambdageek Apr 3, 2024
34e852e
move sample to tool dir
lambdageek Apr 3, 2024
e48e709
Take baselines from the docs/design/datacontracts/data dir
lambdageek Apr 3, 2024
0e8c60e
Add README
lambdageek Apr 3, 2024
cd4f0f0
fix BE
lambdageek Apr 3, 2024
b6b079e
hook up cdac-build-tool to the coreclr build; export DotNetRuntimeCon…
lambdageek Apr 4, 2024
a9b0d7f
try fix win-x86 build warning
lambdageek Apr 4, 2024
dbf0557
paths
lambdageek Apr 4, 2024
74433f1
vebose
lambdageek Apr 4, 2024
ea441b6
fix typo; fixes win32
lambdageek Apr 5, 2024
787b836
cleanup; add contracts.txt
lambdageek Apr 5, 2024
e015e83
add diagram to README
lambdageek Apr 5, 2024
0354723
move implementation notes
lambdageek Apr 5, 2024
da038f3
better verbose output from ObjectFileScraper
lambdageek Apr 5, 2024
f64b548
turn off whole program optimizations for data-descriptor.obj
lambdageek Apr 5, 2024
2e29a10
C++-ify and add real Thread offsets
lambdageek Apr 6, 2024
58162a3
no C99 designated initializers in C++ until C++20
lambdageek Apr 8, 2024
4bb51bf
build data descriptor after core runtime
lambdageek Apr 8, 2024
9700e29
fix gcc build
lambdageek Apr 8, 2024
3f22c7c
fix typo
lambdageek Apr 8, 2024
4aae511
simplify ObjectFileScraper
lambdageek Apr 8, 2024
60738ef
more dependencies
lambdageek Apr 8, 2024
c0dab90
try fix source build
lambdageek Apr 8, 2024
84eaa3b
remove fixme; document
lambdageek Apr 8, 2024
86e5258
Merge commit 'origin/main^' into cdac-contract-tool
lambdageek Apr 9, 2024
cc658c9
invoke 'dotnet cmake-build-tool.dll' instead of 'dotnet run --project'
lambdageek Apr 9, 2024
39fe70d
clean up macro boilerplate
lambdageek Apr 9, 2024
5064deb
platform flags
lambdageek Apr 9, 2024
74000bc
turn off verbose output
lambdageek Apr 9, 2024
76c27bc
can't use constexpr function in coreclr
lambdageek Apr 9, 2024
270fe21
Code review fixups
lambdageek Apr 12, 2024
84e0800
Rename "aux data" to "pointer data"
lambdageek Apr 12, 2024
51da4e3
rename "data-descriptor" to "datadescriptor"
lambdageek Apr 12, 2024
b26f3ac
simplify linking
lambdageek Apr 12, 2024
8f9d1a7
cdac-build-tool don't build dotnet tool; turn on analyzers
lambdageek Apr 12, 2024
87b79b3
update README
lambdageek Apr 12, 2024
b33ad07
rationalize naming; update docs; add some inline comments
lambdageek Apr 12, 2024
c39e6f4
rm whitespace
lambdageek Apr 12, 2024
5ef8428
renamce cdac.h to cdacoffsets.h
lambdageek Apr 12, 2024
9f381be
comments
lambdageek Apr 12, 2024
443f83b
fixup README
lambdageek Apr 12, 2024
9fc2b26
hex offsets; improved formatting
lambdageek Apr 12, 2024
024c566
don't throw in ParseContracts; add line numbers to errors
lambdageek Apr 12, 2024
b0f1f40
spelling
lambdageek Apr 12, 2024
2852d8d
Update README.md
lambdageek Apr 12, 2024
2429bd8
change input format for contracts to jsonc
lambdageek Apr 16, 2024
18a863e
add custom JsonConverter instances for the compact json representation
lambdageek Apr 16, 2024
a49e5e6
code review changes
lambdageek Apr 16, 2024
1f5ba44
inline comments into the data-descriptor-blob C code
lambdageek Apr 16, 2024
2decea4
cleanup
lambdageek Apr 16, 2024
ce18bee
comments and markdown
lambdageek Apr 16, 2024
e143998
Apply suggestions from code review
lambdageek Apr 17, 2024
08cf75d
simplify; bug fix - PointerDataCount include placeholder
lambdageek Apr 17, 2024
0d4d40d
fixup docs
lambdageek Apr 17, 2024
94c56ea
Merge remote-tracking branch 'origin/main' into cdac-contract-tool
lambdageek Apr 18, 2024
3d49b70
Update src/coreclr/tools/cdac-build-tool/JsonConverter/GlobalModelJso…
lambdageek Apr 18, 2024
f845c41
one more set of feedback changes
lambdageek Apr 18, 2024
64430cf
maybe no live ILLink ?
lambdageek Apr 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/design/datacontracts/contract-descriptor.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,9 @@ struct DotNetRuntimeContractDescriptor
uint32_t flags;
uint32_t descriptor_size;
const char *descriptor;
uint32_t aux_data_count;
uint32_t pointer_data_count;
uint32_t pad0;
uintptr_t *aux_data;
uintptr_t *pointer_data;
};
```

Expand Down
4 changes: 4 additions & 0 deletions docs/design/datacontracts/data/empty.jsonc
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
// the empty baseline data descriptor
{
"version": 0
}
2 changes: 2 additions & 0 deletions eng/Subsets.props
Original file line number Diff line number Diff line change
Expand Up @@ -358,6 +358,8 @@
<ProjectToBuild Include="$(CoreClrProjectRoot)tools\aot\crossgen2\crossgen2_inbuild.csproj" Category="clr" />

<ProjectToBuild Condition="'$(TargetOS)' == 'windows' or ('$(TargetOS)' == 'linux' and ('$(TargetArchitecture)' == 'x64' or '$(TargetArchitecture)' == 'arm64')) or '$(TargetOS)' == 'osx'" Include="$(CoreClrProjectRoot)tools\SuperFileCheck\SuperFileCheck.csproj" Category="clr" />

<ProjectToBuild Include="$(CoreClrProjectRoot)tools\cdac-build-tool\cdac-build-tool.csproj" Category="clr" />
</ItemGroup>

<ItemGroup Condition="$(_subset.Contains('+clr.toolstests+'))">
Expand Down
43 changes: 43 additions & 0 deletions src/coreclr/debug/runtimeinfo/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,46 @@ endif()

# publish runtimeinfo lib
install_clr(TARGETS runtimeinfo DESTINATIONS lib COMPONENT runtime)

add_library(cdac_data_descriptor OBJECT datadescriptor.cpp)
# don't build the data descriptor before the VM (and any of its dependencies' generated headers)
add_dependencies(cdac_data_descriptor cee_wks_core)
if(CLR_CMAKE_TARGET_WIN32)
# turn off whole program optimization:
# 1. it creates object files that cdac-build-tool can't read
# 2. we never link cdac_data_descriptor into the final product - it's only job is to be scraped
target_compile_options(cdac_data_descriptor PRIVATE /GL-)
endif()
target_include_directories(cdac_data_descriptor BEFORE PRIVATE ${VM_DIR})
target_include_directories(cdac_data_descriptor BEFORE PRIVATE ${VM_DIR}/${ARCH_SOURCES_DIR})
target_include_directories(cdac_data_descriptor PRIVATE ${CLR_DIR}/interop/inc)

set(GENERATED_CDAC_DESCRIPTOR_DIR "${CMAKE_CURRENT_BINARY_DIR}/cdac")
set(CONTRACT_DESCRIPTOR_OUTPUT "${GENERATED_CDAC_DESCRIPTOR_DIR}/contract-descriptor.c")
if("${CDAC_BUILD_TOOL_BINARY_PATH}" STREQUAL "" OR NOT EXISTS "${CDAC_BUILD_TOOL_BINARY_PATH}")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

- "${CDAC_BUILD_TOOL_BINARY_PATH}" STREQUAL ""
+ NOT CDAC_BUILD_TOOL_BINARY_PATH

it checks both "not defined" and "evaluates to empty string".

message(FATAL_ERROR "No cdac-build-tool set or ${CDAC_BUILD_TOOL_BINARY_PATH} does not exist")
endif()

set(CONTRACT_FILE "${CMAKE_CURRENT_SOURCE_DIR}/contracts.jsonc")

# generate the contract descriptor by running cdac-build-tool
# n.b. this just uses `dotnet` from the PATH. InitializeDotNetCli adds the apropropriate directory
add_custom_command(
OUTPUT "${CONTRACT_DESCRIPTOR_OUTPUT}"
VERBATIM
COMMAND dotnet ${CDAC_BUILD_TOOL_BINARY_PATH} compose -o "${CONTRACT_DESCRIPTOR_OUTPUT}" -c "${CONTRACT_FILE}" $<TARGET_OBJECTS:cdac_data_descriptor>
Copy link
Member

@am11 am11 Apr 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. src/coreclr/build-runtime.sh direct invocation skips requiring dotnet/msbuild, which is necessary for new platform port work. This PR assumes dotnet/msbuild are always available. We should handle that case.

  2. Secondly, can use published cdac-build-tool instead of assembly path so instead of requiring dotnet in PATH, it uses executable (AOT'd, singilefilehost, apphost -based doesn't matter)? If that's not possible (e.g. crossbuild case), then please pass DOTNET_HOST_PATH (set by SDK to the executing dotnet(1)) via cmakeargs as well.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we just make embedding the contract descriptor optional if someone is doing a build without msbuild?

Copy link
Member

@am11 am11 Apr 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If making it optional doesn't affect debugging, that's good. Second part is:

-      <_CoreClrBuildArg Include="-cmakeargs &quot;-DCDAC_BUILD_TOOL_BINARY_PATH=$(RuntimeBinDir)cdac-build-tool\cdac-build-tool.dll&quot;" />
+      <_CoreClrBuildArg Include="-cmakeargs &quot;-DCDAC_BUILD_TOOL_BINARY_PATH='$(RuntimeBinDir)cdac-build-tool/cdac-build-tool.dll'&quot; -cmakeargs &quot;-DCLR_DOTNET_HOST_PATH='$(DOTNET_HOST_PATH)'&quot;" />
-    COMMAND dotnet ${CDAC_BUILD_TOOL_BINARY_PATH} compose -o "${CONTRACT_DESCRIPTOR_OUTPUT}" -c "${CONTRACT_FILE}" $<TARGET_OBJECTS:cdac_data_descriptor>
+    COMMAND "${CLR_DOTNET_HOST_PATH}" "${CDAC_BUILD_TOOL_BINARY_PATH}" compose -o "${CONTRACT_DESCRIPTOR_OUTPUT}" -c "${CONTRACT_FILE}" $<TARGET_OBJECTS:cdac_data_descriptor>

which will resolve to path/to/runtime/.dotnet/dotnet.

Copy link
Member Author

@lambdageek lambdageek Apr 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If making it optional doesn't affect debugging, that's good

in the future, it will make using something like dotnet-sos impossible

I don't understand what scenarios don't have a .NET sdk available. any new platform bringup will start with cross compiling from an existing platform with a .NET sdk and a cross compiler toolchain, right? that case should be handled by the cmake+msbuild infrastructure

I added #101297


-    COMMAND dotnet ${CDAC_BUILD_TOOL_BINARY_PATH} compose -o "${CONTRACT_DESCRIPTOR_OUTPUT}" -c "${CONTRACT_FILE}" $<TARGET_OBJECTS:cdac_data_descriptor>
+    COMMAND "${CLR_DOTNET_HOST_PATH}" "${CDAC_BUILD_TOOL_BINARY_PATH}" compose -o "${CONTRACT_DESCRIPTOR_OUTPUT}" -c "${CONTRACT_FILE}" $<TARGET_OBJECTS:cdac_data_descriptor>

I had something like this before and it broke source build because it was pointing to the repo root .dotnet/dotnet, not to artifacts/sb/.dotnet/dotnet. AFAICT it's fine to just call "dotnet" from the path. eng/commons/build.sh always adds the right SDK dir to the path - it's done by InitializeDotNetCli, and it is source-build aware

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update I'm educating myself over in dotnet/installer#19534 (comment)

I think previously I used something other than DOTNET_HOST_PATH

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated #101297 to use DOTNET_HOST_PATH if it's set

DEPENDS cdac_data_descriptor cee_wks_core $<TARGET_OBJECTS:cdac_data_descriptor> "${CONTRACT_FILE}"
USES_TERMINAL
)

# It is important that cdac_contract_descriptor is an object library;
# if it was static, linking it into the final dll would not export
# DotNetRuntimeContractDescriptor since it is not referenced anywhere.
add_library_clr(cdac_contract_descriptor OBJECT
"${CONTRACT_DESCRIPTOR_OUTPUT}"
contractpointerdata.cpp
)
target_include_directories(cdac_contract_descriptor BEFORE PRIVATE ${VM_DIR})
target_include_directories(cdac_contract_descriptor BEFORE PRIVATE ${VM_DIR}/${ARCH_SOURCES_DIR})
target_include_directories(cdac_contract_descriptor PRIVATE ${CLR_DIR}/interop/inc)
add_dependencies(cdac_contract_descriptor cdac_data_descriptor cee_wks_core)
23 changes: 23 additions & 0 deletions src/coreclr/debug/runtimeinfo/contractpointerdata.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

#include "common.h"

#include <stddef.h>
#include <stdint.h>

#include "threads.h"

extern "C"
{

// without an extern declaration, clang does not emit this global into the object file
extern const uintptr_t contractDescriptorPointerData[];
lambdageek marked this conversation as resolved.
Show resolved Hide resolved

const uintptr_t contractDescriptorPointerData[] = {
(uintptr_t)0, // placeholder
#define CDAC_GLOBAL_POINTER(name,value) (uintptr_t)(value),
#include "datadescriptor.h"
};

}
14 changes: 14 additions & 0 deletions src/coreclr/debug/runtimeinfo/contracts.jsonc
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
//algorithmic contracts for coreclr
// The format of this file is: JSON with comments
// {
// "CONTRACT NAME": VERSION,
// ...
// }
// CONTRACT NAME is an arbitrary string, VERSION is an integer
//
// cdac-build-tool can take multiple "-c contract_file" arguments
// so to conditionally include contracts, put additional contracts in a separate file
{
"SOSBreakingChangeVersion": 1 // example contract: "runtime exports an SOS breaking change version global"
}

Loading
Loading