Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EngSys] Test proxy #15101

Merged
113 commits merged into from
Aug 12, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
113 commits
Select commit Hold shift + click to select a range
1890d91
changing recording to use test-proxy
seankane-msft Jul 19, 2021
c55ef02
updating client and test proxy
seankane-msft Jul 19, 2021
9ef5856
updated Transport Do to send actual request
seankane-msft Jul 19, 2021
4603915
whitespace
seankane-msft Jul 20, 2021
2099dbc
working implementation
seankane-msft Jul 20, 2021
99e3ecf
updating internal recording
seankane-msft Jul 20, 2021
89dea93
now I have a blank recording...
seankane-msft Jul 20, 2021
366abc2
cleaning up
seankane-msft Jul 20, 2021
f4dc6cf
more clean up
seankane-msft Jul 20, 2021
af288f0
transport Do is repeating, proxy is not parsing URL correctly
seankane-msft Jul 20, 2021
22ae67f
small changes
seankane-msft Jul 21, 2021
79b6a2b
merge conflicts
seankane-msft Jul 26, 2021
9e00c51
adding a policy to do the same thing
seankane-msft Jul 26, 2021
8c32067
Merge branch 'track2-tables' of https://github.com/Azure/azure-sdk-fo…
seankane-msft Jul 28, 2021
2f2e990
adding a second test for proxy and transport
seankane-msft Jul 28, 2021
25e262f
further progress with Jeff, switching to only using a policy right now
seankane-msft Jul 28, 2021
756ce72
cleaning up file
seankane-msft Jul 29, 2021
f8965dc
cleaning up
seankane-msft Jul 29, 2021
c06704a
added more debugging, added two required headers that I was missing
seankane-msft Jul 29, 2021
91399e7
proxy working on a single test, need to convert all tests to test proxy
seankane-msft Jul 29, 2021
e236fa1
cleaning up
seankane-msft Jul 29, 2021
b048044
moved over successfully :)
seankane-msft Jul 29, 2021
2abbbcd
more helper functions
seankane-msft Jul 29, 2021
d7d31d9
working for a subtest functionality
seankane-msft Jul 29, 2021
7c3be8d
fixed up a service client test
seankane-msft Jul 30, 2021
18997bc
converting more service tests
seankane-msft Jul 30, 2021
1868088
converted all service client tests
seankane-msft Jul 30, 2021
f5de97a
converted table client tests
seankane-msft Jul 30, 2021
71b10ac
fixed last two client tests
seankane-msft Jul 30, 2021
214e477
entity test
seankane-msft Jul 30, 2021
e61297f
entity tests
seankane-msft Jul 30, 2021
26eb607
converting access policy tests
seankane-msft Aug 2, 2021
992f303
all passing in record mode
seankane-msft Aug 2, 2021
8bb758c
batch tests
seankane-msft Aug 2, 2021
1b9c3bd
liveonly for batch tests
seankane-msft Aug 2, 2021
4e7a0e7
adding a sanitizer and more methods for recording
seankane-msft Aug 2, 2021
d23cb92
adding main method to start and stop the proxy automatically
seankane-msft Aug 2, 2021
5d50021
adding test proxy step to pipeline
seankane-msft Aug 2, 2021
bc5c63a
double dash on version arg
seankane-msft Aug 2, 2021
6d022ed
need a better way to get userful error reporting
seankane-msft Aug 2, 2021
ca7f9d6
explicitly running test-proxy in the background
seankane-msft Aug 2, 2021
43d8c43
adjust the proxy start and install to be compatible with specific pro…
scbedd Aug 2, 2021
5dd22bc
Merge branch 'test-proxy' of https://github.com/seankane-msft/azure-s…
seankane-msft Aug 2, 2021
8580096
dont run test proxy in background
seankane-msft Aug 2, 2021
6178b38
didnt remove an import
seankane-msft Aug 2, 2021
9ed3455
header to proxy_test.go
seankane-msft Aug 2, 2021
38a1a6d
fixes for pipeline
seankane-msft Aug 2, 2021
ecab2a5
added loggin on accident
seankane-msft Aug 2, 2021
fd20e49
use nohup on linux machines
scbedd Aug 2, 2021
f355f82
further simplify nohup usage
scbedd Aug 2, 2021
2e2110c
there are no exes on windows
scbedd Aug 2, 2021
53d3b1c
fixing errors, passing in default env variable
seankane-msft Aug 2, 2021
76053b8
Merge branch 'test-proxy' of https://github.com/seankane-msft/azure-s…
seankane-msft Aug 2, 2021
3390179
force background for linux task. may need another iteration to place …
scbedd Aug 2, 2021
a6190d6
Merge branch 'test-proxy' of https://github.com/seankane-msft/azure-s…
scbedd Aug 2, 2021
82a3230
fake aad credential
seankane-msft Aug 2, 2021
9feac51
Merge branch 'test-proxy' of https://github.com/seankane-msft/azure-s…
seankane-msft Aug 2, 2021
416e257
clean up after test-proxy
scbedd Aug 2, 2021
8f0b219
Merge branch 'test-proxy' of https://github.com/seankane-msft/azure-s…
scbedd Aug 2, 2021
fd8c713
add print for record mode
seankane-msft Aug 2, 2021
df1d4d7
Merge branch 'test-proxy' of https://github.com/seankane-msft/azure-s…
seankane-msft Aug 2, 2021
97fac55
another different attempt at starting the proxy
scbedd Aug 2, 2021
bfc545c
Merge branch 'test-proxy' of https://github.com/seankane-msft/azure-s…
scbedd Aug 2, 2021
06febdd
jobs apparently is an unrecognized command on our linux boxes
scbedd Aug 2, 2021
a30493d
correct dumb typo in run tests
scbedd Aug 2, 2021
f6574b5
replace nuget install with docker
scbedd Aug 2, 2021
3b57585
working fake credentials for client delete entity test
seankane-msft Aug 3, 2021
31de5d4
fixing fake credentials portion, passing locally
seankane-msft Aug 3, 2021
c937f2e
trying an insecureskipverify transport
seankane-msft Aug 3, 2021
2213bd5
debugging statements to find the cwd
seankane-msft Aug 3, 2021
d1a7ab6
removing blank recording
seankane-msft Aug 3, 2021
ad4ad61
leverage proxy tool, not docker
scbedd Aug 4, 2021
2b02ea7
remove apostrophes
scbedd Aug 4, 2021
f53f912
Merge branch 'test-proxy' of https://github.com/seankane-msft/azure-s…
seankane-msft Aug 4, 2021
321800f
merge conflicts
seankane-msft Aug 4, 2021
939289c
moving recordings one directory up
seankane-msft Aug 4, 2021
f2f3ef8
changing directory for test-proxy to start from
seankane-msft Aug 4, 2021
5019877
removing parenthesis
seankane-msft Aug 4, 2021
9dbe47d
adding steps for validating ssl
seankane-msft Aug 4, 2021
a0a9ba7
switching to docker
seankane-msft Aug 4, 2021
ecb9e2b
adding start server script
seankane-msft Aug 5, 2021
f861f5c
working implementation if we can fix the prepending of 'recording' to…
seankane-msft Aug 5, 2021
ab07220
big refactor, moving stuff into recording file, cleaning up proxy_tes…
seankane-msft Aug 5, 2021
a192402
last fixings
seankane-msft Aug 5, 2021
9c97403
convert back to docker. windows and linux images now present
scbedd Aug 6, 2021
49c446b
double condition
scbedd Aug 6, 2021
3bb4b37
moving configuration to a separate file. preparing for eng/common move
scbedd Aug 6, 2021
8e76a04
tier 0 of trust. I don't expect this to work, but it's still worth a …
scbedd Aug 6, 2021
567588a
update condition to use variable syntax
scbedd Aug 6, 2021
6d95e96
guess I'm not using variables. removing
scbedd Aug 6, 2021
e2c78ee
call start-server.ps1
scbedd Aug 6, 2021
fafb7c5
adjust build-test and configure-proxy to run the docker container in …
scbedd Aug 6, 2021
7c4c3d7
wrap in quotes for the container create. it's apparently a bad
scbedd Aug 6, 2021
0219745
adjust the initialize call
scbedd Aug 6, 2021
105e29e
disable vet temporarily
scbedd Aug 6, 2021
808885b
looking for cert file in env variable PROXY_CERT
seankane-msft Aug 6, 2021
c881539
set proxy_cert environment variable to find that certificate
scbedd Aug 6, 2021
797a804
changing to crt file
seankane-msft Aug 6, 2021
b0f82b3
use crt cert
scbedd Aug 6, 2021
69f0e4a
resolve conflicts. use crt
scbedd Aug 6, 2021
de2c846
want to see errors
seankane-msft Aug 6, 2021
bcf19b4
Merge branch 'test-proxy' of https://github.com/seankane-msft/azure-s…
seankane-msft Aug 6, 2021
5018d31
powershell errors
seankane-msft Aug 6, 2021
fbc39c4
correcting how the volume binds to the windows container
scbedd Aug 6, 2021
8865646
Merge branch 'test-proxy' of https://github.com/seankane-msft/azure-s…
scbedd Aug 6, 2021
69258a4
small modification for windows container
scbedd Aug 6, 2021
400a97b
finally got the magic sauce
scbedd Aug 6, 2021
6f63a6c
cleaning up to remove azcore from internal
seankane-msft Aug 9, 2021
8789596
removing recordings that use vcr
seankane-msft Aug 9, 2021
b7942f4
changing location of script
seankane-msft Aug 10, 2021
7f28fae
forgot the stop command
seankane-msft Aug 10, 2021
b41f664
fixing an issue with reading env vars
seankane-msft Aug 11, 2021
eae799e
cleaning up commented code
seankane-msft Aug 12, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 12 additions & 12 deletions eng/pipelines/templates/steps/build-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,34 +28,34 @@ steps:
env:
GO111MODULE: 'on'

- pwsh: |
$modDirs = (./eng/scripts/get_module_dirs.ps1 -serviceDir $(SCOPE))
foreach ($md in $modDirs) {
pushd $md
Write-Host "##[command]Executing go vet ./... in $md"
go vet ./...
}
displayName: 'Vet'
workingDirectory: '${{parameters.GoWorkspace}}'
env:
GO111MODULE: 'on'
- template: configure-proxy.yml
parameters:
ServiceDirectory: ${{ parameters.ServiceDirectory }}

- pwsh: |
$testDirs = (./eng/scripts/get_test_dirs.ps1 -serviceDir $(SCOPE))
foreach ($td in $testDirs) {
pushd $td

$(Build.SourcesDirectory)/eng/scripts/start-server.ps1 start

Write-Host "##[command]Executing go test -run "^Test" -v -coverprofile coverage.txt $td | go-junit-report -set-exit-code > report.xml"
go test -run "^Test" -v -coverprofile coverage.txt . | go-junit-report -set-exit-code > report.xml
go test -run "^Test" -v -coverprofile coverage.txt . > temp.txt
cat temp.txt
cat temp.txt | go-junit-report -set-exit-code > report.xml
# if no tests were actually run (e.g. examples) delete the coverage file so it's omitted from the coverage report
if (Select-String -path ./report.xml -pattern '<testsuites></testsuites>' -simplematch -quiet) {
Write-Host "##[command]Deleting empty coverage file"
rm coverage.txt
}

$(Build.SourcesDirectory)/eng/scripts/start-server.ps1 stop
}
displayName: 'Run Tests'
workingDirectory: '${{parameters.GoWorkspace}}'
env:
GO111MODULE: 'on'
AZURE_RECORD_MODE: 'playback'

- pwsh: |
$coverageFiles = [Collections.Generic.List[String]]@()
Expand Down
22 changes: 22 additions & 0 deletions eng/pipelines/templates/steps/configure-proxy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
parameters:
ServiceDirectory: ''

steps:
- pwsh: |
$certUriPfx = "https://github.com/Azure/azure-sdk-tools/raw/main/tools/test-proxy/docker/dev_certificate/dotnet-devcert.pfx"
$certUriCrt = "https://github.com/Azure/azure-sdk-tools/raw/main/tools/test-proxy/docker/dev_certificate/dotnet-devcert.crt"
$certLocationPfx = "$(Build.SourcesDirectory)/dotnet-devcert.pfx"
$certLocationCrt = "$(Build.SourcesDirectory)/dotnet-devcert.crt"

Invoke-WebRequest `
-Uri $certUriPfx `
-OutFile $certLocationPfx -UseBasicParsing

Invoke-WebRequest `
-Uri $certUriCrt `
-OutFile $certLocationCrt -UseBasicParsing

dotnet dev-certs https --clean --import $certLocationPfx -p "password"

Write-Host "##vso[task.setvariable variable=PROXY_CERT]$certLocationCrt"
displayName: 'Download and Trust Certificate'
66 changes: 66 additions & 0 deletions eng/scripts/start-server.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
param(
[ValidateSet("start", "stop")]
[String]
$mode,
[String]
$targetFolder = "."
)

try {
docker --version | Out-Null
}
catch {
Write-Error "A invocation of docker --version failed. This indicates that docker is not properly installed or running."
Write-Error "Please check your docker invocation and try running the script again."
}

$repoRoot = (Resolve-Path $targetFolder).Path.Replace("`\", "/")
Write-Host $repoRoot

$CONTAINER_NAME = "ambitious_azsdk_test_proxy"
$IMAGE_SOURCE = "azsdkengsys.azurecr.io/engsys/testproxy-lin:1037115"
$Initial = ""

if ($IsWindows -and $env:TF_BUILD){
$IMAGE_SOURCE = "azsdkengsys.azurecr.io/engsys/testproxy-win:1037115"
$Initial = "C:"
}

function Get-Proxy-Container(){
return (docker container ls -a --format "{{ json . }}" --filter "name=$CONTAINER_NAME" `
| ConvertFrom-Json `
| Select-Object -First 1)
}

if ($mode -eq "start"){
$proxyContainer = Get-Proxy-Container

# if we already have one, we just need to check the state
if($proxyContainer){
if ($proxyContainer.State -eq "running")
{
Write-Host "Discovered an already running instance of the test-proxy!. Exiting"
exit(0)
}
}
# else we need to create it
else {
Write-Host "Attempting creation of Docker host $CONTAINER_NAME"
Write-Host "docker container create -v `"${repoRoot}:${Initial}/etc/testproxy`" -p 5001:5001 -p 5000:5000 --name $CONTAINER_NAME $IMAGE_SOURCE"
docker container create -v "${repoRoot}:${Initial}/etc/testproxy" -p 5001:5001 -p 5000:5000 --name $CONTAINER_NAME $IMAGE_SOURCE
}

Write-Host "Attempting start of Docker host $CONTAINER_NAME"
docker container start $CONTAINER_NAME
}

if ($mode -eq "stop"){
$proxyContainer = Get-Proxy-Container

if($proxyContainer){
if($proxyContainer.State -eq "running"){
Write-Host "Found a running instance of $CONTAINER_NAME, shutting it down."
docker container stop $CONTAINER_NAME
}
}
}
Loading