Use GAX generated libraries for Datastore and Bigtable

[dhermes]

@bjwatson WDYT of this? We are currently generating via protoc with our own Makefiles (datastore and bigtable).

[bjwatson]

@dhermes You sure can. Do you mind if I complete this after #2117 (comment)?

BTW, we won't be able to do much more than wrap these calls in our Pythonic surface. My understanding is that we can't do much of our value-add stuff like page streaming with custom methods, and both of these services are full of custom methods (e.g. URLs ending in :<custom_something>).

[tseaver]

@bjwatson From our perspective, dropping the generated code is a pure win, even if we don't gain the other benefits. Even just having the grpc-google-datastore-v1 and grpc-google-bigtable-v2 releases available (no GAX wrappers at all) would be great.

[dhermes]

No rush at all, just wanted to create a formal issue so we don't forget. Good to know :<custom_something> means custom!

Of the 28 methods in datastore/bigtable, 14 are custom, but the true number is 12/12 since the other 16 Bigtable admin methods will have many orders of magnitude fewer requests.

---

BigtableTableAdmin (2/6 custom)

1. CreateTable -> "v2/{parent=projects/*/instances/*}/tables"
2. ListTables -> "v2/{parent=projects/*/instances/*}/tables"
3. GetTable -> "v2/{name=projects/*/instances/*/tables/*}"
4. DeleteTable -> "v2/{name=projects/*/instances/*/tables/*}"
5. ModifyColumnFamilies -> "v2/{name=projects/*/instances/*/tables/*}:modifyColumnFamilies"
6. DropRowRange -> "v2/{name=projects/*/instances/*/tables/*}:dropRowRange"

Bigtable (6/6 custom)

1. ReadRows -> "v2/{table_name=projects/*/instances/*/tables/*}:readRows"
2. SampleRowKeys -> "v2/{table_name=projects/*/instances/*/tables/*}:sampleRowKeys"
3. MutateRow -> "v2/{table_name=projects/*/instances/*/tables/*}:mutateRow"
4. MutateRows -> "v2/{table_name=projects/*/instances/*/tables/*}:mutateRows"
5. CheckAndMutateRow -> "v2/{table_name=projects/*/instances/*/tables/*}:checkAndMutateRow"
6. ReadModifyWriteRow -> "v2/{table_name=projects/*/instances/*/tables/*}:readModifyWriteRow"

BigtableInstanceAdmin (0/10 custom)

1. CreateInstance -> "v2/{parent=projects/*}/instances"
2. GetInstance -> "v2/{name=projects/*/instances/*}"
3. ListInstances -> "v2/{parent=projects/*}/instances"
4. UpdateInstance -> "v2/{name=projects/*/instances/*}"
5. DeleteInstance -> "v2/{name=projects/*/instances/*}"
6. CreateCluster -> "v2/{parent=projects/*/instances/*}/clusters"
7. GetCluster -> "v2/{name=projects/*/instances/*/clusters/*}"
8. ListClusters -> "v2/{parent=projects/*/instances/*}/clusters"
9. UpdateCluster -> "v2/{name=projects/*/instances/*/clusters/*}"
10. DeleteCluster -> "v2/{name=projects/*/instances/*/clusters/*}"

Datastore (6/6 custom)

1. Lookup -> "v1/projects/{project_id}:lookup"
2. RunQuery -> "v1/projects/{project_id}:runQuery"
3. BeginTransaction -> "v1/projects/{project_id}:beginTransaction"
4. Commit -> "v1/projects/{project_id}:commit"
5. Rollback -> "v1/projects/{project_id}:rollback"
6. AllocateIds -> "v1/projects/{project_id}:allocateIds"

[bjwatson]

Good analysis! At least we can provide extra help with methods like ListTables and ListClusters, even if they are low QPS.

[dhermes]

dropping the generated code is a pure win

Exactly this! That was the point of #1384

[bjwatson]

@dhermes I'm now in a position in which I can deliver GAPIC for these APIs, but I want to make sure that you can use them in the near future.

I can now deliver Datastore without much effort, but Bigtable is prone to change in the coming weeks since we're still nailing down our LRO and streaming designs. Nevertheless, I can deliver both if you're planning to layer your code on ours soon after I deliver it (and then give you an update for Bigtable in a few weeks).

Please let me know.

[dhermes]

How do you mean "can use them"? Like we block you somehow from generating them?

[bjwatson]

I mean will you use them sometime in the next week if I spend time on generating them now?

I'm trying not to spend more time on generating APIs than necessary right now, and instead focus on doing things to finalize our surfaces and improve our efficiency at generating APIs. I will do this if you have the bandwidth to consume it.

[lukesneeringer]

This is still on the "would be nice" list, as I just saw the _generated directory in BigTable, but I do not think keeping this issue open is worthwhile tracking at this point.

[dhermes]

@lukesneeringer It's also a bit moot since you want the generated source in our source tree