Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: implement ext_offchain_index_set_version_1 for wasmer runtime #1739

Merged
merged 25 commits into from
Sep 24, 2021
Merged

feat: implement ext_offchain_index_set_version_1 for wasmer runtime #1739

Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
c6d38ac
implement ext_offchain_index_set_version_1
edwardmack Aug 19, 2021
d88d707
Merge branch 'development' into ed/impl_ext_offchain_index_set_v1
edwardmack Aug 20, 2021
99fe2c4
add storage to local
edwardmack Aug 20, 2021
38c0813
Merge branch 'development' into ed/impl_ext_offchain_index_set_v1
edwardmack Aug 23, 2021
cbdb7f2
add baseDB to runtime context
edwardmack Aug 24, 2021
53d11f5
add init base db for tests
edwardmack Aug 24, 2021
9d5094b
Merge branch 'development' into ed/impl_ext_offchain_index_set_v1
edwardmack Aug 24, 2021
caea8b0
add base db to node storage in tests
edwardmack Aug 24, 2021
6284063
Merge branch 'development' into ed/impl_ext_offchain_index_set_v1
edwardmack Aug 24, 2021
9ddda6d
add base db to node storage for sync/test_helpers
edwardmack Aug 24, 2021
cdd0b08
Merge branch 'ed/impl_ext_offchain_index_set_v1' of https://github.co…
edwardmack Aug 24, 2021
8c4b89f
Merge branch 'development' into ed/impl_ext_offchain_index_set_v1
edwardmack Aug 26, 2021
38c99ab
Merge branch 'development' into ed/impl_ext_offchain_index_set_v1
edwardmack Sep 1, 2021
1cdf409
use baseDB instead of creating new db
edwardmack Sep 1, 2021
fcb1238
lint issues
edwardmack Sep 1, 2021
f9c9cdf
address deep source issues
edwardmack Sep 2, 2021
b1864dc
fix lint issues
edwardmack Sep 3, 2021
cc43aa9
Merge branch 'development' into ed/impl_ext_offchain_index_set_v1
edwardmack Sep 7, 2021
defa63f
add db check, rename var
edwardmack Sep 7, 2021
ee82422
Merge branch 'development' into ed/impl_ext_offchain_index_set_v1
edwardmack Sep 10, 2021
b10fd88
add init BaseDB to createRuntimeStorage
edwardmack Sep 10, 2021
a956531
Merge branch 'development' into ed/impl_ext_offchain_index_set_v1
edwardmack Sep 20, 2021
0fc1adf
Merge branch 'development' into ed/impl_ext_offchain_index_set_v1
edwardmack Sep 22, 2021
b70dc1c
Merge branch 'development' into ed/impl_ext_offchain_index_set_v1
edwardmack Sep 23, 2021
8ae7638
Merge branch 'development' into ed/impl_ext_offchain_index_set_v1
edwardmack Sep 24, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions dot/core/test_helpers.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,19 +18,21 @@ package core

import (
"io/ioutil"
"path/filepath"
"testing"

coremocks "github.com/ChainSafe/gossamer/dot/core/mocks"
"github.com/ChainSafe/gossamer/dot/network"
"github.com/ChainSafe/gossamer/dot/state"
"github.com/ChainSafe/gossamer/lib/common"
"github.com/ChainSafe/gossamer/lib/crypto/sr25519"
"github.com/ChainSafe/gossamer/lib/genesis"
"github.com/ChainSafe/gossamer/lib/keystore"
"github.com/ChainSafe/gossamer/lib/runtime"
rtstorage "github.com/ChainSafe/gossamer/lib/runtime/storage"
"github.com/ChainSafe/gossamer/lib/runtime/wasmer"
"github.com/ChainSafe/gossamer/lib/utils"
log "github.com/ChainSafe/log15"

coremocks "github.com/ChainSafe/gossamer/dot/core/mocks"
"github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require"
)
Expand Down Expand Up @@ -105,6 +107,13 @@ func NewTestService(t *testing.T, cfg *Config) *Service {
rtCfg.CodeHash, err = cfg.StorageState.LoadCodeHash(nil)
require.NoError(t, err)

nodeStorage := runtime.NodeStorage{}

nodeStorage.BaseDB, err = utils.SetupDatabase(filepath.Join(testDatadirPath, "offline_storage"), false)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be preferred to use stateSrvc.BaseDB here if stateSrvc != nil

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

require.NoError(t, err)

rtCfg.NodeStorage = nodeStorage
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgmt! just this pice of code that repeats in some places, would be nice to create a test function helper that receives a stateSrvc and returns a NodeStorage

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I created this:

func InitNodeStorage(baseDB runtime.BasicStorage, dataDir string) (runtime.NodeStorage, error) {
    nodeStorage := runtime.NodeStorage{}
    var err error
    if baseDB != nil {
        nodeStorage.BaseDB = baseDB
    } else {
        nodeStorage.BaseDB, err = utils.SetupDatabase(filepath.Join(dataDir, "offline_storage"), false)
    }
    return nodeStorage, err
}

however it seems that I could find a place to put this where it wouldn't cause a cycle import error in one of the files that used it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where did you place it? If you could not find a place to put there is no problem to keep the code as it is.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tried using it in core/test_helpers.go, then sync/test_helpers.go and babe/babe_test.go


cfg.Runtime, err = wasmer.NewRuntimeFromGenesis(gen, rtCfg)
require.NoError(t, err)
}
Expand Down
1 change: 1 addition & 0 deletions dot/services.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,7 @@ func createRuntime(cfg *Config, st *state.Service, ks *keystore.GlobalKeystore,
ns := runtime.NodeStorage{
LocalStorage: localStorage,
PersistentStorage: chaindb.NewTable(st.DB(), "offlinestorage"),
BaseDB: st.Base,
}

codeHash, err := st.Storage.LoadCodeHash(nil)
Expand Down
10 changes: 10 additions & 0 deletions dot/state/base.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,16 @@ func (s *BaseState) LoadCodeSubstitutedBlockHash() common.Hash {
return common.NewHash(hash)
}

// Put stores key/value pair in database
func (s *BaseState) Put(key, value []byte) error {
return s.db.Put(key, value)
}

// Get retrieves value by key from database
func (s *BaseState) Get(key []byte) ([]byte, error) {
return s.db.Get(key)
}

func (s *BaseState) storeSkipToEpoch(epoch uint64) error {
buf := make([]byte, 8)
binary.LittleEndian.PutUint64(buf, epoch)
Expand Down
13 changes: 9 additions & 4 deletions dot/sync/test_helpers.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,12 @@ package sync
import (
"io/ioutil"
"math/big"
"path/filepath"
"testing"
"time"

"github.com/stretchr/testify/mock"

"github.com/ChainSafe/gossamer/dot/state"
syncmocks "github.com/ChainSafe/gossamer/dot/sync/mocks"
"github.com/ChainSafe/gossamer/dot/types"
"github.com/ChainSafe/gossamer/lib/babe"
"github.com/ChainSafe/gossamer/lib/common"
Expand All @@ -34,11 +34,11 @@ import (
"github.com/ChainSafe/gossamer/lib/runtime/wasmer"
"github.com/ChainSafe/gossamer/lib/transaction"
"github.com/ChainSafe/gossamer/lib/trie"
"github.com/ChainSafe/gossamer/lib/utils"
"github.com/ChainSafe/gossamer/pkg/scale"
log "github.com/ChainSafe/log15"
"github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require"

syncmocks "github.com/ChainSafe/gossamer/dot/sync/mocks"
)

// NewMockFinalityGadget create and return sync FinalityGadget interface mock
Expand Down Expand Up @@ -100,6 +100,11 @@ func NewTestSyncer(t *testing.T, usePolkadotGenesis bool) *Service {
rtCfg.CodeHash, err = cfg.StorageState.LoadCodeHash(nil)
require.NoError(t, err)

nodeStorage := runtime.NodeStorage{}
nodeStorage.BaseDB, err = utils.SetupDatabase(filepath.Join(testDatadirPath, "offline_storage"), false)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

require.NoError(t, err)
rtCfg.NodeStorage = nodeStorage

instance, err := wasmer.NewRuntimeFromGenesis(gen, rtCfg) //nolint
require.NoError(t, err)
cfg.Runtime = instance
Expand Down
15 changes: 11 additions & 4 deletions lib/babe/babe_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,24 +21,25 @@ import (
"io/ioutil"
"math/big"
"os"
"path/filepath"
"testing"
"time"

"github.com/ChainSafe/gossamer/dot/core"
"github.com/ChainSafe/gossamer/dot/state"
"github.com/ChainSafe/gossamer/dot/types"
"github.com/ChainSafe/gossamer/lib/babe/mocks"
"github.com/ChainSafe/gossamer/lib/common"
"github.com/ChainSafe/gossamer/lib/crypto/sr25519"
"github.com/ChainSafe/gossamer/lib/genesis"
"github.com/ChainSafe/gossamer/lib/runtime"
rtstorage "github.com/ChainSafe/gossamer/lib/runtime/storage"
"github.com/ChainSafe/gossamer/lib/runtime/wasmer"
"github.com/ChainSafe/gossamer/lib/trie"
"github.com/ChainSafe/gossamer/lib/utils"
log "github.com/ChainSafe/log15"
"github.com/stretchr/testify/require"

"github.com/ChainSafe/gossamer/lib/babe/mocks"
mock "github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require"
)

var (
Expand Down Expand Up @@ -98,8 +99,9 @@ func createTestService(t *testing.T, cfg *ServiceConfig) *Service {
cfg.TransactionState = state.NewTransactionState()
}

testDatadirPath, err := ioutil.TempDir("/tmp", "test-datadir-*") //nolint

if cfg.BlockState == nil || cfg.StorageState == nil || cfg.EpochState == nil {
testDatadirPath, err := ioutil.TempDir("/tmp", "test-datadir-*") //nolint
require.NoError(t, err)

config := state.Config{
Expand Down Expand Up @@ -138,6 +140,11 @@ func createTestService(t *testing.T, cfg *ServiceConfig) *Service {
rtCfg.CodeHash, err = storageState.LoadCodeHash(nil)
require.NoError(t, err)

nodeStorage := runtime.NodeStorage{}
nodeStorage.BaseDB, err = utils.SetupDatabase(filepath.Join(testDatadirPath, "offline_storage"), false)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update.

require.NoError(t, err)
rtCfg.NodeStorage = nodeStorage

cfg.Runtime, err = wasmer.NewRuntimeFromGenesis(gen, rtCfg)
require.NoError(t, err)
}
Expand Down
1 change: 1 addition & 0 deletions lib/runtime/types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ const NodeStorageTypeLocal NodeStorageType = 2
type NodeStorage struct {
LocalStorage BasicStorage
PersistentStorage BasicStorage
BaseDB BasicStorage
}

// InstanceConfig represents a runtime instance configuration
Expand Down
3 changes: 3 additions & 0 deletions lib/runtime/wasmer/exports_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -266,6 +266,9 @@ func TestNodeRuntime_ValidateTransaction(t *testing.T) {
cfg := &Config{}
cfg.Storage = genState
cfg.LogLvl = 4
nodeStorage := runtime.NodeStorage{}
nodeStorage.BaseDB = runtime.NewInMemoryDB(t)
cfg.NodeStorage = nodeStorage

rt, err := NewRuntimeFromGenesis(gen, cfg)
require.NoError(t, err)
Expand Down
35 changes: 23 additions & 12 deletions lib/runtime/wasmer/imports.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ import (
)

//export ext_logging_log_version_1
func ext_logging_log_version_1(context unsafe.Pointer, level C.int32_t, targetData C.int64_t, msgData C.int64_t) {
func ext_logging_log_version_1(context unsafe.Pointer, level C.int32_t, targetData, msgData C.int64_t) {
logger.Trace("[ext_logging_log_version_1] executing...")
instanceContext := wasm.IntoInstanceContext(context)

Expand Down Expand Up @@ -319,7 +319,7 @@ func ext_crypto_ed25519_public_keys_version_1(context unsafe.Pointer, keyTypeID
}

//export ext_crypto_ed25519_sign_version_1
func ext_crypto_ed25519_sign_version_1(context unsafe.Pointer, keyTypeID C.int32_t, key C.int32_t, msg C.int64_t) C.int64_t {
func ext_crypto_ed25519_sign_version_1(context unsafe.Pointer, keyTypeID, key C.int32_t, msg C.int64_t) C.int64_t {
logger.Debug("[ext_crypto_ed25519_sign_version_1] executing...")

instanceContext := wasm.IntoInstanceContext(context)
Expand Down Expand Up @@ -872,7 +872,7 @@ func ext_misc_print_hex_version_1(context unsafe.Pointer, dataSpan C.int64_t) {
}

//export ext_misc_print_num_version_1
func ext_misc_print_num_version_1(context unsafe.Pointer, data C.int64_t) {
func ext_misc_print_num_version_1(_ unsafe.Pointer, data C.int64_t) {
logger.Trace("[ext_misc_print_num_version_1] executing...")

logger.Debug("[ext_misc_print_num_version_1]", "num", fmt.Sprintf("%d", int64(data)))
Expand Down Expand Up @@ -932,7 +932,7 @@ func ext_misc_runtime_version_version_1(context unsafe.Pointer, dataSpan C.int64
}

//export ext_default_child_storage_read_version_1
func ext_default_child_storage_read_version_1(context unsafe.Pointer, childStorageKey C.int64_t, key C.int64_t, valueOut C.int64_t, offset C.int32_t) C.int64_t {
func ext_default_child_storage_read_version_1(context unsafe.Pointer, childStorageKey, key, valueOut C.int64_t, offset C.int32_t) C.int64_t {
logger.Debug("[ext_default_child_storage_read_version_1] executing...")

instanceContext := wasm.IntoInstanceContext(context)
Expand Down Expand Up @@ -979,7 +979,7 @@ func ext_default_child_storage_clear_version_1(context unsafe.Pointer, childStor
}

//export ext_default_child_storage_clear_prefix_version_1
func ext_default_child_storage_clear_prefix_version_1(context unsafe.Pointer, childStorageKey C.int64_t, prefixSpan C.int64_t) {
func ext_default_child_storage_clear_prefix_version_1(context unsafe.Pointer, childStorageKey, prefixSpan C.int64_t) {
logger.Debug("[ext_default_child_storage_clear_prefix_version_1] executing...")

instanceContext := wasm.IntoInstanceContext(context)
Expand All @@ -996,7 +996,7 @@ func ext_default_child_storage_clear_prefix_version_1(context unsafe.Pointer, ch
}

//export ext_default_child_storage_exists_version_1
func ext_default_child_storage_exists_version_1(context unsafe.Pointer, childStorageKey C.int64_t, key C.int64_t) C.int32_t {
func ext_default_child_storage_exists_version_1(context unsafe.Pointer, childStorageKey, key C.int64_t) C.int32_t {
logger.Debug("[ext_default_child_storage_exists_version_1] executing...")

instanceContext := wasm.IntoInstanceContext(context)
Expand Down Expand Up @@ -1036,7 +1036,7 @@ func ext_default_child_storage_get_version_1(context unsafe.Pointer, childStorag
}

//export ext_default_child_storage_next_key_version_1
func ext_default_child_storage_next_key_version_1(context unsafe.Pointer, childStorageKey C.int64_t, key C.int64_t) C.int64_t {
func ext_default_child_storage_next_key_version_1(context unsafe.Pointer, childStorageKey, key C.int64_t) C.int64_t {
logger.Debug("[ext_default_child_storage_next_key_version_1] executing...")

instanceContext := wasm.IntoInstanceContext(context)
Expand Down Expand Up @@ -1348,7 +1348,18 @@ func ext_hashing_twox_64_version_1(context unsafe.Pointer, dataSpan C.int64_t) C
//export ext_offchain_index_set_version_1
func ext_offchain_index_set_version_1(context unsafe.Pointer, keySpan, valueSpan C.int64_t) {
logger.Trace("[ext_offchain_index_set_version_1] executing...")
logger.Warn("[ext_offchain_index_set_version_1] unimplemented")
instanceContext := wasm.IntoInstanceContext(context)
runtimeCtx := instanceContext.Data().(*runtime.Context)

storageKey := asMemorySlice(instanceContext, keySpan)
newValue := asMemorySlice(instanceContext, valueSpan)
cp := make([]byte, len(newValue))
copy(cp, newValue)

err := runtimeCtx.NodeStorage.BaseDB.Put(storageKey, cp)
if err != nil {
logger.Error("[ext_offchain_index_set_version_1] failed to set value in raw storage", "error", err)
}
}

//export ext_offchain_local_storage_clear_version_1
Expand Down Expand Up @@ -1580,9 +1591,9 @@ func storageAppend(storage runtime.Storage, key, valueToAppend []byte) error {
}

// append new length prefix to start of items array
finalVal := append(lengthEnc, valueRes...)
logger.Debug("[ext_storage_append_version_1]", "resulting value", fmt.Sprintf("0x%x", finalVal))
storage.Set(key, finalVal)
lengthEnc = append(lengthEnc, valueRes...)
logger.Debug("[ext_storage_append_version_1]", "resulting value", fmt.Sprintf("0x%x", lengthEnc))
storage.Set(key, lengthEnc)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this updated? I think finalVal is more descriptive, as this isn't an encoded length

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was updated because DeepSource was complaining that append was appending to a variable with a different name. Yes, finalVal is more descriptive, I've update to use that instead.

return nil
}

Expand Down Expand Up @@ -1794,7 +1805,7 @@ func ext_storage_root_version_1(context unsafe.Pointer) C.int64_t {
}

//export ext_storage_set_version_1
func ext_storage_set_version_1(context unsafe.Pointer, keySpan C.int64_t, valueSpan C.int64_t) {
func ext_storage_set_version_1(context unsafe.Pointer, keySpan, valueSpan C.int64_t) {
logger.Trace("[ext_storage_set_version_1] executing...")

instanceContext := wasm.IntoInstanceContext(context)
Expand Down
21 changes: 21 additions & 0 deletions lib/runtime/wasmer/imports_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -409,6 +409,27 @@ func Test_ext_storage_set_version_1(t *testing.T) {
require.Equal(t, testvalue, val)
}

func Test_ext_offline_index_set_version_1(t *testing.T) {
// TODO this currently fails with error could nat find exported function, determine how else to test this
t.Skip()
inst := NewTestInstance(t, runtime.HOST_API_TEST_RUNTIME)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what if change from HOST_API_TEST_RUNTIME to NODE_RUNTIME?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried with NODE_RUNTIME, NODE_RUNTIME_V098, POLKADOT_RUNTIME and DEV_RUNTIME all with same results. I think we need to build a updated test runtime that contains the "rtm_" imports that reflect to call the actual "ext" functions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@EclesioMeloJunior the HOST_API_TEST_RUNTIME is created by the w3f spec team, all it does is directly call the ext_ function that is specified with the rtm_ prefix. however it doesn't contain all the exports we need, so we might need to update it ourselves


testkey := []byte("noot")
testvalue := []byte("washere")

encKey, err := scale.Encode(testkey)
require.NoError(t, err)
encValue, err := scale.Encode(testvalue)
require.NoError(t, err)

_, err = inst.Exec("rtm_ext_offline_index_set_version_1", append(encKey, encValue...))
require.NoError(t, err)

val, err := inst.ctx.NodeStorage.PersistentStorage.Get(testkey)
require.NoError(t, err)
require.Equal(t, testvalue, val)
}

func Test_ext_crypto_ed25519_generate_version_1(t *testing.T) {
inst := NewTestInstance(t, runtime.HOST_API_TEST_RUNTIME)

Expand Down
1 change: 1 addition & 0 deletions lib/runtime/wasmer/test_helpers.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ func setupConfig(t *testing.T, targetRuntime string, tt *trie.Trie, lvl log.Lvl,
ns := runtime.NodeStorage{
LocalStorage: runtime.NewInMemoryDB(t),
PersistentStorage: runtime.NewInMemoryDB(t), // we're using a local storage here since this is a test runtime
BaseDB: runtime.NewInMemoryDB(t), // we're using a local storage here since this is a test runtime
}
cfg := &Config{
Imports: ImportsNodeRuntime,
Expand Down