Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: move all storage to rust #780

Merged
merged 34 commits into from
Jul 5, 2021
Merged

feat: move all storage to rust #780

merged 34 commits into from
Jul 5, 2021

Conversation

Slesarew
Copy link
Contributor

@Slesarew Slesarew commented Jun 10, 2021

This is great rehaul to move all data storage into backend as recommended by RN design and testing good practices guides.

sled is used as db engine; data is stored in InternalStorage for Android and (TBD) for ios.

  • Android db storage
  • ios db storage
  • Store accounts in sled
  • Store metadata in sled
  • Store network specs in sled
  • Manage accounts
  • Manage metadata
  • Manage networks
  • Remove network and account contexts from RN
  • Identity management
  • Encrypt secrets with hardware stored keys
  • Ensure memory safety regarding unencrypted secrets persistence

Optional:

  • Tab view for accounts
  • Options screen rehaul
  • Metadata and types preview screen
  • Network specs preview screen

@Slesarew Slesarew self-assigned this Jun 10, 2021
@Slesarew
Copy link
Contributor Author

Slesarew commented Jul 5, 2021

I'll just squash it here so that everything gets built for preliminary tests; these changes should get into final version anyway.

@Slesarew Slesarew merged commit dc91388 into payload-parser-rust Jul 5, 2021
@Slesarew Slesarew deleted the rust-database branch July 5, 2021 12:27
Slesarew added a commit that referenced this pull request Sep 22, 2021
* fix: update metadata

* docs: created issue for stock metadata updates

* docs: lint

* feat: add generalized metadata storage mwp

* feat: metadata info on network details page (early stub)

* feat: invalid metadata message

* feat: built-in metadata loaded in constants

feat: metadata versions added to constants

* fix!: metadata is actuallty loaded;
removed som non-essential networks due to metadata loading issues

* fix: annoying hooks warning about updating component from another component fixed

* fix: smoother metadata loading; initiation moved to separate from fetching block

* ref: merged networkscontext and registriescontext

* fix: remove process race on init and small fixes

* fix: infinite useEffect cycle in payload card; probably there is more

* feat: show list of available metadata packages for given network

* feat: choose metadata from network settings submenu

* test: fix typos in unit tests

* test: fix tests and some lint/type errors

* feat: Metadata management scanner (#749)

* feat: MWP fast Rust-processed QR scanner frontend

* build: temporarily switch dependency rust-native to side branch

* feat: implement rust code for fountain parsing. Saving point for partial dependencies upgrade attempts

* feat: add metadata through fountain QR codes! (go to network settings>"network name">ManageMetadata>Fast Qr Scanner)

* fix: lint and types

* fix: typo in constants

* fix: multiple minor bugs

* refactor: return rust native libs where they belong (#751)

* ref: return rust native libs where they belong

* docs: lost gitignore

* refactor: move reference animated qr code generator to external repository

* build: move header files in ios build part

* build: add more memory for ios node build

* build: add more memory for ios node build in a different way

* build: revert ios build changes, modify bitrise script to allocate less memory

* build: duplicate headers for ios

* fix: Transaction parser (#754)

* feat: parse generic transaction. Ugly but now works.

* test: increase e2e timeout

* fix: lint

* feat: automatically generate built-in metadata (#755)

* feat: add rust code to automatically fetch and generate built-in metadata constants

* test: add extra paths for linter to ignore

* feat: automatically update constants/networkSpecs.ts with new metadata info

* fix: add blake2 hash to metadata handles; use it as ID if no spec_name or spec_version is available

* fix: blake2 hash in networkSpecs constants

* fix: add hashes for centrifuge and edgeware

* fix: metadata autogeneration lint

* build: update failure_derive to 1.8.0

* build: convenience reminder in build script

* fix: multiple fixes to allow versionless metadata

* test: fix unit tests to recognize versioned metadata

* refactor: faster metadata queries (#759)

* perf: moved metadata handle generator to rust

* refactor: lint and types

* build: try ti disable flipper

* fix: update metadata

* test: increase alert timeout

* test: typo in consts

* fix: merged last commit that was dropped from last squash for some reason

* feat: Metadata management page (#760)

* build: bump polkadot/api version to latest

* fix: red boxes on registering networks

* build: mess with upgrades and lint

* feat: can delete unused metadata (except for built-in probably)

* build: Upgrade polkadot-js 4 7 2 (#761)

* upgrade polkadot and react-native

* test: disabled failing tests temporarily

* test: restore tests

* fix: lint

* test: fix ts-ignore updated rule

* fix: lint and types

* test: bump nodejs version

* test: bump wasm-crypto needed for tests

* fix: update metadata

* build: bump node version for e2e

* build: bump node version in bitrise.yml

* test: teach signing test to scroll down

* test: fix addNewNetwork signing test with scrolling

* test: forgot to import scrolling tap everywhere it's needed

* fix: update westend metadata

* build: build rust for detox

* build: try to fix comment in macro build issue for zeroize_derive

* fix: typu in NetworkContext file

* build: update synstructure rust dep

* build: disable 32-bit ios targets

* build: remoce macro proc so that cross-compile works

* build: very ugly hack to address ios inability to build dynamic libs and rust-lang/cargo#4881

* fix: update metadata

* feat: loading screen stub

* lint

* fix: import react to splashscreen tab

* fix: metadata qr alignment error

* test: increase e2e timeout

* feat: loaded metadata preview screen

* test: switch ios and android e2e tests order

* test: add waiting for loading in beforeall e2e

* fix: lint

* test: typo

* fix: update metadata

* test: preparing for new e2e tests

* docs: bump license year in ts

* docs: bump license year in ios

* docs: bump version year in rust

* fix: gracefully fail to parse transaction if no metadata is available

* test: fix gradle api for detox

* test: remove waiting for alerts

* fix: lint

* feat: slightly prettier loading screen

* fix: lint

* fix: properly handle networks with unset metadata on loading

* fix: #767 and some lint

* fix: lint and types

* feat: slightly prettier transaction parser

* fix: types and lint

* test: attempt to run e2e on bitrise

* test: bump node version on bitrise

* test: swap ios and android testing again

* build: automatically set NDK_HOME with other variables

* test: download last reasonable config from bitrise

* test: increase verbosity in e2e test

* test: increase verbosity for e2e ci test on github

* test: wait for chooser screen after account creation in e2e

* test: this will be reverted - check that ios really clears memory between tests

* fix: lint and ios build

* test: preparing for new tests

* build: ios version fix attempt

* build: fixing ios build

* build: rename xcode folders

* build: update some deps

* build: remove ios/SubstrateSignTests

* fix: reverse ios module renaming mess

* build: ios folder cleanup

* build: move swift-bridge header around

* fix: Bridging header location

* fix: libsigner.a location

* build: try different places for static ios rust lib

* build: remove ndk address and some file duplicates

* build: ios linking

* test: some unfinished test specs added

* chore: update meta_reading

* chore: bump metadata reading version

* fix: gitignore

* fix: gitignore

* feat: rust transaction parser mwp demo

* fix: accept V13 metadata

* feat: stub for rust transaction parser call

* feat: mwp rust transaction parse demo (stub vor visualization)

* feat: add more cards to transaction renderer

* fix: lint and make some things prettier

* build: manually return linking lines to ios project file

* feat: migrate metadata db to insecure storage

* fix: accept V13 metadata

* chore: update metadata

* feat: add templates for cards: loading and error in transaction

* feat: add error messages cards generation and some refactor

* feat!: remove ethereum support, direct connection to transaction parser. Can't sign at the moment.

* refactor: tidy up transaction parser code

* feat: active SIGN button in details screen and some sanitation to remove legacy

* feat: prepare data transfer for signature and signing pin screen

* feat: signing is back! and with all 3 crypto algorithms as well!

* fix: show error messages on pin entry

* fix: remove silly testing message from really hard to find place

* feat: proper pin code error messages and start of rust storage testing (sled vs sqlite?)

* chore: update metadata

* fix: invalid payload screen crashing

* fix: temporarilty disable mortality check until upstream conforms

* fix: data to sign format correction

* feat: payload details look

* test: remove decoders test from RN suite (moved to Rust tests)

* test: remove units conversion test from RN suite (moved to Rust tests)

* fix: choose sled as db

* fix: lint

* fix: some types, lint and legacy removal

* fix: purge ethereum

* fix: ios support and more legacy purging

* ci: Rework Gitlab (#782)

* tighten up build scripts

* rewrite gitlab

* add signing

* fix

* enable

* update gitlab-ci.yml

* become unstuck

* fix

* add debug output

* fix

* fix

* fix

* add artifact

* feat: move all storage to rust (#780)

* FEAT: MFP sled database for android

* feat: network db handling utils

* docs: signing and parsing modules documentation

* fix: gitignore to ignore temporary db files in db handling tests

* refactor: move network specs constants to rust side

* fix: gitignore typo

* fix: gitignoring properly

* test: cleanup db tests locations

* feat: part of bridge to access new network specs storage from RN

* refactor: organize all cards and errors in transaction parsing into enums

* feat: network selector queries db. And some cleanup.

* test: add test feature to show all payload info cards

* feat: MWP metadata signing app example

* feat: MWP fetch and show derivations for given seed and network

* feat: removed RN data storages; app loads and fetches some data - MWP

* feat: prepare fetched metadata for signing and save to db instead of plaintext

* feat: account creation in db and signed metadata checks

* feat: generate metadata qr codes; also bridge Rust to RN

* feat: load type specs and some refactor

* feat: add hw backed key generation section to android

* fix: gitignore

* feat: automatic fetch chainspecs

* fix: create new seed

* fix: mwp address list screen

* feat: replace ethsign with XSalsa20Poly1305; just to save this state - this will be reverted

* feat: native credential-only unlock screen for Android (defunct)

* fix: unhang app on auth

* feat: HW keys for android (auth screen freezes thread, WIP)

* fix: proper thread locks for biometric prompt (Android)

* fix: fetch all seed names

* feat: add seed button

* feat: POC identity cards render

* fix: recover seed now works similar to create seed

* feat: general look for identity cards

* feat: bells and whistles on identity screen

* feat: send public key to UI on fetching relevant identities

* chore: remove some unused java deps

* chore: remove some unused polkadot js deps

* feat: export public account (uos step 1)

* chore: cleanup obsolete screens

* chore: more cleanup

* feat: derivation checker and identity creation backend

* ci: Gitlab build (#784)

* tighten up build scripts

* rewrite gitlab

* add signing

* fix

* enable

* update gitlab-ci.yml

* become unstuck

* fix

* add debug output

* fix

* fix

* fix

* add artifact

* remove test branch

* feat: expose identity creation functions

* feat: expose identity creation functions in RN; handle address collisions

* feat: add new identity

* feat: n+1 feature

* feat: backend to delete identities and suggest their names

* feat: delete identity

* major: tools for creating full signed offline QR updates

* feat: move prepopulated db to assets to copy on init

* feat: remove global context; make TOCnPP condition to populate db; app factory reset should work through deletion of app data now

* refactor: remove old rust code, some general cleanup

* fix: screen refresh on addition of account

* fix: refresh on address deletion

* refactor: RN cleanup

* feat: nice suggestions for seed names

* lint: RN lint and cleanup

* refactor: some types and more cleanup

* feat: automatic TOFU metadata, types and network specs import, fixed signing

* lint: cleanup meta_reading

* fix: signing and upgrading screens fixes

* feat: populate cold database from hot one (and some more cleanup)

* refactor: move all screens to one place

* feat: network settings screen

* feat: remove metadata, remove network

* CI: rework

* CI: poke the pipeline

* CI: metadata

* CI: make it work

* fix: network removal call properly

* fix: empty metadata does not crash; also some refactor and backend for listing all addresses

* fix: rn build android (#785)

* fix: disable broken minification with proguard

* revert: cat in ci

* docs: readme for db_handling

* fix: .h header for ios build

* fix: remove obsolete meta handle generator

* fix: cargo.toml cleanup for main module

* feat: use anyhow to propagate errors to ios

* feat: anyhow all rust code

* refactor: decrease code redundancy by helpers

* fix: rust ios return seedphrase on successful creation

* major: native ios wip dump

* feat: add rust-identicon dependency just to test build

* major: ios account management UI

* ci: remove yarn calls from e2e; should add tests later

* ci: remove ts lint types and unit tests

* ci: ga build script fix

* refactor: cleanup of junk files

* fix: simplify data model and fix some memory leaks and threading

* ci: scheme in actions

* feat: better rust qr parser and PC implementation as devtool

* feat: try to send svg string to ios

* fix: QR parser succeeds 1 frame sooner

* test: 1-frame multiframe decode unit test

* feat: transaction parsing and preview sketch

* feat: history backend

* feat: seed and network settings screens, TC and PP

* feat: bindings for history ios

* feat: ios history ui stub

* refactor: tidy up history in transaction parsing

* feat: rust QR code png renderer

* refactor: consolidate qr constants

* feat: integrate QR code generation into signing

* feat: ios history and hard types in tx and history ui cards

* fix: seriously, output is supposed to be binary-encoded hex symbols?!

* feat: new export pubkey qr code generator

* feat: identicons stub, faster qr videos

* feat: show identities for all seeds

* feat: show all seeds and minor UI fixes

* chore: cleanup RN

* docs: remove ETH, explain legacy situation

* docs: some tutorials

* docs: reference to uos

* fix: signing db checksum mismatch bugfix

* docs: mark how releases are tagged

Co-authored-by: Martin Pugh <martin@parity.io>
Co-authored-by: Martin Pugh <pugh@s3kr.it>
Co-authored-by: Denis P <denis.pisarev@parity.io>
Slesarew added a commit that referenced this pull request Sep 22, 2021
* fix: update metadata

* docs: created issue for stock metadata updates

* docs: lint

* feat: add generalized metadata storage mwp

* feat: metadata info on network details page (early stub)

* feat: invalid metadata message

* feat: built-in metadata loaded in constants

feat: metadata versions added to constants

* fix!: metadata is actuallty loaded;
removed som non-essential networks due to metadata loading issues

* fix: annoying hooks warning about updating component from another component fixed

* fix: smoother metadata loading; initiation moved to separate from fetching block

* ref: merged networkscontext and registriescontext

* fix: remove process race on init and small fixes

* fix: infinite useEffect cycle in payload card; probably there is more

* feat: show list of available metadata packages for given network

* feat: choose metadata from network settings submenu

* test: fix typos in unit tests

* test: fix tests and some lint/type errors

* feat: Metadata management scanner (#749)

* feat: MWP fast Rust-processed QR scanner frontend

* build: temporarily switch dependency rust-native to side branch

* feat: implement rust code for fountain parsing. Saving point for partial dependencies upgrade attempts

* feat: add metadata through fountain QR codes! (go to network settings>"network name">ManageMetadata>Fast Qr Scanner)

* fix: lint and types

* fix: typo in constants

* fix: multiple minor bugs

* refactor: return rust native libs where they belong (#751)

* ref: return rust native libs where they belong

* docs: lost gitignore

* refactor: move reference animated qr code generator to external repository

* build: move header files in ios build part

* build: add more memory for ios node build

* build: add more memory for ios node build in a different way

* build: revert ios build changes, modify bitrise script to allocate less memory

* build: duplicate headers for ios

* fix: Transaction parser (#754)

* feat: parse generic transaction. Ugly but now works.

* test: increase e2e timeout

* fix: lint

* feat: automatically generate built-in metadata (#755)

* feat: add rust code to automatically fetch and generate built-in metadata constants

* test: add extra paths for linter to ignore

* feat: automatically update constants/networkSpecs.ts with new metadata info

* fix: add blake2 hash to metadata handles; use it as ID if no spec_name or spec_version is available

* fix: blake2 hash in networkSpecs constants

* fix: add hashes for centrifuge and edgeware

* fix: metadata autogeneration lint

* build: update failure_derive to 1.8.0

* build: convenience reminder in build script

* fix: multiple fixes to allow versionless metadata

* test: fix unit tests to recognize versioned metadata

* refactor: faster metadata queries (#759)

* perf: moved metadata handle generator to rust

* refactor: lint and types

* build: try ti disable flipper

* fix: update metadata

* test: increase alert timeout

* test: typo in consts

* fix: merged last commit that was dropped from last squash for some reason

* feat: Metadata management page (#760)

* build: bump polkadot/api version to latest

* fix: red boxes on registering networks

* build: mess with upgrades and lint

* feat: can delete unused metadata (except for built-in probably)

* build: Upgrade polkadot-js 4 7 2 (#761)

* upgrade polkadot and react-native

* test: disabled failing tests temporarily

* test: restore tests

* fix: lint

* test: fix ts-ignore updated rule

* fix: lint and types

* test: bump nodejs version

* test: bump wasm-crypto needed for tests

* fix: update metadata

* build: bump node version for e2e

* build: bump node version in bitrise.yml

* test: teach signing test to scroll down

* test: fix addNewNetwork signing test with scrolling

* test: forgot to import scrolling tap everywhere it's needed

* fix: update westend metadata

* build: build rust for detox

* build: try to fix comment in macro build issue for zeroize_derive

* fix: typu in NetworkContext file

* build: update synstructure rust dep

* build: disable 32-bit ios targets

* build: remoce macro proc so that cross-compile works

* build: very ugly hack to address ios inability to build dynamic libs and rust-lang/cargo#4881

* fix: update metadata

* feat: loading screen stub

* lint

* fix: import react to splashscreen tab

* fix: metadata qr alignment error

* test: increase e2e timeout

* feat: loaded metadata preview screen

* test: switch ios and android e2e tests order

* test: add waiting for loading in beforeall e2e

* fix: lint

* test: typo

* fix: update metadata

* test: preparing for new e2e tests

* docs: bump license year in ts

* docs: bump license year in ios

* docs: bump version year in rust

* fix: gracefully fail to parse transaction if no metadata is available

* test: fix gradle api for detox

* test: remove waiting for alerts

* fix: lint

* feat: slightly prettier loading screen

* fix: lint

* fix: properly handle networks with unset metadata on loading

* fix: #767 and some lint

* fix: lint and types

* feat: slightly prettier transaction parser

* fix: types and lint

* test: attempt to run e2e on bitrise

* test: bump node version on bitrise

* test: swap ios and android testing again

* build: automatically set NDK_HOME with other variables

* test: download last reasonable config from bitrise

* test: increase verbosity in e2e test

* test: increase verbosity for e2e ci test on github

* test: wait for chooser screen after account creation in e2e

* test: this will be reverted - check that ios really clears memory between tests

* fix: lint and ios build

* test: preparing for new tests

* build: ios version fix attempt

* build: fixing ios build

* build: rename xcode folders

* build: update some deps

* build: remove ios/SubstrateSignTests

* fix: reverse ios module renaming mess

* build: ios folder cleanup

* build: move swift-bridge header around

* fix: Bridging header location

* fix: libsigner.a location

* build: try different places for static ios rust lib

* build: remove ndk address and some file duplicates

* build: ios linking

* test: some unfinished test specs added

* chore: update meta_reading

* chore: bump metadata reading version

* fix: gitignore

* fix: gitignore

* feat: rust transaction parser mwp demo

* fix: accept V13 metadata

* feat: stub for rust transaction parser call

* feat: mwp rust transaction parse demo (stub vor visualization)

* feat: add more cards to transaction renderer

* fix: lint and make some things prettier

* build: manually return linking lines to ios project file

* feat: migrate metadata db to insecure storage

* fix: accept V13 metadata

* chore: update metadata

* feat: add templates for cards: loading and error in transaction

* feat: add error messages cards generation and some refactor

* feat!: remove ethereum support, direct connection to transaction parser. Can't sign at the moment.

* refactor: tidy up transaction parser code

* feat: active SIGN button in details screen and some sanitation to remove legacy

* feat: prepare data transfer for signature and signing pin screen

* feat: signing is back! and with all 3 crypto algorithms as well!

* fix: show error messages on pin entry

* fix: remove silly testing message from really hard to find place

* feat: proper pin code error messages and start of rust storage testing (sled vs sqlite?)

* chore: update metadata

* fix: invalid payload screen crashing

* fix: temporarilty disable mortality check until upstream conforms

* fix: data to sign format correction

* feat: payload details look

* test: remove decoders test from RN suite (moved to Rust tests)

* test: remove units conversion test from RN suite (moved to Rust tests)

* fix: choose sled as db

* fix: lint

* fix: some types, lint and legacy removal

* fix: purge ethereum

* fix: ios support and more legacy purging

* ci: Rework Gitlab (#782)

* tighten up build scripts

* rewrite gitlab

* add signing

* fix

* enable

* update gitlab-ci.yml

* become unstuck

* fix

* add debug output

* fix

* fix

* fix

* add artifact

* feat: move all storage to rust (#780)

* FEAT: MFP sled database for android

* feat: network db handling utils

* docs: signing and parsing modules documentation

* fix: gitignore to ignore temporary db files in db handling tests

* refactor: move network specs constants to rust side

* fix: gitignore typo

* fix: gitignoring properly

* test: cleanup db tests locations

* feat: part of bridge to access new network specs storage from RN

* refactor: organize all cards and errors in transaction parsing into enums

* feat: network selector queries db. And some cleanup.

* test: add test feature to show all payload info cards

* feat: MWP metadata signing app example

* feat: MWP fetch and show derivations for given seed and network

* feat: removed RN data storages; app loads and fetches some data - MWP

* feat: prepare fetched metadata for signing and save to db instead of plaintext

* feat: account creation in db and signed metadata checks

* feat: generate metadata qr codes; also bridge Rust to RN

* feat: load type specs and some refactor

* feat: add hw backed key generation section to android

* fix: gitignore

* feat: automatic fetch chainspecs

* fix: create new seed

* fix: mwp address list screen

* feat: replace ethsign with XSalsa20Poly1305; just to save this state - this will be reverted

* feat: native credential-only unlock screen for Android (defunct)

* fix: unhang app on auth

* feat: HW keys for android (auth screen freezes thread, WIP)

* fix: proper thread locks for biometric prompt (Android)

* fix: fetch all seed names

* feat: add seed button

* feat: POC identity cards render

* fix: recover seed now works similar to create seed

* feat: general look for identity cards

* feat: bells and whistles on identity screen

* feat: send public key to UI on fetching relevant identities

* chore: remove some unused java deps

* chore: remove some unused polkadot js deps

* feat: export public account (uos step 1)

* chore: cleanup obsolete screens

* chore: more cleanup

* feat: derivation checker and identity creation backend

* ci: Gitlab build (#784)

* tighten up build scripts

* rewrite gitlab

* add signing

* fix

* enable

* update gitlab-ci.yml

* become unstuck

* fix

* add debug output

* fix

* fix

* fix

* add artifact

* remove test branch

* feat: expose identity creation functions

* feat: expose identity creation functions in RN; handle address collisions

* feat: add new identity

* feat: n+1 feature

* feat: backend to delete identities and suggest their names

* feat: delete identity

* major: tools for creating full signed offline QR updates

* feat: move prepopulated db to assets to copy on init

* feat: remove global context; make TOCnPP condition to populate db; app factory reset should work through deletion of app data now

* refactor: remove old rust code, some general cleanup

* fix: screen refresh on addition of account

* fix: refresh on address deletion

* refactor: RN cleanup

* feat: nice suggestions for seed names

* lint: RN lint and cleanup

* refactor: some types and more cleanup

* feat: automatic TOFU metadata, types and network specs import, fixed signing

* lint: cleanup meta_reading

* fix: signing and upgrading screens fixes

* feat: populate cold database from hot one (and some more cleanup)

* refactor: move all screens to one place

* feat: network settings screen

* feat: remove metadata, remove network

* CI: rework

* CI: poke the pipeline

* CI: metadata

* CI: make it work

* fix: network removal call properly

* fix: empty metadata does not crash; also some refactor and backend for listing all addresses

* fix: rn build android (#785)

* fix: disable broken minification with proguard

* revert: cat in ci

* docs: readme for db_handling

* fix: .h header for ios build

* fix: remove obsolete meta handle generator

* fix: cargo.toml cleanup for main module

* feat: use anyhow to propagate errors to ios

* feat: anyhow all rust code

* refactor: decrease code redundancy by helpers

* fix: rust ios return seedphrase on successful creation

* major: native ios wip dump

* feat: add rust-identicon dependency just to test build

* major: ios account management UI

* ci: remove yarn calls from e2e; should add tests later

* ci: remove ts lint types and unit tests

* ci: ga build script fix

* refactor: cleanup of junk files

* fix: simplify data model and fix some memory leaks and threading

* ci: scheme in actions

* feat: better rust qr parser and PC implementation as devtool

* feat: try to send svg string to ios

* fix: QR parser succeeds 1 frame sooner

* test: 1-frame multiframe decode unit test

* feat: transaction parsing and preview sketch

* feat: history backend

* feat: seed and network settings screens, TC and PP

* feat: bindings for history ios

* feat: ios history ui stub

* refactor: tidy up history in transaction parsing

* feat: rust QR code png renderer

* refactor: consolidate qr constants

* feat: integrate QR code generation into signing

* feat: ios history and hard types in tx and history ui cards

* fix: seriously, output is supposed to be binary-encoded hex symbols?!

* feat: new export pubkey qr code generator

* feat: identicons stub, faster qr videos

* feat: show identities for all seeds

* feat: show all seeds and minor UI fixes

* chore: cleanup RN

* chore: clean up old android code

* feat: UI initial commit

* feat: navigation

* feat: rust native calls linking

* docs: code comments everywhere

* feat: onboarding and db populate

* fix: active state for onboard

* feat: network selector

* feat: mwp auth screen invoker

* feat: after all can use encryptedSharedPreferences with biometric

* feat: mwp seed management

* feat: keys list

* feat: png tunnel and key menu

* feat: create and export keys

* feat: camera on homescreen

* feat: qr code parser

* feat: decode all qr messages into clean payload

* feat: transaction signing

* feat: mwp history screen in settings

* feat: history cards stub

* build: decrease minimum sdk level

* feat: some formatting for transaction details cards

* feat: scan progress indicator

* fix: prettier scanner progress bar

* chore: merge ios from master

* fix: compatibility of ios to android changes

Co-authored-by: Martin Pugh <martin@parity.io>
Co-authored-by: Martin Pugh <pugh@s3kr.it>
Co-authored-by: Denis P <denis.pisarev@parity.io>
Slesarew added a commit that referenced this pull request Sep 28, 2021
* feat: Metadata management page (#760)

* build: bump polkadot/api version to latest

* fix: red boxes on registering networks

* build: mess with upgrades and lint

* feat: can delete unused metadata (except for built-in probably)

* build: Upgrade polkadot-js 4 7 2 (#761)

* upgrade polkadot and react-native

* test: disabled failing tests temporarily

* test: restore tests

* fix: lint

* test: fix ts-ignore updated rule

* fix: lint and types

* test: bump nodejs version

* test: bump wasm-crypto needed for tests

* fix: update metadata

* build: bump node version for e2e

* build: bump node version in bitrise.yml

* test: teach signing test to scroll down

* test: fix addNewNetwork signing test with scrolling

* test: forgot to import scrolling tap everywhere it's needed

* fix: update westend metadata

* build: build rust for detox

* build: try to fix comment in macro build issue for zeroize_derive

* fix: typu in NetworkContext file

* build: update synstructure rust dep

* build: disable 32-bit ios targets

* build: remoce macro proc so that cross-compile works

* build: very ugly hack to address ios inability to build dynamic libs and rust-lang/cargo#4881

* fix: update metadata

* feat: loading screen stub

* lint

* fix: import react to splashscreen tab

* fix: metadata qr alignment error

* test: increase e2e timeout

* feat: loaded metadata preview screen

* test: switch ios and android e2e tests order

* test: add waiting for loading in beforeall e2e

* fix: lint

* test: typo

* fix: update metadata

* test: preparing for new e2e tests

* docs: bump license year in ts

* docs: bump license year in ios

* docs: bump version year in rust

* fix: gracefully fail to parse transaction if no metadata is available

* test: fix gradle api for detox

* test: remove waiting for alerts

* fix: lint

* feat: slightly prettier loading screen

* fix: lint

* fix: properly handle networks with unset metadata on loading

* fix: #767 and some lint

* fix: lint and types

* feat: slightly prettier transaction parser

* fix: types and lint

* test: attempt to run e2e on bitrise

* test: bump node version on bitrise

* test: swap ios and android testing again

* build: automatically set NDK_HOME with other variables

* test: download last reasonable config from bitrise

* test: increase verbosity in e2e test

* test: increase verbosity for e2e ci test on github

* test: wait for chooser screen after account creation in e2e

* test: this will be reverted - check that ios really clears memory between tests

* fix: lint and ios build

* test: preparing for new tests

* build: ios version fix attempt

* build: fixing ios build

* build: rename xcode folders

* build: update some deps

* build: remove ios/SubstrateSignTests

* fix: reverse ios module renaming mess

* build: ios folder cleanup

* build: move swift-bridge header around

* fix: Bridging header location

* fix: libsigner.a location

* build: try different places for static ios rust lib

* build: remove ndk address and some file duplicates

* build: ios linking

* test: some unfinished test specs added

* chore: update meta_reading

* chore: bump metadata reading version

* fix: gitignore

* fix: gitignore

* feat: rust transaction parser mwp demo

* fix: accept V13 metadata

* feat: stub for rust transaction parser call

* feat: mwp rust transaction parse demo (stub vor visualization)

* feat: add more cards to transaction renderer

* fix: lint and make some things prettier

* build: manually return linking lines to ios project file

* feat: migrate metadata db to insecure storage

* fix: accept V13 metadata

* chore: update metadata

* feat: add templates for cards: loading and error in transaction

* feat: add error messages cards generation and some refactor

* feat!: remove ethereum support, direct connection to transaction parser. Can't sign at the moment.

* refactor: tidy up transaction parser code

* feat: active SIGN button in details screen and some sanitation to remove legacy

* feat: prepare data transfer for signature and signing pin screen

* feat: signing is back! and with all 3 crypto algorithms as well!

* fix: show error messages on pin entry

* fix: remove silly testing message from really hard to find place

* feat: proper pin code error messages and start of rust storage testing (sled vs sqlite?)

* chore: update metadata

* fix: invalid payload screen crashing

* fix: temporarilty disable mortality check until upstream conforms

* fix: data to sign format correction

* feat: payload details look

* test: remove decoders test from RN suite (moved to Rust tests)

* test: remove units conversion test from RN suite (moved to Rust tests)

* fix: choose sled as db

* fix: lint

* fix: some types, lint and legacy removal

* fix: purge ethereum

* fix: ios support and more legacy purging

* ci: Rework Gitlab (#782)

* tighten up build scripts

* rewrite gitlab

* add signing

* fix

* enable

* update gitlab-ci.yml

* become unstuck

* fix

* add debug output

* fix

* fix

* fix

* add artifact

* feat: move all storage to rust (#780)

* FEAT: MFP sled database for android

* feat: network db handling utils

* docs: signing and parsing modules documentation

* fix: gitignore to ignore temporary db files in db handling tests

* refactor: move network specs constants to rust side

* fix: gitignore typo

* fix: gitignoring properly

* test: cleanup db tests locations

* feat: part of bridge to access new network specs storage from RN

* refactor: organize all cards and errors in transaction parsing into enums

* feat: network selector queries db. And some cleanup.

* test: add test feature to show all payload info cards

* feat: MWP metadata signing app example

* feat: MWP fetch and show derivations for given seed and network

* feat: removed RN data storages; app loads and fetches some data - MWP

* feat: prepare fetched metadata for signing and save to db instead of plaintext

* feat: account creation in db and signed metadata checks

* feat: generate metadata qr codes; also bridge Rust to RN

* feat: load type specs and some refactor

* feat: add hw backed key generation section to android

* fix: gitignore

* feat: automatic fetch chainspecs

* fix: create new seed

* fix: mwp address list screen

* feat: replace ethsign with XSalsa20Poly1305; just to save this state - this will be reverted

* feat: native credential-only unlock screen for Android (defunct)

* fix: unhang app on auth

* feat: HW keys for android (auth screen freezes thread, WIP)

* fix: proper thread locks for biometric prompt (Android)

* fix: fetch all seed names

* feat: add seed button

* feat: POC identity cards render

* fix: recover seed now works similar to create seed

* feat: general look for identity cards

* feat: bells and whistles on identity screen

* feat: send public key to UI on fetching relevant identities

* chore: remove some unused java deps

* chore: remove some unused polkadot js deps

* feat: export public account (uos step 1)

* chore: cleanup obsolete screens

* chore: more cleanup

* feat: derivation checker and identity creation backend

* ci: Gitlab build (#784)

* tighten up build scripts

* rewrite gitlab

* add signing

* fix

* enable

* update gitlab-ci.yml

* become unstuck

* fix

* add debug output

* fix

* fix

* fix

* add artifact

* remove test branch

* feat: expose identity creation functions

* feat: expose identity creation functions in RN; handle address collisions

* feat: add new identity

* feat: n+1 feature

* feat: backend to delete identities and suggest their names

* feat: delete identity

* major: tools for creating full signed offline QR updates

* feat: move prepopulated db to assets to copy on init

* feat: remove global context; make TOCnPP condition to populate db; app factory reset should work through deletion of app data now

* refactor: remove old rust code, some general cleanup

* fix: screen refresh on addition of account

* fix: refresh on address deletion

* refactor: RN cleanup

* feat: nice suggestions for seed names

* lint: RN lint and cleanup

* refactor: some types and more cleanup

* feat: automatic TOFU metadata, types and network specs import, fixed signing

* lint: cleanup meta_reading

* fix: signing and upgrading screens fixes

* feat: populate cold database from hot one (and some more cleanup)

* refactor: move all screens to one place

* feat: network settings screen

* feat: remove metadata, remove network

* CI: rework

* CI: poke the pipeline

* CI: metadata

* CI: make it work

* fix: network removal call properly

* fix: empty metadata does not crash; also some refactor and backend for listing all addresses

* fix: rn build android (#785)

* fix: disable broken minification with proguard

* revert: cat in ci

* docs: readme for db_handling

* fix: .h header for ios build

* fix: remove obsolete meta handle generator

* fix: cargo.toml cleanup for main module

* feat: use anyhow to propagate errors to ios

* feat: anyhow all rust code

* refactor: decrease code redundancy by helpers

* fix: rust ios return seedphrase on successful creation

* major: native ios wip dump

* feat: add rust-identicon dependency just to test build

* major: ios account management UI

* ci: remove yarn calls from e2e; should add tests later

* ci: remove ts lint types and unit tests

* ci: ga build script fix

* refactor: cleanup of junk files

* fix: simplify data model and fix some memory leaks and threading

* ci: scheme in actions

* feat: better rust qr parser and PC implementation as devtool

* feat: try to send svg string to ios

* fix: QR parser succeeds 1 frame sooner

* test: 1-frame multiframe decode unit test

* feat: transaction parsing and preview sketch

* feat: history backend

* feat: seed and network settings screens, TC and PP

* feat: bindings for history ios

* feat: ios history ui stub

* refactor: tidy up history in transaction parsing

* feat: rust QR code png renderer

* refactor: consolidate qr constants

* feat: integrate QR code generation into signing

* feat: ios history and hard types in tx and history ui cards

* fix: seriously, output is supposed to be binary-encoded hex symbols?!

* feat: new export pubkey qr code generator

* feat: identicons stub, faster qr videos

* feat: show identities for all seeds

* feat: show all seeds and minor UI fixes

* chore: cleanup RN

* chore: clean up old android code

* feat: UI initial commit

* feat: navigation

* feat: rust native calls linking

* docs: code comments everywhere

* docs: remove ETH, explain legacy situation

* docs: some tutorials

* docs: reference to uos

* feat: onboarding and db populate

* fix: active state for onboard

* feat: network selector

* feat: some support for metadata v14 (unstable as well as upstream)

* fix: hotfix BitVec u64 for 32bit archs

* feat: mwp auth screen invoker

* feat: after all can use encryptedSharedPreferences with biometric

* feat: docs in transaction preview and user comment for signing

* feat: mwp seed management

* feat: keys list

* feat: png tunnel and key menu

* feat: create and export keys

* feat: camera on homescreen

* feat: qr code parser

* feat: decode all qr messages into clean payload

* feat: transaction signing

* feat: mwp history screen in settings

* feat: history cards stub

* build: decrease minimum sdk level

* feat: some formatting for transaction details cards

* feat: scan progress indicator

* revert: use stable crates until metadata V14 is out

* fix: merge camera progress bar fix

* feat: replace ios navigation with flat custom one

* feat: docs and comments for transactions

* test: check links in docs

* ci: move links check on linux machine actions

* CI: chore

* CI: publish docs then check them

* CI: publish only /docs

* CI: move flag

* CI: use mdBook to compile book

* CI: debug mdbook

* CI: debug mdbook 2

* CI: mdbook latest version

* CI: mdbook dirty hack

* CI: mdbook use output dir

* CI: mdbook SUMMARY

* CI: mdbook builds

* CI: mdbook publish

* CI: linkcheck args

* CI: linkcheck args 1

* CI: linkcheck locally

* CI: mdbook serve in bkg

* CI: mdbook build first

* CI: another linkcheck

* CI: fix links

* CI: accidentially added

* CI: linkchecker fix links

* fix: fix most of settings screen

* docs: copy README to root for github

* chore: removed unused resources (moved some to docs)

* feat: custom focus-managing unified text input for ios

* fix: actually reset camera on home screen button

* feat: delete seed

* feat: search keys bar

* chore: cleanup master merge junk (revert)

Co-authored-by: Martin Pugh <martin@parity.io>
Co-authored-by: Martin Pugh <pugh@s3kr.it>
Co-authored-by: Denis P <denis.pisarev@parity.io>
Slesarew added a commit that referenced this pull request Oct 13, 2021
* test: switch ios and android e2e tests order

* test: add waiting for loading in beforeall e2e

* fix: lint

* test: typo

* fix: update metadata

* test: preparing for new e2e tests

* docs: bump license year in ts

* docs: bump license year in ios

* docs: bump version year in rust

* fix: gracefully fail to parse transaction if no metadata is available

* test: fix gradle api for detox

* test: remove waiting for alerts

* fix: lint

* feat: slightly prettier loading screen

* fix: lint

* fix: properly handle networks with unset metadata on loading

* fix: #767 and some lint

* fix: lint and types

* feat: slightly prettier transaction parser

* fix: types and lint

* test: attempt to run e2e on bitrise

* test: bump node version on bitrise

* test: swap ios and android testing again

* build: automatically set NDK_HOME with other variables

* test: download last reasonable config from bitrise

* test: increase verbosity in e2e test

* test: increase verbosity for e2e ci test on github

* test: wait for chooser screen after account creation in e2e

* test: this will be reverted - check that ios really clears memory between tests

* fix: lint and ios build

* test: preparing for new tests

* build: ios version fix attempt

* build: fixing ios build

* build: rename xcode folders

* build: update some deps

* build: remove ios/SubstrateSignTests

* fix: reverse ios module renaming mess

* build: ios folder cleanup

* build: move swift-bridge header around

* fix: Bridging header location

* fix: libsigner.a location

* build: try different places for static ios rust lib

* build: remove ndk address and some file duplicates

* build: ios linking

* test: some unfinished test specs added

* chore: update meta_reading

* chore: bump metadata reading version

* fix: gitignore

* fix: gitignore

* feat: rust transaction parser mwp demo

* fix: accept V13 metadata

* feat: stub for rust transaction parser call

* feat: mwp rust transaction parse demo (stub vor visualization)

* feat: add more cards to transaction renderer

* fix: lint and make some things prettier

* build: manually return linking lines to ios project file

* feat: migrate metadata db to insecure storage

* fix: accept V13 metadata

* chore: update metadata

* feat: add templates for cards: loading and error in transaction

* feat: add error messages cards generation and some refactor

* feat!: remove ethereum support, direct connection to transaction parser. Can't sign at the moment.

* refactor: tidy up transaction parser code

* feat: active SIGN button in details screen and some sanitation to remove legacy

* feat: prepare data transfer for signature and signing pin screen

* feat: signing is back! and with all 3 crypto algorithms as well!

* fix: show error messages on pin entry

* fix: remove silly testing message from really hard to find place

* feat: proper pin code error messages and start of rust storage testing (sled vs sqlite?)

* chore: update metadata

* fix: invalid payload screen crashing

* fix: temporarilty disable mortality check until upstream conforms

* fix: data to sign format correction

* feat: payload details look

* test: remove decoders test from RN suite (moved to Rust tests)

* test: remove units conversion test from RN suite (moved to Rust tests)

* fix: choose sled as db

* fix: lint

* fix: some types, lint and legacy removal

* fix: purge ethereum

* fix: ios support and more legacy purging

* ci: Rework Gitlab (#782)

* tighten up build scripts

* rewrite gitlab

* add signing

* fix

* enable

* update gitlab-ci.yml

* become unstuck

* fix

* add debug output

* fix

* fix

* fix

* add artifact

* feat: move all storage to rust (#780)

* FEAT: MFP sled database for android

* feat: network db handling utils

* docs: signing and parsing modules documentation

* fix: gitignore to ignore temporary db files in db handling tests

* refactor: move network specs constants to rust side

* fix: gitignore typo

* fix: gitignoring properly

* test: cleanup db tests locations

* feat: part of bridge to access new network specs storage from RN

* refactor: organize all cards and errors in transaction parsing into enums

* feat: network selector queries db. And some cleanup.

* test: add test feature to show all payload info cards

* feat: MWP metadata signing app example

* feat: MWP fetch and show derivations for given seed and network

* feat: removed RN data storages; app loads and fetches some data - MWP

* feat: prepare fetched metadata for signing and save to db instead of plaintext

* feat: account creation in db and signed metadata checks

* feat: generate metadata qr codes; also bridge Rust to RN

* feat: load type specs and some refactor

* feat: add hw backed key generation section to android

* fix: gitignore

* feat: automatic fetch chainspecs

* fix: create new seed

* fix: mwp address list screen

* feat: replace ethsign with XSalsa20Poly1305; just to save this state - this will be reverted

* feat: native credential-only unlock screen for Android (defunct)

* fix: unhang app on auth

* feat: HW keys for android (auth screen freezes thread, WIP)

* fix: proper thread locks for biometric prompt (Android)

* fix: fetch all seed names

* feat: add seed button

* feat: POC identity cards render

* fix: recover seed now works similar to create seed

* feat: general look for identity cards

* feat: bells and whistles on identity screen

* feat: send public key to UI on fetching relevant identities

* chore: remove some unused java deps

* chore: remove some unused polkadot js deps

* feat: export public account (uos step 1)

* chore: cleanup obsolete screens

* chore: more cleanup

* feat: derivation checker and identity creation backend

* ci: Gitlab build (#784)

* tighten up build scripts

* rewrite gitlab

* add signing

* fix

* enable

* update gitlab-ci.yml

* become unstuck

* fix

* add debug output

* fix

* fix

* fix

* add artifact

* remove test branch

* feat: expose identity creation functions

* feat: expose identity creation functions in RN; handle address collisions

* feat: add new identity

* feat: n+1 feature

* feat: backend to delete identities and suggest their names

* feat: delete identity

* major: tools for creating full signed offline QR updates

* feat: move prepopulated db to assets to copy on init

* feat: remove global context; make TOCnPP condition to populate db; app factory reset should work through deletion of app data now

* refactor: remove old rust code, some general cleanup

* fix: screen refresh on addition of account

* fix: refresh on address deletion

* refactor: RN cleanup

* feat: nice suggestions for seed names

* lint: RN lint and cleanup

* refactor: some types and more cleanup

* feat: automatic TOFU metadata, types and network specs import, fixed signing

* lint: cleanup meta_reading

* fix: signing and upgrading screens fixes

* feat: populate cold database from hot one (and some more cleanup)

* refactor: move all screens to one place

* feat: network settings screen

* feat: remove metadata, remove network

* CI: rework

* CI: poke the pipeline

* CI: metadata

* CI: make it work

* fix: network removal call properly

* fix: empty metadata does not crash; also some refactor and backend for listing all addresses

* fix: rn build android (#785)

* fix: disable broken minification with proguard

* revert: cat in ci

* docs: readme for db_handling

* fix: .h header for ios build

* fix: remove obsolete meta handle generator

* fix: cargo.toml cleanup for main module

* feat: use anyhow to propagate errors to ios

* feat: anyhow all rust code

* refactor: decrease code redundancy by helpers

* fix: rust ios return seedphrase on successful creation

* major: native ios wip dump

* feat: add rust-identicon dependency just to test build

* major: ios account management UI

* ci: remove yarn calls from e2e; should add tests later

* ci: remove ts lint types and unit tests

* ci: ga build script fix

* refactor: cleanup of junk files

* fix: simplify data model and fix some memory leaks and threading

* ci: scheme in actions

* feat: better rust qr parser and PC implementation as devtool

* feat: try to send svg string to ios

* fix: QR parser succeeds 1 frame sooner

* test: 1-frame multiframe decode unit test

* feat: transaction parsing and preview sketch

* feat: history backend

* feat: seed and network settings screens, TC and PP

* feat: bindings for history ios

* feat: ios history ui stub

* refactor: tidy up history in transaction parsing

* feat: rust QR code png renderer

* refactor: consolidate qr constants

* feat: integrate QR code generation into signing

* feat: ios history and hard types in tx and history ui cards

* fix: seriously, output is supposed to be binary-encoded hex symbols?!

* feat: new export pubkey qr code generator

* feat: identicons stub, faster qr videos

* feat: show identities for all seeds

* feat: show all seeds and minor UI fixes

* chore: cleanup RN

* chore: clean up old android code

* feat: UI initial commit

* feat: navigation

* feat: rust native calls linking

* docs: code comments everywhere

* docs: remove ETH, explain legacy situation

* docs: some tutorials

* docs: reference to uos

* feat: onboarding and db populate

* fix: active state for onboard

* feat: network selector

* feat: some support for metadata v14 (unstable as well as upstream)

* fix: hotfix BitVec u64 for 32bit archs

* feat: mwp auth screen invoker

* feat: after all can use encryptedSharedPreferences with biometric

* feat: docs in transaction preview and user comment for signing

* feat: mwp seed management

* feat: keys list

* feat: png tunnel and key menu

* feat: create and export keys

* feat: camera on homescreen

* feat: qr code parser

* feat: decode all qr messages into clean payload

* feat: transaction signing

* feat: mwp history screen in settings

* feat: history cards stub

* build: decrease minimum sdk level

* feat: some formatting for transaction details cards

* feat: scan progress indicator

* revert: use stable crates until metadata V14 is out

* fix: merge camera progress bar fix

* feat: replace ios navigation with flat custom one

* feat: docs and comments for transactions

* test: check links in docs

* ci: move links check on linux machine actions

* CI: chore

* CI: publish docs then check them

* CI: publish only /docs

* CI: move flag

* CI: use mdBook to compile book

* CI: debug mdbook

* CI: debug mdbook 2

* CI: mdbook latest version

* CI: mdbook dirty hack

* CI: mdbook use output dir

* CI: mdbook SUMMARY

* CI: mdbook builds

* CI: mdbook publish

* CI: linkcheck args

* CI: linkcheck args 1

* CI: linkcheck locally

* CI: mdbook serve in bkg

* CI: mdbook build first

* CI: another linkcheck

* CI: fix links

* CI: accidentially added

* CI: linkchecker fix links

* fix: fix most of settings screen

* docs: copy README to root for github

* chore: removed unused resources (moved some to docs)

* feat: custom focus-managing unified text input for ios

* fix: actually reset camera on home screen button

* feat: delete seed

* feat: search keys bar

* feat: different crypto types

* fix: make all cards show public again

* chore: update db in ios

* docs: fix missing links and remove more outdated stuff

* feat: move all remaining logic away from FFI boundary file

* chore: update db

* docs: remove generate message tutorial link

* chore: cleanup side effects from master merge (revert)

* fix: parser-formatter bugs

* fix: history new identity fields in iOS

* fix: respect QR code versions below 10

* fix: return docs to transaction cards and format them slightly better

* fix: remove ios autosuggest on sensitive fields

* feat: orphan add key when seed is selected

* fix: add network should update db

* feat: swipe to change exported key

* fix: safer array bound check

* feat: markdown support for terms and conditions

* feat: markdown in transaction docs

* chore: fix merging artefacts

* chore: return cargo.lock

* chore: change rococo genhash

* fix: proper network genesis hash in seed export

Co-authored-by: Martin Pugh <martin@parity.io>
Co-authored-by: Martin Pugh <pugh@s3kr.it>
Co-authored-by: Denis P <denis.pisarev@parity.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant