v0.50.0: Integrate frame-decode, redo storage APIs and break up Error.

[jsdw]

Intro / Plan

This is the first PR towards integrating subxt and subxt-historic into a single library capable of working well with current and historic blocks.

The main gist of this PR is incorporating new storage APIs into Subxt inspired from subxt-historic, breaking apart the big Error type returned everywhere into smaller types (while also improving the overall error feedback), and integrating frame-decode to handle all decoding/encoding logic in subxt_core.

I would propose that given the extent of the changes that this merge will bring, that we target a new version—v0.50.0—to signify the extent of the changes.

I would propose the following steps around this:

• This release will live on a v0.50.0 branch while we get the changes needed ready.
• Once ready, we'll do an initial release, publish it and ask for feedback. We'll have a window (eg 3 months) under which we will support both v0.50.0 and previous version to give people time to migrate. (We'll also write a migration guide to cover the big things)
• We'll bugfix etc on the current master branch, so v0.4x.x releases may still occur in the meantime to push out features anf fixes. subxt-historic will be maintained on this branch as needed.
• After the mgiration window is up we will merge v0.50.0 into master and maintain only this version going forwards.

Changes in this PR

With that all said, the main changes in this PR:

• Redo storage APIs to better handle static and dynamic values (learning from subxt-historic API work). Make use of frame_decode and traits like IntoEncodableValues and IntoDecodableValues for easier usage.
• Redo error handling; instead of returning a single Error everywhere, we now return a more specific error from each function. These more specific errors can all be convetred into Error via ?. We do this for more specific / better error reporting while maintaining 99% of the simplicity of a single error type.
• Integrate frame-decode into subxt_core and have it handle all encoding / decoding things.

Many smaller changes which stem from these:

• Alter some other address/payload traits to make use of IntoEncodableValues etc traits for easier encoding and consistency with storage API redo.
• Alter subxt::foo::dynamic functions so that instead of only accepting/returning scale_value::Values, the user can specify the types. (plan is to make simple str/(str,str) usage available for that Value type instead, so the dynamic fns become about picking types.
• Change how storage entry information is stored in metadata to make using it more efficient (and simpler) and make it easy to lean on frame_decode for core logic.
• Add proper Yes/Maybe/No util types/trait rather than ad hoc Yes and () types.
• Redo storage codegen to accomodate new APIs, and other codegen to accomodate address/payload trait changes and generally consistify things.
• Add missing std features that caused some compile issue.
• Change some functions from just returning scale_value::Value to returning whatever type the user wants (inc the former) so long as it impls DecodeAsFields. For consistency these are renamed to .decode_as_fields() too.
• Remove subxt_core::metadata mod to avoid lots of code dupe and lean more on frame_decode traits instead everywhere; no more DecodeWithMetadata and DecodedValueThunk as they aren't compatible with using frame_decode for things.
• Add get_bytes for constants and custom values to return byutes, now that DecodedValueThunk is no longer a thing (more efficient anyway).
• While working on storage APIs, remove a layer of indirection that turned out not to be helpful (enum of IsMap or IsPlain storage entry).
• References to Addresses/Payloads are valid Addresses/Payloads too now, and they are taken by value everywhere, which means one can pass references or owned addresses/payloads to any relevant function. Preserves ability to pass refs in for storage queries which would otherwise need to be passed by value now.
• Update metadata conversion from v14/v15/v16 to accommodate above changes.
• Update the subxt book (docs) to accomodate these changes.

Not done here but I'd like to do / think about doing in followup PRs:

• Move .at/.at_latest higher up as in subxt-historic, so step 1: create client, step 2: client.at(block) / client.at_latest(), step 3: client_at_latest.storage().entry(..) / client_at_latest.constants().. etc. Submitting TX and subscribing to blocks are exceptions and will not live under this.
• Runtime APIs and View Function APIs could also be adapted to be consistent with Storage APIs. Advantage aside from consistency would be reducing codegen by moving argument handling into runtime code like with storage APIs. Would also enable ("Metadata", "metadata_versions") like addresses for easy dynamic access and removing need to use dynamic API unless you want to specify types up front (again like storage APIs).
• More work towards removing functions from subxt_metadata::Metadata and relying more on frame_decode traits. Less important in the new plan but less functionality exposed here == more flexibility with how metadata is stored.
• Storage iter stream is impl Stream; can we make this nicer? (It has a lifetime req)

Notes for reviewers

A lot of the changes are just related to Error -> MoreSpecificError everywhere, and then additions to imporove the error reporting in many of these places.

Main things that are worth looking at:
- subxt/examples/* - have a look and see the main API changes via the examples here.
- core/src/storage/* - new storage APIs in subxt-core
- subxt/src/storage/* - new storage APIs in subxt proper.
- subxt/src/error/mod.rs - the new errors.
- subxt/src/book/* - documentation updates to accomodate the changes (do they make enough sense for now?)
- codegen/src/api/storage.rs - codegen changes for new storage APIs (other codegen changes may also be worth a glance; generally aiming to ismplify/consistify things here)

From the main changes, anything that seems dumb or could be done better or just general questions you have around why things are done in any given way would be very welcome, but ofc anything you want to remark on is :)

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	cargo/​scale-info-legacy@​0.2.3 ⏵ 0.2.4			-3
	cargo/​scale-value@​0.18.1
	cargo/​frame-decode@​0.10.0 ⏵ 0.11.1

View full report

[jsdw]

I'm going to merge this to be the base of the v0.50.0 branch for now. This will give something to build on :)

[TarikGul]

Hey so I actually looked through the PR to get familiar with it, and it looks good to me! Thanks for the progress on this :)

[jsdw]

That's amazing, thank you!