Aaron Luhning, Author at Vital Point AI

All posts by Aaron Luhning

Protected: NEAR Dapp Storage With Textile ThreadsDB

This content is password protected. To view it please enter your password below:

Building a Decentralized Community Platform on NEAR

In mid-June 2020, I joined the NEAR Guild Program. NEAR Guilds take many forms but in essence, they are communities supporting the NEAR ecosystem in some way.  My guild is Vital Point Guild and it's objective is to help people learn to develop using NEAR protocol by focusing on building and releasing projects for real world issues.

Once accepted, guild leaders become part of the NEAR guild community and are provided with a community platform to interact, share ideas and ask questions.  We are rewarded with platform tokens for one off bounty and routine regular engagement which will eventually be exchangeable for NEAR tokens.  

It's up to us what we do with the tokens, but I believe the majority of us will be investing them back into our guilds, rewarding our own members for their contributions to help build out the ecosystem.

Rise of an Idea

Tribe - cloud based community platform

The community platform currently used by NEAR for leader guild engagement portal is Tribe - a cloud based solution.  It's white-label meaning it can be branded to an organization's needs and hosts a wide array of features focused on creating lively and engaging communities.  Of note, it incorporates a community token that can be distributed to an organization's members.

On joining the guild program and accessing the portal, I immediately thought it kind of odd that a layer 1 blockchain protocol like NEAR intent on ushering in a Web 3.0 world with decentralization uses Web 2.0 tools to build its communities.

I know Web 2.0 and Web 3.0 can co-exist nicely, but given the speed of NEAR transactions and inexpensive gas usage, it seemed very possible that one could build a decentralized community platform with the same gamification and engagement features as Tribe directly on NEAR.  The one thing I wasn't too sure on was how to store all the data associated with such an application - so I stopped thinking about it and carried on with other things. 

Until...

Along Comes EthGlobal and HackFS

A few weeks later - mid July 2020 - I found out about HackFS - an Eth Global Hackathon.  Having competed in two previous EthGlobal events - both outstanding - I was immediately interested in the focus of this event - IPFS, Filecoin and decentralized storage.  

Taking a look at the participating sponsors, I learned about Textile.io and suddenly saw the very real possibility of integrating NEAR with Textile, thus solving my storage dilemma, to create a decentralized social media community platform native to NEAR - potentially even as a future replacement to Tribe and other centralized community platform offerings.

At very least, I reasoned that even if nobody else turns out to be interested in such a product - I wanted it built for my guild - Vital Point Guild.  Investing the time and effort into building such a decentralized community platform on NEAR complete with NEAR wallet/token integration serves a couple purposes:

  • provides basis of a real world project for our upcoming VP Guild Academy that I will use to teach new NEAR based developers how to tackle real-world issues;
  • provides my guild with a platform that is completely customizable and not limited to a set of features inherent in existing platforms;
  • gives everyone using it ownership of any data provided - they can leave it, delete it, or move it at will; 
  • provides community components that will be useful to integrate into other products I'm building such as Paralog; and
  • while not Ethereum based - provided a great project to work on for the HackFS hackathon given the decentralized storage nature of the application.

Sidenote...

in my opinion, hackathons are awesome events to level-up your development and blockchain skills.  You get introduced to new technologies, access the experts/mentors, and generally just come out the other end so much better and further ahead than when you went in...

Of course I signed up...

It was a no-brainer and and I have spent the majority of my free time over the last month building out my vision.  As these things usually go - what I've got after 30 days of part-time work isn't even half of what I envisioned and hoped to achieve 30 days ago.

Things happen, life gets in the way and the tech you're building on constantly changes (testnet timeouts and well intentioned api/sdk updates consume hours of your life to track down new issues and update code that you just managed to get working - sarcasm intended ).

Despite the challenges, I'm super happy with the learning that happened and even though it's not anywhere near perfect or production ready, it's a fantastic start and proves that a decentralized community platform built on NEAR and Textile (IPFS) is both doable and useable.

Whether it's enough to win any HackFS prizes is another story altogether...guess we'll find out in a week or so, but given the quality of talent, the teams, and projects that come out of EthGlobal hackathons - won't be placing any bets on myself.  Some extremely talented folks out there.  I just count myself lucky to be able to glean bits and pieces of their intelligence here and there.

Here's My HackFS Submission

Part of the HackFS judging is a 4 min video detailing the project to be followed up with a 3-4 minute Q&A session.  I can tell you it is super hard to get a project you've spent 30 days working on into a 4 min backgrounder and demo.

Word of Advice...

if you're in a hackathon needing a video, leave a couple days to get it together - took a lot longer than expected.

There is a live VP Guilds demo available - deployed on Fleek (which is also extremely cool, super easy to use and a great alternative to things like Netlify if you want to deploy to IPFS).  Fleek also uses Textile - definitely recommend considering them for your future projects.

So, how did I build it?

You may have been following another post I wrote recently regarding using NEAR accounts as identities for Textile.  Central to my whole plan was linking NEAR accounts to Textile threadsDB threadIds.  If you're not familiar with Textile - think of it as a database provider.  You can run your own on your hardware or use their hub - which I've opted to do for the hackathon.  In future, I understand it's trivial to leave the hub and run everything through my own instances if so desired.

Step 1:  Associate NEAR accounts to Textile ThreadIDs

First thing I needed to do was use a NEAR account to authenticate against Textile to link a database (threadId and identity) to the NEAR account.  Like the NEAR wallet does now, the Dapp stores that important info in browser local storage and queries an authentication server to get a token that allows the user to interact with their Textile user database (thread).

Textile allows both user and app level databases (threads) to be created.  User level threads mean data put there is only available to the user whereas app level threads are created with an account key that makes any data in those threads available to the whole app.  From a decentralized community standpoint, this is important because I wanted users to have the option of determining exactly what info they shared with everyone vs what they keep private for themselves.  I'll come back to that in a minute.

Step 2:  Make sure nobody loses their data

Storing things in browser local storage works fine until that storage gets cleared - then the user loses access to their data forever - not good.  To get around that, I not only store the identity and threadId in browser local storage, but I get an encryption key from the authentication server and encrypt and store the textile identity and threadId in PersistentMaps on NEAR.  

As an aside...

It was never my intention to stand-up an authentication server to provide that encryption key.  I spent many, many hours trying to figure out how to prove identity using some kind of signing in a way that I could use that as the encryption mechanism - never quite got there and had to move on...  Probably revisit at some point.

Thus, when a user revisits the Dapp, we first search their local cache for an identity.  If not there, we query the NEAR chain for it.  If it still does not exist - it's either because it's the first time the user has used the app or I've blown away the old contract and updated it (of course, that won't happen when it's live and the keys are removed).  At any rate, the identity is retrieved and local storage is restored so the Dapp can function properly.

Step 3:  Start making the community platform useful

With the identity, linkage to NEAR account, and authentication sign in/sign out stuff sorted out - let's turn our attention to the Dapp features itself.  

Right now, it's pretty underwhelming - simply manages user profiles, allows pers to create and submit news or posts, comment on, edit, update, and/or delete those posts, see lists of all users, and see their current NEAR wallet balance.  Nothing earth-shattering, but the cool thing is that it's all done on IPFS through Textile and the functionality all comes from the NEAR contract powering it.

The profiles is something I'll definitely be coming back to in the future.  See that as a standalone composable component that could be built out for NEAR in much the same way 3Box does for Ethereum.

Managing data accessibility

I mentioned earlier that I wanted to ensure that people have a way to keep their data private or publish it for the dapp to aggregate and display.  I've done that by creating an app level database and individual user databases (threads in Textile speak).  

If a user writes a post and toggles it to publish, it gets saved in both the user and app databases.  If they subsequently mark it as unpublished or delete it, the app and user databases get updated accordingly.  Doing these gives the dapp the ability to track published and draft posts per user.  I believe Textile is working on more fine-grained access control which might supplant this design in future - but that's how I've got it working now.

Link to data off chain - unique IDs to IPFS through Textile

I'm currently using the NEAR chain to store hash ids to their respective post data, comment data, profile data and so on on IPFS through the Textile hub.  Pretty sure that's normal for most blockchain apps and by doing so, I limit what is getting stored on chain to the tombstone/important info.  

Data integrity...

The Dapp currently generates a verification hash for any data that gets posted.  Why I built this in - I'm not 100% sure.  I think I intend to use it in future to verify that info is in the original state it was provided when recalled from IPFS via Textile.  Not really sure I even need to bother with this as I believe IPFS CIDs naturally ensure the content hasn't changed (otherwise the CID would have changed...).  Regardless, it's currently baked in - not doing much, but it's there.

Textile threads refer to collections which are basically database table schemas.  I build one for each type of data the Dapp needs to keep track off and they're linked together via ids just like any relational database would work.  The Dapp currently has collections for user profiles, comments, news posts, categories, and so on... all the typical stuff you'd find in a WordPress installation.

Step 4:  Deploy it and start getting feedback

I chose to deploy on Fleek and was incredibly, pleasantly surprised.  Pretty much identical to what you can do on Netlify, but it puts everything on IPFS.  Took about 10 min to learn the platform and get the app up and functioning.  How is that not the future of hosting?

In Closing

Because HackFS is an EthGlobal hackathon, I want to quickly address why I didn’t choose to build this on Ethereum.  

While Eth 2.0 may change the equation, in my opinion it is unreasonable to believe that people will choose a community platform that doesn’t offer the same general performance at the same general price point as existing centralized platforms.  Ethereum simply costs too much and takes too long to be useful for the community platform I envision without using a layer 2 solution.

When HackFS launched, I was just becoming familiar with NEAR protocol and thought it was a great opportunity to explore it more in depth. NEAR is now what Eth 2.0 may be able to achieve in the future. It’s fast and cheap and has some other features that I believe are essential to adoption of any decentralized social media type of application.  

Even though I’m building this on NEAR – there is an Eth/NEAR bridge which was part of my original vision - just wasn't able to get it implemented by submission time.  I think everyone benefits when everything works together.

Would love to hear what you think - leave a comment/feedback.  Until next time, happy developing/building/blockchaining or whatever you do.

Near Textile Integration

NEAR Accounts and Access Keys as Identities for Textile Integration

A huge foundational component of a couple of the DApps I’m currently building on NEAR are centered on user interactions and contributions in a community setting.  One of them is a project I started this week that will even be a submission for HackFS.  In attempting to architect them in a decentralized fashion, I need the ability to use NEAR accounts as identity providers to other decentralized services – specifically Textile for IPFS database and file storage.

A Quick Textile Primer

Textile is proving to be very useful for what I’m doing with NEAR.  I think it’s a normal pattern to store high value info on chain and then link to a hash of information requiring more storage or of lesser importance stored on IPFS or some other centralized or decentralized service (like OpenSea does with NFT images) to save on costs associated with contract storage. 

Textile caught my attention because of their threadsDB service.  It allows us to build application or user-level databases that contain collections.  These collections are basically database tables defined by a JSON schema that hold instances or records.

It’s basically the same construct one can find in most centralized databases of many of today’s apps – like a WordPress installation.  Textile even suggests that working with it will feel familiar to developers who have worked with technologies like MongoDB. 

The instance data inserted into a ThreadsDB collection is encrypted by default and is good for meta data type info.  Larger files including images, videos, and other datasets will typically utilize another offering they provide called Buckets.  Together, they are an abstraction on IPFS that I see as opening the door to upgrading and improving many existing web 2.0 apps to web 3.0 versions while enabling further innovation.  In addition to the Textile Hub which I’m going to talk about more below, Textile has a fourth service – Powergate that adds Filecoin storage onto all of this.  I’m currently envisioning use for it as a long-term storage/availability option in my DApps to ensure users can retrieve their data (think backups or archives).

Using NEAR as an Identity Provider

As alluded to - Textile has a hub which is a portal that allows developers to access IPFS and Filecoin (eventually). 

The hub makes it easy to manage and update, persist, deploy, and scale information on IPFS.  It requires an identity to function and because it supports public-key infrastructure (PKI) any identity provider that operates within those parameters can be integrated (3Box, Metamask, uPort, Blockstack or derive your own).

The general flow of how the identity gets authorization to work is as follows:

  • A user attempts to sign-in providing their public key (can be an app level user account key or an organization level key that provisions user keys to allow per-user data storage)
  • The Dapp interacting with Textile creates a one-time challenge for the user
  • The user signs the challenge with their private key to create credentials
  • The Dapp verifies the credentials and allows access

This is far outside my field of expertise, but I believe the whole token generation/credential issuance setup here is typically done by a backend API server with access to a secret key.  The Textile hub is a token generation endpoint and also does credential verification.

How I am achieving NEAR integration with Textile

It’s very easy to generate an identity using the Libp2p crypto library which is made avail in the @textile/threads-core library.

import {Libp2pCryptoIdentity} from '@textile/threads-core';


async function example () {

   /** Random new identity */

   const identity = await Libp2pCryptoIdentity.fromRandom()


   /** Convert to string. */

   const identityString = identity.toString()


   /** Restore an identity object from a string */

   const restored = Libp2pCryptoIdentity.fromString(identityString)

}

I started there and initially thought about using browserlocalstorage to store the plain text identity string created by libp2p.  However, I needed a way to link that identity string to the user’s NEAR account – otherwise I’d get a new identity string every time I logged into the app.  That would mean any info I had thrown into collections with threadIds linked to that identity would no longer be accessible.

I decided to create that association on the NEAR blockchain by creating a data model (persistentMap) to store the Textile identity string with the account sending it to be stored (current user’s account name).  

So, at this point, with a bit of logic in the client – the Dapp successfully looked for an existing identity for the currently logged in user.  It either retrieved it from the blockchain or if an identity didn’t exist (new user) the Dapp created a new random identity using Libp2p and stored it on chain for that user to retrieve the next time they logged in.  I also cached it in browserlocalstorage to speed things up – adding that as an initial check for the identity into the Dapp as well.

Happy at First...

At this point, I was pretty stoked, the Dapp was functioning how I wanted allowing different users to login, use an existing or new identity to retrieve a token/credentials from Textile and post/retrieve data from their own user-level databases on IPFS.  My happiness eroded though when I thought a bit about security and the fact that the user identities were being stored in plain text on chain and in browserlocalstorage.  I needed to take things a bit further.

That’s when I started searching for some kind of stable key or data in the NEAR account that I could use as an encryption key to encrypt the identities before they get stored on chain and decrypt them when retrieved for use. 

I eventually achieved the encryption/decryption objective, but the only option I could find at the time was the user’s NEAR public key of one of their full access keys – which, of course, doesn’t increase security at all – just adds another step that some determined hacker needs to figure out before eventually deciphering the encrypted identity.

NEAR Experts to the Rescue

I spent the next several days trying to figure out how NEAR access keys worked, reading and re-reading the documentation here and here, trying things out and ultimately ended up pretty damn confused about how NEAR was handling things.  For instance, I found:

  1. New contract level access keys were created every time an account logged in and out – authorizing the Dapp each time - and it wasn’t a behaviour restricted to my Dapp.  It happened with the default create-near-app template as well.  In my mind, it seemed odd and after going through and deauthorizing 80+ access keys in my account that it generated from various testing, I questioned why the wallet or dapp wouldn’t just check to see if an access key existed and use that.  I noticed there was a getAccessKeys() method – figured it would be simple enough to do a check or perhaps add a deauthorization step into the logout flow to delete the key after use.
  2. A number of full access keys were appearing in my NEAR account wallet – seemingly at random with much less frequency than the contract access keys – and I wasn’t really sure why.

By this time, I was also close to insanity as I couldn’t find any functions or methods or anything in the NEAR account or wallet objects that I could get at that wouldn’t eventually change or disappear.  I was coming up short in finding something I could use for that encryption key and I was doing my best to avoid having to setup a central server to act as an API to issue that key (assuming, with my limited knowledge of that subject that that was what I needed to do). 

Luckily, the wonderfully knowledgeable people at NEAR including Matt Lockyer, Kendall Cole,  and Mike Purvis got me on a call and pointed me in a new direction.     

To be clear...

The willingness of these fine gentlemen to help and decipher what I was trying to say is a big deal.  While I’m improving and doing my best to grasp all the tech here, I didn’t study computer science or programming and am completely self-taught.


All of this is so far outside of what I normally do in my day-to-day professional life that I wasn’t quite sure whether I had a good enough grasp of the jargon/tech to be able to explain my issues in a way that they’d be able to understand.  From my perspective, it went better than I thought it might in no small part to the patience and understanding they showed.  


They may have a completely different perspective… 

First, they answered my questions above about the access keys:

(Note that I’m paraphrasing my understanding – if anyone wants to expand on this in the comments below, feel free in case it’s more nuanced or I’m saying something that isn’t really the case):

  1. New access keys are created on every login/logout because NEAR doesn’t have enough information to decide when to delete a contract key.  Erring on the side of caution, they let them accumulate so as not to deauthorize a key that a Dapp needs in some fashion.  They indicated they’re looking into how to make this more intuitive in the wallet (perhaps things like indicating when last time that key was used to sign something).
  2. Full access keys are generated anytime a user uses a recovery link or accesses the account from a different machine (i.e, desktop, laptop, mobile, etc…).  Again, the old ones aren’t deleted because it doesn’t know if they’re every going to be needed again.

While that turned on a couple light bulbs, it still didn’t help with my encryption key problem.

Remember that I’m looking for something coming out of NEAR to use as an encryption key per user that wasn’t going to change all the time.  The full access key explanation actually made things worse when considering that there are different keys for every different user device.  As I need my Dapps to be avail anywhere on any device, my mind started visualizing immense problems figuring out how to maintain/synch multiple copies of the same data to correspond to every new key that would be created.

Throughout all this, I was starting to think the easy button would be just to have the user create their own encryption key – much like a password.  I quickly dismissed that as a viable course of action as I think one of the huge advantages of NEAR is the ability of the user to interact with a Dapp without passwords and all the associated security issues that come with it.

But Wait, There's More...

Thankfully, this is where Matt Lockyer started talking about something he had done with Ethereum that, quite frankly, went over my head.  But I think the jist of it was that he used a NEAR account to sign some kind of meaningless data – thus proving his identity and then used that signed transaction (or a sha256 hash of it to achieve the right length) as the component needed in whatever it was he was interfacing with (sorry Matt – things are a bit fuzzy for me at this point). 

He also revealed that under the hood NEAR maintains a key somewhere (I believe that is part of the 2FA or recovery system) that serves the purpose of linking all these full access keys together – meaning if I could sign/verify something against that underlying key – it wouldn’t matter what device was being used – the derived encryption key would be the same.

Something isn't quite right...in my head

As an aside – while it helps my situation now, it raises a question I may ask someday about centralization as wherever those keys are stored seems like a pretty nice target and while it was mentioned they are backed up, etc… looks like a point of potential failure – I could be dead wrong on this – something that I intend to understand at some point.

Matt pointed me to a couple sections of code in the NEAR contract helper that can help me achieve what I need to do (below) and I’m hopeful that this approach is going to give me what I need to be able to encrypt and safely store the Textile identity string on the NEAR chain.

This code comes from near's near-contract-helper

So that's where I'm currently at.  I’ll come back and update this page with my solution once I eventually manage to do it.  

The call was yesterday, I work all day, not as super technically proficient as all you wizards out there, and thus, I’m still working on it.  The good news though is that having already spent several days ripping what little hair I have left out on this problem to get to this point, at least I feel like I’m headed in the direction of a workable solution.

Coming Soon:  Part 2 - The Solution...

>