A huge foundational component of a couple of the DApps I’m currently building on NEAR are centered on user interactions and contributions in a community setting. One of them is a project I started this week that will even be a submission for HackFS. In attempting to architect them in a decentralized fashion, I need the ability to use NEAR accounts as identity providers to other decentralized services – specifically Textile for IPFS database and file storage.
A Quick Textile Primer
Textile is proving to be very useful for what I’m doing with NEAR. I think it’s a normal pattern to store high value info on chain and then link to a hash of information requiring more storage or of lesser importance stored on IPFS or some other centralized or decentralized service (like OpenSea does with NFT images) to save on costs associated with contract storage.
Textile caught my attention because of their threadsDB service. It allows us to build application or user-level databases that contain collections. These collections are basically database tables defined by a JSON schema that hold instances or records.
It’s basically the same construct one can find in most centralized databases of many of today’s apps – like a WordPress installation. Textile even suggests that working with it will feel familiar to developers who have worked with technologies like MongoDB.
The instance data inserted into a ThreadsDB collection is encrypted by default and is good for meta data type info. Larger files including images, videos, and other datasets will typically utilize another offering they provide called Buckets. Together, they are an abstraction on IPFS that I see as opening the door to upgrading and improving many existing web 2.0 apps to web 3.0 versions while enabling further innovation. In addition to the Textile Hub which I’m going to talk about more below, Textile has a fourth service – Powergate that adds Filecoin storage onto all of this. I’m currently envisioning use for it as a long-term storage/availability option in my DApps to ensure users can retrieve their data (think backups or archives).
Using NEAR as an Identity Provider
As alluded to - Textile has a hub which is a portal that allows developers to access IPFS and Filecoin (eventually).
The hub makes it easy to manage and update, persist, deploy, and scale information on IPFS. It requires an identity to function and because it supports public-key infrastructure (PKI) any identity provider that operates within those parameters can be integrated (3Box, Metamask, uPort, Blockstack or derive your own).
The general flow of how the identity gets authorization to work is as follows:
This is far outside my field of expertise, but I believe the whole token generation/credential issuance setup here is typically done by a backend API server with access to a secret key. The Textile hub is a token generation endpoint and also does credential verification.
How I am achieving NEAR integration with Textile
It’s very easy to generate an identity using the Libp2p crypto library which is made avail in the @textile/threads-core library.
import {Libp2pCryptoIdentity} from '@textile/threads-core';
async function example () {
/** Random new identity */
const identity = await Libp2pCryptoIdentity.fromRandom()
/** Convert to string. */
const identityString = identity.toString()
/** Restore an identity object from a string */
const restored = Libp2pCryptoIdentity.fromString(identityString)
}
I started there and initially thought about using browserlocalstorage to store the plain text identity string created by libp2p. However, I needed a way to link that identity string to the user’s NEAR account – otherwise I’d get a new identity string every time I logged into the app. That would mean any info I had thrown into collections with threadIds linked to that identity would no longer be accessible.
I decided to create that association on the NEAR blockchain by creating a data model (persistentMap) to store the Textile identity string with the account sending it to be stored (current user’s account name).
So, at this point, with a bit of logic in the client – the Dapp successfully looked for an existing identity for the currently logged in user. It either retrieved it from the blockchain or if an identity didn’t exist (new user) the Dapp created a new random identity using Libp2p and stored it on chain for that user to retrieve the next time they logged in. I also cached it in browserlocalstorage to speed things up – adding that as an initial check for the identity into the Dapp as well.
Happy at First...
At this point, I was pretty stoked, the Dapp was functioning how I wanted allowing different users to login, use an existing or new identity to retrieve a token/credentials from Textile and post/retrieve data from their own user-level databases on IPFS. My happiness eroded though when I thought a bit about security and the fact that the user identities were being stored in plain text on chain and in browserlocalstorage. I needed to take things a bit further.
That’s when I started searching for some kind of stable key or data in the NEAR account that I could use as an encryption key to encrypt the identities before they get stored on chain and decrypt them when retrieved for use.
I eventually achieved the encryption/decryption objective, but the only option I could find at the time was the user’s NEAR public key of one of their full access keys – which, of course, doesn’t increase security at all – just adds another step that some determined hacker needs to figure out before eventually deciphering the encrypted identity.
NEAR Experts to the Rescue
I spent the next several days trying to figure out how NEAR access keys worked, reading and re-reading the documentation here and here, trying things out and ultimately ended up pretty damn confused about how NEAR was handling things. For instance, I found:
- New contract level access keys were created every time an account logged in and out – authorizing the Dapp each time - and it wasn’t a behaviour restricted to my Dapp. It happened with the default create-near-app template as well. In my mind, it seemed odd and after going through and deauthorizing 80+ access keys in my account that it generated from various testing, I questioned why the wallet or dapp wouldn’t just check to see if an access key existed and use that. I noticed there was a getAccessKeys() method – figured it would be simple enough to do a check or perhaps add a deauthorization step into the logout flow to delete the key after use.
- A number of full access keys were appearing in my NEAR account wallet – seemingly at random with much less frequency than the contract access keys – and I wasn’t really sure why.
By this time, I was also close to insanity as I couldn’t find any functions or methods or anything in the NEAR account or wallet objects that I could get at that wouldn’t eventually change or disappear. I was coming up short in finding something I could use for that encryption key and I was doing my best to avoid having to setup a central server to act as an API to issue that key (assuming, with my limited knowledge of that subject that that was what I needed to do).
Luckily, the wonderfully knowledgeable people at NEAR including Matt Lockyer, Kendall Cole, and Mike Purvis got me on a call and pointed me in a new direction.
To be clear...
The willingness of these fine gentlemen to help and decipher what I was trying to say is a big deal. While I’m improving and doing my best to grasp all the tech here, I didn’t study computer science or programming and am completely self-taught.
All of this is so far outside of what I normally do in my day-to-day professional life that I wasn’t quite sure whether I had a good enough grasp of the jargon/tech to be able to explain my issues in a way that they’d be able to understand. From my perspective, it went better than I thought it might in no small part to the patience and understanding they showed.
They may have a completely different perspective…
First, they answered my questions above about the access keys:
(Note that I’m paraphrasing my understanding – if anyone wants to expand on this in the comments below, feel free in case it’s more nuanced or I’m saying something that isn’t really the case):
- New access keys are created on every login/logout because NEAR doesn’t have enough information to decide when to delete a contract key. Erring on the side of caution, they let them accumulate so as not to deauthorize a key that a Dapp needs in some fashion. They indicated they’re looking into how to make this more intuitive in the wallet (perhaps things like indicating when last time that key was used to sign something).
- Full access keys are generated anytime a user uses a recovery link or accesses the account from a different machine (i.e, desktop, laptop, mobile, etc…). Again, the old ones aren’t deleted because it doesn’t know if they’re every going to be needed again.
While that turned on a couple light bulbs, it still didn’t help with my encryption key problem.
Remember that I’m looking for something coming out of NEAR to use as an encryption key per user that wasn’t going to change all the time. The full access key explanation actually made things worse when considering that there are different keys for every different user device. As I need my Dapps to be avail anywhere on any device, my mind started visualizing immense problems figuring out how to maintain/synch multiple copies of the same data to correspond to every new key that would be created.
Throughout all this, I was starting to think the easy button would be just to have the user create their own encryption key – much like a password. I quickly dismissed that as a viable course of action as I think one of the huge advantages of NEAR is the ability of the user to interact with a Dapp without passwords and all the associated security issues that come with it.
But Wait, There's More...
Thankfully, this is where Matt Lockyer started talking about something he had done with Ethereum that, quite frankly, went over my head. But I think the jist of it was that he used a NEAR account to sign some kind of meaningless data – thus proving his identity and then used that signed transaction (or a sha256 hash of it to achieve the right length) as the component needed in whatever it was he was interfacing with (sorry Matt – things are a bit fuzzy for me at this point).
He also revealed that under the hood NEAR maintains a key somewhere (I believe that is part of the 2FA or recovery system) that serves the purpose of linking all these full access keys together – meaning if I could sign/verify something against that underlying key – it wouldn’t matter what device was being used – the derived encryption key would be the same.
Something isn't quite right...in my head
As an aside – while it helps my situation now, it raises a question I may ask someday about centralization as wherever those keys are stored seems like a pretty nice target and while it was mentioned they are backed up, etc… looks like a point of potential failure – I could be dead wrong on this – something that I intend to understand at some point.
Matt pointed me to a couple sections of code in the NEAR contract helper that can help me achieve what I need to do (below) and I’m hopeful that this approach is going to give me what I need to be able to encrypt and safely store the Textile identity string on the NEAR chain.
This code comes from near's near-contract-helper
So that's where I'm currently at. I’ll come back and update this page with my solution once I eventually manage to do it.
The call was yesterday, I work all day, not as super technically proficient as all you wizards out there, and thus, I’m still working on it. The good news though is that having already spent several days ripping what little hair I have left out on this problem to get to this point, at least I feel like I’m headed in the direction of a workable solution.