Firebase is proprietary

[cben]

The Firebase backend is proprietary so you can't run your own instance of the full stack.
At this stage, I would not trust myself with running a secure & reliable backend, so using Firebase is better for Mathdown end-users.

OTOH, there is little need to change anything about Firebase and being able to develop as a purely client-side code makes forking and testing much easier - a user can try any fork without exporting/importing user data.
"Running" a fork is as simple as loading the static HTML from anywhere - local checkout, github pages, cloud9 workspace etc.

Any free solution should preserve the ability to fork and use without migrating data.

---

Best free-backend alternatives to Firepad:

• https://github.com/Operational-Transformation/ot.js/ is mature, well documented, has CodeMirror binding.
  Includes node.js server code; no persistence layer?!
• https://github.com/share/ShareJS is simple and has (currently incomplete - ShareJS with Codemirror ? josephg/ShareJS#124)) CM binding.
  Includes node.js server code, needs MongoDB.

For other firebase-like functionality (if we add any), meteor.com looks most similar in spirit, and even provides free hosting for server-side part of the app (arbitrary node.js?).

[cben]

For more firebase alternatives, look at http://nobackend.org/solutions.html. Haven't found a directly usable alternative yet.

[JohnBerg60]

Hood.ie looks promissing!

[cben]

Yay, first comment not by me in this project :-)

Thanks for the link. First impressions:

• I love the Offline First stance.
• From 5 min of googling there is no Hood.ie based real-time collaborative editor.
  According to Real-time collaborative textarea? hoodiehq/hoodie#167 one should use e.g. ShareJS for real-time editing and only use Hood.ie for periodic snapshots.
• It seems I'd need to host my own CouchDB [and ShareJS] server. As mentioned above I don't currently have the time or skill to do that reliably.

[cben]

Turns out the client-side firebase.js was only MIT-licensed by mistake, now corrected:
https://groups.google.com/forum/#!topic/firebase-talk/pAklVV3Whw8

[cben]

Short-term I'm staying on Firebase, long-term I'd love a fully open alternative.

In the medium-term, I'm thinking of combining Firebase for most users with some story for self-hosted backend, even if not scalable.
Specifically, running on Sandstorm.io could be sweet.

[cben]

https://swarmjs.github.io/articles/todomvc looks promising as a firepad alternative.
Based on CRDT (covergent/commutative/conflict-free replicated data types) unlike "some slap-together per-column last-write-wins solution, like Firebase and many others are using". Didn't read enough to know if it's really superior; at first glance their offline story sounds better.

Reading list:
https://github.com/gritzko/swarm/blob/master/lib/Text.js
http://christophermeiklejohn.com/crdt/2014/07/22/readings-in-crdts.html
A comprehensive study of Convergent and Commutative Replicated Data Types
TreeDoc
Merging OT and CRDT algorithms

[tohagan]

I recommend you do some more reading on CouchDb - specifically look at it's Eventual Consistency model and Conflict Management which I think is a mature well thought out solution that you can algorithmically customise as per your app requirements.

CouchDb can replicate and scale from offline mobile device (via PouchDb) => single server => replicated servers => clustered => geospatial replication. Since it's a multi-master database engine, any of these replicated instances can receive both reads and updates. Update conflicts are also retained and replicated in a tree per object that tracks revision ancestry for eventual resolution via a conflict resolution service that you build.

CouchDb + PouchDb (inside Hood.ie) is a killer combo. CouchDb 1.7 is a solid 5 year old technology. Couch 2.0 is now in developer preview and will bring BigCouch clustering (thanks to Cloudant) and much more.

No you don't need to host CouchDb yourself. Have a look at Cloudant.com (IBM), Couchappy.com and IrisCouch.com . Hosting and configuring it all yourself is now much easier and cheaper than it used to be using places like DigitalOcean.com (tons of step by step guides to setup and secure your host) and Hood.ie will also likely help here when it matures. You might want to also look at Docker.

Best of all it's Apache 2.0 or MIT licence - no propriety lock in here!

[cben]

Thanks for the pointers.

Is there any ready implementation of collaborative text editing on top of [CP]ouchDB?
I'm afraid I won't have the time (nor expertise) to roll my own any time soon; at best adapt an existing one, e.g. port Firepad (which is open source) on top of non-Firebase storage — but even that is unlikely to happen in a year...

I'm also eyeing https://github.com/mizzao/meteor-sharejs implementing OT on top of Meteor collections (some more discussion at https://trello.com/c/tkBErvIk/39-operational-transformation).

[tohagan]

You might look at the Delta Pouch plugin for PouchDb

• https://github.com/redgeoff/delta-pouch
  "A PouchDB plugin for partial updates that uses the every-doc-is-a-delta storage pattern. You can use delta pouch to enable conflict-free collaborative editing of the same docs."

Factoryng - AngularJS Adapter

• https://github.com/redgeoff/factoryng

"An all-in-one AngularJS factory that wraps PouchDB and Delta Pouch"
It's also a factory for Firebase in case you change your mind.

Other useful plugins

• http://pouchdb.com/external.html

[cben]

Thanks. I'll need to dig deeper as Delta Pouch is not extensively documented. "conflict-free editing of same docs" is promising but I'm getting the suspicion it's using the Mongo/Couch jargon sense of "doc" = json object, and conflict-free merely means different fields in the json structure can be modified simultaneously
[Source: http://stackoverflow.com/a/25050819/239657 seemed to be the origin of Delta Pouch, with merge() being for (var i in obj2) { obj1[i] = obj2[i]; }; the current code is similar.]

OT indeed represents text as a sequence of write-once deltas, but there is more to it. Not much

Note to self: also look at http://derbyjs.com/, an outgrowth of ShareJS which certainly does support OT.

---

I should note that moving off Firebase is a long-term desire but at the current rate I'm unlikely to touch it anytime soon (say not in a year). There are many things to fix that are more immediately embarrassing.

[cben]

Time to upgrade to non-ancient firebase.
For license they point to https://www.firebase.com/terms/terms-of-service.html
I'm not fully sure what their terms-of-use mean for an otherwise open-source app. My impression is that it's OK in practice and that their intention has never been to prevent open-source apps.
Two points that jump at me:
- (3.2) I grant the Company rights to do basically anything with my Application (for the sole purpose of allowing them to provide Software and Services). I'm fine with that, as I'm releasing my work under a permissive license anyway.
- It's not clear whether the license is addressed ("You") to me as the one whose credit card is on file / to me and anybody who forks mathdown as the app developer / or to all end users.

[tohagan]

Just don't include Firebase keys/logins in your source code (you shouldn't be doing this anyway). Rather, include in your README the instructions for signing up with Firebase and generating and configuring the keys. Most developers do this via an environment variable as this is the typical way to configure keys in a cloud hosted service. That way, they then must sign up and agree to the Firebase terms of service and you're not involved in that legal relationship.

[tohagan]

Personally I'd avoid any hosting company that offered any of the Firebase 3.x terms. The last one 3.4 is a serious "turn off" for me. These guys are sharks.

[cben]

Good ideas.
I'm going to mail Firebase for clarifications.

This issue is by far the most — the only — one people comment on.
But I won't soon have time to work on another backend and certainly not to run one myself.
Patches of course welcome :-)

[jamesgibson14]

I am looking into changing the Firepad database to be a plugin, so that you can choose/build your own data-store. I will also be working on a Meteor/MongoDb data-store as the first alternative.

[cben]

@jamesgibson14 Cool!
Firepad's interaction with Firebase is already confined to lib/firebase-adapter.js.
A couple threads that shed light on it: FirebaseExtended/firepad#163 , https://groups.google.com/forum/#!topic/firepad-io/z5-pF2BFyN8

It seems firepad doesn't need much beyond atomic append (on collision one succeeds and others have to retry). It does use onDisconnect for presence & cursors.

[cben]

More random thoughts:

I've played today with Sandstorm. It certainly makes self-hosting fun.

I've read up a lot on CRDTs.

• All things equal, a CRDT sounds preferable to OT for self-hosting as it makes backing up and moving between hosts trivial. It also passes the "desert island test": a group of people who have lost connectivity could set up a temporary server on their island, collaborate with that, and later seamlessly reconnect to the "central" server. In simple words, OT ~ SVN, CRDT ~ Git. Distributed => more practical freedom.
• I'm not sure what's the state of CRDT collaborative editors in practice (beyond academic papers). Victor Grishchenko's Swarm is still a moving target. Found a couple others on github [1, 2] but they don't look very serious.
• Looking at the algorithms it seems to me OT might be better at intent preservation. Example: when 2 people insert at the same place and keep typing there, OT can be aware of what the user saw when he typed and has reasonable chance of cleanly separating the 2 texts, while all CRDTs I saw would intermix their letters more or less chronologically. I might be wrong.

[amark]

Hey, I'm in the middle of writing a collaborative writing tool using my Open Source Firebase replacement (http://github.com/amark/gun). The editor will use CRDTs and DAGs instead of OT, because OT requires a centralized server unfortunately as @cben already noted.

I have a single-user demo up here: http://db.marknadal.com/random/write.html (Warning! This is out of my Dropbox folder so it might vanish or be broken at any given moment). Hope you guys might find it helpful since this thread seems to be a compiling list for alternatives/ideas, just wanted to share. Vice-versa, if anybody would like to help me out on building it, I would love that! Thanks. :)

[cben]

Eureka!
SageMathCloud has become fully open source. It has its own realtime collaborative editing implementation. I hear it's not hard to self-host on a VM.
Therefore, one way to achieve fully self-hosted mathdown is to integrate it into SMC.

That means (1) focusing on fixing CodeMirror-MathJax, and integrating that first whereever math can be edited (2) integrating any remaining markdown styling/editing goodness?
It does sound like deconstructing this project.

If my target audience is scientists, that'd make great sense. SMC already does lots of other stuff they need, including collaborative latex editing.
If my target audience is hackers/writers who love markdown, it may be too heavyweight.

Food for thought for me...
[This is not purely technical but where I personally want to invest effort in next couple months. I wanted to "finish" mathdown to some degree I'll be comfortable with - what I call "beta", advertise it a bit, then get a job and shift mathdown to a longer-term slower side project. What's a good goal for this last sprint? Solving "firebase is proprietary" is not important to me in the short term, but it's a cute bonus if I was considering SMC anyway.]

Other places I've dreamt of integrating CodeMirror-MathJax are IPython notebook and ShareLaTeX (but the latter uses Ace, CodeMirror would be a big re-work). Gitit might also be nice.

[cben]

I hear SMC is not hard to self-host on a VM.

Emmm, no.
https://github.com/sagemathinc/smc/blob/master/INSTALL.md
It is currently hard to self-host. The instructions are scarily long. Each part is simple but there are a dozen parts. The instructions are currently focused on Google cloud engine (which is where SMC runs in prod). It's probably actually not hard to get everything running on one machine but there is no quick start doc/script.

[yasserf]

Just came across this because of looking into operational transform, but you might in theory be able to deepstream.io ;) (disclaimer: core contributer)

https://deepstream.io/