Build Guide

[sizzlemctwizzle]

Building this thing is starting to get complex with the addition of S3. We should put together a step by step guide of how to build and run the site locally so that developers don't have to try and figure it out on their own.

[jerone]

And add a reference in the README.md

[sizzlemctwizzle]

Yeah, maybe.
Packages that MUST be installed:

npm install express toobusy mu2 mongoose connect-mongo formidable async aws-sdk passport passport-github

Then you need to install (and ruby too): https://github.com/jubos/fake-s3

[cletusc]

I just did something like this for another project I am working on. See my contributing guide for some ideas.

[jerone]

@cletusc That's perfect, only add fake-s3 for development and it's finished.

[sizzlemctwizzle]

Also change the bit about installing the package to forking, cloning your fork and then adding the main repo as remote upstream in git. Then mention about creating issues, branches for them, and submitting a pull request. Some of our contributors might be unfamiliar with git, so link to some guide (I actually wrote my own git crash course for another project that we could use; it's very short and to the point).

[Watilin]

I'm currently struggling with this point after I saw my fork didn't get automatically updated. Fortunately, Github's fork help explains it pretty well. In fact I was wondering what will happen to Dexmaster's fork – which is based on mine – once I get mine updated.

[cletusc]

FYI, I did have some notes about commits in the original style guide that you could probably adapt as well.

[sizzlemctwizzle]

Forks don't update automatically. His fork will remain in its current state until he updates it.

[Martii]

Just a note for those building on a localdomain I read a comment here to use this fork for toobusy.

Whether or not this works better I don't know yet... still having issues with $ node app.js not running the base site as is even with toobusy require commented out and a function surrogate returning false.

Type $ npm -l for a listing of all npm commands including picking other variants of node_modules for $ npm install... Another shortcut directly is $ npm install -h. For future reference I will be using $ as current user and # as root... so it's obvious not to type that particular character set... and occasionally I'll be using $ sudo... so if you don't have sudoers set up you might want to.

[Zren]

If we don't need to test with toobusy, we could move the import/require to be inside if (env == production) {} in app.js

To get around it atm (on Windows), I just made ./node_modules/toobusy/ and copied ./node_modules/toobusy/package.json from the master branch. Then made ./node_modules/toobusy/index.js with module.exports = function(){ return false; };

[Martii]

we could move the import/require to be inside if (env == production) {} in app.js

+1 I'm all for not having to do any surrogates for localdomain testing. :)

---

@Zren,
Which exact node.js version ($ node -v and $ npm -v) are you using in your non-production environment? I still can't get a few things to work and this would be helpful please.

$ node app.js
Failed to load c++ bson extension, using pure JS version

Error: Most middleware (like session) is no longer bundled with Express and must be installed separately. Please see https://github.com/senchalabs/connect#middleware.
    at Function.Object.defineProperty.get (~/node_modules/express/lib/express.js:89:13)
    at module.exports (~/node_modules/connect-mongo/lib/connect-mongo.js:30:39)
    at Object.<anonymous> (~/repo/git/oujs/martii/OpenUserJS.org/app.js:3:42)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Function.Module.runMain (module.js:497:10)
    at startup (node.js:119:16)
    at node.js:902:3

[Zren]

node --version
v0.10.28

npm@1.4.9

I have Python3 as a default python nowadays, so I had to tell NPM to use Python2 link.

npm config set python python2.7

[Martii]

Thanks... I can't get it working for some uknown reason. Works with simple Hello, World scripts but the moment bson is introduced it crashes.

$ gcc --version
gcc (Linaro GCC 4.7-2014.01) 4.7.4 20140107 (ROSA)

$ g++ --version
g++ (Linaro GCC 4.7-2014.01) 4.7.4 20140107 (ROSA)

$ python -V
Python 2.7.6

$ npm -v
1.4.3

$ node -v
v0.10.26

$ ruby --version
ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]

$ gem list --local

*** LOCAL GEMS ***
...
fakes3 (0.1.5.2)
...

I have Python including the plugin for it from my distro and currently at v0.10.26 (Do have your version (latest stable) and the pre from GH but that one makes things much worse LOL)

[Zren]

I have a compiler setup too (forget how I did it though).

http://stackoverflow.com/a/23190258

Mentions installing gcc/g++/make as well as python2.

[sizzlemctwizzle]

Yeah you need those things to compile the C++ version of bson. The JS one
doesn't work in my experience.
On May 30, 2014 5:51 PM, "Chris Holland" notifications@github.com wrote:

I have a compiler setup too (forget how I did it though).

http://stackoverflow.com/a/23190258

Mentions installing gcc/g++/make as well as python2.

—
Reply to this email directly or view it on GitHub
#60 (comment)
.

[Zren]

There's also this: http://stackoverflow.com/a/23873467

npm install -g node-gyp
npm update

[Martii]

Unfortunately it appears to build okay:

$ npm install bson
npm WARN package.json mu2@0.5.20 No repository field.
... (clipped GET requests and status but all 200) ...
> bson@0.2.8 install ~/node_modules/bson
> (node-gyp rebuild 2> builderror.log) || (exit 0)

bson@0.2.8 node_modules/bson
└── nan@1.0.0

Result for:

$ sudo npm -g install node-gyp
... (clipped GET requests and status but all 200) ...
/usr/bin/node-gyp -> /usr/lib/node_modules/node-gyp/bin/node-gyp.js
node-gyp@0.13.1 /usr/lib/node_modules/node-gyp
├── which@1.0.5
├── osenv@0.1.0
├── graceful-fs@2.0.3
├── rimraf@2.2.8
├── semver@2.2.1
├── nopt@2.2.1 (abbrev@1.0.5)
├── mkdirp@0.5.0 (minimist@0.0.8)
├── minimatch@0.3.0 (sigmund@1.0.0, lru-cache@2.5.0)
├── npmlog@0.0.6 (ansi@0.2.1)
├── fstream@0.1.25 (inherits@2.0.1, mkdirp@0.3.5)
├── glob@3.2.11 (inherits@2.0.1)
├── tar@0.1.19 (inherits@2.0.1, block-stream@0.0.7)
└── request@2.36.0 (json-stringify-safe@5.0.0, forever-agent@0.5.2, aws-sign2@0.5.0, qs@0.6.6, oauth-sign@0.3.0, tunnel-agent@0.4.0, node-uuid@1.4.1, mime@1.2.11, tough-cookie@0.12.1, http-signature@0.10.0, hawk@1.0.0, form-data@0.1.2)

[Zren]

This is my npm list https://gist.github.com/Zren/97f4e4a1da9f82aef3b3 I'm using express@3.5.1 and bson@0.2.7

[Martii]

Hmmm quite a bit of difference in versioning. Here's my current $ npm list

---

Inside bson there is:

$ cat builderror.log
Traceback (most recent call last):
  File "/usr/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py", line 18, in <module>
    sys.exit(gyp.script_main())
AttributeError: 'module' object has no attribute 'script_main'
gyp ERR! configure error 
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:337:16)
gyp ERR! stack     at ChildProcess.EventEmitter.emit (events.js:98:17)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (child_process.js:797:12)
gyp ERR! System Linux 3.10.34-nrj-desktop-3rosa
gyp ERR! command "node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/user/node_modules/bson
gyp ERR! node -v v0.10.26
gyp ERR! node-gyp -v v0.12.2
gyp ERR! not ok 

These "error" messages are a bit unclear to me right now.

[Zren]

Thought so. Express 4.x breaks a lot of things, as it no longer uses connect (for middleware).

https://github.com/OpenUserJs/OpenUserJS.org/blob/master/package.json#L7
https://gist.github.com/Martii/6bc1504ddd39d33e6e0f#file-gistfile1-sh-session-L14

Downgrade. Or delete/rename ./node_modules and do npm install.

[Martii]

Downgrade.

I'm only about 24 hours into using node so I'm not sure how to do this manually yet.

Or delete/rename ./node_modules and do npm install.

That yields this:

...
npm ERR! toobusy@0.2.4 install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the toobusy@0.2.4 install script.
npm ERR! This is most likely a problem with the toobusy package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-gyp rebuild
npm ERR! You can get their info via:
npm ERR!     npm owner ls toobusy
npm ERR! There is likely additional logging output above.

...

Interesting thing is that I'm on my local branch that uses toobusy-js ... so not quite sure why and where it got the original toobusy reference.

---

EDIT: A-ha in ./package.json

Reset package.json to use toobusy-js

Yields this:

$ node app.js
Failed to load c++ bson extension, using pure JS version
js-bson: Failed to load c++ bson extension, using pure JS version

(We need a debug message here saying that the server should be running)

[Martii]

Okay it seems to be working now at http://localhost:8080. Don't know exactly when I'm supposed to run ./fakes3.sh too which causes an error btw:

$ ./fakes3.sh
./fakes3.sh: line 2: fakes3: command not found

Thank you for your assistance @Zren... now to go try to see what you've been doing with the UI ;) Looks very promising!

@sizzlemctwizzle
You may want to try out toobusy-js production side for a few minutes and see if it bombs anything... seems to be working localhost wise here.

[Zren]

Oh you got it working? Grats. If anyone else runs into problems trying to install toobusy for their dev machine, run npm install Zren/fake-node-toobusy and skip the fuss.

[Zren]

fakes3.sh runs https://github.com/jubos/fake-s3 which requires you to install Ruby first.

sudo apt-get install ruby
gem install fakes3

[Martii]

which requires you to install Ruby first.
...
gem install fakes3

Ahh missed this part... Thanks! Ruby is installed by default in this distro just not that gem file. I'm still not sure what this does btw.... it appears that I have a connection to a mongoDB so I see scripts on OUJS via localhost... so what does this do?

I'm going to be helping sizzle build the docs for devs here which is why I'm asking all of this fuss. (besides being green to node) :) I also use $ sudo urpmi not apt-get, or pacman, or the bazillion other distro package managers so we'll need to account for that.

Misc question: Does something like console.log or dump(msg) exist/work to terminal/console to tell the devs that the server is running? I just guessed that it was and went to the url.

[Zren]

fakes3 is the service that stores static files, which is what is used to store userscripts/libraires. You can run the site without it, you'll just timeout when viewing the scriptViewSourcePage / scriptEditSourcePage / etc which deal with the source code.

By default, you connect to a nodejistsu dev db that (assumeably) sizzle setup. You can set the CONNECT_STRING env variable to a mongodb uri to connect to another db.

Eg: This is my localhost mongodb setup. I made a folder called scripts in the project root, then added /scripts to ./.git/info/exclude which is basically a per repo .gitignore. Then made ./scripts/runlocal.

export CONNECT_STRING="mongodb://localhost:27017/openuserjsorg"
node app.js

Which I can call from the project root with scripts/runlocal. Note that I'll have ./fakes3.sh and mongodb running seperately as well.

Yes, console.log is basically stdout. Since nodejistsu probably has it's own router logging, there's no logging in localhost at all. I added https://github.com/Zren/OpenUserJS.org/blob/ui/app.js#L59 for dev purposes myself.

[Martii]

You can run the site without it, you'll just timeout when viewing the scriptViewSourcePage / scriptEditSourcePage / etc which deal with the source code. ... By default, you connect to a nodejistsu dev db that (assumeably) sizzle setup.

$ ./fakes3.sh
Loading FakeS3 with ~/repo/git/oujs/martii/OpenUserJS.org/fakeS3 on port 10001 with hostname s3.amazonaws.com

Timing out when viewing any source on the default dev db here with current HEAD with the mod for toobusy-js ... fakes3.sh with and without that server running too.

I'll see if I can go install mongoDB... I think that was in my distro packages and give your config a whirl.

[sizzlemctwizzle]

FakeS3 stores scripts locally. You can't view the source of scripts you
didn't add since they aren't stored on your machine.
On May 30, 2014 9:00 PM, "Marti Martz" notifications@github.com wrote:

You can run the site without it, you'll just timeout when viewing the
scriptViewSourcePage / scriptEditSourcePage / etc which deal with the
source code. ... By default, you connect to a nodejistsu dev db that
(assumeably) sizzle setup.

Timing out when viewing source on the default dev db here with current
HEAD with the mod for toobusy-js. fakes3.sh with and without that server
running too. I'll see if I can go install mongoDB... I think that was in my
distro packages and give your config a whirl.

—
Reply to this email directly or view it on GitHub
#60 (comment)
.

[Zren]

If you're installing a mongodb server, you'll run into a problem when starting up. You'll need to manually set some mongodb values (GitHub ClientId and ClientSecret). You can see the values on the dev db by going to /admin/api. Manually setting them on the new db is trickier. I ended up installing a mongodb GUI, but we should probably make a setup script.

Here's one I put together now:

var mongoose = require('mongoose');

var Strategy = require('./models/strategy.js').Strategy;

var githubClientId = process.argv[2];
var githubClientSecret = process.argv[3];
var connectStr = process.argv[4] || process.env.CONNECT_STRING || 'mongodb://localhost:27017/openuserjsorg';

var dbOptions = { server : { socketOptions : { keepAlive: 1 } } };
var db = mongoose.connection;

// Connect to the database
mongoose.connect(connectStr, dbOptions);
db.on('error', console.error.bind(console, 'connection error:'));
db.once('open', function () {
  var githubStrategy = new Strategy({
    id: githubClientId,
    key: githubClientSecret,
    name: 'github',
    display: 'GitHub',
  });

  githubStrategy.save(function (err, fluffy) {
    if (err) {
      console.error(err);
    } else {
      console.log('Saved');
    }

    db.close();
  });

});

// These ids/secrets are fake. Grab existing values from the dev db at http://localhost:8080/admin/api
// Usage: node setup.js 8asdf98a9fds09sd9f89 0asdf09asfd0asfd98sdafa09g7fd9gds7g9fdg7
// Usage: node setup.js 8asdf98a9fds09sd9f89 0asdf09asfd0asfd98sdafa09g7fd9gds7g9fdg7 mongodb://localhost:27017/openuserjsorg

Might make one that clones all strategies on the dev db later.

[Zren]

I found out how to create GitHub client_id/client_key today. You need to go to Account Settings > Applications > Register New Application or click here.

For the Authorization callback URL, enter the following (case sensitive) url:
https:// + yourdomain.com + /auth/github/callback/
Eg: https://nameless-hollows-5487.herokuapp.com/auth/github/callback/
Eg: https://localhost:8080/auth/github/callback/

[supahgreg]

For posterity: if you want to manually create the MongoDB collection for Strategy it must be named exactly "strategies".

Mongoose lower-cases and pluralizes the model name to get the name it will use for collections.

So (assuming a DB named oujs) it looks like:

use oujs
db.createCollection("strategies")
db.strategies.insert({id: "your_GitHub_client_ID", key: "your_GitHub_secret", name: "github", display: "GitHub"})

References:
http://mongoosejs.com/docs/api.html#index_Mongoose-model
https://github.com/LearnBoost/mongoose/blob/master/lib/utils.js

[supahgreg]

Also, since I'm not using localhost I needed to set AUTH_CALLBACK_BASE_URL=my_base_URL to get OAuth working as expected.

[jerone]

@sizzlemctwizzle commented on 29 nov. 2013 21:17 CET:

Building this thing is starting to get complex with the addition of S3. We should put together a step by step guide of how to build and run the site locally so that developers don't have to try and figure it out on their own.

I think we have a very comprehensive list of steps by now:

• https://github.com/OpenUserJs/OpenUserJS.org/blob/master/README.md
• https://github.com/OpenUserJs/OpenUserJS.org/blob/master/CONTRIBUTING.md
• https://github.com/OpenUserJs/OpenUserJS.org/blob/master/STYLEGUIDE.md