Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

benchmark(gabe-fs-markdown-images): add img benchmark #29009

Merged
merged 1 commit into from
Jan 18, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 74 additions & 0 deletions benchmarks/gabe-fs-markdown-images/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*

# Runtime data
pids
*.pid
*.seed
*.pid.lock

# Directory for instrumented libs generated by jscoverage/JSCover
lib-cov

# Coverage directory used by tools like istanbul
coverage

# nyc test coverage
.nyc_output

# Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files)
.grunt

# Bower dependency directory (https://bower.io/)
bower_components

# node-waf configuration
.lock-wscript

# Compiled binary addons (http://nodejs.org/api/addons.html)
build/Release

# Dependency directories
node_modules/
jspm_packages/

# Typescript v1 declaration files
typings/

# Optional npm cache directory
.npm

# Optional eslint cache
.eslintcache

# Optional REPL history
.node_repl_history

# Output of 'npm pack'
*.tgz

# dotenv environment variable files
.env*

# gatsby files
.cache/
public

# Mac files
.DS_Store

# Yarn
yarn-error.log
.pnp/
.pnp.js
# Yarn Integrity file
.yarn-integrity
yarn.lock

generated_articles
generated_images
generated_image_pools
21 changes: 21 additions & 0 deletions benchmarks/gabe-fs-markdown-images/LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
The MIT License (MIT)

Copyright (c) 2015 Gatsbyjs

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
63 changes: 63 additions & 0 deletions benchmarks/gabe-fs-markdown-images/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Baseline Gatsby Benchmark: fs + markdown + images

This is a baseline benchmark site in the Gabe project.

This site in particular tracks Markdown performance for individual files per page that also have an image (not part of the markdown).

The site can generate an arbitrary amount of super simple pages. Each page has a small header, a quote, and two small paragraphs of random text. No images, because we want to benchmark Markdown.

The results of this benchmark can be compared to the results of the `gabe-fs-markdown` benchmark, to see a tentative impact of using images in markdown.

## Install

Run `yarn` or `npm install`

## Usage

Unlike most other gabe benchmarks, the generation part is a little more complex because it will generate image file pools first and then copy images from those pools into their destination.

### Image generation

Image generation is rather expensive. The default size for 128k can take 2 hours single threaded. For that reason, the image generation can use workers instead.

Recommended way for larger pages is to first generate all the images up to the amount you're going to use. These pools will persist across benchmarks so it's a one time cost:

For example; to generate 128k 100x100 images using 8 worker threads:

```
C=8 W=100 H=100 N=128000
```

This will require an up to date node because workers aren't available in node 10.13, you'll get a warning if that's the case.

The files will be generated in `generated_image_pools/jpg/wxh`. If `C` is not set then it will only add images and assume the existing images are already properly incrementally numbered, without gaps.

If `C` is set (and used) then it will regenerate all images regardless and use that many workers to divide the work.

### Image usage

When you run the benchmark, or generate the random content files, it will first check whether the pools have a sufficient amount of images. If they don't then the image pool is amended (see above).

Once the pool contains enough images for a given type/dimension, the random `.md` files are generated and for each file an image is copied from the pool as well. The copying of images is a lot faster.

It's important to note that the pool will persist between benchamrk runs, while the randomly generated content does not.

### Running the benchmark

Either way, you can start a benchmark run using the following. If the pool doesn't exist or does not have enough images, images will be generated:

```shell
W=100 H=200 N=1000 M=2 yarn bench
```

- `N=1000`: instructs the run to build a site of 1000 pages
- `M=2`: instructs nodejs to use up to 2gb of memory for its long term storage
- `W=100`: use images that are 100px wide
- `H=200`: use images that are 200px high
- `C=8`: (optional) force regenerate the image pool for given size and use 8 worker threads while doing so. Only need to do this once per image type+dimension.
- Deletes generates files from previous run
- Generates `N` pages with pseudo-random content, copies one image from pool per page generated
- Runs `gatsby clean`
- Runs `gatsby build`

The default `yarn bench` will build 512 pages with 1gb memory.
27 changes: 27 additions & 0 deletions benchmarks/gabe-fs-markdown-images/gatsby-config.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
module.exports = {
siteMetadata: {
title: `Gatsby FS Markdown Benchmark for Gabe`,
description: "A blog like no other blog",
author: "Bob the Blogger",
},
plugins: [
`gatsby-transformer-remark`,
'gatsby-plugin-image',
{
resolve: `gatsby-source-filesystem`,
options: {
name: `blog`,
path: `${__dirname}/generated_articles`,
},
},
{
resolve: `gatsby-source-filesystem`,
options: {
name: `img`,
path: `${__dirname}/generated_images`,
},
},
'gatsby-plugin-sharp',
'gatsby-transformer-sharp',
],
}
43 changes: 43 additions & 0 deletions benchmarks/gabe-fs-markdown-images/gatsby-node.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
const path = require(`path`)

const blogPost = path.resolve(`./src/templates/blog-post.js`)

exports.createPages = async ({ graphql, actions }) => {
const { createPage } = actions

const result = await graphql(`
query {
allMarkdownRemark {
nodes {
id
frontmatter {
slug
title # used in prev/next
}
}
}
}
`)

if (result.errors) {
throw result.errors
}

const posts = result.data.allMarkdownRemark.nodes

posts.forEach(({ id, frontmatter: { slug } }, index) => {
const previous = index === posts.length - 1 ? null : posts[index + 1]
const next = index === 0 ? null : posts[index - 1]

createPage({
path: slug,
component: blogPost,
context: {
id,
slug,
previous,
next,
},
})
})
}
Loading