DynoSRCSan Francisco

  • 167 Commits
  • 163 Pushes
  • 65 Deploys
Launch Site

DynoSRC

By DynoSrc

Screencast
Quick Intro

Eliminate HTTP requests for JavaScript files and serve differential updates to your users on the fly. No, really. Like, seriously.

Description

i'll show you how far the rabbit hole goes

Minimize HTTP Requests

DynoSRC loads JavaScript files inline in your HTML response, then stores them in localStorage. You can even inline the calls to the DynoSRC client lib in your HTML response, eliminating all HTTP requests for JavaScript on your site.

Differential Updates

Normally, if a JS asset on your site changes, your users will have to download the entire file again even though just a fraction of it changed. DynoSRC sends down differentials updates so changes to large files don't require full downloads.

To expand on what we are doing here:

Mobile traffic is taking over, and all aspects of performance are critical. We decided to focus on JavaScript download performance and see how far down the rabbit hole we could go. DynoSRC addresses three problems with JS download perf:

Problem 1: Mobile caches are much smaller than on desktop. Your code is more likely to be evicted from your users' cache before your next release/deployment, so return users often end up redownloading a previously cached file.

To address this, DynoSRC stores your JS files in localStorage. What's cool about this is that not only do you have a 5mb cache all to yourself, you know if something is cached or not rather than "flying blind" so to speak.

Problem 2: Mobile connections are slow. But it's mostly a latency problem -- a lot of people have 4G with faster download speeds than their home internet connections. Even on 4G, there is typically a ~200ms overhead for each HTTP request, which gets worse if it involves DNS lookups or redirects.

Limiting HTTP requests on mobile is important, and one way of doing that is to inline your JS into your HTML page. But that quickly becomes unwieldy for anything more than a snippet of code, as your users will be redownloading that code with every page load.

DynoSRC solves this by storing a manifest of what the user has cached in localStorage into a cookie. On each page load, the server checks the manifest, and inlines any JS files the client has not yet downloaded into the HTML. These files are then cached into localStorage and the manifest updated.

Problem 3: When you have a release/deployment and files are version bumped, your users have to redownload those entire files -- even if only a small amount of lines changed for a couple of bug fixes.

DynoSRC solves this by serving up differential updates. The server checks the manifest to see what version of a file the client has, then creates a diff between the latest version of the file and the version in the client. In this case, the inline JS file sent down to the client is simply a diff. The client applies the diff against its copy of the file, then updates its copy in localStorage and the manifest.

In the case of ready-to-deploy files, where multiple files have been concatenated together, obfuscated, and minified. The server will detect minified files and beautify them, creating a diffable file.

Other cool features of DynoSRC include:

  • Ability to serve your own files from disk, serve files from github repos, or even through bower.
  • Choose your own destiny: Don't want to serve files and diffs inline in your HTML? DynoSRC supports diffing through AJAX or JSONP, too.
  • dynoProxy: Normally you can't share localStorage between domains, but by using an iframe proxy and JSONP, you can establish a central server for DynoSRC that can!
  • More to come!

Check out the source on github if you're curious or want to contribute!

Judging Instructions

We have only tested in Chrome (stable), so use that. Visit http://dinosrc.it/ in your browser to get an overview of how DynoSRC works, then check out the test drive page to see an example of client-side diff patching and storage of JavaScript modules.

Visit the "Getting Started" page to see how to leverage DynoSRC in your own NodeJS projects. We've published the npm module to npm and github (development on master locked until competition ends).

What they Used

DynoSRC computes diffs by forking a git process on the server. We hope to tie in js-git in the future when it supports the diffing capabilities required, making the project completely contained w/in node and node modules!

Other libraries used here: http://dinosrc.it/credits

Who

Votes

Your Vote

Voting is now closed.

Other Votes

  • (25)
  • judge

    Interesting solution to a widespread problem. Curious to see if there are any realworld constraints from using localstorage. I like that there's a good attempt on marketing pages, but those could still use more polish.

  • judge
  • judge

    I would love to see this proven in the wild. I'm curious how well the localstorage stuff + diff merging works in the real world.

  • contestant

    USP

    Really approach and well explained!

  • contestant

    SAPO

    I personally don't like the idea of apps being stored on local storage. Apart from that, it seems like a good bandwidth saver. Can't be much more complete than publishing an npm :) .

    • josheverett
      contestant

      Hi Luis. Thanks for your vote! Could you elaborate a bit on what you don't like about the idea of storing JS files in localStorage? What sort of information do you think is better suited to localStorage? We'd love to hear your thoughts!

      Thanks! :)

    • luismreis
      contestant

      Hi Josh, it's nothing specific to your project. I just imagine my total local storage growing uncontrollably if every site places its code there.

  • judge

    This is very impressive work, and something that could be become extremely popular for web/mobile developers I think.

    I am incredibly impressed that you both wrote the utility and had a live demo site all in one weekend. The demo site and tutorials were very clear. I noted that you could also roll versions back in time.

    A few thoughts on this for moving forward-- 1. you'll need some kind of error detection && recovery mechanism (e.g. if the diffs somehow end up with a corrupt js, just reload everything). I could easily see this as being extremely difficult to debug, particularly if only certain users are getting corrupt javascript files as a result of some issue. It could be a real customer support nightmare.

    1. It would be nice to have this integrated in with a CDN like Amazon's cloudfront for instance. Versioning / update management in CDNs is kind of painful, and an extension of your ideas here might be able to solve some of the CDN file versioning issues, etc.

    Great work!

  • judge

    Joyent

    I don't think forking out to the GIT CLI is a good design in the long term, but it's an interesting proof of concept that deals with a real issue on Mobile platforms.

    • diurnalist
      contestant

      Thanks for the vote! We agree re: the git process. We're following Tim Caswell's js-git project and are looking forward to when he ports over the diffing implementations so we can leverage that and reduce that integration hurdle. We are also looking at other diffing algorithms. We actually evaluated a few for the Knockout but found that forking a git process was the most straightforward way to achieve a reasonable level of performance when the input sizes were significant. Most of the other implementations fell over when the files reached a certain size :p

  • judge

    Sencha

  • judge

    Square

    "The server will detect minified files and beautify them, creating a diffable file." I was wondering how you were gonna handle that. :)

    I'm concerned about using the diff endpoint as an attack vector on the server.

    Cool hack!

    • diurnalist
      contestant

      Thanks for the vote! Can you elaborate a bit on your security concerns? I'd like to understand more so we can work to address any vulnerabilities in future updates.

      If the diff endpoint was not secured against traffic from other origins, I can see how a lot of traffic could overwhelm the servers as it struggles to compute all the diffs. We do cache the diffs once they are computed, which mitigates this issue a bit, but then there is a concern of running out of server memory. For the purposes of the Knockout we didn't focus much on tightening these bolts, but we're going to be iterating on the library to address issues like this in the near future :)

  • judge

    Pluralsight

    Interesting idea but seems inappropriate for a Node Knockout project.

    • josheverett
      contestant

      Hi Geoffrey, could you explain why you feel our entry is inappropriate for Node Knockout?

      We were in constant contact with the organizers (onsite at Joyent HQ) during the competition, and they didn't seem to share this view at all. :/

    • By "inappropriate" I just meant that I'm used to seeing more full featured end-to-end apps rather than pure backend or API projects. No complaints about the possible usefulness of the library!

  • judge
  • contestant

    CARFAX

    Great idea! Just curious, did you guys collect any data on how much it actually sped up page load times?

  • contestant

    Great work! Very useful tool.

  • contestant

    One of the most original ideas I've seen in a while, great job guys!

    • ryanstevens
      contestant

      Thanks! We had a lot of fun hacking on this project and I'm glad to see people like yourself embrace the proof of concept so well. Look for the official github repo to have lots of activity on it once the competition is over.

  • contestant

    Looks like a cool idea. 5MB is a pretty good amount to play with. I imagine you would have fallbacks in place for browsers that don't support localStorage.

  • judge

    Twilio

    I think this is a cool concept - I've not seen anything out there which is working with JS asset diffs in this way. I also think it's pretty incredible how much the team got done over a weekend, there are a lot of ideas in here with potential. I would be curious to see the before/after numbers with a few test sites to see how much is actually gained through this approach versus using a combination of server-side tools like browserify and maybe Basket.js or other localStorage JS caching alternatives on the client.

    Great work, very cool! Also, great job explaining the value prop in the video, and I like your pixel dinosaur :)

    • diurnalist
      contestant

      Thanks for your vote! We didn't have enough time to give enough due diligence towards establishing some really solid performance benchmarks. That's something that we'll be rolling out, and using as a guideline towards figuring out how to further optimize various use-cases. We felt it would be better to leave detailed benchmarks out altogether (except with some mentions of theoretical gains) rather than hack together some data points that may turn out to be misleading. Instead we focused on hammering on some core use cases and making the API and implementation solid.

      Definitely keep an eye on the Github repo after the competition, where we'll be updating both the module (adding more tests and such), as well as beefing up the demo site to include more performance benchmarks and example use cases so newcomers can grok the library with ease.

      Cheers, Jason

    • Reimertz
      contestant

      Mr. Dynosaur like you back.

  • judge

    CratePlayer

    • diurnalist
      contestant

      Thanks for the vote! I was wondering if I could explain more clearly what we built during the hackathon so you can get a sense of what we completed. The repo at diurnalist/dynosrc was 100% built during the knockout. We've locked development on the branches there for the duration of judging. We also developed the website as a test/example application. The website actually uses dynosrc to deliver the simple JS (jquery included) that drive the demos.

      Let me know if you have any other questions about what the project encapsulates!

      Cheers, Jason

  • judge

    Esri

    Interesting entry

  • contestant

    e-conomic

    might be useful

  • contestant

    This is super impressive. I especially liked the demo page that showed me how things kind of work. Can't wait to see where else you take this.

  • judge

    Brandcast

    super useful product solving a real problem for mobile. Clean design, great implementation + open sourced already. Great Job guys!

  • contestant

    Printly.pk

    smart!!

  • judge

    Nice job!

  • contestant

    Accolite

    I am unable to make out what is done during hackathon in this as the link provided (http://dinosrc.it/) doesn't seem to be a node knockout link.

    I wanted to not give any vote until someone helped me understand. But since voting is mandatory to post the comment, I am putting a vote of 1 in each ladder except for innovation as I do think the idea is pretty cool. Would definitely reconsider changing my votes if someone can help me understand what was done during the hackathon.

    Also on the overall implementation of the idea, doesn't forking a diff process introduce slowness? Have you guys done any comparisons with how it compares to download latency?

    Also how are differential updates applied to version stored in localstorage?

    • josheverett
      contestant

      Hi Ashutosh, thanks for checking out our entry!

      The link provided (http://dinosrc.it/) is our node knockout project - the site is there to explain the tool and show its use in action. You can view our entry at the "normal" NKO URL here:

      http://waving.2013.nodeknockout.com/

      These are exactly the same site, hitting the exact same server, just different domain names pointing to the same place.

      Hope this helps. Cheers!

      Josh

    • josheverett
      contestant

      Sorry, missed your further questions.

      I'm not 100% sure what you mean by "forking a diff process", but all the diffs are only generated once then cached on the server. Even when not cached, we found that having the node server pull two versions of a file from github, diff them, and send them to the client typically took less than 150ms.

      The diffs are applied in the client using the applyPatch function from Kevin Decker's jsdiff library.

      Josh

    • ashutosh3
      contestant

      Thanks for responding Josh. Checked out and idea is indeed promising and sounds like you guys achieved a lot in 2 days.

      Checked out the diff patching code as well on client.

      Pretty solid overal.

  • contestant

    Hacker School

    Cool idea! I'm excited to see if you can overcome some of the issues re: localstorage that people have posed.

  • judge

    Freelancer

    Looking forward to trying this out on my next project.

    • ryanstevens
      contestant

      @pgte, we are talking every day on how to actually make this module battle tested enough for teams to use in production. We certainly believe this knockout hack proved it can be done and this technique needs to happen in the mainstream.

      This is the official repo https://github.com/diurnalist/dynosrc, we have decided until the competition we will not be accepting pull requests so the perception of our hack isn't altered. We feel this is a way to "lock in" the code during the competition to remain fair to other entries who can't / didn't open source their project outside of the walls of the knockout servers. This is all to say expect LOTS of activity on this repo once the competition is over.

  • contestant

    I so wanted this to do hot code swapping. But reading your use cases I can totally see how useful it could be to limit latency-induced user interface issues!

  • contestant

    Voxer

    Looks great. Might start incorporating it in my code.

    • ryanstevens
      contestant

      Let us know how it goes! Obviously we "web app" to showcase the module was the only thing that actually used it- so we're waiting for some real feedback from the community how to actually harden the API / solution. If you in SF, hit us up and we'll pair with you to get it going.

  • contestant

    wow

    wow such idea very innovate

    diffs seem less interesting (on rare updates, invalidating is fine) and the more interesting ideas proposed don't appear to be complete/working

    • diurnalist
      contestant

      :)

      Yes, the diffing part of it is probably only useful in certain use-cases. Rare updates have the benefit of being able to just cache heavily and then version bump assets on a deploy. In that case, this library would be helpful instead to give you the automatic inlining of assets on a mobile-served page.

      The proxy thing is finicky right now and is intended to be more of a conversation piece than anything. If you have 3rd party cookies blocked you're going to have problems, I think. Also make sure you try it at http://dinosrc.it/proxy-demo without the www - think that might make a difference, but I can't remember why right now :p - we might have not set it up entirely right, that push happened pretty close to the deadline!

    • diurnalist
      contestant

      Never mind, I'm full of it - the domain shouldn't matter. I bet it's the 3rd party cookies setting on your browser. Browsers treat localStorage in that same category of 3rd-party persisted data, so you won't be able to automatically write to the proxy'd domain's storage in most circumstances w/ that protection turned on. Theoretically there are times when it won't be blocked, such as if you've visited the proxy'd domain before in the same browsing session (I think that's how Safari and latest FF do it) - but in my experience it's hard to rely on.

  • contestant

    BMK Solutions

    Brilliant idea. Will be sure to try this out :)

    • diurnalist
      contestant

      Thanks! Let us know if you have any issues or suggestions on how to make it fold into your existing projects with greater ease.

  • contestant

    Ancestry.com

    Really interesting idea and very clear explanation of how it works! Definitely going to try this out soon

    • diurnalist
      contestant

      Thanks for the feedback - let us know what you think when you get a chance to try it out on your own time.

  • judge

    Idea whose time has come. Due to nature of the idea, the demo is a bit hard to grock. Would like to see some better visualization of what is going on behind the scenes when diffs are loaded and applied. Not clear how the code gets from localStorage back into the VM.

    • What about minified diff? I don't think minified code will diff well.
    • How can we ensure security?
    • ryanstevens
      contestant

      These are great questions. So one aspect of our hack is we do in fact try to address magnification. Essentially our module detects if a file is minified (has a small amount of lines with a LOT of characters per line) and appends semi colons with new line characters. This way our diff can be on a line by line basis. In theory a minifier that is consistent would not produce noise in a diff between to files except around the LOC that actually changed.

      I do not feel there are any more security concerns from this solution than traditional script serving models. The files themselves are stored in local storage, which inherits the same security as cookies and other locked down aspects under a domain. So scripts cannot be used from other domains.

    • diurnalist
      contestant

      @mhevery to provide a bit more clarity, here's what happens when a patch is applied with the dynoSrc client library via a call like this:

      dynoSrc.apply("moduleName", "1.0.1", "git diff -- ... diff ...");

      • dynoSrc pulls cached copy of JS file from localStorage, if any
      • If there was already an asset in localStorage, the incoming diff is patched in and the asset is saved back to localStorage for future requests.
      • If there was no asset cached, the diff isn't actually a diff, as that would be inefficient. Rather, we just send the whole file contents down. The contents are then immediately cached in localStorage.
      • Finally, we set a cookie is set on the client to mark the module at the new version (this is how we know what diffs to send clients when the page is rendered serverside.)

      We did try to address minified files as @ryanstevens attested to. There is some debate about the merits of minification given how good gzip is on its own, and we wanted the tool to be agnostic to your decisions there.

      That was the main design goal of the library: don't be opinionated, and offer a flexible library that can support various paradigms. You can use the server component to generate diffs that you apply w/ your own JS code, or you can write your own tool to send down diffs on a different stack than node and still use our client library to do the patch-ups and caching.

      Cheers Jason

  • contestant

    BrandExtract

    This is really cool. I can learn a lot from the code. Combined with WebSocket and we may eliminate requests altogether.

    • diurnalist
      contestant

      Thanks for the feedback! You can read all the source for the module on our Github.

  • contestant

    Spatial Automation Lab -- University of Wisconsin, Madison / 3D Systems / Bespoke Innovations

    Interesting idea, though I am not sure how this compares to bundling everything in one gigantic script file. I guess if you are frequently updating components on the site, then this type of approach might be a lot better.

    • diurnalist
      contestant

      Thank you for the vote and feedback!

      One gigantic script file is actually where diffs would work best, we think! Since there is a bit of overhead for a diff, just b/c of the structure required to encapsulate the information of added vs. removed lines, the benefits are bigger when your changeset is small relative the entire file size.

      Consider if the JS being thrown down to the page is ~30K lines. If you roll out an update that changes only a few functions, a client's diff against their previous copy might only be ~100 lines.

      The diff is only one part of our solution, though we thought it was pretty cool. Simply by providing a mechanism for caching the JS assets via localStorage, the system will often prevent a client from having to download a JS file altogether.

      Cheers - Jason

IMPORTANT DATES

REGISTRATION
SEP 17
COMPETITION
NOV 9-11 UTC
JUDGING
NOV 11-17
WINNERS
NOV 18

Thank you to our Platinum Sponsors