Minimize HTTP Requests
Normally, if a JS asset on your site changes, your users will have to download the entire file again even though just a fraction of it changed. DynoSRC sends down differentials updates so changes to large files don't require full downloads.
To expand on what we are doing here:
Problem 1: Mobile caches are much smaller than on desktop. Your code is more likely to be evicted from your users' cache before your next release/deployment, so return users often end up redownloading a previously cached file.
To address this, DynoSRC stores your JS files in localStorage. What's cool about this is that not only do you have a 5mb cache all to yourself, you know if something is cached or not rather than "flying blind" so to speak.
Problem 2: Mobile connections are slow. But it's mostly a latency problem -- a lot of people have 4G with faster download speeds than their home internet connections. Even on 4G, there is typically a ~200ms overhead for each HTTP request, which gets worse if it involves DNS lookups or redirects.
Limiting HTTP requests on mobile is important, and one way of doing that is to inline your JS into your HTML page. But that quickly becomes unwieldy for anything more than a snippet of code, as your users will be redownloading that code with every page load.
DynoSRC solves this by storing a manifest of what the user has cached in localStorage into a cookie. On each page load, the server checks the manifest, and inlines any JS files the client has not yet downloaded into the HTML. These files are then cached into localStorage and the manifest updated.
Problem 3: When you have a release/deployment and files are version bumped, your users have to redownload those entire files -- even if only a small amount of lines changed for a couple of bug fixes.
DynoSRC solves this by serving up differential updates. The server checks the manifest to see what version of a file the client has, then creates a diff between the latest version of the file and the version in the client. In this case, the inline JS file sent down to the client is simply a diff. The client applies the diff against its copy of the file, then updates its copy in localStorage and the manifest.
In the case of ready-to-deploy files, where multiple files have been concatenated together, obfuscated, and minified. The server will detect minified files and beautify them, creating a diffable file.
Other cool features of DynoSRC include:
Check out the source on github if you're curious or want to contribute!
DynoSRC computes diffs by forking a git process on the server. We hope to tie in js-git in the future when it supports the diffing capabilities required, making the project completely contained w/in node and node modules!
Other libraries used here: http://dinosrc.it/credits
Voting is now closed.
Hi Luis. Thanks for your vote! Could you elaborate a bit on what you don't like about the idea of storing JS files in localStorage? What sort of information do you think is better suited to localStorage? We'd love to hear your thoughts!
Hi Josh, it's nothing specific to your project. I just imagine my total local storage growing uncontrollably if every site places its code there.
Thanks for the vote! We agree re: the git process. We're following Tim Caswell's js-git project and are looking forward to when he ports over the diffing implementations so we can leverage that and reduce that integration hurdle. We are also looking at other diffing algorithms. We actually evaluated a few for the Knockout but found that forking a git process was the most straightforward way to achieve a reasonable level of performance when the input sizes were significant. Most of the other implementations fell over when the files reached a certain size :p
Thanks for the vote! Can you elaborate a bit on your security concerns? I'd like to understand more so we can work to address any vulnerabilities in future updates.
If the diff endpoint was not secured against traffic from other origins, I can see how a lot of traffic could overwhelm the servers as it struggles to compute all the diffs. We do cache the diffs once they are computed, which mitigates this issue a bit, but then there is a concern of running out of server memory. For the purposes of the Knockout we didn't focus much on tightening these bolts, but we're going to be iterating on the library to address issues like this in the near future :)
Hi Geoffrey, could you explain why you feel our entry is inappropriate for Node Knockout?
We were in constant contact with the organizers (onsite at Joyent HQ) during the competition, and they didn't seem to share this view at all. :/
By "inappropriate" I just meant that I'm used to seeing more full featured end-to-end apps rather than pure backend or API projects. No complaints about the possible usefulness of the library!
Thanks! We had a lot of fun hacking on this project and I'm glad to see people like yourself embrace the proof of concept so well. Look for the official github repo to have lots of activity on it once the competition is over.
Thanks for your vote! We didn't have enough time to give enough due diligence towards establishing some really solid performance benchmarks. That's something that we'll be rolling out, and using as a guideline towards figuring out how to further optimize various use-cases. We felt it would be better to leave detailed benchmarks out altogether (except with some mentions of theoretical gains) rather than hack together some data points that may turn out to be misleading. Instead we focused on hammering on some core use cases and making the API and implementation solid.
Definitely keep an eye on the Github repo after the competition, where we'll be updating both the module (adding more tests and such), as well as beefing up the demo site to include more performance benchmarks and example use cases so newcomers can grok the library with ease.
Mr. Dynosaur like you back.
Thanks for the vote! I was wondering if I could explain more clearly what we built during the hackathon so you can get a sense of what we completed. The repo at diurnalist/dynosrc was 100% built during the knockout. We've locked development on the branches there for the duration of judging. We also developed the website as a test/example application. The website actually uses dynosrc to deliver the simple JS (jquery included) that drive the demos.
Let me know if you have any other questions about what the project encapsulates!
Hi Ashutosh, thanks for checking out our entry!
The link provided (http://dinosrc.it/) is our node knockout project - the site is there to explain the tool and show its use in action. You can view our entry at the "normal" NKO URL here:
These are exactly the same site, hitting the exact same server, just different domain names pointing to the same place.
Hope this helps. Cheers!
Sorry, missed your further questions.
I'm not 100% sure what you mean by "forking a diff process", but all the diffs are only generated once then cached on the server. Even when not cached, we found that having the node server pull two versions of a file from github, diff them, and send them to the client typically took less than 150ms.
The diffs are applied in the client using the applyPatch function from Kevin Decker's jsdiff library.
Thanks for responding Josh. Checked out and idea is indeed promising and sounds like you guys achieved a lot in 2 days.
Checked out the diff patching code as well on client.
Pretty solid overal.
@pgte, we are talking every day on how to actually make this module battle tested enough for teams to use in production. We certainly believe this knockout hack proved it can be done and this technique needs to happen in the mainstream.
This is the official repo https://github.com/diurnalist/dynosrc, we have decided until the competition we will not be accepting pull requests so the perception of our hack isn't altered. We feel this is a way to "lock in" the code during the competition to remain fair to other entries who can't / didn't open source their project outside of the walls of the knockout servers. This is all to say expect LOTS of activity on this repo once the competition is over.
Let us know how it goes! Obviously we "web app" to showcase the module was the only thing that actually used it- so we're waiting for some real feedback from the community how to actually harden the API / solution. If you in SF, hit us up and we'll pair with you to get it going.
Yes, the diffing part of it is probably only useful in certain use-cases. Rare updates have the benefit of being able to just cache heavily and then version bump assets on a deploy. In that case, this library would be helpful instead to give you the automatic inlining of assets on a mobile-served page.
The proxy thing is finicky right now and is intended to be more of a conversation piece than anything. If you have 3rd party cookies blocked you're going to have problems, I think. Also make sure you try it at http://dinosrc.it/proxy-demo without the
www - think that might make a difference, but I can't remember why right now :p - we might have not set it up entirely right, that push happened pretty close to the deadline!
Never mind, I'm full of it - the domain shouldn't matter. I bet it's the 3rd party cookies setting on your browser. Browsers treat
localStorage in that same category of 3rd-party persisted data, so you won't be able to automatically write to the proxy'd domain's storage in most circumstances w/ that protection turned on. Theoretically there are times when it won't be blocked, such as if you've visited the proxy'd domain before in the same browsing session (I think that's how Safari and latest FF do it) - but in my experience it's hard to rely on.
Thanks! Let us know if you have any issues or suggestions on how to make it fold into your existing projects with greater ease.
Thanks for the feedback - let us know what you think when you get a chance to try it out on your own time.
These are great questions. So one aspect of our hack is we do in fact try to address magnification. Essentially our module detects if a file is minified (has a small amount of lines with a LOT of characters per line) and appends semi colons with new line characters. This way our diff can be on a line by line basis. In theory a minifier that is consistent would not produce noise in a diff between to files except around the LOC that actually changed.
I do not feel there are any more security concerns from this solution than traditional script serving models. The files themselves are stored in local storage, which inherits the same security as cookies and other locked down aspects under a domain. So scripts cannot be used from other domains.
@mhevery to provide a bit more clarity, here's what happens when a patch is applied with the dynoSrc client library via a call like this:
dynoSrc.apply("moduleName", "1.0.1", "git diff -- ... diff ...");
We did try to address minified files as @ryanstevens attested to. There is some debate about the merits of minification given how good gzip is on its own, and we wanted the tool to be agnostic to your decisions there.
That was the main design goal of the library: don't be opinionated, and offer a flexible library that can support various paradigms. You can use the server component to generate diffs that you apply w/ your own JS code, or you can write your own tool to send down diffs on a different stack than node and still use our client library to do the patch-ups and caching.
Thanks for the feedback! You can read all the source for the module on our Github.
Spatial Automation Lab -- University of Wisconsin, Madison / 3D Systems / Bespoke Innovations
Thank you for the vote and feedback!
One gigantic script file is actually where diffs would work best, we think! Since there is a bit of overhead for a diff, just b/c of the structure required to encapsulate the information of added vs. removed lines, the benefits are bigger when your changeset is small relative the entire file size.
Consider if the JS being thrown down to the page is ~30K lines. If you roll out an update that changes only a few functions, a client's diff against their previous copy might only be ~100 lines.
The diff is only one part of our solution, though we thought it was pretty cool. Simply by providing a mechanism for caching the JS assets via localStorage, the system will often prevent a client from having to download a JS file altogether.
Cheers - Jason