I try to convince my team that the node.js ecosystem has gotten into a stage where it cannot be used for security/financial applications because the sheer amount of dependencies pose an inherent threat. I advocate for Go because of the tendency to less and easier reviewable dependencies. Nobody except me seems to see a problem there, despite me being able to point out specific security incidents.
I am wondering if I am missing something obvious here and would value any opinion.
I don't think the NPM ecosystem is in a great security position, but the pushback I'd make here is that you should be reviewing your Go dependencies as well.
If you're looking at the NPM ecosystem and saying, "the number of dependencies is problematic because it takes a long time to review them", I agree with you. If you're looking at the Go ecosystem and saying, "there are fewer dependencies, so I don't need to review them", then that's a security antipattern.
The better way to phrase this to your team is that you need to review dependencies, period. The NPM ecosystem isn't problematic because it introduces a new requirement to review dependencies, it's problematic because reviewing dependencies is harder. You can use NPM all you want for security-sensitive code. You just have to spend the extra time to review dependencies, which means bringing in new dependencies will be much slower than in a shallower ecosystem like Go's.
That's the point of OP's article. You can't skip reviewing dependencies in any ecosystem, in part because all of our ecosystems across the board have crap sandboxing and permissions, but also because of license issues and quality issues like the author found. There is no shortcut for that, whether you're using Ruby, Go, Python, whatever. You have to review your dependencies.
"the number of dependencies is problematic because it takes a long time to review them"
The dependencies themselves are generally smaller and more self-contained. Arguably it is easier to review since the side effects and cross-module behaviors are much more pronounced (it's generally deemed a "bad thing" by the community for modules to make unnecessary global actions)
My point being that "NPM is bad for security" is probably the wrong take to convince a software team to avoid the ecosystem.
Instead, phrase it as, "well obviously we need to review our dependencies, so why use an ecosystem where that's hard?" Don't phrase it as a security problem, phrase it as a development time and overhead problem.
Or better yet, let your team use NPM as long as they review the dependencies, and if your dependency tree is really a problem, your team will naturally become annoyed by the extra work and will avoid large dependencies on their own without a lot of extra prompting.
My biggest issue with the massive dependencies is not licensing or security. Both are important, but there is an even bigger block.
My biggest issue is with things in the node ecosystem is breakage, especially React libraries. I was following a NetNinja tutorial series on Youtube. I had to use a Firebase-Redux library for a combined React/Redux/Firebase series. There was a security vulnerability with something down the stack only a few months after the tutorial was out. So to use the updated version I had to use the beta version of a Firebase library which broke the tutorial code despite the library change being a point release like 1.4.0-beta2. Either 1.3.0 to 1.4.0 or 1.2.0 to 1.4.0 would be consider a breaking release for this particular dev.
We have a large group of dependencies chained upon each other with maintainers that have different support commitments and styles. React might be the de facto JS system, but it relies on a large system of these 3rd party packages to do deeper functionality which can so easily break things. I already had to peg React itself to an older version to use this tutorial when it was months old.
I realized at this point how much of a pain it would be to maintain this code in my portfolio if I did something with it. Let alone a paid product. I stopped learning React at this point (or at least learning React+Redux+Firebase).
Ruby sorta has this issue too, but it feels the Node ecosystem is much more accelerated. The Rails framework dominates the gem ecosystem so much usually gems break functionality by the version of Rails they support. So gems often will not update or warn of breaking changes based on the Rails version or ActiveRecord which follows Rails versioning. This makes gem dependencies much more manageable.
The node ecosystem does not have a large dominating framework like Rails. Even if you considered React that framework, Facebook breaks it whenever they please. 16.4 which sound like a minor version update break things. While Rails 4.2 was supported for 4 years. jQuery might be years before it breaks support.
similarly, I refuse to use JS libs that have ridiculous dependency trees (most annoyingly, including Webpack, which means my Vue setup is more interesting than it needs to be).
I was explaining why to a friend, and it appears that no-one takes this threat seriously, not even in secure/financial apps.
Like you, I wonder if I'm missing something obvious. Why does the npm dependency trust nightmare give me the screaming heeby-jeebies but everyone else thinks it's all perfectly fine?
It's not just you. Screaming heebie-jeebies is exactly the correct response.
People who don't worry about it tell themselves that nothing that matters very much is coded in javascript, obvious exceptions notwithstanding.
Maybe the only practical way left to avoid it is to avoid javascript for things that matter, and to avoid products constructed with javascript for uses that matter. This might be an unpopular observation, but that does not make it wrong.
It is possible that WASM can fix much of this, at the cost of exposing ourselves to what might be extreme risks inherent in WASM itself. And of course, to whatever dependencies are brought in for the language we compile to WASM.
Indeed, rust is a likely wasm choice and it is working on an npm nightmare of its own with crates.io. A blessed crates pack is sorely needed to fill out the intentionally thin stdlib in cases you can't afford a dependency tree 20 layers deep for all the reasons mentione on this thread.
It's unfortunate that Rust is so tied to cargo, and unfortunate that cargo insists on allowing multiple versions of a dependency. If you point out that cargo is poorly designed everyone just yells and you and says you don't understand.
Nonetheless I have run into the diamond dependency problem in practice. Their solution is unprincipled and there's no way to turn off the default behavior.
Do you mean passing a struct from e.g. clap 2 into clap 3? Yeah that won't work directly, nor should it in my opinion. If it is frequently required, projects can and should provide a compatibility layer which does any necessary conversions.
That would seem to completely disallow any compatibility layers, which seems like a bad outcome. Unless you had some sort of solution in mind for that?
I agree that the error message sucks, for what it's worth.
> The second biggest flaw is that it couples package management to a programming language, despite those being almost completely unrelated concerns.
Yes it's frustrating. I wish it was more decoupled, maybe with package standards like Haskell tried to do with Cabal.
It makes it hard to use Rust as a drop-in replacement for C. I can add a few C files to a Haskell project without much problem, but most rust code goes via cargo.
I don't think anyone thinks it's perfectly fine (well, maybe newer people in the industry). I just don't think anyone sees a viable alternative.
By adopting any semblance of a modern front-end stack, you're going to be pulling in a huge dependency tree. So, your choices are:
A) Don't do modern front-end development. This, despite being very popular with the HN crowd, is a challenging option. Losing the power of popular frameworks makes it difficult to recruit new developers, slows down developer velocity, and can cause tech debt to accrue more quickly.
B) Validate every dependency. For a skilled security reviewer, this may be possible. For an average developer however, reviewing thousands of dependencies is unlikely to be a good use of time. There are some improvements, automated tools that can be run, but the unfortunate reality is that the NPM attack surface is massive and I don't know of any techniques for producing any truly reliable security assessment.
C) Depend only on packages that you trust and trust them to validate their dependencies. Obviously, if this transitive trust breaks down at any layer then a vulnerability is introduced.
In practice, I've only ever seen individuals and companies do C. I think if there was some tooling around B, that would be the best option. I see a lot of room for disruption here - if there was a way to assign trust to a dependency and then a package-manager level construct that would only allow trusted dependencies to be installed, that might work.
What are you using instead of Webpack? I have a very old personal project still using Webpack 2 and would love a simpler alternative that isn’t just downloading library dist files and checking them in to my project repo. I don’t even need babel or minification or any of that, just simple dependency version management and fetching.
Granted this only covers the initial package for each of these, there are usually a bunch of extra plugins or cli packages that you'll need to work with each of these but I also found the dependency tree of each of the added Rollup plugins to be less than the other 2.
Because even in 2020, even in " secure/financial apps" most programmers have security and lic legality as a secondary (or even forth level) concerns, they just want to "make something cool"
They play lip service to it being secure, and/or license compliant
I would argue the reason developers don’t have concerns around licensing is partly due to lack of knowledge around the differences between licenses at a fundamental level and partly due to a “why should I care?” mentality at least at a subconscious level, if not higher.
A got a taste of it yet again this past weekend when I commented here on HN about GPL and AGPL being licenses that corporations often have concerns with.
There's also Java and .NET for those who want better IDE support, better dependency management (without recursively downloading hundreds of packages), and a more traditional language and more fully featured language.
PS: I came to statically typed languages as an adult, I actually disliked those languages before I had the lightbulb moment, so I think I might even be more qualified than everyone who hasn't managed to enjoy both sides ;-)
Actually, again as someone who has some real life experience with a number of different systems, including
- pip for Python,
- NPM,
- Nuget
- and Maven (and Ant)
Maven is by far my favorite, despite XML.
In fact, give how small pom files are and how little you have to deal with them (if you know what you are doing) I find it amazing how many comments I have had to read about "XML-hell" etc.
Rust is on this path as well, the small standard library pretty much forces people into using a ton of tiny dependencies. Compiling a rust project is exactly like using npm and seeing a never ending list of deps
The underlying problem is the lack of curation. Micro-dependencies that are widely used and have been reviewed for quality should be promoted to a group of standard packages. They'd still be dependencies like any other but they'd be officially maintained and installed by default.
Another problem with npm: every package gets its own copy of its dependencies. Not sure if Rust does that.
No, you are basically right, but the number of nodes in the dependency tree doesn't really mean that you really have to review all of those. Usually you end up with a big basket of actual dependent projects, and with some versions for them (which leads to the big explosion of the number of nodes in the dep tree).
Naturally it should be easy to specify a whitelist of licenses. (Of course then one has to decide whether to trust the package.json-s.)
That said, security review is hard for any ecosystem. Go probably has inherent advantages compared to the JS ecosystem, simply by virtue of being younger, having a real standard library, being more focused (no browser vs nodeJS issues) etc.
PS: there are projects that aim to do collaborative audit/review for Rust ( https://github.com/crev-dev/cargo-crev ) there should be something like that for the JS world. also there's the NPM "report vulnerability" feature.
I'm not a web dev, not a JavaScript fan, but out of pure curiosity I've been playing with the idea of doing my next side project in node, just to get an idea of what modern Js feels like.
But the whole dependency hell, left-pad and the likes are a real turnoff. So what I'd probably end up with is coding in pure Js without a package manager or "build system". Basically like you did PHP in the 90s. The question is whether I'd end up with anything remotely sane (already assuming it doesn't involve any crypto or similarly complex).
It is certainly possible to do JS development in a more controlled way.
My advice is to avoid most of the JS build tools (i.e. grunt, gulp) by using npm scripts (basically shell commands defines in `package.json`) to trigger build actions, or to start smaller, more specialised tools. At least then you will understand how your build works and won't have an extra layer of buggy plugins screwing things up.
Also being conservative and critical of the dependencies you take on is a great idea. Many smaller dependencies are not worth it. Bigger popular libraries like lodash are often a better deal. Unfortunately this micro-module philosophy and "publish random trash to npmjs.com" attitude has created a huge amount of crud packages.
I've had success organising a bigger project into multiple node packages inside the same git repository and using yarn's workspaces feature to tie it altogether. I avoid the complexity of publishing those (private) packages on npmjs.com. yarn's workspaces make it possible for my main app to find its (internal) package dependencies in the same git repo.
is certainly possible to do JS development in a more controlled way.
My advice is to avoid most of the JS build tools (i.e. grunt, gulp) by using npm scripts (basically shell commands defines in `package.json`) to trigger build actions, or to start smaller, more specialised tools. At least then you will understand how your build works and won't have an extra layer of buggy plugins screwing things up.
Also being conservative and critical of the dependencies you take on is a great idea. Many smaller dependencies are not worth it. Bigger popular libraries like lodash are often a better deal.
I've had success organising a bigger project into multiple node modules inside the same git repository and using yarn's workspaces feature to tie it altogether. I avoid the complexity of publishing those (private) modules on npmjs.com. yarn's workspaces make it possible for my main app to find its (sub-)module dependencies in the same git repo.
I have a project with a build system that is basically cat src/js/* > dist/bundle.js , it's refreshing when you have to deal with setting up webpack at work.
It’s slowly expanding, but JS has an anemic standard library. The node platform adds a little, but yo’re still miles away from a typical standard library like Java or python.
If you want to have 0 dependencies, you will have to write a lot of code to do things that would just be a line or two in most other languages.
The problem is not with anemic standard library. The problem is with lack of commonly accepted set of libraries. Java is not so good. You can't even copy one stream to another in Java 8. There's no built-in DI. There's no built-in ORM. Even built-in logging is so bad that everyone uses something different. Hell, no built-in JSON in 2019. But there's set of community-adopted libraries and everyone's using them. You need more utility methods? There's Guava (and, sometimes, apache commons). There's no left-pad library. You need DI? There's Spring. You need ORM? There's Hibernate. You need logging? Well, here's some competition between logback and log4j and that's about it. And even that competition has plenty of adapters which allows to write library code that works fine with both.
So JS community should throw left-pad in the window and declare underscore.js as a library of choice, so every other library will depend on it and that's about it.
I tried vanilla JS three years ago. It got so tedious after just one page that I decided to try and use Elm instead. I have not regretted it once. In contrast to the hacky Npm ecosystem, Elm feels like I'm working with somebody who values my mental sanity a great deal.
Modern JS is the current ECMAScript standard. Typescript is not "modern JS", any more than Coffeescript was, any more than the compile-to-js language that will probably replace it in a week.
You're confusing the tool for the language the tool operates on, akin to claiming modern C++ is Visual Studio.
Coffee-script was obviously always a fad. It basically traded legibility for ergonomics. Adding nothing of particular value to maintainability and longevity.
TypeScript otoh is just ES with types. It moves in the completely opposite direction, adding just enough syntax to get a proper type system in place (a pretty good one to boot).
If anything I would expect it’s syntax additions to be adopted as the standard and directly supported by runtimes. Despite static typing not being that useful for interpreters.
Unless runtimes will completely phase out es, focusing on wasm. (But that will take decades, so not really an either or thing)
Not using TS means leaving a lot of useful tools on the floor.
The ECMAScript standardization is process is invaluable, after all TS builds upon it. But TS provides a saner subset (and adds a very productive type system), without that I wouldn't touch "modern" JS even with a stick.
They both have flaws. NPM due to the cascading mass of transient dependencies; go (historically) due to pulling dependencies from the master branch of a repo. All of those can change from underneath you.
In any case you can’t avoid proper auditing and pen-testing. Even if your app code is rock solid, it doesn’t mean your infrastructure is safe. Or even your office building or the data centre serving the code. Or your employees and their credentials.
All it takes is a decision to use mongodb or redis without auth and to expose the ports to the internet, and the most secure runtime in the world won’t save you then.
Or a bit of social engineering so the attacker is making perfectly legit requests in your perfectly secured app.
Dependencies are cattle, not pets. There's nothing wrong with having zillions of them; what you need is good tools to manage them in bulk.
In the JVM ecosystem, projects are only allowed in the Maven Central repository if they have their license documented in a machine readable fashion there; it's then trivial to check what licenses you're depending on (and there are several plugins available for doing so). I'm amazed other ecosystems don't offer the same.
That doesn't make any sense. That analogy works for mostly-identical computers, where if your software won't run on one computer you can just use another mostly-identical computer.
Almost by definition dependencies are not interchangeable. You can't replace a routine to pad strings with a routine that is a web server. Or a matrix multiplier. Even dependencies that do the same overall job almost never have the same API. Heck, even the same dependency often ends up with a different incompatible API over time as versions change.
> There's nothing wrong with having zillions of them; what you need is good tools to manage them in bulk.
Every added dependency is a risk. Each unintentional vulnerability in each dependency increases the number of vulnerabilities that might be exploitable in your system. And that's just the unintentional vulnerabilities.
Practically all systems provide no useful sandboxing between dependencies, so if any one of your transitive dependencies is malicious, then your entire system is malicious.
Every new dependency also brings in potential license issues, per this article. I think it's unacceptable to have a scarefest about the AGPL, GPL, or LGPL; for a vast number of applications those licenses are just fine. The bigger risks are software that has no license at all, which are a legal risk for any project that uses them until governments change international treaties involving copyright (which is not likely any time soon). But it's certainly true that various licenses are not acceptable for certain situations, and every new dependency increases the risks of licensing problems.
Having no dependencies is absurd; it's uneconomic to build everything from scratch. But every time you add a dependency you need to think about the trade-off; it is sometimes wise to not reuse something.
... Which also happen keep their own cattle that you're still responsible for.
It's not so bad in languages with solid standard libraries. In Python projects I might have 20 direct deps, ~50 indirect.
In a real JS project I'm building, I have 17 direct, 3829 indirect. The JS standard library is so damned thin that everything pulls in some random version of the kitchen sink.
yarn list | sed -E 's/.*─ //' | sort -u | wc -l # minus 2
In situations like that your job of auditing licenses, updates, sec issues, etc balloons exponentially with each new dependency.
Tooling should absolutely be used, but it still doesn't perform the job of working out whether or not you want to upgrade a component, or whether or not you're likely to have suffered a security breach, or how to report on how well audited your dependencies are.
"Servers are cattle, not pets" relies on the servers being substantially identical, cloned like bananas from a single source where each is as good as the other.
Dependencies are, if they are to be useful at all, all different. Dependencies are suppliers, in the business sense. Having lots of dependencies loaded at runtime is like a modern just-in-time giant supply chain; it lets you take advantage of efficiencies in exchange for being more brittle.
Or they are like BOM items on a circuit board. Part of the original drive to "componentise" software came from people experienced in electronic engineering; you don't have to reinvent the transistor, you just buy them at a cost of a few dollars for a reel of thousands. But experienced designers will still try to:
- choose more-common components wherever possible
- ensure there are multiple sources for a component
- reduce the overall number of BOM lines, which reduces supply risk and inventory cost
The software world would go completely bananas if the cost for dependencies was not exactly zero. Imagine having to license left-pad.
> There's nothing wrong with having zillions of them...
There's nothing wrong until something goes wrong an now you're royally screwed. With zillion dependencies you are at a mercy of zillion maintainers, and none of them has any obligation to you. They can break backwards compatibility in patch releases, introduce subtle behavior changes, steer the project in an unexpected direction or abandon it altogether.
I’m a bit torn on this. I have most of my experience in the .NET ecosystem, where dependencies are a lot more manageable. However, if something breaks, you’re screwed a lot harder, because it’s not so easy to replace a large library, and there are very likely fewer well-maintained alternatives than there would be on NPM.
In total, I find it hard to deny how productive the NPM ecosystem can be, despite my philosophical objections to the way the community is run. Am I crazy here?
You aren't alone in this. The Node/NPM/JS scene is churning out code and innovations like there's no tomorrow, that's something to admire.
What I feel they are missing is a community process to consolidate things. You don't need three generations of ten incompatible solutions for a given problem - after some iterations, things should consolidate into one or two more or less standardized libs that don't break existing code at every damn point release.
> You aren't alone in this. The Node/NPM/JS scene is churning out code and innovations like there's no tomorrow, that's something to admire.
I don't find churning out code admirable, and I also don't think I've seen any true innovation come out of the NPM scene (bar innovation in the browser/JS space itself, which I think isn't a good measure as it's mostly just working around limitations that shouldn't be there in the first place).
That goes into the direction of my thinking. I am concerned about transitive security issues. It is impossible to check in node dependencies into version control (size/binaries). They have a lock file to pin versions, but dependencies that are downloaded upon each build are are not reproducible from my point of view. With Go, it’s easy to vendor and check in, it’s also straight forward to review them. There have been examples of targeted attacks using npm packages and that is something I am very concerned about.
People move billions with a node.js application we develop and the company will eventually be liable if the system is compromised through a targeted attack.
On a different note, I think the ecosystem moves too fast, packages and versions are getting deprecated and new ones getting released constantly. I have the feeling that the whole ecosystem is targeted towards building small MVP apps, not relying a long-term business on it. Maybe I am too harsh here, but that is a frustration growing for years now. I am happy to be proven wrong.
Not a huge fan of node or anything but npm lock files do pin to a hash. Also in commercial world you're going to be pulling through nexus or some other cache to reduce bandwidth use and developer downtime.
Are there other reproducibility concerns I should be worrying about? Are you thinking npm modules with native code or that (this does happen!) actively pull other stuff during build? Most of those do their own pinning but agree the whole thing is messy.
> There's nothing wrong with having zillions of them
That is a scary point of view. We have forgotten that there is no such thing as a zero-cost abstraction, apparently, and shovelware developers are now employed by enterprises and write enterprise software...
CPUs aren't getting faster. Software is getting slower a LOT faster than hardware is getting faster, these days, and people are still apparently perfectly fine with adding dependency upon dependency upon abstraction upon abstraction and it's adding up extremely quickly.
A good first step to addressing this is to favor a small copy & paste operation over a small dependency. Lots of people will shriek at the idea of this, but I promise you, all the problems with copying and pasting code are nowhere nearly as severe as all the problems with dependencies. Working to avoid problems you don't have results in creating problems that you definitely do have.
This protects you against licences you don't expect, but not against malicious or subverted dependencies.
Since a dependency can generally do anything your application has privileges for, widely depended-on libraries are an attractive target. A cattle approach means more dependencies, and it being easier for new ones to sneak in.
> what you need is good tools to manage them in bulk
How do you manage reputation and relationships in bulk? There are transactional costs here. Dependencies created by maintainers with impeccable reputations are a much smaller risk than arbitrary dependencies created by arbitrary maintainers.
You require releases to be signed by maintainers (again, something maven central enforces and other repositories ought to), and then you have a notion of maintainer identity and can decide which you trust (again something that plugins let you do). If there are still too many maintainers then you can use the GPG web of trust approach, as e.g. Debian does, and see which maintainers are part of trusted organisations.
No (and I don't think any code repository does or could; what would a "proof of security" actually prove?). It does require machine-checkable signing of all releases.
>Nobody except me seems to see a problem there, despite me being able to point out specific security incidents.
This seems to apply to security in general. Security these days is doing the minimum amount needed to check some boxes that you are now secure. I suspect a lot of this is driven by incentives. There is few negatives to an individual to use bad security over good security, and the costs of good security means less of the 'good stuff' being developed which results in worse reviews and less prestige. And if people are hacked, the blame is primarily placed on the hackers with little danger to the developers and often even less danger to the company (they might have to spend 5% of the budget they saved on PR to repair their image).
I'm not sure how to fix the issue with prioritization of security. My first guess would be by changing the incentives to companies so they bear the liability in identity theft instead of the user (the very concept of identify theft is a trick to blame the end consumer instead of either the business leaking data or the business giving away money without verifying if data is accurate).
I am wondering if I am missing something obvious here and would value any opinion.