I still love nginx, but it's a damn shame they gate some really useful features behind their nginx plus license, namely service discovery via DNS SRV records, cache purge requests and request queueing for upstreams.
I think they have perfectly selected features for plus. Service discovery, request queueing etc. are only required above a certain size of backend, and in this way almost 100% of the hobby sites are fine with open source version. And the hobby sites are getting reliability and security that could only comes with paid service for free.
No personal offense intended, but I really /hate/ this justification. "Open Source" should not be equated with "Hobby Projects". It should be possible to run large-scale important things on pure open source.
My employer (Wikimedia, which is a non-profit) has been running all our server infra on pure open source for many years, for very principled reasons. We even publish all of our internal infrastructure config in public git repos as well. We're an open-source-only shop that's doing very non-hobby work. We have thousands of servers, we run our own CDN on multiple continents, we have somewhere around a billion unique users per month, and our average global rate of inbound HTTP requests is around 135K/second.
However, it keeps getting harder to stick to our open source principles every year as more projects shift towards open-core models with this kind of "Anyone who needs feature X is surely a big corporation making billions who doesn't mind running closed source software" type of thinking. It's not just bad for us, it's bad for the whole larger open source and Internet ecosystems if all the important bits for running at scale are locked up and hidden away in closed-source software.
This is a rather weird point to make, considering the way Wikipedia has handled the collection and usage of funds. Investing in its technical infrastructure would be a less controversial usage of their money than what they are doing currently.
As a theoretical exercise, I'm not sure that's strictly true. If you're reliant on donations, then income is never guaranteed. If you suddenly fall short of targets, maybe you have to lay some staff off. But stopping your license fees means you then have to pay someone to switch to an alternative.
Avoiding recurring payments you can't easily stop makes reasonable financial sense to me. Particularly for a business that doesn't sell anything.
As a theoretical exercise, you could also choose to "lavish" funds on the developers of the OSS platform instead of a recurring license fee for use, accomplishing a more outreach oriented use of funds than paying non-profit "employees" north of market with remarkable packages. If funds get tight, you can simply stop laving on the OSS, while enjoying ongoing use. Such a license can definitely be arranged -- I've done it at scale.
Let’s say you need a feature in nginx. You can either pay for an nginx plus license, or fork the oss nginx and develop the feature in-house.
Many people assume that developing it in-house is a one-time cost. However, in my experience, code is not an asset but a liability for which you will have to pay interest in the form of maintenance. More specifically, in this case there will most likely need to be adjustments made to the in-house extension as the oss nginx version it was built on top is being updated over the years.
My point here is that you should probably treat in-house code as a recurring cost, because in practice it usually is.
The question then becomes which recurring cost is higher: The license or the in-house maintenance?
I wholeheartedly agree with you, and I am so glad that wikimedia is pushing to stay truly open source.
Accepting these "sprinkles on top" licenses is highly foolish. I do hope that the true free and open source licensed software prevails in the server space. As many as possible must continue to hold out and push forward with new better and better scalable fully open source software.
> My employer (Wikimedia, which is a non-profit) has been running all our server infra on pure open source for many years, for very principled reasons
Curious how deep does it go? Do you only use firmware in server which has open source code. Do you require all drivers to be open source?
> It should be possible to run large-scale important things on pure open source.
It is definitely possible. With enough developer time, you could create copy of any software. Every paid or open core software runs on the philosophy that it is cheaper to pay them rather than to build them.
Which brings to the core issue, developer time is not free or cheap. The fact is that software quality and reliability is dependent on the money that the software has means paid software quality in general should be higher. While wikimedia could get enough money through donation, no one donates to open source projects that could be counted in full time developer salary, likely not even wikipedia(correct me if wrong).
nginx opensource is more than enough to have a very capable solution.
All bells and whistles that are fenced off are more or less for people who don't want to bother engineering their infrastructure properly. Or can't do that for a reason.
The only thing that may really be needed is name resolution inside an upstream. But you can work around it in many cases.
If there is something you need in OSS - consider asking in a mailing list. Best case - you'll have a workaround (an unlikely outcome: feature will be moved to OSS). Worst case - you lose nothing, don't you?
It's quite difficult to get a decent feedback on a thing like a webserver. You can't add a pop-up window or email your users. You have to rely on an active feedback. A vague comment somewhere in the internet doesn't count as such.
You are correct, but you are an outlier. The vast majority of organizations who are serving the quantity of requests trust Wikimedia is serving, are commercial entities.
DevExpress is closed source, but they give sources to customers under a promise that they won't leak. Maybe nginx can do something like that? Or openresty has a module for your needs?
Have you tried contacting nginx and explain the usecase/problem as you did here?
A lot of software companies have different or free licenses for cases like yours.
In Wikimedia's case it's more likely a matter of principle than cost. They have plenty of money despite what the donation popups make it seem. They want open source because it's open, not because it's cheap.
AFAICT nginx has a true open-source edition, which anybody can fork. Was there enough interest in the community in developing SCH features in the open-source code base?
Most large projects make money. Wikipedia is one of them (they have more than sufficient funds). If you are a company that writes open source software you have to earn money somehow. I think aiming paid features at customers that make money (i.e. large enterprises) is logical.
I like to overbuild my own things as a way to build a skillset, and I've basically only been doing that on software where only extremely specialized features are gated (e.g. Vault's HSM service, can't justify to afford one of those anyway).
I'm sure you like your site, but to be honest anything that doesn't make at least $3,675 per year is not a business, it's a hobby. Nobody said that hobbies are insignificant, they just don't (necessarily) make money.
If it were $3675/business/year, I might agree. But $3675/server-instance/year is insanely expensive for many very real businesses. That's $300/month! Imagine a business like Shopify, but with one server instance per customer. No way can you afford to do that if you have to buy nginx pro for each instance. Unit economics still matter to lots of businesses.
That said, you can run quite a big site without any of these features just fine. Also nginx is open source and you can implement these features yourself, either using C or Lua, relatively easily as well.
nginx is a reverse proxy. You're not meant to be running dozens of instances, you're supposed to stick one instance in front of a whole collection of application instances. Your nginx costs don't scale with your server load, they scale with your infrastructure complexity.
These prices (per instance!) add up; there are many other pieces of software clamoring for their share, hardware costs, networking costs, labor costs. It's death by a thousand cuts.
You're saying anything that makes under $4k a year isn't a business, but you're using it to justify that anything that makes under $8k a year isn't a business. Because you're earmarking those profits to go to a vendor, which then are not profits.
Are people not entitled to make $300 a month because they owe Nginx for a bot throttling feature? Can you run a small website without throttling these days?
> Are people not entitled to make $300 a month because they owe Nginx for a bot throttling feature?
Where is this attitude of entitlement coming from? Are people entitled to use Nginx's code for free? Yes, free software is a wonderful thing, and I do believe that most software should be free. But the reality, in this world, is that software developers need to eat, and relying solely on donations only works for a handful of projects (as does relying on selling support). Open-core is a reasonable middle ground.
If somebody wants to reimplement that feature on top of Nginx and release it under a free license, there's absolutely nothing stopping them. If you want to use an already written and well-integrated version from Nginx themselves, then you can pay up.
It's not a significant amount of money for any business that gets this amount of traffic. Being this stingy on software your business relies on deserves derision.
It has been built by extending Nginx with an OpenResty-based DNS resolver instead. You could even extract it away from Kong itself and use it standalone.
And, come to think of it, some additional great features would be automatic TLS certs via Let's Encrypt and maybe even being able to use a shared cache with multiple instances of nginx.
I put off adding TLS certs to my personal website for years. It's just a few static files served by nginx on a server - there wasn't a great reason to bother, I thought.
It took me almost exactly 3 minutes start to finish with letsencypt. Where 'start' was "I should stop putting that off" and typing "letsencrypt.com" into a browser, and 'finish' was nginx serving up https:// on all my domains. I'm genuinely curious what nginx could possibly do to improve the situation there.
Caddy targets single instances. No coordination is required to avoid receiving receive multiple certificates for multiple instances for same domain.
Nginx has a profit from big business that would require coordination to avoid multiple certificates. For comparison, it is worth noting that to ensure such coordination, Traefik requires an Enterprise plan and a separate agent. Such a business also often has mechanisms for storing certificates, so they do not necessarily want to integrate it into Nginx.
With Caddy, you can cluster easily by either sharing its filesystem data storage across machines, or configuring a different storage driver (redis, consul, db, etc.) and they will automatically coordinate for cert issuance. Caddy writes a lock file to the storage which prevents multiple instances from stepping on eachother's toes.
Still irritated that they ship it with an unprotected admin endpoint enabled by default on localhost:2019 that eats JSON and allows reconfiguration of the webserver like adding new sites that enable further attacks.
Put "{ admin off }" as a separate block in the root Caddyfile to disable it.
Our view is that localhost:2019 is inherently protected though - that only allows requests from the same machine. If you're running the machine with shared users, then of course it's up to you to further secure things.
That said, see https://github.com/caddyserver/caddy/issues/5317, we are considering changing the default, but that would be a breaking change although it would likely be a transparent change for most users. The default that makes the most sense depends on the platform and installation method, which is why it's complicated.
SSRF is a thing, just because you trust the code doesn't mean you've eliminated all security risks. The tools we use in the industry should all have secure defaults.
Glad to hear you're considering changing the default!
It would be enough for me if Caddy generated a password (that's hard for attackers to predict) on first launch, set that in a config file it has write access to (autosave.json for example), and then required Basic auth using this password unless the configuration specified otherwise. My problem is that this endpoint is entirely unauthenticated.
Assuming that everyone on local host should have admin access to the web server is an extremely bad default, yes. Listening on ports that the user has not requested is bad enough but doing so without any authentication on that interface is bonkers.
What is the problem here, that is any different from allowing untrusted code on your machine? Yes, if you don't trust your other users you need to lock down your system. That's not new or unique to Caddy, and indeed you can lock down Caddy too. Once the machine is popped locally it's already game over, authentication can't prevent that.
Large enterprise codebases will almost always have some security vulnerabilities in them but are still considered trusted code. Caddy is a fine alternative for ingress to such applications if you ask me, maybe a little on the immature side, but it's certainly getting there. An admin might absolutely pick Caddy for ingress in order to keep config simple, writing a Caddyfile, and ending up with the aforementioned admin endpoint enabled by mistake.
Defense works in layers. Punching through one of those layers makes securty worse even if that layer was not the first or the primary defense.
The problem here is that a priviledged process (and yes, a process that has access to Port 80 and 443 is priviledged even if it doesn't run as root) gives unauthenticated control to less priviledged processes.
With good security design you don't lock things down as needed but instead start fully locked down and open up only the accesss you really need.
Yes -- hugely underrated. Memory safety vulnerabilities are the cause of 60-70% of exploits [0] including the infamous Heartbleed. Caddy is not susceptible to these types of vulnerabilities. That we continue to deploy C code to the edge -- including software we think is hardened like nginx or Apache or OpenSSL -- boggles my mind.
Lots of folk have mentioned Caddy, but I'd like to also mention Traefik which kind of does the same thing but I found it to be less of a pain in the arse.
> And, come to think of it, some additional great features would be automatic TLS certs via Let's Encrypt and maybe even being able to use a shared cache with multiple instances of nginx.
That said, with servers like Caddy supporting automatic TLS and even Apache httpd having built in support now with mod_md, Nginx feels more like an outlier for not supporting ACME out of the box: https://httpd.apache.org/docs/2.4/mod/mod_md.html
For my personal sites I actually use Apache because it's pretty good and has lots of modules (including OpenID Connect support for Keycloak etc.), but Nginx has always been really easy to install and setup, especially for serving static files (e.g. built front end apps) or being a basic reverse proxy.
That said using HTTP-01 instead of DNS-01 across multiple nodes with the same hostname is needlessly complex in most cases and DNS-01 doesn't always have the best support in web servers (Apache in particular is just like: "Yeah, we can run whatever script you want, but the contents and integrating with your DNS server is up to you").
Does Certbot still pull in 40 odd dependancies? I wanted to use it but ugh, the amount of extra libraries it wanted to pull in. acme.sh does it all with a single bash script (plus a few supporting binaries, to be fair)
Not sure, I think I've only used the version they distribute on pip. Which can be a bit obnoxious if you have to build the python cryptography package for your target os/platform but if you don't it's a fairly minor install.
There are quite a few other options besides certbot, I was just suggesting it as a "install + one liner" for adding auto renewing ssl certs to a server.
There's literally zero work with Caddy. I'm not sure I understand what you're trying to say.
Also, there's massive advantages to having TLS issuance built into the webserver, such as proper OCSP stapling, having an active process to trigger renewals as soon as a revocation happens (hearing about it via OCSP), solving the ACME TLS-ALPN challenge without extra steps, and unique features like On-Demand TLS that many SaaS companies are relying on to provide a custom domains feature to their customers. None of those things are possible unless it's tightly integrated in the webserver.
I'm just daydreaming here, completely understand they have a business to run and they probably thought long and hard about which features to put into which license. No criticism intended, just something that eventually might catch up on them if another http server/proxy can offer everything that nginx does plus the missing items.
Oh, I thought their only product was plus and I didn't realize that the comment was referring to something else! Which proves the point the comment was making very well!! So from what I understand, they also sell regular ngnix support
What support and services do you actually need though. This would create a perverse incentive to make the software more complex and unreliable and then selling the resources to fix it for you.
Vs just actually selling the software and having it work flawlessly with minimal work.
If people want that badly enough under a FOSS license, there's nothing really stopping them from implementing it themselves, as an add-on or a fork of the BSD-licenced codebase.
If it's actually gated, not enough people care enough to get together and make and maintain a better one themselves.
I found that most of the time it was worth the time to invest a few hours/days into implementing the missing feature with Lua/OpenResty, if there wasn't already an Open Source extension available on GitHub.
The thing is, they aren't 'needs' just nice-to-haves that are provided for free with other open-source software. Thus, when trying to choose technology for personal projects, people don't choose your software. They choose a 'competitor.' Thus when it comes time to choose software with a green-field or startup, the non-free solution isn't chosen because devs aren't familiar with it.
Literally, at work (a startup), we use Traefik (which is horrible imho) instead of nginx (which I've used literally everywhere else) because the devs who originally started this project had never used it on a personal project.
So the issue isn't paying for it, it is about making it useful enough for personal projects that people use it for personal projects that it then gets used on professional projects. Right now, it is barely useful for personal projects unless you know what you're doing.
This is a great point. People use Caddy because for simple cases the configuration is seemingly "easy", and more importantly it includes Let's Encrypt out of the box. I've tried to argue with these users that nginx has many powerful (and necessary) features, to no avail. Traefik is a similar story: It integrates neatly with Consul for service discovery so it's a go-to tool for TLS termination in front of a service mesh.
Do I personally need DNS SRV support? No, I have a templated Nomad config that will re-render and reload the nginx config if the consul upstreams change. Setting this up though is definitely a bigger hurdle than just specifying a Consul service as an upstream.
I mean, if nginx offered a free personal license that worked on 5 servers (which would match up to Ubuntu Pro's free personal license), I'd be all over that. I have 3 servers and I'm very unlikely to dish out $500 a month, per server, to get nginx+, but I could sell that to my employer if I was already familiar with the technology or could convince my coworkers to go give it a try and see what they thought.
It seems they're more interested in high-touch sales (which is unlikely to happen on a dev team), vs. organic growth.
There is of course the issue of consistency: you can't be said to believe in free software (which is an ideology based around the rights of users) if you operate a proprietary software vendor.
A vendor that releases some free software and some proprietary software is a proprietary software vendor. (This describes, for example, both Microsoft and nginx.)
Yes, but people have to be ABLE to pay for your software. I simply cannot afford nginx+ for a hobby, and I'm a "free" asset when I make decisions for a company that CAN pay. Also, a startup may use the 'personal' license pre-revenue, then when it is time to scale up, they have to pay. Their features are coupled to your product, and they are going to pay because they have to (or rewrite a lot of code), a 'trial' simply isn't going to allow this to happen.
Maybe "these users" are not as braindead as you think? nginx still defaults to settings that made sense in 2005. I usually go for caddy now not because I am not familiar with nginx, but because it doesn't require me to drop hundreds of lines of config across a dozen of includes to get a properly configured HTTP server. Automation doesn't make much sense in our use case, and it gets tiring maintaining a separate ansible playbook just for nginx setup.
Caddy has sane defaults for many settings and I only need to add a couple of response headers, drop the domain name, and where to proxy the requests to. It takes care of maintaining things like the list of TLSv1.2 ciphersuites.
Adding more domains takes two lines of config thanks to parameterized includes — another important thing nginx misses that has to be implemented manually via scripts using something like envsubst (or again, full-on automation which is problematic if many of your servers are just a little bit different from each other — for good reasons).
Or maybe, just maybe, devs are sick of software that pretends it’s still 1995. Nginx is awful to work with, and has an awful community.
The cherry on top is the OpenResty “community”. You know the one that call you an idiot for not pretending Lua is orgasmically good; the one that declares you “just” implement thousands of lines of code because there’s no possible way a useful library could exist; the one with the ecosystem full of awful, worse than js leftpad garbage that hasn’t been updated in three years.
Not all of us are enterprise customers with enterprise budgets. There is no middle option. It starts at 3675 annually per instance. That’s more than the price of most of my servers
We do pay them, for a bunch of boxes. But I can't afford to run their plus version in front of every microservice. Or even in front of our entry point server. That's 5 times as many machines, plus staged versions of the app, plus other services running on the same boxes.
If you're running anything that is not multithreaded, you're running a reverse proxy in front of it. And even then you can still improve server availability by running a reverse proxy.
I love paying for software features that I need, but anything over $100 a month is too much for my personal budget for side projects. For example, I really want to run HA Traefik with Let's Encrypt shared across three instances in my datacenter rack, but Traefik Enterprise is many times more than what I can afford to pay, so I make due with one instance because the service discovery features for Nomad are fantastic. Same for Hashicorp Vault. I'd pay $100/mo for an Enterprise Vault for the HSM integration.
I really wish companies would come up with SMB pricing to help us side project hackers out so we could grow into paying for the bigger plans. Also, I understand why they don't. SMB support is a huge PITA.
> Don’t shame an amazing open source project for trying to be financially sustainable.
While one can talk about corporate contributions and basically ownership of some open source projects (the F5/Nginx case included), I also think that the funding is a big challenge for many projects out there.
