A100 is available for $1.03 from GCP and V100 is available at $.27 per hour from Alibaba according to CloudOptimizer [1]

[1] https://cloudoptimizer.io

Just a side note here, it seems that these are spot instances, whereas all our VMs are non-interruptible. So there is a bit of difference (e.g. you probably wouldn't do a weeklong Blender render on a GCP VM if it could be interrupted and lose your work, whereas you can definitely run it on TensorDock because our VMs are reserved).

Of course, you can set up data checkpointing to save your data, but overall, it is a bit of an extra hassle to run on spot/interruptible instances, and if you do get interrupted, you are wasting valuable time waiting for stock to free up again.

Kind of sad because a Blender render is exactly the right usecase as it doesn’t really matter that much if it takes a little bit longer.

I have never seen this tool. Thanks for sharing!

The Alibaba price you cited is for interruptible/spot. For V100 on-demand (uninterruptible) Oracle is least expensive from that list at $1.275/hr.

Thanks, I created this tool to exactly find cheapest GPUs then expanded to everything else.

Interruptible is sufficient for model training which is what these high end GPUs are typically used for (and you want cheapest possible)

Wow, cool! Yes - interruptible can be very cheap... I'll add it to our backlog so we do that instead of idly mining

I was wondering, do you happen to have an API for listing servers? We're launching a marketplace later in August (https://www.tensordock.com/product-marketplace), and we expect pricing to be really really cheap. Like #1 in industry cheap while retaining.

Interruptible, if we add that, would probably be even less than those prices listed.

It'd be really cool if we could auto-update availabilities of GPU servers through an API so that we can list our servers on your tool as well :)

If your prices really end up being industry cheapest let me know and we can make it happen.

Didn't realize you made this tool. It's super useful.

Some unsolicited feedback, if you're still actively developing:

- You should consider some of the lesser known cloud providers (e.g. Coreweave/Lambda Labs/TensorDock).

- Add information about whether the servers support NVLink/NVswitch. For example, A100s come in 3 flavors: PCIe without NVLink bridges, PCIe with NVLink bridges, and SXM with NVlink/NVSwitch fabric.

There are hundreds of smaller providers, each having a different API , if having it at all, so this is not possible for a one man operation. CloudOptimizer is already the largest cloud comparison tool on the web (12 cloud providers listed)

>TensorDock doesn't actually own any of the server

Vaguely recall reading they're using a mixed model, but OP will hopefully confirm

Isn't that for only one quarter of the A100?

Yes... a full A100 (+ reasonable CPU etc) looks closer to 2€.

Somebody should build "cheap" service to EU. Quick googling and Paperspace is only one with datacenter at EU. Those GDPR requirements are now rather nasty so you usually cannot move customer/third-party data outside EU.

We will consider! We are working with two potential suppliers in the area... expect more news by the end of the year if those deals go through! :)

Thanks for the comment! Quick answer: it's complicated.

Long answer: We own a substantial amount of compute ourselves, as far as Singapore where we have fully-owned hardware at Equinix SG1. We started with our own crypto mining operation just outside of Boston, but as wholesale consumers approached us in 2020 due to pandemic surges, we added business internet and power backups. Suddenly we were operating a rudimentary "office data center." Two large reseller sites sell on our fully-owned hardware. Boston's electric costs are very high ($0.23/kWh), so we're gradually moving more hardware to tier 3/4 data centers that are cheaper on a per-unit basis.

But, we partner with 3rd parties too (4 large scale 1000+ GPU operators each to be exact) to resell their compute. This is also how we'll enter Europe... we're working closely with an existing supplier that colocates servers at Hydro66 in Sweeden and another at Scaleway Paris. We provide the software, they provide the hardware, and we pay them a special rate based on the volume we're doing. Partnering with others is the only way we can handle large scale without insanely high capex costs (that being said, we do get preferential pricing as an NVIDIA Inception Program member, which we take advantage of for our own fully-owned hardware).

We have a doc in it here: https://docs.tensordock.com/infrastructure/reservable-instan...

We're also working on a marketplace (client site: https://www.tensordock.com/product-marketplace, host site: https://www.tensordock.com/host). We expect a beta version to be up and running in the next ~2 weeks. With this, we'll have a script that hosts will use to install a small version of OpenStack. Then, they set prices, and customers can deploy directly on that hardware. By aggregating all these hosts together on the same marketplace, we hope we can slash the price of compute.

So far, owning our own hardware has allowed us to negotiate better rates and enter markets where previous services don't exist (namely, Singapore, where we sell subscription servers with 1070 GeForce cards for $150/month — unheard of pricing for an APAC city). Eventually, we hope there'll be suppliers in every city selling on our marketplace or core cloud product so that we can really become the #1 place for ML startups to provision compute. In a way, we want to be the Amazon of cloud computing. Amazon, in a way, created a global marketplace. Yes, they sell their own products, but they also sell others' products. By doing so, you know that you're getting a good deal on whatever you buy. We want to end up being the same thing for compute, but that's still a few years off :)

TL;DR - we own a lot of hardware, and we resell a lot of hardware. But in the future, we want to focus on the reselling aspect to truly be able to nail the user experience and handle demand surges while maintaining low costs.

And data security? Can you answer that part of the question?

Most of my ML training is done using personal data or sensitive documents, and I have not found a cheap provider yet that I can use.

Yes. Curious on this too.

Lambda/Paperspace/Coreweave do own their own servers (Lambda being the cheapest). That alleviates some security concerns.

It all depends on your security requirements though.

Whoops, apologies for missing this! For our core cloud product, we only partner with established providers. Large-scale compute wholesalers with $5m+ of compute each in secure data centers. These companies' entire businesses are built on selling secure compute to customers like us and other medium/large businesses. Basically, this isn't some random dedicated server host off of LowEndTalk :)

We have data protection agreements with all of them, and we can also do bare metal machines on request so that you have full control over your physical machine.