I'll need to write up how the scheduler works at some point, but it's heavily based on these papers[0][1]. It solves two different problems. First, it lets us throttle resource-intensive queries. Second, it enables work stealing. If you have idle cores on your machine, we'll assign those cores to running queries to help speed them up. That means if you have an over-provisioned machine, we'll make use of the extra capacity to speed your queries up.
Interesting. It would need some work to handle all the workloads I’ve faced where you have two applications with different priorities (ie oltp or grid workloads and analytics).
In those cases you want to make sure that BI users do not interrupt the transaction processing by assigning them a set of cores, a priority, and work mem limits for example.
Looks doable without major changes.
We don't expose the priorities right now, but we have the priorities decay over time. That way faster queries get prioritized over long running queries. That should achieve the behavior you're looking for.
Yes for this particular use case it might help (no per-application aggregated temp file limit though). Also my grid workloads generate thousands of threads running queries with lots of small updates (job queues, calculation results..). I wish there was a way for postgresql to isolate those connections to a limited set of cores (or number of cores).
To avoid completely saturating the server.
Right now we advise using s3 and storing results in files, and asynchronously loading them with batch processes.
The thing is you are going to be comoditized on this too. And it’s going to be faster to migrate off your database with AI. You are going to need to have a plan.
Committer here. I would love to hear more about this scenario.
Is the proposal to be able to export pg_dump formatted data on some schedule or trigger, entirely hosted in PostgreSQL and with timeouts? There are already extension that can export to blob/file storage and can be combined with pg_durable or pg_cron, so I assume the challenge is pg_dump compatible data export from SQL running in the database?
Just monitor it and you’re done. I’ve delivered and maintained hundreds of pg instances and never faced this issue. There is so much literature about it that at some point no one even slightly skilled will face it.
This is just anecdote, colliding with documented database behavior, who is not an issue on Oracle, SQL Server, or IBM DB2.
PostgreSQL explicitly documents xid wraparound as a failure mode that can lead to catastrophic data loss and says vacuuming is required to prevent it. Near exhaustion, it will refuse commands.
The mainstream media reported on Iryna’s murder as soon as there was video, but it has been a constant subject of reporting in Charlotte since it happened with immediate political ramifications.
I don’t read Twitter, but I do read my local news. I’m not quite sure that anyone is better off now that her murder is being nationally reported, to be honest.
It took CNN three days after the video was already circulating and getting huge traction; before they reported on it themselves.
four days for NYT.
This is after the video had already been circulating for days and received a lot of attention, the killing happened on August 22nd and the video has been going around since the 5th of september: https://www.mediaite.com/media/conservatives-call-out-media-...
These same outlets reported on George Floyds death effectively immediately.
Mark Duggan was shot in London and the US MSM picked it up faster.
Not aware of anything regarding local news, but when one killing reaches international news and the other has to be already organically international news via social media before reporting happens: people start to make presumptions.
PostgreSQL has historically been bad at managing the noisy neighbor problem, but with thread pools, and io priorities, it can be solved.
Has this been tackled here ?