#359 — June 10, 2020

Read on the Web

Postgres Weekly

10 Common Postgres Errors— Some quick fire common errors and warnings to watch out for, with symptoms and solutions, around things like memory, disk space, and permissions.

Ibrar Ahmed (Percona)

A Tool to Make EXPLAIN ANALYZE Plans More Readable— A long standing (over 11 years!) Web-based tool that lets you paste in the result of EXPLAIN ANALYZE queries and see a more easily understandable version. Why are we linking it again? It’s had a bunch of updates.

Hubert Lubaczewski, Łukasz Lewandowski

Whitepaper: Business Case for Professional Support— Learn the importance of Professional Support for your mission-critical PostgreSQL systems & how it can benefit your company. It increases database performance, helps scale, distributes data, reduces costs, saves you from known pitfalls, and more.

2ndQuadrant Services sponsor

PostgreSQL Load Balancing with HAProxy— I’m a huge fan of the haproxy TCP and HTTP proxy/load balancer, but admittedly haven’t used it for Postgres before.. This post demonstrates using HAProxy with Severalnines’ ClusterControl for load balancing Postgres much in a similar way as you might an HTTP server.

Severalnines

Multi-Master Replication Solutions for Postgres— Horizontally scaling Postgres has been enough of a challenge over the years that entire companies (e.g. Citus Data) have been founded to make it easier. But there are various ways to do multi-master replication nonetheless, and this post links to several approaches, whether open or closed source, free or paid.

Ibrar Ahmed (Percona)

Deduplication in Postgres 13 B-Tree Indexes— PostgreSQL v13 introduces deduplication of entries for B-tree indexes. This article describes the new feature and demonstrates it with a simple test.

Laurenz Albe

A Step-by-Step Way to Backup a Heroku Postgres Database to an AWS S3 Bucket— Another variation on this that’s worth doing IMO is turning on S3 versioning so you can easily get rolling backups and keep a consistent filename.

Paweł Urbanek

Locating the Postgres Configuration File— SHOW config_file is the key here, but it might take a little more work if using Docker, say.

Luca Ferrari

What Are Failover Slots?

Craig Ringer

Visualize Postgres Performance In Real-Time With Datadog— Datadog’s Postgres OOTB dashboard visualizes data on latency, errors, read/write capacity, and throttled request in a single pane of glass.

Datadog sponsor

pgsql-http: An HTTP Client Extension for Postgres— If you need to make HTTP requests direct from Postgres, this is one way to go and links against libcurl.

Paul Ramsey

PgHero 2.5: A Performance Dashboard for Postgres— Built in Ruby. And, yes, we have a newsletter for that. 😁

Andrew Kane

SQLancer: A Tool for Detecting Logic Bugs in Database Systems— Considered a ‘research prototype’ for now, SQLancer’s job is to stress a database system into returning inconsistent or illogical results. Written in Java, it supports several databases (including Postgres).

Manuel Rigger and Zhendong Su

#360 — June 17, 2020

Read on the Web

Postgres Weekly

Jepsen Finds a Bug in Postgres 12.3— The name ‘Jepsen’ is sure to stir emotions in any database server creator as their analyses of distributed systems (such as databases!) often reveal all sorts of edge cases and flaws. And so it goes with Postgres with a bug in serializable isolation being found. This is technical stuff, but it’s great to see systems like Postgres placed under such rigorous analysis.

Kyle Kingsbury

How One Word in Postgres Unlocked a 9x Performance Improvement— The creator of a personal finance tool experienced a user whose data caused a flood of INSERTs large enough to cause a problem. Here’s the tale of how a simple RETURNING clause enabled a huge optimization in the process.

James Long

Webinar | Distributed SQL: A Modern, Cloud-Native PostgreSQL— Hitting challenges scaling PostgreSQL, deploying it in the cloud, or using it with microservice architecture? A Distributed SQL database might be a better fit for your workload. Tune in tomorrow, June 18, for an architecture review and live demo.

COCKROACH LABS sponsor

Citus 9.3 Released: The Postgres Horizontal Scaling Extension— Citus is the now Microsoft-owned Postgres extension that turns Postgres into a distributed database that can be more powerfully used across multiple servers. 9.3 improves the distributed SQL support, now with full support for window functions.

citusdata

XgeneCloud: Instant REST and GraphQL APIs on any Database— Where ‘any’ includes Postgres, naturally :-) There’s quite a lot of pieces to this and they seem to be forming a company around the tech, but it’s open source nonetheless.

xgenecloud

Understanding User Management in PgBouncer— PgBouncer is a popular connection proxy and pooler for Postgres, but it doesn’t just pass things straight through to Postgres, it can handle authentication directly itself. This article clears up how that works.

Peter Eisentraut

Controlling Runaway Queries with Statement Timeouts— A well tuned database on good hardware is easily capable of many thousands of queries per second but long lived queries can quickly cause things to back up. Here’s one defense against queries that want to live forever.

Craig Kerstiens

A Free, Multi-Node, Petabyte Scale, Time-Series Database on Postgres— Read more about Timescale’s multi-node time-series database and why they’re making it available for free.

Timescale sponsor

Controlling Server Variables at Connection Time— A brief tip, this. For example: psql 'options=-cwork_mem=100MB dbname=test'

Bruce Momjian

Composite Data Type Performance Issues in Postgres

Hans-Jürgen Schönig

Running pgbackrest on FreeBSD— pgbackrest is a backup and restore system for Postgres.

Luca Ferrari

pq: The Pure Go(lang) Postgres Driver for database/sql— Now supports GSS authentication.

Blake Mizerany and contributors

#361 — June 24, 2020

Read on the Web

Postgres Weekly

'10' Things Postgres Could Improve— An in-progress four part series (part 2 here) covering topics like transaction ID issues and replication.

Shaun Thomas

📊 PDF: It's Time for a JSON/JSONB Great Unification— This dense, technical slidedeck is for you if you’re really into your JSON and you really care about Postgres’s JSON capabilities. It digs into JSON support in a forthcoming SQL standard and what future versions of Postgres can do to support this by unifying JSON and JSONB into one new data type.

Oleg Bartunov

We Help Customers Speed Up Postgres Queries By 1000x. Learn How— With pganalyze, companies like Atlassian are able to speed up their queries by orders of magnitude. In this ebook, we share our best practices for optimizing Postgres performance.

pganalyze sponsor

Postgres 13 Beta 1 Now in Amazon RDS Database Preview Environment— RDS lets you run a managed Postgres deployment on AWS and the Preview Environment lets you play with pre-release versions if you want to test your tooling, etc.

Amazon Web Services

How to Force a Table to Have Just One Row— Bruce demonstrates how to force a table to have at most one row by creating a unique expression index on a constant, with no column name references.

Bruce Momjian

EnterpriseDB Rebrands to EDB— EnterpriseDB are a popular company in the Postgres space so if you now see 'EDB' anywhere, well.. it's them :-)

EnterpriseDB / EDB

SQL Trickery: Hypothetical Aggregates— You can use this technique to determine the rank of a hypothetical value within an existing set.

Hans-Jürgen Schönig

tuned, Postgres, and You— tuned is a dynamic adaptive system tuning daemon (from Red Hat) that tunes system settings dynamically depending on usage. This post demonstrates creating a tuned profile for Postgres in particular.

Douglas J Hunley

ltree vs. WITH RECURSIVE— This is a follow up to Hans-Jürgen’s article on hierarchical queries.

Hans-Jürgen Schönig

Your Data Is Your Business— PGX is a full-service database consultancy focused on PostgreSQL data systems, on any platform or hosting environment.

PostgreSQL Experts, Inc. sponsor

Wrapping Db2 with Postgres— Db2 is a family of database products developed by IBM and focused on enterprise use. If you need to migrate Db2 data to Postgres, db2_fdw provides a way to do it.

Marcelo Diaz

Reasons to Migrate from Oracle to Postgres— This is truly a case of ‘preaching to the choir’ in this newsletter, but if you need any more reasons when chatting to others.. 😄

Kirk Roybal

pg_auto_failover: Automated Failover and High-Availability Extension— Monitors and manages automated failover for a Postgres cluster.

Citus Data

#362 — July 1, 2020

Web Version

Postgres Weekly

Using the system_stats Extension to Monitor Postgres— system_stats is a new extension (out of EDB, formerly EnterpriseDB) made up of several stored procedures you can use to monitor Postgres (or, more accurately, the server it’s running on).

Dave Page

Testing Extensions with GitHub Actions— pgxn-tools provides a Docker image with scripts to install and run any version of Postgres and this can be brought together with a CI/CD platform like GitHub Actions to test Postgres extensions on multiple Postgres versions.

David E. Wheeler

Whitepaper: BDR - Advanced Clustering & Scaling for PostgreSQL— Learn more about Advanced Clustering & Scaling for PostgreSQL with BDR including use cases, architectures, features & benefits. BDR provides multi-master replication with AlwaysOn availability, worldwide clusters, & Cloud Native deployments for PostgreSQL databases.

2ndQuadrant PostgreSQL Products sponsor

Calculating Differences Between Rows in SQL— If you have lots of rows that represent changing values (such as time series data, say) being able to calculate the differences between the values over time could be useful, here are some approaches.

Hans-Jürgen Schönig

PDF: Running Postgres in Kubernetes— A slidedeck based rundown of the top options for running Postgres in Kubernetes and some of the features contained within each.

Lukas Fittl

13 Tips to Improve Insert Performance on Postgres and TimescaleDB— The first five tips are related to Postgres in general, with the remainder being specific to TimescaleDB, a time series data extension for Postgres.

Michael Freedman

On Join Strategies and Performance— It’s worth understanding the three forms of join strategy Postgres uses when joining relations and how you can influence them with indexes.

Laurenz Albe

Can Case Comparison Be Controlled?— “Let’s go over some Postgres case-precision behaviors like the handling of character strings, identifiers, and keywords.” A handy primer to case comparison issues here.

Bruce Momjian

Timescale Cloud: Hosted, High-Performance PostgreSQL— Now available in 75+ regions across 3 major clouds. Analytics, performance, and scale. 👉 Start free, no credit card required.

Timescale sponsor

Postgres Anonymization on Demand— If your database stores personally identifiable information – and it probably does! – anonymization can help protect that data while still keeping the database useful. This articles digs into an approach for applying anonymization globally, covering all applications, with minimal code rewrites.

Achilleas Mantzios

Setting Up ora2pg for an Oracle to Postgres Migration— I suspect if you have access to an Oracle database, you might not be looking around for tutorials randomly, but if you are.. and if you need to migrate the data to Postgres.. ora2pg and this tutorial can help.

Yorvi Arias

Authenticating pgpool II with LDAP

Ahsan Hadi

A Story of Indexing JSONB Columns— A neat mix of story and tip when it comes to working with JSONB at scale.

Vsevolod Solovyov

#363 — July 8, 2020

Web Version

Postgres Weekly

Using the system_stats Extension to Monitor Postgres— system_stats is a new extension (out of EDB, formerly EnterpriseDB) made up of several stored procedures you can use to monitor Postgres (or, more accurately, the server it’s running on).

Dave Page

Generating Random Strings and Integers That Actually Aren’t— A look at creating “random-looking” coupon codes or similar strings in a PL/pgSQL function.

Josh Williams

We’ve Helped 100s of Customers Speed Up Postgres. Learn Our Best Practices— With pganalyze, companies like Robinhood are able to speed up their queries by orders of magnitude. In this ebook, we share our best practices for optimizing Postgres performance.

pganalyze sponsor

Bitnine Looks to Scale Postgres with AGE— AgensGraph is a transactional graph database built on Postgres but has now morphed into a new project called AGE and is being incubated by Apache.

Datanami

Amazon RDS Now Works on AWS Outposts— Let’s untangle this. RDS (Relational Database Service) is AWS’s service for running Postgres (and MySQL) in the cloud. AWS Outpost is a way to take the AWS experience into your own datacenter. So now you can use AWS’s cloud database service on AWS’s hardware but within your own physical environment. OK - got it!

AWS

Using CREATE DOMAIN to Create New Data Types— If you want to create more advanced server side data type constraints than the default data types can provide, you can create your own data types (domains).

Hans-Jürgen Schönig

Generated Columns vs. Triggers in Postgres 12— Postgres 12 introduced the idea of generated columns - columns which are generated from other columns’ values. This post compares the performance against using triggers to similar ends.

Emanuel Calvo and Anthony Sotolongo

More Postgres 13 Features: Deduplication in B-Tree Indexes— Postgres 13, due later this year, has a feature that can decrease the size of B-tree indexes by way of avoiding duplication. This post includes a contrived example to show off the difference.

Hamid Akhtar

Enterprise-Ready PostgreSQL | EDB Supercharges PostgreSQL— A portfolio of service offerings to help you on your PostgreSQL journey. Data migration. Hybrid cloud. High availability.

EDB sponsor

How Storj Migrated from Postgres to CockroachDB: Bulk Loading Performance

Jessica Grebenschikov

plpgsql_check: A PL/pgSQL Code Checker— A specialized tool for when you want to find any errors lurking in your PL/pgSQL functions. Support for functions with arguments of polymorphic types has just been added.

Pavel Stehule

Noisia: A 'Harmful Workload Generator' for Postgres— Creates things like deadlocks, transactions that do nothing and queries that produce on-disk temporary files. Why? For testing, stress-testing your setup, etc. Use carefully and with caution.

Lesovsky Alexey

#364 — July 15, 2020

Web Version

Postgres Weekly

PGX: Build Postgres Extensions with Rust— A framework for building Postgres extensions with Rust rather than the C or C++ you might be more used to. Why is this a good thing? Rust is becoming pretty popular in more cutting edge circles lately and the language’s safety features make it a compelling choice.

ZomboDB

JSONB: A Container of Types— Bruce reminds us that JSONB columns don’t just have to hold collections like arrays or maps/hashes, you can directly store strings, numbers and booleans too.

Bruce Momjian

Achieve High Availability with PostgreSQL [Whitepaper]— Critical business applications require availability of their backend database cluster. Discover how to setup & deploy enterprise-grade High Availability of up to four 9s. Learn the features and benefits in the Highly Available Postgres Clusters Whitepaper.

2ndQuadrant PostgreSQL Products sponsor

How to Find a Postgres Bug Born in 1997— The tale of discovering a bug hidden away inside Postgres’s documentation or circle type (depending on your point of view) for the past 23 years.

David Zhang

How to SCRAM with pgBouncer— Not only can you see how pgBouncer (the connection pooler) uses SCRAM authentication but also what SCRAM authentication is and how it works.

Jonathan S. Katz

What Are Postgres Templates?— We’ve covered this in one of our ‘tips of the week’ before, but if you’re unaware of what template databases are and how they’re used, this is a quick primer.

Angelico de los Reyes

Foreign Data Wrappers: Postgres's Secret Weapon?— Foreign data wrappers (often called FDWs, for short) lets you query remote databases directly from your PostgreSQL instance and here’s how that works alongside Splitgraph, a toolkit for building and working with “data images”.

Artjoms Iškovs (Splitgraph)

Distributed SQL Beats Polyglot Persistence for Building Microservices— Polyglot persistence is the silent killer of release agility; distributed SQL is the answer for today’s multicloud era.

YugabyteDB sponsor

PgTyped: Typesafe SQL in TypeScript— Use raw SQL in TypeScript with guaranteed type-safety.

Adel Salakh

pgsodium: A Postgres Extension for Using libsodium— libsodium is a crypto library for encryption, decryption, signatures, password hashing and more. We first linked when it was in alpha but it’s now 1.0 :-)

Michel Pelletier

#365 — July 22, 2020

Web Version

Postgres Weekly

Recreating YikYak with PostgreSQL— YikYak was an anonymous social network that used your location to show you posts 5km around you (we can’t see how that could possibly cause problems.. 😂) and you can recreate the underlying principle of its operation using Postgres’s geographical coordinate support.

Adam Fallon

pgwatch2 v1.8.0 Released— pgwatch2 is a popular Postgres monitoring tool and it now has Pgpool-II, Postgres 13 and TimescaleDB metrics storage support.

Kaarel Moppel

Real-Time Postgres Performance Monitoring— Collect out-of-the-box and custom Postgres metrics and correlate them with data from across your distributed infrastructure and applications. Try it free with Datadog for 14 days.

Datadog sponsor

Partitioning a Large Table Without a Long-Running Lock— You’ve got a huge table, you need to partition it, but you need that table to remain available to your app.. what do you do? Andrew Dunstan has a recipe to follow.

Andrew Dunstan

Getting More Performance for LIKE and ILIKE Statements— Our tip of the week (below) covers pattern matching and LIKE, but what if it’s not enough and you want to speed up performance? There are solutions!

Hans-Jürgen Schönig

GROUPING SETS and NULL Values— Our tip in issue 359 was about grouping sets, a way to perform grouping within a query that’s more complex than simple GROUP BY column can do. Bruce looks at the relationship between using GROUPING SETS and null values.

Bruce Momjian

Representing Dates, Times and Intervals in Postgres— Postgres comes with a bunch of built-in date and time related data types. But why should you use them over strings or integers? Here’s an overview looking at the why, plus advice on how to do so effectively.

RapidLoop

💻Live Webinar: 5 Ways to Improve Postgres Insert Performance— Join us on Aug 19 to learn five simple, yet powerful, techniques to supercharge your PostgreSQL ingest performance - plus live demos and pro tips for each tactic.

Timescale sponsor

Examining the Postgres Catalog with Python— Mark Ryan looks at how to get the most out of database metadata - by writing a program to automatically extract the information held within.

Towards Data Science

Generating & Managing Postgres Schema Migrations with Spring Boot JPA

Muhammad Haroon

▶  Using PEM to Improve Performance in Postgres: The Postgres Tuning Wizard and Performance Diagnostics— An hour-long webinar running through how to use EDB’s Postgres Enterprise Manager (PEM) GUI. (Note: the audio is a bit muffled generally.)

EDB

#366 — July 29, 2020

Web Version

Postgres Weekly

What is 'Fill Factor' and How Does It Affect Performance?— The ‘fill factor’ defines just how tightly packed table storage is. For example, when a row is updated, the ‘new’ row might end up in a different page of storage due to size limitations, whereas a less tightly packed table would allow the update to happen almost in situ.

Kaarel Moppel

A SQL Style Guide— We linked to this a couple of years ago but Bruce Momjian has reminded us of this handy SQL style guide to ensure legible and maintainable queries.

Simon Holywell

[Case Study] Successful Migration to PostreSQL by Global Gaming Company— Learn how 2ndQuadrant helped International Gaming Technology (IGT) successfully Migrate to PostgreSQL from an expensive and proprietary DBMS to deliver high quality gaming experiences worldwide, as well as how they've experienced zero outages since the switch.

2ndQuadrant PostgreSQL Services sponsor

Unicode Normalization in Postgres 13— If you know nothing about what ‘fully composed’ or ‘fully decomposed’ means when it comes to Unicode or what Postgres does when considering the equality of Unicode strings, this is a short and sweet introduction to some ideas you might not need to know in detail but should probably know about.

Peter Eisentraut

Avoiding Passwords in Log Files— “Because Postgres uses SQL queries to manage user accounts, including password assignment, it is possible for passwords to appear in the server logs..”

Bruce Momjian

A Safer Price Type in Postgres— Is it worth modelling prices by creating a domain/type rather than just sticking to cents in a number column? Vados experiments.

Vados

Postgres 13's EXPLAIN Now Includes WAL Information— You’ll be able to see how many WAL records are created and similar info.

Luca Ferrari

Monitor and Visualize Postgres Database Metrics End-To-End— Monitor PostgreSQL performance end-to-end with OOTB, customizable, drag-and-drag dashboards in Datadog. Try it free.

Datadog sponsor

Installing TimescaleDB on macOS with Postgres.app— TimescaleDB is a popular time series data Postgres extension and Postgres.app is an equally popular way to run Postgres on the Mac. This tutorial brings the two together.

Prathamesh Sonpatki

Using Custom Types with Citus and Postgres, From Popular Hack to Transparent Feature— How custom types work with Citus and how user-defined PostgreSQL types are automatically propagated to all the nodes in a Citus cluster.

Nils Dijk

▶  Postgres Performance Tuning and Optimization— An hour-long talk that focuses on tuning configuration settings (both in Postgres and on Linux generally) and what things they each affect.

Ibrar Ahmed (Percona)

What It Was Like Speaking Online at EuroPython 2020— A speaker shares a little of the experience of giving a talk at an online conference.

Paolo Melchiorre

GoodJob: A New Postgres-Based, Multithreaded, Background Job System for Ruby on Rails— Ben calls GoodJob a “second-generation” backend because it focuses on compatibility with ActiveJob. It’s suited for use cases queuing fewer than 1 million jobs/day.

Ben Sheldon

#367 — August 5, 2020

Web Version

Postgres Weekly

Postgres 13 Incompatibilities To Be Be Aware Of— Postgres 13 will be in general release soon (it’s currently in beta) and while Postgres tends to be reasonably backward compatible over time, there are some changes worth being aware of before you make a migration, as seen here.

Ibrar Ahmed

How to Securely Authenticate with SCRAM in Postgres 13— A tutorial on setting up SCRAM-based password authentication. Interesting not just as a tutorial but also a primer on what SCRAM is and why channel binding improves security overall.

Jeff Davis

How to Tune PostgreSQL GUC Parameters— This article discusses GUC parameters that can be used for PostgreSQL tuning and how to configure them for improved performance. Take a look.

EDB sponsor

Measuring the Difference Between Dates— “What is the difference between two dates? You would think there was one answer, but there isn’t.” An interesting look at how there are multiple ways to get the difference between two dates or timestamps in Postgres and how they can return slightly different answers.

Bruce Momjian

effective_cache_size: A Practical Example— Some insights and a practical example of what the effective_cache_size setting does.

Hans-Jürgen Schönig

Announcing pgBackRest for Azure: Fast, Reliable Postgres Backups— Backups are a key staple of running any database. pgBackRest aims to be a fast, reliable, easy-to-use backup and restore solution with the ability to seamlessly scale to the largest databases, and now there’s official support for Azure.

Craig Kerstiens

Connection Pooling in Pgpool-II— Pgpool-II is a connection pooling tool for Postgres – this post looks at the basics of its operation.

B Peng

Best-Practices on How to Speed Up Your Postgres Queries. Free eBook— Companies like Robinhood and Atlassian are able to speed up their queries by orders of magnitude. This eBook shares our best practices for optimizing Postgres performance.

pganalyze sponsor

▶  How to use SSL in Postgres The Right Way: Encrypt Your Data in Transit

Kirill Shirinkin

Continuous Backups using WAL-G— WAL-G lets you manage the archiving of Postgres data and backups and can also let you restore a database to its state at a particular moment in time.

Angelico de los Reyes

Computing INTERVAL Values— Bruce seems to be on a date and time streak this week, and here he looks at some quirks of date/time intervals.

Bruce Momjian

▶  High Performance HTAP with Postgres & Hyperscale (Citus)

Marco Slot and Claire Giordano

#368 — August 12, 2020

Web Version

Postgres Weekly

Postgres Pagination Approaches— Most webapp users don’t want to see a list of thousands of items at once, so pagination is often adopted as a way to ‘page’ through a larger group of items in more manageable chunks. There are several ways to achieve this with Postgres with various tradeoffs.

Bruce Momjian

A pg_stat_statements Troubleshooting 'Hack'— If you want to get a useful ‘trick up your sleeve’ for troubleshooting with pg_stat_statements, you might enjoy Kaarel’s story.

Kaarel Moppel

[Whitepaper] AlwaysOn Postgres— Discover how to achieve AlwaysOn availability in PostgreSQL using BDR - groundbreaking technology from 2ndQuadrant. AlwaysOn guarantees up to six 9s of availability for worldwide PostgreSQL database clusters running mission critical business applications.

2ndQuadrant PostgreSQL Products sponsor

Use NATURAL FULL JOIN to Compare Two Tables in SQL— “NATURAL JOIN is syntax sugar for joining using all the shared column names of the two tables, and the FULL JOIN makes sure we can retrieve also the columns that are not matched by the join predicate.”

Lukas Eder

Backtraces in Postgres 13— Postgres 13 introduces a simple but useful capability to log a stack trace into the server logs when an error is reported.

Amit Khandekar

Postgres High Availability: Considerations and Candidates— The first in a promised series of posts looking at various ways to introduce high availability concepts to Postgres. This time, RepMgr, Patroni, PAF, and PgPool-II are mentioned.

Hamid Akhtar

eBook: The Most Important Events to Monitor in Your Postgres Logs— In this eBook, we are looking at the Top 6 Postgres log events for monitoring query performance and preventing downtime.

pganalyze sponsor

On Partitioning Improvements in Postgres 13

Ahsan Hadi

Avoiding the Pitfalls of BRIN Indexes

John Porvaznik

A Crash Course on Postgres for R Users— R is a popular statistical computing environment and language, so you might find it useful to use Postgres from it.

Pachá

Working with a JSONB Array of Objects in Postgres

Rob Tomlin

Going Beyond jsonb? A Generalized, Unstructured Data Type for Postgres— Álvaro wonders if regular JSON support is enough and if there should be something beyond it in Postgres – a generalized unstructured data type, if you will.

Álvaro Hernández

plpgsql_check Now Supports Tracing— plpgsql_check is a linter for Pl/pgSQL code.

Pavel Stěhule

pgagroal 0.8.0: A High Performance Postgres Connection Pool— 0.8 brings failover and systemd support.

Red Hat Inc.

#369 — August 19, 2020

Web Version

Postgres Weekly

Postgraphile: Quickly Get a GraphQL API for Your Postgres Database— Work on v5 of this popular Node.js tool is underway, but meanwhile we get a 4.8 release which supports ‘enum tables’ and all of Postgres’s built-in geometric types. If you want to offer a Postgres database up over a GraphQL-based API, it’s worth checking out.

Graphile

Postgres 12.4, 11.9, 10.14, 9.6.19, 9.5.23, and 13 Beta 3 Released— A flurry of Postgres releases to fix a variety of bugs but, more importantly, two security vulnerabilities (in versions 9.5-12 only). We’re also reminded that Postgres 9.5 will stop receiving fixes after February 2021.

PostgreSQL Global Development Group

Real-Time Postgres Performance Monitoring— Collect out-of-the-box and custom PostgreSQL metrics and correlate them with data from your distributed infrastructure, applications, and logs. Gain real-time performance insights and set intelligent alerts with Datadog. Start a free trial.

Datadog sponsor

Building a Recommendation Engine Inside Postgres with Python and Pandas— Learn how you can leverage Python and Pandas from directly inside Postgres to build your own recommendation engine.

Craig Kerstiens

An Introduction to Recursive Queries— Recursive queries (built using WITH) let you do interesting things with SQL and aren’t too difficult to understand, particularly with the simple practical examples given here.

Laurenz Albe

SQLite 3.33.0 Released (With a Postgres-Inspired Feature)— We don't often cover SQLite in this newsletter, but when the world’s most heavily used database engine gains support for UPDATE FROM and aims to be compatible with Postgres's implementation, why not? 😄

SQLite Team

Timescale Cloud Extends to 76 Regions Across 3 Clouds— TimescaleDB essentially extends Postgres with significant time-series data functionality and Timescale Cloud is the creator’s commercial ‘TimescaleDB in the cloud’ service.

Timescale

Envoy 1.15 Introduces a New Postgres Extension with Monitoring Support— Envoy is a popular service proxy for cloud applications, and it can now ‘speak’ Postgres’ wire protocol! This post digs into why this plugin was developed, what functionality it currently implements, and what the roadmap looks like for future releases.

CNCF

Running Multiple PgBouncer Instances with systemd— PgBouncer is a connection pooler that runs as a single process, but with a little work you can run multiple instances for different virtual hosts.

Peter Eisentraut

A Look at Keyset Pagination— Last week, Bruce covered pagination approaches in general and is now focusing on one technique, using LIMIT and OFFSET alongside WHERE. He’s also dug into how it works in practice while items are being inserted or deleted.

Bruce Momjian

CockroachDB: Scalable, Distributed PostgreSQL— Never shard a Postgres instance again. Meet CockroachDB, the Distributed SQL database that’s naturally resilient and provides out-of-the-box scalability.

Cockroach Labs sponsor

Iterators in Postgres with Lateral Joins— “When the keyword LATERAL is added to your join the output will now apply the right hand part of the join to every record in the left part of the join.”

Steve Pousty

How We Used Postgres Extended Statistics to Achieve a 3000x Speedup— Big numbers like that always make me put my cynical eyeglasses on, but in this case it’s a neat technical tale of encouraging Postgres to do the right thing.

Jared Rulison

PostGIS 3.0.2, 2.5.5, and 2.4.9 Released— Minor bug fix and performance enhancement releases for the popular geospatial extension.

PostGIS Developers

#370 — August 26, 2020

Web Version

Postgres Weekly

Why Postgres 13 Will Be A 'Lucky' Release— See why the author thinks that why 13, often maligned as an ‘unlucky’ number, will be lucky for Postgres it adds features like incremental sort, parallel vacuum, and improves performance of B-tree indexes.

Jonathan S. Katz

synchronous_commit Options and Synchronous Standby Replication— synchronous_commit lets you customize when a transaction commit can be acknowledged back to the client as successful which has implications in replicated systems.

Jobin Augustine

[Whitepaper] Best Practices to Harden your Database— Learn the fundamental concepts of Security for PostgreSQL Databases & develop a deep understanding of industry best practices. Discover how to secure your Postgres cluster & keep your data safe. Have confidence your database is adequately protected from attacks.

2ndQuadrant Whitepapers sponsor

Improvements for Handling Large Number of Connections Coming to Postgres 14— Note that this is about Postgres 14 so you’re going to be waiting a while (as 13 isn’t even out yet!) but handling large numbers of connections is a perennial issue for many so these links to some discussion threads may be of interest.

Hubert depesz Lubaczewski

Query 40K+ Datasets Through a 'Data Delivery Network' With Any Postgres Client— Splitgraph offers what they call a ‘data delivery network’ that acts like a distributed SQL caching proxy compatible with the Postgres wire protocol. In this demo it lets you query over 40,000 datasets with SQL using the psql or other Postgres clients you know and love.

Splitgraph

Calculating Standard Deviation with SQL— As is often the case, Postgres offers a function for this task: stddev().

Bruce Momjian

A Look at TLS Related Updates Coming in Postgres 13— Supporting a minimum of TLS 1.2, Postgres 13 brings a few improvements when it comes to secure connections including support for channel binding.

Cary Huang

How to Setup Postgres on an IPv6 Enabled Network— If you’re confident with IPv6 already, this goes into more depth than you’ll need, but if not.. you may find some useful tips here.

David Zhang

Faster CI/CD for All Your Software Projects - Try Buildkite ✅— See how Shopify scaled from 300 to 1800 engineers while keeping their build times under 5 minutes.

Buildkite sponsor

pg-costop: Vector Arithmetic and Weighted, Variably Randomized Cosine Similarity Search— Mathemetically this is a bit beyond me but it’s a set of PL/pgSQL functions for working with vectors to calculate cosine proximity / similarity rankings.

turbo

pg-shortkey: YouTube-like Short IDs as Postgres Primary Keys— “This installs a trigger and type which allow you to use YouTube-like short IDs (e.g. 1TNhBqYo-6Q) as Postgres Primary Keys. Just like YouTube IDs, ‘SHORTKEY’ IDs are fixed length and URL-safe.”

turbo

QuestDB: A Performance-Focused 'NewSQL' Time-Series Database System— A relational database (that isn’t built on Postgres but does support its wire protocol) focused on fast time-series data processing. Built in Java.

QuestDB Limited

Pgpool-II 4.1.3, 4.0.10, 3.7.15, 3.6.22 and 3.5.26 Released— Bug fixes all round for all maintained versions of the popular connection pooling system.

PgPool Development Group

#371 — September 2, 2020

Web Version

Postgres Weekly

An Overview of Trusted Extensions in PostgreSQL 13— Postgres supports extensions as a way to easily introduce new features and functions dynamically, but in Postgres 13 there’ll be the concept of a trusted extension which will allow any user with CREATE privileges on the current database to bring in an extension without superuser privileges.

Nidhi Bansal

Tuning Autovacuum for Postgres Databases— Most of the time you don’t need to tweak the settings for autovacuum as Postgres will generally do the right thing, but sometimes the default configuration just isn’t enough and this article will help you with some approaches for tuning things.

Laurenz Albe

Best-Practices on How to Speed Up Your Postgres Queries. Free eBook— Companies like Robinhood and Atlassian are able to speed up their queries by orders of magnitude. This eBook shares our best practices for optimizing Postgres performance.

pganalyze sponsor

Three Easy Things To Remember About Postgres Indexes— Basic ideas, but worth reviewing if you’re not an index expert. Indexes can speed things up beyond simple queries, indexes aren’t always used even if you think they should be, and indexes add maintenance and storage overhead you need to be prepared to manage.

Kat Batuigas

Parallelism Comes to VACUUM in Postgres 13— One of the noteworthy new features in Postgres 13 will be the ability to speed up the execution time of index vacuuming. Masahiko provides a quick example of the benefits.

Masahiko Sawada

How JSON Aggregate Functions Are Pretty Cool— Rather than extract data with SQL queries and then convert it into JSON, you could get Postgres to do the heavy lifting instead and here’s how.

Knut Hühne

The Need for External Compression Methods in Postgres— If you could specify different compression systems for different tables or columns, based upon the most appropriate one for the type of data stored, that would be pretty neat.. and it seems the Postgres team are beginning to discuss such a potential feature in Postgres.

Amit Khandekar

Using Envoy Proxy’s New PostgreSQL & TCP Filters to Collect SQL Statistics— The new PostgreSQL filter from Envoy Proxy makes it easy for developers and ops engineers to collect SQL statistics.

YugabyteDB sponsor

Benchmarking Checkpointing in Postgres— Evaluating checkpointing with varying log sizes from 1GB to 100GB.

Vadim Tkachenko

Full-Text Search Battle: Postgres vs Elasticsearch— Less a battle and more a basic comparison of techniques with a simple benchmark, but may be of interest.

Rocky Warren

Pitfalls and Quirks of Logical Replication in Postgres 12— “I’d like to share some thoughts after setting up a logical replication in a large-ish (one terabyte) production database.”

Elephant Tamer

#372 — September 9, 2020

Web Version

Postgres Weekly

A Look at B-Tree Index Deduplication in Postgres 13— B-tree indexes are the default type of index created in Postgres so any changes to their operation is likely to have a lot of knock-on effects. Deduplicating these indexes, as possible in the forthcoming Postgres 13, will help keep these indexes smaller and has performance implications (most likely lower I/O usage at a cost of minor CPU increase, but with higher overall performance in most cases).

Ryan Lambert

How to Get the Best Out of Postgres Logs— Postgres’s logging system is very tunable and there are lots of parameters to fiddle with. This post covers some basic practices for getting the most out of a Postgres server’s logs and what you can tweak.

Sadequl Hussain

[Whitepaper] Business Case for Professional Support— Learn the importance of Professional Support for your mission-critical PostgreSQL systems & how it can benefit your company. Discover how it increases database performance, helps scale, distributes data, reduces costs, saves you from known pitfalls, and more.

2ndQuadrant Services sponsor

What’s New in the Citus 9.4 Extension to Postgres— Citus transforms Postgres into a distributed database, distributing your data and your SQL queries across multiple nodes. v9.4 improves EXPLAIN ANALYZE, has some performance and safety improvements, and can now calculate percentiles at scale using the t-digest extension.

Marco Slot (Citus Data)

Generating a Normal Distribution in SQL— Postgres’s tablefunc extension provides a variety of functions that return tables, including sets of normally distributed random values.

Hans-Jürgen Schönig

PostGIS and the Geography Type— The PostGIS geography type is a geospatial type that understands coordinates as spherical coordinates (in latitude and longitude) and here’s a basic introduction to them (and one of the reasons to use PostGIS too, really).

Paul Ramsey

Best-Practices on How to Speed Up Your Postgres Queries. Free eBook— Companies like Robinhood and Atlassian are able to speed up their queries by orders of magnitude. This eBook shares our best practices for optimizing Postgres performance.

pganalyze sponsor

Tuning Postgres on ZFS— “The main reason to use ZFS instead of ext4/xfs is compression. With reasonable configuration you can achieve 3-5x compression ratio using LZ4. That means that LZ4 compresses 1 terabyte of data down to ~300 gigabytes.”

Uptrace

Building Microservices with Deno, Reno, and Postgres— Deno is a server-side JavaScript runtime built on top of V8 (a bit like Node, but not) and Reno is a routing library for Deno apps.

James Wright

Mining for Logic Bugs in the Citus Extension to Postgres with SQLancer— One of those things you’re unlikely to need to do, but it’s nice to know how such problems are approached. SQLancer is a tool we’ve linked to before that helps you detect logic-related bugs in database systems.

Nazli Ugur Koyluoglu

#373 — September 16, 2020

Web Version

Postgres Weekly

How (and Why) GitLab Upgraded from Postgres 9.6 to 11— Some of the GitLab team explain the precise maintenance process they took to execute a major version upgrade of Postgres on their systems. And, even better, they recorded the whole 2 hour process so you can see how it was actually conducted! It doesn’t make for exciting viewing, but it’s a very uncommon look behind the ops curtain at a major company.

GitLab

7 Best Practice Tips for Bulk Data Loading— Anyone who’s managed a database has needed to import a large quantity of data at some point, and anyone who’s tried this has rapidly discovered there are both fast and slow ways to do it 😉 This post looks at some some best practices for bulk importing data into Postgres databases.

Sadequl Hussain

Quickly Identify Slow-Running Postgres Queries in Datadog— Improve PostgreSQL performance in real-time with actionable alerts on slow-running queries and bottlenecks. Start proactively monitoring your PostgreSQL databases with a Datadog free trial.

Datadog sponsor

A Few Hidden Gems of Postgres 13— With all the blog posts we’re seeing each week, it feels like Postgres 13 is the most anticipated release ever 😄 Here, Jonathan reflects on a handful of ‘hidden gems’ that’ll be turning up in the release.

Jonathan S. Katz

Picking Between Joins or Subqueries in Postgres: Lessons Learned— The author was advised “Try to use joins rather than subqueries.” Here, he works through an example of why and where the benefits are.

Steve Pousty

Index Improvements Coming to Postgres 13— We’ve linked to numerous articles over the past few months about how indexes will be improved in Postgres 13 but this rounds up six of the biggest benefits in one place complete with code examples.

Ibrar Ahmed and Jobin Augustine

Which Partition Contains a Specific Row?— If you’re using hash partitioning, can you determine which partition contains a given row? Yes, you can.

Gabriele Bartolini

Using Postgres to Offload Real-Time Reporting and Analytics from MongoDB— Weighs up the advantages and disadvantages of moving read-heavy analytics off a primary MongoDB database to PostgreSQL.

Shawn Adams

Faster CI/CD for All Your Software Projects - Try Buildkite ✅— See how Shopify scaled from 300 to 1800 engineers while keeping their build times under 5 minutes.

Buildkite sponsor

How To Install and Run Postgres using Docker

J Shree

pgtools: A Visual Way to Monitor Database Events in Real Time— A very new project built in Python and using Vue.js on the client side.

Lukas Loeffler

allballs? A Rather Odd Time Keyword in Postgres..— There's a ‘childish behavior ahead’ warning on this one.. 😆 but via this tweet I learnt that Postgres (still) supports allballs as a keyword to mean 00:00:00. We’ll let you guess why, or you can read this explanation on the mailing list. Amazingly, as of the 13 betas, it's still there.

Postgres GitHub Repo

#374 — September 23, 2020

Web Version

Postgres Weekly

A Battleship Game, Implemented with Postgres— See SQL taken to the next level with a working game running within Postgres, complete with a creative way of taking player input.

Firemoon777

PostgreSQL 13 Release Candidate 1 Released— We link to things about Postgres 13 all the time, if you hadn’t noticed, and it’s shaping up to be a huge release – so it’s great to see it close to completion with this first RC. The beta release notes cover the essentials, but we’ll do a full roundup of features at the final release.

PostgreSQL Global Development Group

Highway to Zero Downtime PostgreSQL Upgrades— Get a comprehensive walk-through of how to perform a "near" zero downtime upgrade using pglogical in this free webinar. Learn how logical decoding presents a whole new world of opportunities for upgrades that require a very small amount of downtime.

2ndQuadrant PostgreSQL Webinars sponsor

Crunchy Bridge: The Newest 'Postgres As A Service'— Crunchy Data is the latest company to get on the ‘Postgres as a managed service’ bandwagon with Crunchy Bridge which is available on AWS and Azure (and supports migration and replication between the two).

Craig Kerstiens (Crunchy Data)

Diary of an Engineer: Delivering 45x Faster Percentiles using Postgres, Citus, and t-digest— Nils had a problem to solve for a customer but couldn’t meet their SLA of 30 seconds and didn’t have the customer’s data to experiment with.. nonetheless, he found a creative way to estimate which types of percentile calculations would meet their SLA and used t-digest to do it.

Nils Dijk (Microsoft)

Postgres 13's LIMIT ... WITH TIES— WITH TIES is a new SQL standard feature being implemented in Postgres that causes LIMIT (or FETCH FIRST) clauses to not just cut off at a specified limit but to also include rows with values that tie with the final one(s).

Álvaro Herrera

Lessons Learned from Running Postgres 13: Better Performance, Monitoring & More— We took a look at smaller indexes with B-Tree Deduplication, Parallel VACUUM, improved WAL Usage Stats, and more.

Pganalyze sponsor

How 'HOT' Updates Yield Better Performance— An introduction to a feature included first with Postgres 8.3 but which, allegedly, are not properly covered in the docs. HOT updates (Heap Only Tuple) occur behind the scenes and improve performance in certain situations where lots of UPDATEs occur.

Laurenz Albe

AWS Aurora Postgres Versions 'Vanished' for Days, Customers Claim— Greg Clough, a software engineer who uses AWS, noticed that several Postgres versions on AWS Aurora ‘vanished’ last week (in the sense they couldn’t be deployed – existing datbases didn’t disappear). Most now appear to be back, but it’s a curious story.

The Register

📄  Postgres and the Artificial Intelligence Landscape— It’s just slides for now (though a talk was given) but Bruce’s slides often provide value even on their own.

Bruce Momjian

Exploring PL/Python: Turning Postgres Table Data Into a NumPy Array

Kat Batuigas

Why RudderStack Used Postgres Over Apache Kafka for a Streaming Engine— Kafka was a natural fit for what RudderStack, a data platform, does, but they found enough negatives about it to build their own queueing system on top of Postgres instead.

RudderStack

#375

Web Version

Postgres Weekly

Postgres 13 Released— Just shy of a year after Postgres 12 was released, 13 is here focused on evolutionary steps forward for our favorite database. The release notes provide the required laundry list of new and tweaked features, but we'll cover some of the bigger things here with links to relevant articles:

• Deduplication in B-tree indexes
• Parallelism for VACUUM
• Trusted extensions
• Use SELECT gen_random_uuid() without needing an extension
• CREATE STATISTICS can now help with plans with IN/ANY operations
• postgres_fdw now supports certificate authentication
• TLS improvements generally
• ..and other hidden gems!

PostgreSQL Global Development Group

🏆 Top 5 PostgreSQL Extensions— We round up a few of our & TimescaleDB community members’ must-have PG extensions, complete with why we 💛 them, install instructions, sample queries & pro tips to get you started.

Timescale sponsor

A Quick Look at Postgres 13 RC1's Query Performance— These benchmarks were done before the final Postgres 13 release (and took three days to run!) but should continue to stand up. The results are broadly (but not unanimously) positive for the new release versus 12.4.

Kaarel Moppel

Operating Postgres at Scale: Saving Space (Basically) for Free— With over 100 terabytes of data in play on their Postgres clusters, any efficiency gains possible are a big win for Braintree. Here’s the story of how they saved around 10% of disk space with ‘very little effort’ by re-ordering table columns.

James Coleman (Braintree)

When to Deploy or Upgrade to a New Major Postgres Release— It’s rarely practical to upgrade to the latest and greatest version of something (cough, cough, Postgres 13) as soon as it comes out but how should you approach the inevitable future upgrade?

Andrew Dunstan

Simple Anomaly Detection Using Plain SQL— “Using some high school level statistics and a fair knowledge of SQL, I implemented a simple anomaly detection system that works.”

Haki Benita

Using Postgres and pgRouting To Explore The Smooth Waves of Yacht Rock— pgRouting is a powerful geospatial routing extension for Postgres and PostGIS usually used for pathfinding/mapping/direction applications. This post creatively attempts to use it to find the most influential Yacht rock artist.

John Porvaznik

We Look at What’s New in Postgres 13: Better Performance, Monitoring & More— Smaller indexes with B-Tree Deduplication, Extended Statis Improvements, Parallel VACUUM, improved WAL Usage Stats, and more.

Pganalyze sponsor

Debugging PL/pgSQL with GET STACKED DIAGNOSTICS— GET STACKED DIAGNOSTICS makes debugging PL/pgSQL code a lot easier but isn’t super well known. This post will show what it does and how you can make use of it.

Hans-Jürgen Schönig

How to Configure SCRAM and MD5 Authentication in Pgpool-II— Pgpool-II is a popular Postgres connection pooler.

Bo Peng

DuckDB: An Embeddable SQL OLAP Database System— Built in C++, DuckDB bills itself (geddit?) as ‘SQLite for Analytics’ and has bindings for C/C++, Python, and R. We’ve mentioned this in our Database Weekly several times, but it’s just struck me that it may complement use cases you have for Postgres too.

CWI Database Architectures Group

#376

Web Version

Postgres Weekly

sqlbench: Measures and Compares The Execution Time of SQL Queries— "The main use case is benchmarking simple CPU-bound query variants against each other during local development." Postgres-only for now but pull requests for other databases are welcomed. Written in Go.

Felix Geisendörfer

Using CTEs to Do a Binary Search of Large Tables with Non-Indexed Correlated Data— An interesting situation here. A query had to be optimized but no changes to the underlying schema were allowed at all, including the generation of indexes. Some clever thinking was required!

David Christensen

Managed PostgreSQL Hosting on GCP Now Available at ScaleGrid— ScaleGrid is the easiest way to manage PostgreSQL on Google Cloud Platform. Features such as Superuser Access, Custom Extensions and Slow Query Analysis. Bring Your Own Cloud (BYOC) to reduce hosting costs, or use our dedicated hosting. Try for free.

ScaleGrid sponsor

EDB Acquires 2ndQuadrant— “Company buys other company” is rarely an exciting story but EDB (formerly EnterpriseDB) and 2ndQuadrant are big in the Postgres space and with this acquisition EDB has pledged to “push Postgres even further.”

Ed Boyajian (EDB)

What is Flexible Server in Azure Database for PostgreSQL?— Flexible Server is a new deployment option for Azure Database for PostgreSQL that aims to provide more fine-grained control over scaling and cost optimization (think things like stop/start, burstable compute, setting custom maintenance windows).

Sunil Agarwal (Microsoft)

zheap: Reinvented Postgres Storage for Better Bloat— Table ‘bloat’ is when a table or indexes grow in size without the actual underlying data reflecting this. zheap is a way to keep such bloat under control with a storage engine capable of running UPDATE-intense workloads more efficiently.

Hans-Jürgen Schönig

Some VACUUM and ANALYZE Best Practice Tips— A look at two important and commonly used features which can cause confusion.. which means a few best practices could come in very useful.

Sadequl Hussain

Why You Might Migrate Your Heroku Postgres to AWS RDS— If you’re using Heroku, their Postgres offering is pretty fantastic, but here are some arguments for not using it which, if you’re in Europe or worried about GDPR, may even have regulatory consequences.

Paweł Urbanek opinion

dropdb --force, a New Postgres 13 Feature— Super keen to drop a database ASAP even if clients are connected to it? Now you can 😂 You can also use FORCE with DROP DATABASE.

Ibrar Ahmed

Best-Practices on How to Speed Up Your Postgres Queries. Free eBook— We share our learnings from helping companies like Atlassian, Robinhood, and others speed up their queries.

pganalyze sponsor

How to Fix Postgres Performance Issues with PG Extras— An introduction to a tool you can use to spot Postgres related issues in Node, Elixir and Ruby contexts. node-postgres-extras is the repo you want.

Paweł Urbanek

Postgres 13's Features Distilled— A quick, easily read roundup of Postgres 13’s main features if you didn’t catch up with our issue last week.

Kovid Rathee

dbcrossbar: Move Large Datasets Between Different Databases and Formats— Copy tabular data between databases, CSV files and cloud storage. Written in Rust.

Faraday, Inc.

Arctype: A Desktop SQL Client for Postgres and MySQL— When I saw someone tweet that this is the ‘most beautiful client I’ve used’, I had to take a look at this. spreadsheet-style data editing. Beekeeper Studio is another worth checking out that we might write more about soon.

Arctype

#377 — October 14, 2020

Web Version

Postgres Weekly

Optimizing Storage with pg_squeeze— A look at an extension that offers periodic, automatic and transparent fixing of tables that exceed a ‘bloat threshold’. "No more need for VACUUM FULL– pg_squeeze has it all."

Hans-Jürgen Schönig

Analyzing the Limits of Connection Scalability— Postgres can handle thousands of connections but high numbers of concurrent clients isn’t one of its strong points. This post looks at how things could be improved long-term and where the problems creep in.

Andres Freund

Crunchy Bridge: More Postgres Power, Less Administration— A better option for your Postgres hosting. With a rich set of extensions like PL/Python + SciPy/NumPy/Pandas and more on demand, Crunchy Bridge is made for modern apps. Built by Postgres experts and the team who brought you Heroku Postgres.

Crunchy Data sponsor

▶  The State of (Full) Text Search in Postgres 12— The audio isn’t great but nonetheless this is a great primer to different text search techniques in Postgres starting from the absolute basics (e.g. LIKE and regexp_match) and moving on to Postgres’s full-fat FTS features.

Jimmy Angelakos

On the Community Impact of EDB's 2ndQuadrant Purchase— Last week we mentioned that EDB (formerly EnterpriseDB) acquired 2ndQuadrant– this makes it a bit of a heavyweight in the Postgres space, and Bruce ponders the risks especially as over half the Postgres core team (including Bruce) is now from one parent company (this then turned into a Hacker News discussion.)

Bruce Momjian

Making 2 Million Ancient Usenet Posts Available with Postgres and Python— Usenet was a fascinating distributed discussion system whose heyday was in the 80s and 90s and there are various archives of posts made to it (such as someone asking what Postgres is in 1989!).

Jozef Jarosciak

Scaling Row Level Security to Group Roles— Row level security lets you grant privileges to selected rows for selected users but what about groups of users or ‘roles’? Enter pg_has_role.

Elephant Tamer

Postgres Monitoring for Developers: The DBA Fundamentals— A look at several stats you might keep an eye on when managing a Postgres database. pgmonitor is the sort of tool you can use to track these things.

Jonathan S. Katz

Measuring the Memory Overhead of a Postgres Connection— This relates directly to the connection scalability post featured above but may be of separate interest.

Andres Freund

Some Query Caching and Load Balancing Tools— Summarizes some query caching and load balancing options available to use with Postgres, including pgpool-II, Apache Ingest, Heimdall Data, HAProxy, and Bucardo.

Viorel Tabara

What Is This Mythical Zero Downtime Database Migration to the Cloud?— Join us for a CIO led panel discussion on the issues and risks and the technology available to streamline the experience.

ScaleArc sponsor

Multitenancy with Postgres Schemas: Key Concepts Explained— If you have an interest in this topic, their What Surprised Us with Postgres-Schema Multitenancy article may also be of use.

Tomasz Wróbel

What's New in pg_auto_failover 1.4— pg_auto_failover is an open source extension that manages automated failover for Postgres clusters. This release was sent to me with the note: “In this new 1.4 release Dimitri has added multiple standby support, which is a really big deal.”

Dimitri Fontaine

#378 — October 21, 2020

Web Version

Postgres Weekly

The Surprising Impact of Medium-Size Texts on Performance— You’ve got your small text (usernames, emails), large text (entire documents), and your ‘medium’ text (comments, descriptions).. While TOAST brings efficiencies to storing larger documents, medium-sized columns can make rows very wide and affect performance disproportionately. A rather neat deep dive complete with examples.

Haki Benita

30 Years of Continuous Postgres Development with Bruce Momjian— It’s a brief interview, but Bruce shares a few tidbits about the Postgres development process, quality, and how features make it in.

Scott Grant

[Whitepaper] Achieving High Availability with PostgreSQL— Critical business applications require availability of their backend database cluster. Discover how to setup & deploy enterprise-grade High Availability of up to four 9s. Learn the features and benefits in the Highly Available Postgres Clusters Whitepaper.

2ndQuadrant PostgreSQL Products sponsor

PostgresConf.CN and PGConf.Asia 2020 Taking Place Online (November 17-20)
China PostgreSQL Association

Tuning Your Postgres Database for High Write Loads— If you set up a database as a proof of concept or to fit an initially small workload, problems can occur once things scale up including warnings like “checkpoints are occurring too frequently”. What’s going on and what can be tweaked?

Tom Swartz

How Postgres Stores 'null' Values— Good news – you almost certainly don’t need to know this, but intrigue always gets the better of me..

Movead Li

Multicorn: Python Powered Foreign Data Wrappers— Mulitcorn is a FDW but which comes with a variety of customizable subwrappers (for things like SQLAlchemy, RSS, working with the file system, or SQLite) that you can adjust.. or write your own.

Kozea

pgFormatter: A Postgres SQL Syntax Beautifier— ..that can work from the terminal or on a Web server via CGI. Demo here.

Gilles Darold

Prisma’s Data Guide: PostgreSQL— Introductory PostgreSQL tutorials: learn how to configure and use Postgres to take advantage of its best features.

Prisma sponsor

Logical Replication Upgrade in Postgres
James Chanco Jr.

Working with Data Consistency Issues in Logical Replication
Elephant Tamer

How to Upgrade PostgreSQL 11 to PostgreSQL 12 with Zero Downtime— using logical replication.