Philip

Backing up and Restoring Postgres using pg_basebackup

by

There are several great tools available that handle backing up and managing the backups of your PostgresQL database. It is really important to understand the underlying process that these tools use though as well as the standard postgres commands are that you would need to run in case you ever need to do it manually.

There are 2 types of backups that you can take in Posgres, logical and physical. Logical backups are in the form if the SQL statements necessary to recreate the database (not necessarily in a human readable form). The 2 tools to take logical backups are pg_dump and pg_dumpall.

Read more

What is the pg_clog and the clog

by

There are several directories named log in a Postgres installation.

You have pg_xlog, pg_log and pg_clog.

These are all important but I’ll talk about the others another time.

Pg_clog is the commit log. It is generally a small folder that you should never have a reason to look at. (note that from version 10 of postgres the pg_clog directory is being renamed to pg_xact I will continue to refer to it as pg_clog in this document but the functioning of both is the same)

Importantly, you can never delete anything from that directory. If you do your database will become unusable and you will need to recreate it from a backup.

Read more

Setting up a Postgres test cluster in vagrant

by

Most of the time its fine to test things out on your local machine and local installation of Postgres but often, you want more flexibility, the ability to quickly reset to a known starting point and the ability to try out different and more complex server architectures.

That is where Vagrant comes in. I use it to quickly set up Postgres clusters at different versions and different configurations.

You can find all of these in my repo at https://github.com/philmcc/postgres_clusters

For example here is a set up to allow you to set up a 3 node cluster with 1 leader and 2 followers all on postgres 9.4 and using replication slots:

Read more

What is apache Cassandra?

by

Cassandra is a fast distributed database.
It has several defining features:

  • Built in high availability. – Any node can handle read and write requests and your data is replicated to x nodes so regardless of which node (or even a data center) goes down, you will still have access to read and write your data.
  • Linear Scalability. – Doubling the number of (identical) nodes should double the write performance. Its basically as simple as that was all nodes can handle all operations and there is no central control.
  • Predictable performance. (i.e. doubling the number of identical nodes should double the write throughput)
  • no single point of failure. -nodes can go down and come back up without the front end application becoming aware of it.
  • Multiple Data Centres catered for and taken advantage of out as standard.
  • Built to run on commodity hardware – so you can run it on lots of $1000 servers rather than 1 or 2 $100000 servers.
  • Easy to manage operationally. – The system is designed to need very little ops input.

Read more

Relational Databases and Big Data workloads.

by

This intro to Cassandra is taken from the DataStax course. I don’t necessarily agree with everything – particularly their take on what a traditional RDBMS can and can’t do but I have included their view here for completeness.

Cassandra is designed for ‘Big Data’ workloads. Im order to understand the characteristics of Big Data, lets first define ‘Small Data’:

This would typically be a volume of storage that would fit on 1 machine and a RDBMS is typically fine and able to handle the number of operations and the quantity of data. The system will support a number of concurrent users in the hundreds. It fully supports ACID.

When you want to scale such a system, you are going to do it vertically first – with a bigger host, more RAM or processors.

Can Relational databases support big data?

Read more

How to make a Decision

by

How to Make a Decision
How to Make a Decision

I am slightly embarrassed to admit that I have often avoided making decisions throughout my life. That’s not to say that I haven’t made things happen but I have often avoided putting my foot down and choosing a path one way or the other, preferring instead to allow circumstances to make the decision myself. That’s not to say that I haven’t subconsciously pushed circumstances one way or another, but it’s not the same and does not give you the same benefits as being decisive and telling the world what you want.

The thing is that an argument could be made that the whole point of life is to make decisions (if we take any spiritual questions out of the equation). The biological definition of life is that ‘life is irritable’ meaning that the environment does something and life does something back. This is true for all life – a plant decides on some level to move towards the light and if it were to stop doing it, it would die. The same is true for us on a much more complex level. If we were to stop making decisions, we would not get up, eat or drink and we would soon die.

Read more

How to live 10 Years Longer by Playing Games

by

Live 10 years LongerPlus be happier and have fewer regrets

“I’m a gamer so I like to have goals. I like special missions and secret objectives, so here’s my special mission for this talk. I’m going to try to increase the lifespan of everyone in this room by seven and a half minutes.”
– Jane McGonigal

Jane Mcdonigal gave a talk at TED recently that I loved. Its backed by some real research and gives us a way to not only live longer but to enjoy life while we live it and to avoid the top 5 deathbed regrets which are:

Top 5 deathbed regrets:

Read more