CVS log for pgsql/src/backend/storage/ipc/shmem.c

Up to [PostgreSQL CVS Repository] / pgsql / src / backend / storage / ipc

Request diff between arbitrary revisions - Display revisions graphically

Keyword substitution: kv
Default branch: MAIN

Update copyright for the year 2010.

Update copyright for 2009.

Remove the last vestiges of the MAKE_PTR/MAKE_OFFSET mechanism.  We haven't
allowed different processes to have different addresses for the shmem segment
in quite a long time, but there were still a few places left that used the
old coding convention.  Clean them up to reduce confusion and improve the
compiler's ability to detect pointer type mismatches.

Kris Jurka

Update copyrights in source tree to 2008.

Update CVS HEAD for 2007 copyright.  Back branches are typically not
back-stamped for this.

Redesign the patch for allocation of shmem space and LWLocks for add-on
modules; the first try was not usable in EXEC_BACKEND builds (e.g.,
Windows).  Instead, just provide some entry points to increase the
allocation requests during postmaster start, and provide a dedicated
LWLock that can be used to synchronize allocation operations performed
by backends.  Per discussion with Marc Munro.

pgindent run for 8.2.

Replace strncpy with strlcpy in selected places that seem possibly relevant
to performance.  (A wholesale effort to get rid of strncpy should be
undertaken sometime, but not during beta.)  This commit also fixes dynahash.c
to correctly truncate overlength string keys for hashtables, so that its
callers don't have to anymore.

Add support for loadable modules to allocated shared memory and
lightweight locks.

Marc Munro

Add support to dynahash.c for partitioning shared hashtables according
to the low-order bits of the entry hash value.  Also make some incidental
cleanups in the dynahash API, such as not exporting the hash header
structs to the world.

Remove 576 references of include files that were not needed.

Allow include files to compile own their own.

Strip unused include files out unused include files, and add needed
includes to C files.

The next step is to remove unused include files in C files.

Update copyright for 2006.  Update scripts.

Rearrange backend startup sequence so that ShmemIndexLock can become
an LWLock instead of a spinlock.  This hardly matters on Unix machines
but should improve startup performance on Windows (or any port using
EXEC_BACKEND).  Per previous discussion.

Get rid of the SpinLockAcquire/SpinLockAcquire_NoHoldoff distinction
in favor of having just one set of macros that don't do HOLD/RESUME_INTERRUPTS
(hence, these correspond to the old SpinLockAcquire_NoHoldoff case).
Given our coding rules for spinlock use, there is no reason to allow
CHECK_FOR_INTERRUPTS to be done while holding a spinlock, and also there
is no situation where ImmediateInterruptOK will be true while holding a
spinlock.  Therefore doing HOLD/RESUME_INTERRUPTS while taking/releasing a
spinlock is just a waste of cycles.  Qingqing Zhou and Tom Lane.

Re-run pgindent, fixing a problem where comment lines after a blank
comment line where output as too long, and update typedefs for /lib
directory.  Also fix case where identifiers were used as variable names
in the backend, but as typedefs in ecpg (favor the backend for
indenting).

Backpatch to 8.1.X.

Re-run pgindent, fixing a problem where comment lines after a blank
comment line where output as too long, and update typedefs for /lib
directory.  Also fix case where identifiers were used as variable names
in the backend, but as typedefs in ecpg (favor the backend for
indenting).

Backpatch to 8.1.X.

Standard pgindent run for 8.1.

Fix LWLockAssign() so that it can safely be executed after postmaster
initialization.  Add spinlocking, fix EXEC_BACKEND unsafeness.

Convert the arithmetic for shared memory size calculation from 'int'
to 'Size' (that is, size_t), and install overflow detection checks in it.
This allows us to remove the former arbitrary restrictions on NBuffers
etc.  It won't make any difference in a 32-bit machine, but in a 64-bit
machine you could theoretically have terabytes of shared buffers.
(How efficiently we could manage 'em remains to be seen.)  Similarly,
num_temp_buffers, work_mem, and maintenance_work_mem can be set above
2Gb on a 64-bit machine.  Original patch from Koichi Suzuki, additional
work by moi.

Modify hash_search() API to prevent future occurrences of the error
spotted by Qingqing Zhou.  The HASH_ENTER action now automatically
fails with elog(ERROR) on out-of-memory --- which incidentally lets
us eliminate duplicate error checks in quite a bunch of places.  If
you really need the old return-NULL-on-out-of-memory behavior, you
can ask for HASH_ENTER_NULL.  But there is now an Assert in that path
checking that you aren't hoping to get that behavior in a palloc-based
hash table.
Along the way, remove the old HASH_FIND_SAVE/HASH_REMOVE_SAVED actions,
which were not being used anywhere anymore, and were surely too ugly
and unsafe to want to see revived again.

Remove an unused variable `ShmemBootstrap', and remove an obsolete
comment. Patch from Alvaro.

Tag appropriate files for rc3

Also performed an initial run through of upgrading our Copyright date to
extend to 2005 ... first run here was very simple ... change everything
where: grep 1996-2004 && the word 'Copyright' ... scanned through the
generated list with 'less' first, and after, to make sure that I only
picked up the right entries ...

Arrange to preallocate all required space for the buffer and FSM hash
tables in shared memory.  This ensures that overflow of the lock table
creates no long-lasting problems.  Per discussion with Merlin Moncure.

Pgindent run for 8.0.

Update copyright to 2004.

Drops in the CreateProcess calls for Win32 (essentially wrapping up the
fork/exec portion of the port), and fixes a handful of whitespace issues

Claudio Natoli

ShmemInitHash forgot to specify HASH_ALLOC flag bit in its hash_create
call.  You'd think this would cause some problems, but because of the
way hash_create is coded, the only side-effect was creation of a useless
memory context for the hashtable.

[ This description should have been on the earlier fork/exec
  commit, but I am adding it now so it is in CVS.]

The patch basically is a slight rearrangement of the code to allow
fork/exec on Unix, with the ultimate goal of doing CreateProcess on
Win32.  The changes are:

        o  Write out postmaster global variables and per-backend
variables to be read by the exec'ed backend

        o  Mark some static variables as global when exec is used so
then can be dumped from postmaster.c, marked NON_EXEC_STATIC

        o  Remove value passing with -p now that we have per-backend
file

        o  Move some pointer storage out of shared memory for easier
dumping.

        o  Modified pgsql_temp directory cleanup to handle per-database
directories and the backend exec directory under datadir.


Claudio Natoli

This patch is the next step towards (re)allowing fork/exec.

Claudio Natoli

$Header: -> $PostgreSQL Changes ...

Ensure that all places that are complaining about exhaustion of shared
memory say 'out of shared memory'; some were doing that and some just
said 'out of memory'.  Also add a HINT about increasing max_locks_per_transaction
where relevant, per suggestion from Sean Chittenden.  (The former change
does not break the strings freeze; the latter does, but I think it's
worth doing anyway.)

Message editing: remove gratuitous variations in message wording, standardize
terms, add some clarifications, fix some untranslatable attempts at dynamic
message building.

Arrange to align shared disk buffers on at least 32-byte boundaries,
not just MAXALIGN boundaries.  This makes a noticeable difference in
the speed of transfers to and from kernel space, at least on recent
Pentiums, and might help other CPUs too.  We should look at making
this happen for local buffers and buffile.c too.  Patch from Manfred Spraul.

Update copyrights to 2003.

Error message editing in backend/storage.

Pass shared memory id and socket descriptor number on command line for
fork/exec.

pgindent run.

Update copyright to 2002.

Create an internal semaphore API that is not tied to SysV semaphores.
As proof of concept, provide an alternate implementation based on POSIX
semaphores.  Also push the SysV shared-memory implementation into a
separate file so that it can be replaced conveniently.

Change made to elog:

o  Change all current CVS messages of NOTICE to WARNING.  We were going
to do this just before 7.3 beta but it has to be done now, as you will
see below.

o Change current INFO messages that should be controlled by
client_min_messages to NOTICE.

o Force remaining INFO messages, like from EXPLAIN, VACUUM VERBOSE, etc.
to always go to the client.

o Remove INFO from the client_min_messages options and add NOTICE.

Seems we do need three non-ERROR elog levels to handle the various
behaviors we need for these messages.

Regression passed.

Ensure that all direct uses of spinlock-protected data structures use
'volatile' pointers to access those structures, so that optimizing
compilers will not decide to move the structure accesses outside of the
spinlock-acquire-to-spinlock-release sequence.  There are no known bugs
in these uses at present, but based on bad experience with lwlock.c,
it seems prudent to ensure that we protect these other uses too.
Per pghackers discussion around 12-Dec.  (Note: it should not be
necessary to worry about structures protected by LWLocks, since the
LWLock acquire and release operations are not inline macros.)

pgindent run on all C files.  Java run to follow.  initdb/regression
tests pass.

Further cleanup of dynahash.c API, in pursuit of portability and
readability.  Bizarre '(long *) TRUE' return convention is gone,
in favor of just raising an error internally in dynahash.c when
we detect hashtable corruption.  HashTableWalk is gone, in favor
of using hash_seq_search directly, since it had no hope of working
with non-LONGALIGNable datatypes.  Simplify some other code that was
made undesirably grotty by promixity to HashTableWalk.

Another round of cleanups for dynahash.c (maybe it's finally clean of
portability issues).  Caller-visible data structures are now allocated
on MAXALIGN boundaries, allowing safe use of datatypes wider than 'long'.
Rejigger hash_create API so that caller specifies size of key and
total size of entry, not size of key and size of rest of entry.
This simplifies life considerably since each number is just a sizeof(),
and padding issues etc. are taken care of automatically.

Implement new 'lightweight lock manager' that's intermediate between
existing lock manager and spinlocks: it understands exclusive vs shared
lock but has few other fancy features.  Replace most uses of spinlocks
with lightweight locks.  All remaining uses of spinlocks have very short
lock hold times (a few dozen instructions), so tweak spinlock backoff
code to work efficiently given this assumption.  All per my proposal on
pghackers 26-Sep-01.

Get rid of PID entries in shmem hash table; there is no longer any need
for them, and making them just wastes time during backend startup/shutdown.
Also, remove compile-time MAXBACKENDS limit per long-ago proposal.
You can now set MaxBackends as high as your kernel can stand without
any reconfiguration/recompilation.

pgindent run.  Make it all clean.

Change Copyright from PostgreSQL, Inc to PostgreSQL Global Development Group.

Significant cleanups in SysV IPC handling (shared mem and semaphores).
IPC key assignment will now work correctly even when multiple postmasters
are using same logical port number (which is possible given -k switch).
There is only one shared-mem segment per postmaster now, not 3.
Rip out broken code for non-TAS case in bufmgr and xlog, substitute a
complete S_LOCK emulation using semaphores in spin.c.  TAS and non-TAS
logic is now exactly the same.
When deadlock is detected, "Deadlock detected" is now the elog(ERROR)
message, rather than a NOTICE that comes out before an unhelpful ERROR.

Put external declarations into header files.

Fix confusion between sizeof(long) and sizeof(long*), as well as
failure to MAXALIGN the start of shmem allocable space.  No reports
of trouble here, just compulsive tidiness.

First phase of memory management rewrite (see backend/utils/mmgr/README
for details).  It doesn't really do that much yet, since there are no
short-term memory contexts in the executor, but the infrastructure is
in place and long-term contexts are handled reasonably.  A few long-
standing bugs have been fixed, such as 'VACUUM; anything' in a single
query string crashing.  Also, out-of-memory is now considered a
recoverable ERROR, not FATAL.
Eliminate a large amount of crufty, now-dead code in and around
memory management.
Fix problem with holding off SIGTRAP, SIGSEGV, etc in postmaster and
backend startup.

Remove unused include files.  Do not touch /port or includes used by defines.

Ye-old pgindent run.  Same 4-space tabs.

Shared-memory hashtables have non-extensible directories, which means
it's a good idea to choose the directory size based on the expected
number of entries.  But ShmemInitHash was using a hard-wired constant.
Boo hiss.  This accounts for recent report of postmaster failure when
asking for 64K or more buffers.

Add:

  * Portions Copyright (c) 1996-2000, PostgreSQL, Inc

to all files copyright Regents of Berkeley.  Man, that's a lot of files.

XLOG (also known as WAL -:)) Bootstrap/Startup/Shutdown.
First step in cleaning up backend initialization code.
Fix for FATAL: now FATAL is ERROR + exit.

Several changes here, not very related but touching some of the same files.
* Buffer refcount cleanup (per my "progress report" to pghackers, 9/22).
* Add links to backend PROC structs to sinval's array of per-backend info,
and use these links for routines that need to check the state of all
backends (rather than the slow, complicated search of the ShmemIndex
hashtable that was used before).  Add databaseOID to PROC structs.
* Use this to implement an interlock that prevents DESTROY DATABASE of
a database containing running backends.  (It's a little tricky to prevent
a concurrently-starting backend from getting in there, since the new
backend is not able to lock anything at the time it tries to look up
its database in pg_database.  My solution is to recheck that the DB is
OK at the end of InitPostgres.  It may not be a 100% solution, but it's
a lot better than no interlock at all...)
* In ALTER TABLE RENAME, flush buffers for the relation before doing the
rename of the physical files, to ensure we don't get failures later from
mdblindwrt().
* Update TRUNCATE patch so that it actually compiles against current
sources :-(.
You should do "make clean all" after pulling these changes.

 Move some system includes into c.h, and remove duplicates.

Remove unused #includes in *.c files.

Clean up #include in /include directory.  Add scripts for checking includes.

1. xact.c: update comments about changing MyProc->xid and MyProc->xmin.
2. varsup.c:ReadNewTransactionId(): don't read nextXid from disk -
   this func doesn't allocate next xid, so ShmemVariableCache->nextXid
   may be used (but GetNewTransactionId() must be called first).
3. vacuum.c: change elog(ERROR, "Child item....") to elog(NOTICE) -
   this is not ERROR, proper handling is just not implemented, yet.
4. s_lock.c: increase S_MAX_BUSY by 2 times.
5. shmem.c:GetSnapshotData(): have to call ReadNewTransactionId()
   _after_ SpinAcquire(ShmemIndexLock).

1. Additional fix against ERROR:  Child itemid marked as unused
   in CommitTransaction().
2. Changes in GetSnapshotData().

pgindent run over code.

Clean up compile errors and warnings, cf Billy Allie's
complaints (and some of my own).

1. Vacuum is updated for MVCC.
2. Much faster btree tuples deletion in the case when first on page
   index tuple is deleted (no movement to the left page(s)).
3. Remember blkno of new root page in BTPageOpaque of
   left/right siblings when root page is splitted.

Retrofit hashtable and shared-mem-size-estimation bug fixes
into REL6_4.

Repair some pretty serious problems in dynahash.c and
shared memory space allocation.  It's a wonder we have not seen bug
reports traceable to this area ... it's quite clear that the routine
dir_realloc() has never worked correctly, for example.

Change my-function-name-- to my_function_name, and optimizer renames.

Cleanup of source files where 'return' or 'var =' is alone on a line.

SET TRANSACTION ISOLATION LEVEL ...
LOCK TABLE IN ... MODE
...implemented

Serialized mode works!

Initial MVCC code.
New code for locking buffer' context.

OK, folks, here is the pgindent output.

Renaming cleanup, no pgindent yet.

Use Snapshot in heap access methods.

VariableCache (next XID generator) is placed in shmem.

Rename shared memory area.

Update backend flowchart.

Rename BindingTable to ShmemIndex.

More cleanups for compiler warnings.

Fix for hang after postmaster restart.  Add new proc_exit and
shmem_exit to replace exitpg().

Rename BindingTab to BindingTable.

cleanups

Remove un-needed braces around single statements.

Cleanup.

ASSERT fixes.

pgindent run before 6.3 release, with Thomas' requested changes.

Goodbye ABORT.  Hello ERROR for all errors.

Change elog(WARN) to elog(ERROR) and elog(ABORT).

Inline memset() as MemSet().

Used modified version of indent that understands over 100 typedefs.

Another PGINDENT run that changes variable indenting and case label indenting.  Also static variable indenting.

Massive commit to run PGINDENT on all *.c and *.h files.

Remove more (void) and fix -Wall warnings.

From: "D'Arcy J.M. Cain" <darcy@druid.net>
Subject: [PATCHES] backend/storage/ipc/shmem.c

  - strname is only used if assert checking is on

More stuff to make it compile on various ports.

TransactionIdIsInProgress is here now and gives quality answer
by scanning PROC structures of all running backend.

All external function definitions now have prototypes that are checked.

More compile cleanups

Compile and warning cleanup

Clean up Makefile(s)

Add #include "postgres.h" as required

Remove #include "c.h" *and* "utils/elog.h" as applicable

Added needed include file.

Postgres95 1.01 Distribution - Virgin Sources

Initial revision