unix tool for removing duplicates from a large .pgn file?

Main

Date: 02 Feb 2007 16:03:23
From: Folkert van Heusden
Subject: unix tool for removing duplicates from a large .pgn file?

Hi,

I'm collecting as much chess matches played (by humans) as possible.
Now I have around 400MB of pgn-files and I would like to remove the
inevitable duplicates.
Does anyone know a tool for unix (linux/macos x/bsd/aix/etc.) doing so?

--
--------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com

Date: 02 Feb 2007 21:26:21
From: Dave (from the UK)
Subject: Re: unix tool for removing duplicates from a large .pgn file?

Folkert van Heusden wrote:
> Hi,
>
> I'm collecting as much chess matches played (by humans) as possible.
> Now I have around 400MB of pgn-files and I would like to remove the
> inevitable duplicates.
> Does anyone know a tool for unix (linux/macos x/bsd/aix/etc.) doing so?
>
png-extract

http://www.cs.kent.ac.uk/people/staff/djb/pgn-extract/

will do it for you. It has no GUI, will do what you want.

ChessDB (based on Scid, but developed further), will do too.

http://chessdb.sourceforge.net/

it has a GUI interface, so is quite different in use. Depends what you
want.

--
Dave (from the UK)

Please note my email address changes periodically to avoid spam.
It is always of the form: [email protected]
Hitting reply will work for a few months only - later set it manually.

http://chessdb.sourceforge.net/ - a Free open-source Chess Database

Date: 02 Feb 2007 23:42:58
From: Folkert van Heusden
Subject: Re: unix tool for removing duplicates from a large .pgn file?

Gian-Carlo/Dave: thanks! pgn-extract does exactly what I was looking for!

Date: 02 Feb 2007 18:49:32
From: Gian-Carlo Pascutto
Subject: Re: unix tool for removing duplicates from a large .pgn file?