Open chess database format?

Main

Date: 31 Jul 2008 08:21:05
From: Andy Duplain
Subject: Open chess database format?

Does anyone else see the need for an open binary chess database
format? While PGN is widely supported and obviously useful for
transmitting games via e-mail, websites, etc., it isn't a database
format, rather a concatenation of games. A binary format would be
required to provide a database, which can provide the data structure
nececessary to allow searches of game criteria and search positions,
etc.

Anyone have any ideas on how to progress on this subject?

Cheers,
Andy

Date: 13 Aug 2008 21:25:30
From: Simon Krahnke
Subject: Re: Open chess database format?

* David Richerby <[email protected] > (14:59) schrieb:

> <game>
> <player color=white firstname=Garratoly lastname=Karpsparov rating=10000/>

That's not XML!

mfg, simon .... l

Date: 05 Aug 2008 12:03:39
From: Andy Duplain
Subject: Re: Open chess database format?

On 5 Aug, 12:44, "Odd Gunnar Malin" <[email protected] > wrote:
> "Andy Duplain" <[email protected]> skrev i meldingnews:da2757f7-9c=
[email protected]...
>
> > Does anyone else see the need for an open binary chess database
> > format? =A0While PGN is widely supported and obviously useful for
> > transmitting games via e-mail, websites, etc., it isn't a database
> > format, rather a concatenation of games. =A0A binary format would be
> > required to provide a database, which can provide the data structure
> > nececessary to allow searches of game criteria and search positions,
> > etc.
>
> > Anyone have any ideas on how to progress on this subject?
>
> > Cheers,
> > Andy
>
> On the bottom of the pgn standard text there is a description of a pgc
> format.
>
> But for fast search you need to index the database so if the games are in
> binary or ascii format isn't too important. For faster header search you
> only need to create a list (index) of start position of each game, but fo=
r
> position search you maybe need to generate a tree. I don't think it is to=
o
> important that these index or tree files are in an open format because
> individual program have individual needs so generating them by the progra=
m
> itself when you import a gamefile seems better.
>
> Another theme is extending of the gamefile (pgn or pgc) to add some more
> feature to the game, comment before move, null move, a draw offer, and mu=
ch
> more. Maybe up to todays standard this should be some sort of xml but you
> will then miss the important feature of reading the game without a progra=
m,
> eg. printing it out and play it on a chessboard. If you do a search for
> chess and xml I'm sure you find some work done here.
>
> Odd Gunnar Malin

Well I think it is important the moves are stored in binary form,
simply to allow games to be modified and written back to the database
- something that is not trivial with PGN for example. I think if a
single, open, format was used then this would benefit all - for
example if you create your index/tree/key/whatever in one program you
want to be able to distribute that and use it in another - this is
especially true of theme/opening keys as they have to be created in a
creative manner and cannot be blindly created from game data with no
user interaction.

Size is, of course, still important, regardless of disk sizes and
internet connection speeds these days, and binary provides the best
options here.

Cheers,
Andy

Date: 13 Aug 2008 14:10:01
From: David Richerby
Subject: Re: Open chess database format?

Andy Duplain <[email protected] > wrote:
> Well I think it is important the moves are stored in binary form,
> simply to allow games to be modified and written back to the
> database - something that is not trivial with PGN for example.

And how do you think that a binary format will make that any easier?
Ultimately, editing a chess game is going to change the space required
to store it and that's just as much of a problem for a binary format
as for a text-based format.

> I think if a single, open, format was used then this would benefit
> all - for example if you create your index/tree/key/whatever in one
> program you want to be able to distribute that and use it in another
> - this is especially true of theme/opening keys as they have to be
> created in a creative manner and cannot be blindly created from game
> data with no user interaction.

But a key is an index to a specific dataset. Without the dataset, the
key is meaningless. So, if you and I are manipulating the same
dataset, we may as well use the same database program to do that. I
realise that's not ideal but it doesn't strike me as a killer problem
in need of a heavyweight solution.

> Size is, of course, still important, regardless of disk sizes and
> internet connection speeds these days, and binary provides the best
> options here.

PGN compresses like a dream so this isn't so much of an issue, I
think.

Dave.

--
David Richerby Miniature Painting (TM): it's like a
www.chiark.greenend.org.uk/~davidr/ Renaissance masterpiece but you can
hold in it your hand!

Date: 05 Aug 2008 13:44:36
From: Odd Gunnar Malin
Subject: Re: Open chess database format?

"Andy Duplain" <[email protected] > skrev i melding
news:da2757f7-9c0d-4ce2-bd9e-3bd870c090eb@k30g2000hse.googlegroups.com...
> Does anyone else see the need for an open binary chess database
> format? While PGN is widely supported and obviously useful for
> transmitting games via e-mail, websites, etc., it isn't a database
> format, rather a concatenation of games. A binary format would be
> required to provide a database, which can provide the data structure
> nececessary to allow searches of game criteria and search positions,
> etc.
>
> Anyone have any ideas on how to progress on this subject?
>
> Cheers,
> Andy

On the bottom of the pgn standard text there is a description of a pgc
format.

But for fast search you need to index the database so if the games are in
binary or ascii format isn't too important. For faster header search you
only need to create a list (index) of start position of each game, but for
position search you maybe need to generate a tree. I don't think it is too
important that these index or tree files are in an open format because
individual program have individual needs so generating them by the program
itself when you import a gamefile seems better.

Another theme is extending of the gamefile (pgn or pgc) to add some more
feature to the game, comment before move, null move, a draw offer, and much
more. Maybe up to todays standard this should be some sort of xml but you
will then miss the important feature of reading the game without a program,
eg. printing it out and play it on a chessboard. If you do a search for
chess and xml I'm sure you find some work done here.

Odd Gunnar Malin

Date: 13 Aug 2008 13:59:19
From: David Richerby
Subject: Re: Open chess database format?

Odd Gunnar Malin <[email protected] > wrote:
> But for fast search you need to index the database so if the games
> are in binary or ascii format isn't too important. For faster header
> search you only need to create a list (index) of start position of
> each game, but for position search you maybe need to generate a
> tree. I don't think it is too important that these index or tree
> files are in an open format because individual program have
> individual needs so generating them by the program itself when you
> import a gamefile seems better.

Indeed.

> Another theme is extending of the gamefile (pgn or pgc) to add some
> more feature to the game, comment before move, null move, a draw
> offer, and much more.

PGN can already do comments. Null-moves could be added just by
allowing, say, `--' as a move; draw offers could be added through the
existing `Numerical Annotation Glyph' mechanism.

Other obvious things that could be added are engine evaluations and
clock data. Fritz does this by adding specially-formatted comments,
which is always an option for `private' extensions to any standard.

> Maybe up to todays standard this should be some sort of xml but you
> will then miss the important feature of reading the game without a
> program, eg. printing it out and play it on a chessboard. If you do
> a search for chess and xml I'm sure you find some work done here.

Ugh. I can see no reason to want to use XML for this. XML is
designed for hierarchically-organized data and chess data just isn't
very hierarchical, with the exception of analysis trees, which are
already adequately dealt with by PGN. What advantage is there in
converting

[white "Karpsparov, Garratoly"]
[black "Kramchnoi, Viktimir"]
[whiteelo "10000"]
[blackelo "10001"]

1.e4 c5 (1... e5 {is good but isn't Kasparov's style}) 0-1

to

<game >
<player color=white firstname=Garratoly lastname=Karpsparov rating=10000/ >
<player color=white firstname=Viktimir lastname=Kramcnhoi rating=10001/ >

<gamescore result=0-1 >
<move >e4</move>
<move >c5</move
<variation >
<move >e5</move><comment>is good but isn't Kramchnoi's style</comment>
</variation >
</gamescore >
</game >

?

Dave.

--
David Richerby Solar-Powered Apple (TM): it's like
www.chiark.greenend.org.uk/~davidr/ a tasty fruit but it doesn't work in
the dark!

Date: 03 Aug 2008 14:09:45
From: Andy Duplain
Subject: Re: Open chess database format?

On 31 Jul, 16:24, Paolo Casaschi <[email protected] > wrote:
> scid is THE open source chess database software and defines its own binar=
y
> format.http://scid.sf.net
>
> Paolo
>
>
>
> On Thu, 31 Jul 2008, Andy Duplain wrote:
> > Does anyone else see the need for an open binary chess database
> > format? =A0While PGN is widely supported and obviously useful for
> > transmitting games via e-mail, websites, etc., it isn't a database
> > format, rather a concatenation of games. =A0A binary format would be
> > required to provide a database, which can provide the data structure
> > nececessary to allow searches of game criteria and search positions,
> > etc.
>
> > Anyone have any ideas on how to progress on this subject?
>
> > Cheers,
> > Andy- Hide quoted text -
>
> - Show quoted text -

There is more than one open source chess database, and just because a
program is open source it doesn't mean the binary format is open. I
was looking for some initiative that allowed people that have
experience with chess database formats to contribute to the format,
allowing it become more powerful than anything currently available.
Does the author of SCID seek contribution in the format?

Andy

Date: 13 Aug 2008 13:31:27
From: David Richerby
Subject: Re: Open chess database format?

Andy Duplain <[email protected] > wrote:
> There is more than one open source chess database

True but SCID seems to be the one that's most talked-about.
(Admittedly, round here that's partly because there's a certain amount
of controversy about Pascal's version of it.)

> and just because a program is open source it doesn't mean the binary
> format is open.

I don't see how, unless SCID uses a closed-source library to do its
database management.

> Does the author of SCID seek contribution in the format?

SCID's author, Shane Hudson, has not worked on the project for a few
years, now. There are a couple of forks being actively developed.

Dave.

--
David Richerby Gigantic Painting (TM): it's like a
www.chiark.greenend.org.uk/~davidr/ Renaissance masterpiece but it's huge!

Date: 13 Aug 2008 18:53:05
From: Pascal
Subject: Re: Open chess database format?

David Richerby a �crit :
> Andy Duplain <[email protected]> wrote:
>> There is more than one open source chess database
>
> True but SCID seems to be the one that's most talked-about.
> (Admittedly, round here that's partly because there's a certain amount
> of controversy about Pascal's version of it.)

Usssssh ! don't wake up the trolls ;-) !

FYI, there seems to be no fork of Scid really maintained those days (I
may be wrong), and Scid (the Genuine one), is (see
http://scid.sourceforge.net).

If you look at
http://sourceforge.net/mailarchive/forum.php?forum_name=scid-users
for latest months activity you will see that Scid's development is up
and well.

To come back to this post's subject, a format must proove its efficiency
first, then (and only then) *maybe* become a de facto standard (only if
this format is far better than all others). The first phase of the
process (development and qualification) is certainly many years long.

Pascal

Date: 07 Aug 2008 13:14:47
From: Pascal
Subject: Re: Open chess database format?

Andy Duplain a �crit :

> There is more than one open source chess database, and just because a
> program is open source it doesn't mean the binary format is open. I
> was looking for some initiative that allowed people that have
> experience with chess database formats to contribute to the format,
> allowing it become more powerful than anything currently available.
> Does the author of SCID seek contribution in the format?

Scid binary format is not perfect (for example a game is limited to 32
kB, which in some extrem cases is not enough). But it is pretty efficient.
If anybody :
- either can improve this format
- or produce another format that is better (fucntionality, size,
performance, ...)
he's certainly much welcome to contribute and post his result.

Unfortunately in that field good ideas are not enough : they must go
through the implementation and tests phases. Then good ideas sometimes
look bad (I already experienced that myself tweaking chess data).

Pascal

Date: 31 Jul 2008 16:24:04
From: Paolo Casaschi
Subject: Re: Open chess database format?

scid is THE open source chess database software and defines its own binary
format.
http://scid.sf.net

Paolo

On Thu, 31 Jul 2008, Andy Duplain wrote:

> Does anyone else see the need for an open binary chess database
> format? While PGN is widely supported and obviously useful for
> transmitting games via e-mail, websites, etc., it isn't a database
> format, rather a concatenation of games. A binary format would be
> required to provide a database, which can provide the data structure
> nececessary to allow searches of game criteria and search positions,
> etc.
>
> Anyone have any ideas on how to progress on this subject?
>
> Cheers,
> Andy
>