It is currently Wed Dec 01, 2021 2:56 am

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 2 posts ] 
Author Message
Offline
 Post subject: Problems with exchanging SGF files
Post #1 Posted: Sun Jul 18, 2021 2:57 am 
Dies with sente

Posts: 112
Liked others: 135
Was liked: 28
Rank: British 3 kyu
KGS: thirdfogie
Problems have occurred when a friend tries to read SGF files prepared by me and
emailed to him. I use Quarry, which only runs on Linux. He uses sgfeditor,
which only runs on Windows, so far as I can tell. The error report is
typically "It didn't run", which presumably reflects what sgfeditor told him.
Fortunately, I use CGOBAN3 as my interface to KGS. That program contains
an SGF editor which has the same foibles as sgfeditor, up to now. Unlike
sgfeditor, CGOBAN3 gives useful error reports including the line number within
the offending SGF file.

The following list is mainly reconstructed from memory, and may itself contain
errors. Apologies for any undeserved slights on sgfeditor. My reference for
SGF format is here. https://www.red-bean.com/sgf/sgf4.html

Problem 1

It is possible to save an SGF file after analysis with Lizzie and an AI. The
results of the analysis are stored in LZ[] fields. The sgfeditor program
failed with no error report when it read such a field. (No error report
reached me, at least.) The fix was to use the sgfstrip program (from sgf-utils
for Linux) to strip the LZ codes as follows.

Code:
sgfstrip LZ < raw.sgf > stripped.sgf

Problem 2

Quarry sometimes outputs an empty field such as C[] or LB[]. It is not clear
how this happens: possibly from attempts by the user to delete an existing
comment or label, usually because it was in the wrong place. The sgfeditor
program fails with no error report (to me) when it finds one of these. The fix
was to run the sgfc (SGF checker) program on the output from sgfstrip, as
follows.

Code:
sgfc -e stripped.sgf file-for-email

Problem 3

Running sgfc to work around Problem 2 eventually introduced Problem 3,
which is best explained using some sample SGF lines.

Code:
C[thirdfogie [?\]: Blah, blah, blah, blah\
blah, blah, blah, blah, blah, bl.]BL[1778.673];W[dd]C[JoeCasey [-\]: Bl\
ah, blah, blash, blah, blah, blah???]WL[1756.487];B[pj]C[thirdfogie [?\\
]: Blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, bl\
blah, blah, blah, blah, blah, blah, blah, blah, blah, bl.]

How did this happen? The original games were played on KGS, which precedes
each in-game comment with a string such as thirdfogie [?]: or JoeCasey [-]: to
identify the commenter's account and grade. The use of square brackets in a
comment field is asking for trouble because they must be escaped wth a
backslash like this [?\]. All works well until such a string gets broken at
the end of a line, as in the example. The sgfc program decided to write the [?\\
string at the end of the third line. The sgfeditor program failed with no error
report (to me) when it found that string.

The immediate work-around was to use the -L option with sgfc. That option
also produces a more human-friendly layout which may help with future debugging.
It solved the problem with the [?] string from KGS, but might not fix it if
square brackets appeared elsewhere in a comment. As a backup, my conversion script
now checks for any occurrence of \\ and issues a warning if it finds one.

It is not clear to me whether a [?\\ string at the end of a line is valid
SGF. Quarry handles it with no problems. Lizzie truncates the displayed
comment at that point, but recovers and processes the rest of the SGF file.
Other programs just give up.

Problem 4

Compressed point lists are described in section 3.5.1 of the SGF
standard. They can be output by Quarry for set-up stones, but are not
understood by some other programs. (I forgot the details.) In my opinion,
compressed point lists are not worth the extra complexity they introduce.

I hope this post is useful to someone else. Most likely, no-one else attracts
bugs like I do. But at least I have a record of how things stood in July 2021.

Top
 Profile  
 
Offline
 Post subject: Re: Problems with exchanging SGF files
Post #2 Posted: Sun Jul 18, 2021 3:06 pm 
Dies with sente

Posts: 83
Liked others: 5
Was liked: 29
thirdfogie wrote:
Problems have occurred when a friend tries to read SGF files prepared by me and emailed to him. ... The following list is mainly reconstructed from memory. My reference for SGF format is here. https://www.red-bean.com/sgf/sgf4.html

That SGF standard is outdated. Some common constructs do not occur. Other things do occur but cause problems today.

Some years ago I scraped a few million SGF files from the net, and published statistics and recommendations, see https://homepages.cwi.nl/~aeb/go/misc/sgf.html.

Quote:
Problem 1 ...
The fix was to use the sgfstrip program (from sgf-utils for Linux) to strip the LZ codes as follows.
Code:
sgfstrip LZ < raw.sgf > stripped.sgf

Maybe sgfutils?

Quote:
Problem 2
Quarry sometimes outputs an empty field such as C[] or LB[].
The fix was to run the sgfc (SGF checker) program.

This sgfc was written to check conformance to the outdated SGF spec. It can harm your SGF files. Avoid it. Does
Code:
sgf < raw.sgf > cooked.sgf
solve your problem?

Quote:
Running sgfc to work around Problem 2 eventually introduced Problem 3 ... It is not clear to me whether a [?\\ string at the end of a line is valid SGF.

"Whitespace is ignored everywhere, except inside a Value. Inside a Value whitespace is significant, but an escaped newline is ignored."

Quote:
Problem 4
Compressed point lists are described in section 3.5.1 of the SGF
standard.

They are rarely used. As you say, many programs do not understand them.


This post by aeb was liked by: Joaz Banbeck
Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 2 posts ] 

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group