Did you tried to use the option "-z 0" to disable
compression? If that does not exhibit the problem, would
you be so kind and run again using "--compress-algo=bzip2"?

Thanks

From: harry_b@mm.st
To: bug-any@bugs.gnupg.org
Cc: wk@gnupg.org, Gnats <gnats@trithemius.gnupg.org>, gnupg-hackers@gnupg.org,
gnats-admin@trithemius.gnupg.org
Subject: Re: gnupg/537
Date: Tue, 13 Sep 2005 08:40:11 +0200

Hi Werner,

I don't use gpg's internal compression method since my application must
handle the compressed data itself anyway.

The bug occurs if I run:

• gpg --encrypt --sign FILE
• gpg --decrypt FILE.gpg

on a gzipped file. The problem does not occur if I try these steps on a)
the plain text file or b) the same file differently gzipped.

I assume it has something to do with the binary data which is used, like
size or byte order.

When I run gpg --debug-all with such a buggy file I get this trace:
[...]
gpg: DBG: iobuf-14.0: close `file_filter(fd)'
gpg: DBG: /home/harry/.gnupg/pubring.gpg: close fd 6
gpg: DBG: fd_cache_close (/home/harry/.gnupg/pubring.gpg) used existing slot
gpg: DBG: finish_lookup: checking key XXXXXXXX (all)(req_usage=0)
gpg: DBG: using key XXXXXXXX
gpg: DBG: cache_user_id: already in cache
gpg: DBG: iobuf-13.0: close `file_filter(fd)'
gpg: DBG: /home/harry/.gnupg/pubring.gpg: close fd 7
gpg: DBG: fd_cache_close (/home/harry/.gnupg/pubring.gpg) used existing slot
gpg: Good signature from "my key <mail@address>"
gpg: DBG: free_packet() type=6
gpg: DBG: mpi_free
[...]
gpg: DBG: dummy m_size called
gpg: DBG: mpi_free_limb_space of size 0
gpg: DBG: free_packet() type=8
gpg: [don't know]: invalid packet (ctb=14)
gpg: DBG: free_packet() type=18
gpg: DBG: iobuf-1.2: underflow: req=8192
gpg: DBG: iobuf-1.2: underflow: got=0 rc=-1
gpg: DBG: iobuf-1.2: pop `(null)' in underflow (!len)
gpg: DBG: iobuf chain: 1.0 `file_filter(fd)' filter_eof=0 start=4675
len=4675
gpg: DBG: iobuf-1.0: underflow: eof
gpg: DBG: iobuf-1.0: close `file_filter(fd)'
gpg: DBG: check-data-410-1.data.gpg: close fd 3
gpg: DBG: fd_cache_close (check-data-410-1.data.gpg) new slot created
random usage: poolsize=600 mixed=4 polls=0/28 added=140/2464

outmix=0 getlvl1=0/0 getlvl2=0/0

secmem usage: 1408/3968 bytes in 2/9 blocks of pool 5472/32768

Is this of any help?

Harry

--On Monday, September 12, 2005 08:54:07 PM +0200 wk@gnupg.org wrote:

Synopsis: invalid packet error

State-Changed-From-To: open->chatting
State-Changed-By: werner
State-Changed-When: Mon, 12 Sep 2005 20:54:07 +0200
State-Changed-Why:
Did you tried to use the option "-z 0" to disable
compression? If that does not exhibit the problem, would
you be so kind and run again using "--compress-algo=bzip2"?

From: Werner Koch <wk@gnupg.org>
To: bug-any@bugs.gnupg.org
Cc: harry_b@mm.st
Subject: Re: gnupg/537
Date: Tue, 13 Sep 2005 12:15:09 +0200

You might still be using the compression; there is some detection code
for already compressed data but that is not fully reliable - only -z0
disable the compression step reliable.

From: harry_b@mm.st
To: Werner Koch <wk@gnupg.org>
Cc: bug-any@bugs.gnupg.org
Subject: Re: gnupg/537
Date: Wed, 14 Sep 2005 08:30:51 +0200

--==========955DBE32511545E7A793==========
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Hi Werner,

it seems like that solved my problem!!
When I use --compress-level=3D0 the test seems to run forever without=20
failuer. Right now I am running a cycle of 100000 tests and it seems that=20
the problem is gone!

Thanks alot for your help!!

Harry

--On Tuesday, September 13, 2005 12:15:09 PM +0200 Werner Koch=20
<wk@gnupg.org> wrote:

You might still be using the compression; there is some detection code
for already compressed data but that is not fully reliable - only -z0
disable the compression step reliable.

--=20

1024D/40F14012 18F3 736A 4080 303C E61E 2E72 7E05 1F6E 40F1 4012

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GIT/S dx s: a C++ ULS++++$ P+++ L+++$ !E W++ N+ o? K? !w !O !M
V PS+ PE Y? PGP+++ t+ 5-- X+ R+ !tv b++ DI++ D+ G e* h r++ y++
------END GEEK CODE BLOCK------

--==========955DBE32511545E7A793==========
Content-Type: application/pgp-signature
Content-Transfer-Encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFDJ8ObfgUfbkDxQBIRAuV6AJ9vnO3TeZRSWwaGzuedD9kNcb5YKgCeJort
0wePEgzqwJX40aeGH2dw2Vg=

lRkg

-----END PGP SIGNATURE-----

--==========955DBE32511545E7A793==========--

From: Werner Koch <wk@gnupg.org>
To: harry_b@mm.st
Cc: bug-any@bugs.gnupg.org
Subject: Re: gnupg/537
Date: Wed, 14 Sep 2005 10:24:11 +0200

On Wed, 14 Sep 2005 08:30:51 +0200, harry b said:

When I use --compress-level=0 the test seems to run forever without
failuer. Right now I am running a cycle of 100000 tests and it seems
that the problem is gone!

Fine, you should do that - it is also faster than trying to compress
already compessed data.

However, I'd like to figure out what the real reasons is. In
particular whether it is gpg or the compression library. If you have
a little bit of spare (CPU) time could you please try again using
--compress-algo=bzip2 but without --compress-level ? If this also
works, we can be pretty sure that the culprit is the zlib compression
library.

Thanks,

Werner

From: harry_b@mm.st
To: Werner Koch <wk@gnupg.org>
Cc: bug-any@bugs.gnupg.org
Subject: Re: gnupg/537
Date: Sun, 18 Sep 2005 21:25:07 +0200

--==========DDD2E1D20B62E4124FE4==========
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Hi Werner,

sorry, it took a while until I got time to write a little script to verify=20
the weird behaviour.

I tried to use bzip2 instead of gzip (again without telling gpg what it got =

to compress) and it fails the same way.

What I do is:

1. create some test data (some random XML in my case)
2. gzip/bzip2 -9 the data and name it file1.data
   1. run the command: echo "PWD" | GNUPGHOME=3D./tests GPG_AGENT_INFO=3D =

gpg=20
--no-tty --recipient=3D"somekey" --passphrase-fd 0 --armour --sign =
--encrypt=20
--output=3D"file2.data" "file1.data"

4. echo "PWD" | GNUPGHOME=3D./tests GPG_AGENT_INFO=3D gpg --no-tty=20

--passphrase-fd 0 --decrypt --output=3D"file3.data" "file2.data"

5. diff --brief file1.data file3.data

With this setup, gpg fails every now and then to run the decompression =
step.

Next thing I'll try is what happens when gpg compresses the data itself. I=20
assume it does not happen then.

Is there anything else I can do to track this problem down some more?

Harry

--On Wednesday, September 14, 2005 10:24:11 AM +0200 Werner Koch=20
<wk@gnupg.org> wrote:

On Wed, 14 Sep 2005 08:30:51 +0200, harry b said:

When I use --compress-level=3D0 the test seems to run forever without
failuer. Right now I am running a cycle of 100000 tests and it seems
that the problem is gone!

Fine, you should do that - it is also faster than trying to compress
already compessed data.

However, I'd like to figure out what the real reasons is. In
particular whether it is gpg or the compression library. If you have
a little bit of spare (CPU) time could you please try again using
--compress-algo=3Dbzip2 but without --compress-level ? If this also
works, we can be pretty sure that the culprit is the zlib compression
library.

Thanks,

Werner

--=20

1024D/40F14012 18F3 736A 4080 303C E61E 2E72 7E05 1F6E 40F1 4012

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GIT/S dx s: a C++ ULS++++$ P+++ L+++$ !E W++ N+ o? K? !w !O !M
V PS+ PE Y? PGP+++ t+ 5-- X+ R+ !tv b++ DI++ D+ G e* h r++ y++
------END GEEK CODE BLOCK------

--==========DDD2E1D20B62E4124FE4==========
Content-Type: application/pgp-signature
Content-Transfer-Encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFDLb8afgUfbkDxQBIRAuK7AJ4rE/Xr4j7oVLZBwyORoybtw2upNACfUb27
b3UC3QL2gwkTNUo8BbUFA4Y=

R6aA

-----END PGP SIGNATURE-----

--==========DDD2E1D20B62E4124FE4==========--

From: harry_b@mm.st
To: Werner Koch <wk@gnupg.org>
Cc: bug-any@bugs.gnupg.org
Subject: Re: gnupg/537
Date: Tue, 27 Sep 2005 14:42:42 +0200

--==========3774EEA59CBCA8CE2E20==========
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Hi Werner,

sorry to bother you again. The problem seems not to be gone - it just=20
happens less frequently.

I have another case where I want to gpg-encrypt a gzipped data file and=20
when I decrypt it I get the error.

The steps are:

$ GNUPGHOME=3D./src/tests GPG_AGENT_INFO=3D gpg=20
--recipient=3D"cpm@testdomain.org" --armour --sign --encrypt=20
--compress-level=3D0 --output=3Dfile.enc file.bin
$ GNUPGHOME=3D./src/tests GPG_AGENT_INFO=3D gpg --decrypt test > /dev/null
gpg: WARNING: unsafe permissions on homedir `./src/tests'=20

You need a passphrase to unlock the secret key for
user: "Console Password Manager (This is just a test key.)=20
<cpm@testdomain.org>"
2048-bit ELG-E key, ID A6F6FD5B, created 2005-04-12 (main key ID C74DEDAD)

gpg: encrypted with 2048-bit ELG-E key, ID A6F6FD5B, created 2005-04-12

"Console Password Manager (This is just a test key.)=20

<cpm@testdomain.org>"
gpg: [don't know]: invalid packet (ctb=3D45)
gpg: Signature made Tue 27 Sep 2005 02:36:55 PM CEST using DSA key ID=20
C74DEDAD
gpg: Good signature from "Console Password Manager (This is just a test=20
key.) <cpm@testdomain.org>"
gpg: [don't know]: invalid packet (ctb=3D2d)

Is there anything I can do to track this down? I can create any number of=20
test files to create this error - it just might take some time. :-)

Harry

--On Tuesday, September 13, 2005 12:15:09 PM +0200 Werner Koch=20
<wk@gnupg.org> wrote:

You might still be using the compression; there is some detection code
for already compressed data but that is not fully reliable - only -z0
disable the compression step reliable.

--=20

1024D/40F14012 18F3 736A 4080 303C E61E 2E72 7E05 1F6E 40F1 4012

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GIT/S dx s: a C++ ULS++++$ P+++ L+++$ !E W++ N+ o? K? !w !O !M
V PS+ PE Y? PGP+++ t+ 5-- X+ R+ !tv b++ DI++ D+ G e* h r++ y++
------END GEEK CODE BLOCK------

--==========3774EEA59CBCA8CE2E20==========
Content-Type: application/pgp-signature
Content-Transfer-Encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFDOT5FfgUfbkDxQBIRAkyWAKCmbGUCO7V0fMAN1eeUk8lI6Paj9ACfdEo+
S2CfWKm+OSR2ml8jOEPVbpQ=

9NfA

-----END PGP SIGNATURE-----

--==========3774EEA59CBCA8CE2E20==========--

Florian Weimer found the problem and provided a fix.

Disk crash on my development box. Will take a while until I can check in the fix.

Ah well, the fix fixes the problem but introduces other problems. Needs further
analysis.

Finally fixed in trunk (-r 4279). Needs to be ported to 1.4.

We used to let parse-packet.c handle the MDC packet but this
turned out to be a problem with compressed packets: With old
style packets there is no length information available and
the decompressor uses an implicit end. However we can't know
this implicit end beforehand (:-) and thus may feed the
decompressor with more bytes than actually needed. It would
be possible to unread the extra bytes but due to our weird
iobuf system any unread is non reliable due to filters
already popped off. The easy and sane solution is to care
about the MDC packet in encr-data.c and never pass it to the
packet parser.

Backported fix to gnupg 1.4.5 (patch included)

D24: 94_fix-537.diff