From perbu at varnish-software.com  Mon Oct  1 07:45:17 2012
From: perbu at varnish-software.com (Per Buer)
Date: Mon, 1 Oct 2012 09:45:17 +0200
Subject: memory issue when running varnishncsa
In-Reply-To: <CAOCtP1fT6MvuTBKQvHUOi44UNzzoHoVNp688PYpUx4=FaGn_Hw@mail.gmail.com>
References: <CAOCtP1d82eFiUXSksvZVMcTiDRANMhNd7hzL_q3n2DHu9=60zg@mail.gmail.com>
 <CAOXZevDtzxUyxKAouAy2bur5TvbMGcq6xctzM5eU9pT6aov5Vw@mail.gmail.com>
 <CAOCtP1fT6MvuTBKQvHUOi44UNzzoHoVNp688PYpUx4=FaGn_Hw@mail.gmail.com>
Message-ID: <CAOXZevC72sbpu5DQ_w8s07-qVEsMeRYK6w_9xsLdra=DvujPZQ@mail.gmail.com>

Hi Zheng,

When you're writing stuff to disk you are increasing pressure on the VM.
The VM might try to do something smart and decide it needs a lot more
memory for caching the log files you're writing to. That might starve
Varnish.

I would run it again and have Munin or something similar recording what is
happening so you'll know.

Per.

On Fri, Sep 28, 2012 at 5:31 PM, Zheng Liu <huozhe at gmail.com> wrote:

> Per,
>
> Thanks for the info. I remember I was seeing the ram usage by varnishncsa
> did not change a lot. I didn't take any screen shot, so I may remember
> wrong. Let me try it one more time and see what happened. Could it be the
> log dumping to disk can't keep up with the insertion rate, so varnish has
> to allocate more for the in-memory logging? Just a guess.
>
> thanks,
> Zheng
>
> On Fri, Sep 28, 2012 at 6:46 AM, Per Buer <perbu at varnish-software.com>wrote:
>
>> Hi,
>>
>> On Thu, Sep 27, 2012 at 5:47 PM, Zheng Liu <huozhe at gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I tried to use varnishncsa to generate log on disk, so I can have
>>> another daemon tail and analyze it. However, when I enable it, I see
>>> varnishd starts to use more and more memory. I am using varnish-3.0.3 on
>>> Ubuntu 11.10. The box has 190G and I configured varnish to use 160G. Its
>>> footprint has been stable at 180G for months. Then after I started
>>> varnishncsa, within 5 hours, all the memory was gone and it starts to use
>>> swap. I wonder if this is expected behavior, and we are suppose to allow
>>> more head room during logging with varnishncsa?
>>>
>>
>> There have been some reports of a varnishleak in varnishncsa - but in the
>> cases I've seen it's varnishncsa  itself which balloons up. Would that be
>> the case here aswell?
>>
>> --
>> Per Buer
>> Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
>> *Varnish makes websites fly!*
>> Whitepapers <http://www.varnish-software.com/whitepapers> | Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
>> Twitter <https://twitter.com/varnishsoftware>
>>
>>
>>
>


-- 
Per Buer
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121001/641bbb0a/attachment.html>

From nick.tailor at gmail.com  Mon Oct  1 15:52:28 2012
From: nick.tailor at gmail.com (nick tailor)
Date: Mon, 1 Oct 2012 08:52:28 -0700
Subject: memory issue when running varnishncsa
In-Reply-To: <CAOXZevDtzxUyxKAouAy2bur5TvbMGcq6xctzM5eU9pT6aov5Vw@mail.gmail.com>
References: <CAOCtP1d82eFiUXSksvZVMcTiDRANMhNd7hzL_q3n2DHu9=60zg@mail.gmail.com>
 <CAOXZevDtzxUyxKAouAy2bur5TvbMGcq6xctzM5eU9pT6aov5Vw@mail.gmail.com>
Message-ID: <CA+8iBq8w5xhFE7yUeR5yPa8KZR1aGMx9WmAXjw6L0qeTM5bCkw@mail.gmail.com>

Did you do any OS updates?

Cheers

Nick Tailor

On Fri, Sep 28, 2012 at 6:46 AM, Per Buer <perbu at varnish-software.com>wrote:

> Hi,
>
> On Thu, Sep 27, 2012 at 5:47 PM, Zheng Liu <huozhe at gmail.com> wrote:
>
>> Hi,
>>
>> I tried to use varnishncsa to generate log on disk, so I can have another
>> daemon tail and analyze it. However, when I enable it, I see varnishd
>> starts to use more and more memory. I am using varnish-3.0.3 on Ubuntu
>> 11.10. The box has 190G and I configured varnish to use 160G. Its footprint
>> has been stable at 180G for months. Then after I started varnishncsa,
>> within 5 hours, all the memory was gone and it starts to use swap. I wonder
>> if this is expected behavior, and we are suppose to allow more head room
>> during logging with varnishncsa?
>>
>
> There have been some reports of a varnishleak in varnishncsa - but in the
> cases I've seen it's varnishncsa  itself which balloons up. Would that be
> the case here aswell?
>
> --
> Per Buer
> Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
> *Varnish makes websites fly!*
> Whitepapers <http://www.varnish-software.com/whitepapers> | Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
> Twitter <https://twitter.com/varnishsoftware>
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121001/04b78099/attachment.html>

From vladimir.efros at gmail.com  Mon Oct  1 20:22:07 2012
From: vladimir.efros at gmail.com (Vladimir Efros)
Date: Mon, 1 Oct 2012 16:22:07 -0400
Subject: Varnish Security
Message-ID: <CAL=as-_8dOB3fo6nmmyJvKErxcDgxTeaASUJF4+BFD8psUS27A@mail.gmail.com>

Hi,

Varnish security *main.vcl* contains the following directory:
#clear all internal variables

include "/etc/varnish/security/build/variables.vcl";

and
#fallthrough: clear all internal variables on security.vcl_recv exit

include "/etc/varnish/security/build/variables.vcl";

but /etc/varnish/security/build/variables.vcl is not included into the git.
I commented it out, and it is working fine but where can I get
/etc/varnish/security/build/variables.vcl?

Thanks,

Vladimir
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121001/20ecbb97/attachment.html>

From kelvin1111111 at gmail.com  Tue Oct  2 02:10:29 2012
From: kelvin1111111 at gmail.com (Kelvin Loke)
Date: Tue, 2 Oct 2012 10:10:29 +0800
Subject: Upgrade Varnish will not replace the libraries
Message-ID: <CAAct57y=Ob43dgxwdQdHLF6PDXURjLiExQ8i_VAA-d-NWYER8g@mail.gmail.com>

Hi,

Just thought of sharing it to have some suggestion/enhancement on
Varnish upgrade.

Using Ubuntu 10.04 LTS 64-bit, with Varnish 3.0.2. When I tried to
upgrade Varnish to 3.0.3, It seems that it cannot upgrade the Varnish
shared libraries automatically.

# apt-get install varnish
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  varnish-doc
The following packages will be upgraded:
  varnish
1 upgraded, 0 newly installed, 0 to remove and 105 not upgraded.
Need to get 531kB of archives.
After this operation, 49.2kB of additional disk space will be used.
Get:1 http://repo.varnish-cache.org/ubuntu/ lucid/varnish-3.0 varnish
3.0.3-1~lucid [531kB]
Fetched 531kB in 3s (156kB/s)
(Reading database ... 46434 files and directories currently installed.)
Preparing to replace varnish 3.0.2-1~1lucid1 (using
.../varnish_3.0.3-1~lucid_amd64.deb) ...
 * Stopping HTTP accelerator varnishd                     [ OK ]
Unpacking replacement varnish ...
dpkg: warning: unable to delete old directory '/var/lib/varnish':
Directory not empty
Processing triggers for ureadahead ...
ureadahead will be reprofiled on next reboot
Processing triggers for man-db ...
Setting up varnish (3.0.3-1~lucid) ...
Installing new version of config file /etc/init.d/varnishncsa ...
Installing new version of config file /etc/default/varnishlog ...
Installing new version of config file /etc/default/varnishncsa ..
                                                              .

Configuration file `/etc/default/varnish'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : background this process to examine the situation
 The default action is to keep your current version.
*** varnish (Y/I/N/O/D/Z) [default=N] ?
 * Starting HTTP accelerator varnishd                     [ OK ]

# dpkg -l | grep varnish
ii  libvarnishapi1                  3.0.2-1~1lucid1
                                                                shared
libraries for Varnish
ii  varnish                         3.0.3-1~lucid

state of the art, high-performance web accel

Delete the /var/lib/varnish manually before the upgrade will do the
trick, but would be better to have it done in automated way :)



From tfheen at varnish-software.com  Tue Oct  2 09:13:49 2012
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Tue, 2 Oct 2012 11:13:49 +0200
Subject: Varnish sometimes delays requests for minutes
In-Reply-To: <1348013499-sup-8011@geror.local>
References: <1348013499-sup-8011@geror.local>
Message-ID: <20121002091348.GB21856@err.no>

> The hung requests only happen on pages that have expired caches, but not every
> request to that page while it is expired will trigger a hang.  That is, I have
> a difficult time reproducing it, and kinda have to just watch requests in the
> corner of my eye while going about other duties.
> 
> I am fairly certain that these hangs are not the fault of the backend, but not
> completely sure - mark me down at 95% certainty.

My bet would be on the backend, but it's hard to tell without
varnishlog.

Something like varnishlog -m 'ReqEnd:([\d.]+\s){4}\d{2,}\.' will get you
just requests that take more than 10s to complete, which should
hopefully be easier to read than the all the raw requests.

If you have trouble deciphering a varnishlog record, please post it here
and we can try to help you.

Cheers,
-- 
Tollef Fog Heen
Technical lead, Varnish Software
t: +47 21 98 92 64



From kacperw at gmail.com  Wed Oct  3 17:21:39 2012
From: kacperw at gmail.com (Kacper Wysocki)
Date: Wed, 3 Oct 2012 19:21:39 +0200
Subject: Varnish Security
In-Reply-To: <CAL=as-_8dOB3fo6nmmyJvKErxcDgxTeaASUJF4+BFD8psUS27A@mail.gmail.com>
References: <CAL=as-_8dOB3fo6nmmyJvKErxcDgxTeaASUJF4+BFD8psUS27A@mail.gmail.com>
Message-ID: <CABaBnj6ij0on_Gc9J=LMbGtNzt6F=mTM641mD5PRsA30DzX=WQ@mail.gmail.com>

On Mon, Oct 1, 2012 at 10:22 PM, Vladimir Efros
<vladimir.efros at gmail.com> wrote:

> but /etc/varnish/security/build/variables.vcl is not included into the git.
> I commented it out, and it is working fine but where can I get
> /etc/varnish/security/build/variables.vcl?

run 'make' in the vcl/ directory to build the variables.vcl file.



From sahmed1020 at gmail.com  Wed Oct  3 19:31:05 2012
From: sahmed1020 at gmail.com (S Ahmed)
Date: Wed, 3 Oct 2012 15:31:05 -0400
Subject: invalidating (ban) a cache problematically, real life maximum?
Message-ID: <CAG2rwuNDxvHWFXC_Y3sn2hEd6y9=tSHJqsqU+3uRhJhSdgkphg@mail.gmail.com>

I was reading:
https://www.varnish-software.com/static/book/Cache_invalidation.html

My main question is, what sort of # of ban requests can one realistically
send to varnish to invalid caches? (individual items, not ranges etc.)

Say a site like tumblr is using varnish, which has thousands and thousands
of blogs, and so many blog entries are being updated via comments etc.  How
realistic would sending 1K ban's be to varnish?

Maybe I have misunderstood this, but I'm guessing their are ruby/java
client libraries that when content gets updated in a web application, I
could then send a request using the ruby/java client library to ban a
specific item from cache right?

If there are client libraries, where can they be found?  I'm looking for
both ruby and java.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121003/fd0729f9/attachment.html>

From vstavrinov at gmail.com  Thu Oct  4 12:06:22 2012
From: vstavrinov at gmail.com (Vladimir Stavrinov)
Date: Thu, 4 Oct 2012 16:06:22 +0400
Subject: fetch & delivery simutaneously
Message-ID: <20121004120622.GA6393@mana.playfast.ru>


Hi all

I am using squid for caching big (1Gb and more) files. Now I am
considering of migration to varnish. But I don't see there the feature
that squid has by default: if the file is missing in cache it starts
fetching it from backend and delivering to client at the same time. So
client don't wait at all and begins receiving the file immediately. But
with varnish client should wait until file have come into cache
completely. For big files it is not acceptable. In most cases client
closes connection becouse of timeout or by users action.

My question is: are there configuration solutions of this problem, or
this feature is not implemented yet at all?

-- 

***************************
##  Vladimir Stavrinov
##  vstavrinov at gmail.com
***************************




From shib4u at gmail.com  Thu Oct  4 12:16:37 2012
From: shib4u at gmail.com (Shibashish)
Date: Thu, 4 Oct 2012 17:46:37 +0530
Subject: fetch & delivery simutaneously
In-Reply-To: <20121004120622.GA6393@mana.playfast.ru>
References: <20121004120622.GA6393@mana.playfast.ru>
Message-ID: <CAOV_DHj6jvF0ZyftSJDHd5AaBi-vkHQUUeuWR+20PuOQc_GR1Q@mail.gmail.com>

On Thu, Oct 4, 2012 at 5:36 PM, Vladimir Stavrinov <vstavrinov at gmail.com>wrote:

>
> Hi all
>
> I am using squid for caching big (1Gb and more) files. Now I am
> considering of migration to varnish. But I don't see there the feature
> that squid has by default: if the file is missing in cache it starts
> fetching it from backend and delivering to client at the same time. So
> client don't wait at all and begins receiving the file immediately. But
> with varnish client should wait until file have come into cache
> completely. For big files it is not acceptable. In most cases client
> closes connection becouse of timeout or by users action.
>
> My question is: are there configuration solutions of this problem, or
> this feature is not implemented yet at all?
>
> --
>
> ***************************
> ##  Vladimir Stavrinov
> ##  vstavrinov at gmail.com
> ***************************
>
>
Streaming is available in varnish 3.0
https://www.varnish-software.com/blog/streaming-varnish-30
<https://www.varnish-software.com/blog/streaming-varnish-30>


ShiB.
while ( ! ( succeed = try() ) );
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121004/89a36080/attachment.html>

From vstavrinov at gmail.com  Thu Oct  4 13:40:24 2012
From: vstavrinov at gmail.com (Vladimir Stavrinov)
Date: Thu, 4 Oct 2012 17:40:24 +0400
Subject: fetch & delivery simutaneously
In-Reply-To: <CAOV_DHj6jvF0ZyftSJDHd5AaBi-vkHQUUeuWR+20PuOQc_GR1Q@mail.gmail.com>
References: <20121004120622.GA6393@mana.playfast.ru>
 <CAOV_DHj6jvF0ZyftSJDHd5AaBi-vkHQUUeuWR+20PuOQc_GR1Q@mail.gmail.com>
Message-ID: <20121004134024.GB6393@mana.playfast.ru>

On Thu, Oct 04, 2012 at 05:46:37PM +0530, Shibashish wrote:

>    Streaming is available in varnish 3.0
>    https://www.varnish-software.com/blog/streaming-varnish-30

Thanks. But this way it can serve single client only. There is no such
restriction for squid. And why this should be done with special
configuration? As I already mention, the squid do this by default, more
over, there are no way to switch this feature off. And this is reasonable
solution, because I don't see any case where we need to make client
hold on, while requested data are fetching from backend.

-- 

***************************
##  Vladimir Stavrinov
##  vstavrinov at gmail.com
***************************




From vstavrinov at gmail.com  Thu Oct  4 14:55:54 2012
From: vstavrinov at gmail.com (Vladimir Stavrinov)
Date: Thu, 4 Oct 2012 18:55:54 +0400
Subject: fetch & delivery simutaneously
In-Reply-To: <20121004134024.GB6393@mana.playfast.ru>
References: <20121004120622.GA6393@mana.playfast.ru>
 <CAOV_DHj6jvF0ZyftSJDHd5AaBi-vkHQUUeuWR+20PuOQc_GR1Q@mail.gmail.com>
 <20121004134024.GB6393@mana.playfast.ru>
Message-ID: <20121004145554.GC6393@mana.playfast.ru>

On Thu, Oct 04, 2012 at 05:40:24PM +0400, Vladimir Stavrinov wrote:

> restriction for squid. And why this should be done with special
> configuration? As I already mention, the squid do this by default, more

I see, it is because of second limitation: the fetch are going on with
the speed of client. But this means, this feature in it's current state
is useless. It is obvious we prefer to hold on all client, but fetch
with maximum speed, rather than force all but one clients to wait much
more, while one client completes downloading on low speed.

-- 

***************************
##  Vladimir Stavrinov
##  vstavrinov at gmail.com
***************************




From vstavrinov at gmail.com  Thu Oct  4 15:14:33 2012
From: vstavrinov at gmail.com (Vladimir Stavrinov)
Date: Thu, 4 Oct 2012 19:14:33 +0400
Subject: fetch & delivery simutaneously
In-Reply-To: <20121004145554.GC6393@mana.playfast.ru>
References: <20121004120622.GA6393@mana.playfast.ru>
 <CAOV_DHj6jvF0ZyftSJDHd5AaBi-vkHQUUeuWR+20PuOQc_GR1Q@mail.gmail.com>
 <20121004134024.GB6393@mana.playfast.ru>
 <20121004145554.GC6393@mana.playfast.ru>
Message-ID: <20121004151433.GE6393@mana.playfast.ru>

On Thu, Oct 04, 2012 at 06:55:54PM +0400, Vladimir Stavrinov wrote:

> I see, it is because of second limitation: the fetch are going on with
> the speed of client. But this means, this feature in it's current state
> is useless. It is obvious we prefer to hold on all client, but fetch
> with maximum speed, rather than force all but one clients to wait much
> more, while one client completes downloading on low speed.

And finally, right solution is here:

http://repo.varnish-cache.org/test/3.0.2+streaming/deb/varnish_3.0.2+streaming2-1lucid1_amd64.deb

With this package nobody will wait.

-- 

***************************
##  Vladimir Stavrinov
##  vstavrinov at gmail.com
***************************




From vstavrinov at gmail.com  Thu Oct  4 14:55:54 2012
From: vstavrinov at gmail.com (Vladimir Stavrinov)
Date: Thu, 4 Oct 2012 18:55:54 +0400
Subject: fetch & delivery simutaneously
In-Reply-To: <20121004134024.GB6393@mana.playfast.ru>
References: <20121004120622.GA6393@mana.playfast.ru>
 <CAOV_DHj6jvF0ZyftSJDHd5AaBi-vkHQUUeuWR+20PuOQc_GR1Q@mail.gmail.com>
 <20121004134024.GB6393@mana.playfast.ru>
Message-ID: <20121004145554.GC6393@mana.playfast.ru>

On Thu, Oct 04, 2012 at 05:40:24PM +0400, Vladimir Stavrinov wrote:

> restriction for squid. And why this should be done with special
> configuration? As I already mention, the squid do this by default, more

I see, it is because of second limitation: the fetch are going on with
the speed of client. But this means, this feature in it's current state
is useless. It is obvious we prefer to hold on all client, but fetch
with maximum speed, rather than force all but one clients to wait much
more, while one client completes downloading on low speed.

-- 

***************************
##  Vladimir Stavrinov
##  vstavrinov at gmail.com
***************************




From g.kaimlo at gmail.com  Thu Oct  4 15:12:22 2012
From: g.kaimlo at gmail.com (Kaim)
Date: Thu, 04 Oct 2012 17:12:22 +0200
Subject: fetch & delivery simutaneously
In-Reply-To: <20121004151433.GE6393@mana.playfast.ru>
References: <20121004120622.GA6393@mana.playfast.ru>
 <CAOV_DHj6jvF0ZyftSJDHd5AaBi-vkHQUUeuWR+20PuOQc_GR1Q@mail.gmail.com>
 <20121004134024.GB6393@mana.playfast.ru>
 <20121004145554.GC6393@mana.playfast.ru>
 <20121004151433.GE6393@mana.playfast.ru>
Message-ID: <506DA756.70008@gmail.com>

On 10/04/2012 05:14 PM, Vladimir Stavrinov wrote:
> On Thu, Oct 04, 2012 at 06:55:54PM +0400, Vladimir Stavrinov wrote:
>
>> I see, it is because of second limitation: the fetch are going on with
>> the speed of client. But this means, this feature in it's current state
>> is useless. It is obvious we prefer to hold on all client, but fetch
>> with maximum speed, rather than force all but one clients to wait much
>> more, while one client completes downloading on low speed.
>
> And finally, right solution is here:
>
> http://repo.varnish-cache.org/test/3.0.2+streaming/deb/varnish_3.0.2+streaming2-1lucid1_amd64.deb
>
> With this package nobody will wait.
>

You have git repo for 3.0.3 version :
https://github.com/mbgrydeland/varnish-cache-streaming/tree/streaming-3.0.3



From vstavrinov at gmail.com  Thu Oct  4 17:27:00 2012
From: vstavrinov at gmail.com (Vladimir Stavrinov)
Date: Thu, 4 Oct 2012 21:27:00 +0400
Subject: fetch & delivery simutaneously
In-Reply-To: <506DA756.70008@gmail.com>
References: <20121004120622.GA6393@mana.playfast.ru>
 <CAOV_DHj6jvF0ZyftSJDHd5AaBi-vkHQUUeuWR+20PuOQc_GR1Q@mail.gmail.com>
 <20121004134024.GB6393@mana.playfast.ru>
 <20121004145554.GC6393@mana.playfast.ru>
 <20121004151433.GE6393@mana.playfast.ru> <506DA756.70008@gmail.com>
Message-ID: <20121004172700.GA4007@terra.00>

On Thu, Oct 04, 2012 at 05:12:22PM +0200, Kaim wrote:

> You have git repo for 3.0.3 version :
> https://github.com/mbgrydeland/varnish-cache-streaming/tree/streaming-3.0.3

Is this special branch for full streaming support?. Does this means
there will no such support in regular 3.0.3 release? First, I have
installed 3.0.2 from Debian official repository, but it have partial
only support for streaming. Then I replaced it with mentioned above
3.0.2+streaming2-1lucid1 with full streaming support. So, I think it is
special branch for full support. When full streaming will come with
regular release?



***************************
###  Vladimir Stavrinov
###  vstavrinov at gmail.com
***************************




From hugo.cisneiros at gmail.com  Thu Oct  4 17:38:48 2012
From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch))
Date: Thu, 4 Oct 2012 14:38:48 -0300
Subject: fetch & delivery simutaneously
In-Reply-To: <20121004172700.GA4007@terra.00>
References: <20121004120622.GA6393@mana.playfast.ru>
 <CAOV_DHj6jvF0ZyftSJDHd5AaBi-vkHQUUeuWR+20PuOQc_GR1Q@mail.gmail.com>
 <20121004134024.GB6393@mana.playfast.ru>
 <20121004145554.GC6393@mana.playfast.ru>
 <20121004151433.GE6393@mana.playfast.ru> <506DA756.70008@gmail.com>
 <20121004172700.GA4007@terra.00>
Message-ID: <CA+KACLmJ5CNv=0U+uDUrUsDZ_Jr_bHhv5-o=QEqwpUUx5AZL0w@mail.gmail.com>

On Thu, Oct 4, 2012 at 2:27 PM, Vladimir Stavrinov <vstavrinov at gmail.com> wrote:
> On Thu, Oct 04, 2012 at 05:12:22PM +0200, Kaim wrote:
>
>> You have git repo for 3.0.3 version :
>> https://github.com/mbgrydeland/varnish-cache-streaming/tree/streaming-3.0.3
>
> Is this special branch for full streaming support?. Does this means
> there will no such support in regular 3.0.3 release? First, I have
> installed 3.0.2 from Debian official repository, but it have partial
> only support for streaming. Then I replaced it with mentioned above
> 3.0.2+streaming2-1lucid1 with full streaming support. So, I think it is
> special branch for full support. When full streaming will come with
> regular release?

It is now a separate branch, but may be incorporated in the future on
the official release branch (as some e-mails and pages from
varnish-cache says, maybe 3.1). Give it a try, it is pretty good work.
I began using for a few days and it's working very well.

-- 
[]'s
Hugo
www.devin.com.br



From aanchal.gera at gmail.com  Fri Oct  5 12:30:05 2012
From: aanchal.gera at gmail.com (Aanchal Gera)
Date: Fri, 5 Oct 2012 18:00:05 +0530
Subject: Stats of Hit/Miss for a particular URL
In-Reply-To: <CAFf4ghYN-upo0EVqXH8MQjKiOiGnjoZW3fa=A5GdA6v7iSeRdw@mail.gmail.com>
References: <CAFf4ghYN-upo0EVqXH8MQjKiOiGnjoZW3fa=A5GdA6v7iSeRdw@mail.gmail.com>
Message-ID: <CAFf4ghZkVbJf7d-FK5QdOg67Q2RpvrDpue=9OmgZXtXmJ8p1Tw@mail.gmail.com>

Hi All

I am using ESI Tags with varnish.

How can I check with varnishstat/varnishlog if a particular URL ever hits
the cache. And it if it does, how can I see the Hit/Miss ratio.

Regards,
Aanchal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121005/97e019ec/attachment.html>

From themadindian at yahoo.com  Fri Oct  5 12:45:21 2012
From: themadindian at yahoo.com (Mad Indian)
Date: Fri, 5 Oct 2012 05:45:21 -0700 (PDT)
Subject: No subject
Message-ID: <1349441121.10239.YahooMailNeo@web163906.mail.gq1.yahoo.com>

Hi, I just upgraded to 3.0.3 from 3.0.2 and now on some of my pages that 301 are now causing 503s, I'm getting this error in the varnishlog

4 Gzip ? ? ? ? c u F - 0 0 0 0 0
? ? 4 FetchError ? c TestGunzip error at the very end
? ? 4 VCL_call ? ? c error deliver
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121005/fee583b1/attachment.html>

From tfheen at varnish-software.com  Sat Oct  6 11:40:11 2012
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Sat, 6 Oct 2012 13:40:11 +0200
Subject: Stats of Hit/Miss for a particular URL
In-Reply-To: <CAFf4ghZkVbJf7d-FK5QdOg67Q2RpvrDpue=9OmgZXtXmJ8p1Tw@mail.gmail.com>
References: <CAFf4ghYN-upo0EVqXH8MQjKiOiGnjoZW3fa=A5GdA6v7iSeRdw@mail.gmail.com>
 <CAFf4ghZkVbJf7d-FK5QdOg67Q2RpvrDpue=9OmgZXtXmJ8p1Tw@mail.gmail.com>
Message-ID: <20121006114011.GA8504@err.no>

]] Aanchal Gera

> How can I check with varnishstat/varnishlog if a particular URL ever hits
> the cache. And it if it does, how can I see the Hit/Miss ratio.

varnishlog -m RxURL:/url will show you the log records for that url.
Hit/miss ratio isn't calculated for you, you need to keep track of that
yourself.

Cheers,
-- 
Tollef Fog Heen
Technical lead, Varnish Software
t: +47 21 98 92 64



From samirdevaliya at gmail.com  Mon Oct  8 06:54:09 2012
From: samirdevaliya at gmail.com (samir devaliya)
Date: Mon, 8 Oct 2012 12:24:09 +0530
Subject: Installation Prerequisites related query
Message-ID: <CABfdyPYCRAXjLVNWTNyVUdQ0poNc1Pv_htKp3en_MV8RWYM2Og@mail.gmail.com>

Hi,

Is it Possible to install varnish on debian 3.1 (sarge)

if yes kindly provide the version of varnish which can be installed

Thanks & Regards
Samir Devaliya
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121008/6c9d6a8f/attachment.html>

From aanchal.gera at gmail.com  Mon Oct  8 08:33:55 2012
From: aanchal.gera at gmail.com (Aanchal Gera)
Date: Mon, 8 Oct 2012 14:03:55 +0530
Subject: No-cache but still a HIT
Message-ID: <CAFf4ghZU=dLk972RDWz0ve_Ktv4X6ivm6EORc-0gWk0sZYRq1w@mail.gmail.com>

Hi All

I have set the Cache-Control directives as "no-cache" for an ESI fragment.
Even then, I receive a HIT from Varnish. How is this possible?


   1. *ResponseHeaders:-
   Accept-Ranges:* bytes
   2. Age:
   16
   3. cache-control:
   no-cache, private
   4. clientip:
   112.196.23.118
   5. Connection:
   keep-alive
   6. Content-Encoding:
   gzip
   7. Content-Length:
   20
   8. Content-Type:
   text/html; charset=UTF-8
   9. Date:
   Mon, 08 Oct 2012 08:26:14 GMT
   10. Server:
   Apache
   11. Vary:
   Accept-Encoding
   12. Via:
   1.1 varnish
   13. X-Cache:
   HIT
   14. X-Varnish:
   1454051837 1454051829


Regards,
Aanchal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121008/ff235668/attachment.html>

From apj at mutt.dk  Mon Oct  8 08:38:31 2012
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Mon, 8 Oct 2012 10:38:31 +0200
Subject: No-cache but still a HIT
In-Reply-To: <CAFf4ghZU=dLk972RDWz0ve_Ktv4X6ivm6EORc-0gWk0sZYRq1w@mail.gmail.com>
References: <CAFf4ghZU=dLk972RDWz0ve_Ktv4X6ivm6EORc-0gWk0sZYRq1w@mail.gmail.com>
Message-ID: <20121008083831.GK4805@nerd.dk>

On Mon, Oct 08, 2012 at 02:03:55PM +0530, Aanchal Gera wrote:
> 
> I have set the Cache-Control directives as "no-cache" for an ESI fragment.
> Even then, I receive a HIT from Varnish. How is this possible?

You'll have to implement no-cache in VCL yourself, it's not honored by default.

-- 
Andreas



From aanchal.gera at gmail.com  Mon Oct  8 08:45:32 2012
From: aanchal.gera at gmail.com (Aanchal Gera)
Date: Mon, 8 Oct 2012 14:15:32 +0530
Subject: No-cache but still a HIT
In-Reply-To: <20121008083831.GK4805@nerd.dk>
References: <CAFf4ghZU=dLk972RDWz0ve_Ktv4X6ivm6EORc-0gWk0sZYRq1w@mail.gmail.com>
 <20121008083831.GK4805@nerd.dk>
Message-ID: <CAFf4ghYxNbv3j6_ER7qZDG-CE59sABLGdgnW4JzK5_q=Ybxn=g@mail.gmail.com>

Ohh, Is it!!!

So, what would be the best way to tell Varnish that I don't want to cache
an ESI fragment?

Regards,
Aanchal

On Mon, Oct 8, 2012 at 2:08 PM, Andreas Plesner Jacobsen <apj at mutt.dk>wrote:

> On Mon, Oct 08, 2012 at 02:03:55PM +0530, Aanchal Gera wrote:
> >
> > I have set the Cache-Control directives as "no-cache" for an ESI
> fragment.
> > Even then, I receive a HIT from Varnish. How is this possible?
>
> You'll have to implement no-cache in VCL yourself, it's not honored by
> default.
>
> --
> Andreas
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121008/1ad9e26e/attachment.html>

From rowan at jetboy.co.uk  Mon Oct  8 10:12:00 2012
From: rowan at jetboy.co.uk (Rowan @ Jetboy)
Date: Mon, 8 Oct 2012 11:12:00 +0100
Subject: Error 413 Request Entity Too Large
In-Reply-To: <CABfdyPYCRAXjLVNWTNyVUdQ0poNc1Pv_htKp3en_MV8RWYM2Og@mail.gmail.com>
References: <CABfdyPYCRAXjLVNWTNyVUdQ0poNc1Pv_htKp3en_MV8RWYM2Og@mail.gmail.com>
Message-ID: <002001cda53d$5b81e540$1285afc0$@co.uk>

I've had a handful of complaints from my website users that they're seeing
the error below when trying to access an installation of the MyBB forum.

 

Error 413

Request Entity Too Large

Guru Meditation:

XID: 190197240

Varnish cache server

 

 

I spoke to one customer who was experiencing this error using IE8/XP (which
I couldn't replicate on various IE8/XP PCs), and we tried restarting Varnish
3 with:

 

-p http_req_hdr_len=8192 \

-p http_resp_hdr_len=8192 \

 

in the startup DAEMON OPTS. Unfortunately it didn't clear the error for her.
Is there anything else I should be trying?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121008/b021eba8/attachment.html>

From omegabk at gmail.com  Mon Oct  8 10:20:27 2012
From: omegabk at gmail.com (Omega BK)
Date: Mon, 8 Oct 2012 12:20:27 +0200
Subject: No-cache but still a HIT
In-Reply-To: <CAFf4ghYxNbv3j6_ER7qZDG-CE59sABLGdgnW4JzK5_q=Ybxn=g@mail.gmail.com>
References: <CAFf4ghZU=dLk972RDWz0ve_Ktv4X6ivm6EORc-0gWk0sZYRq1w@mail.gmail.com>
 <20121008083831.GK4805@nerd.dk>
 <CAFf4ghYxNbv3j6_ER7qZDG-CE59sABLGdgnW4JzK5_q=Ybxn=g@mail.gmail.com>
Message-ID: <50522E9D-771F-48C5-8727-FCE42B55B535@gmail.com>

Varnish takes care of the s-maxage, you have to send it on your header to specify the duration of the TTL

Best.


Sent from my iPhone

On 8 oct. 2012, at 10:45, Aanchal Gera <aanchal.gera at gmail.com> wrote:

> Ohh, Is it!!!
> 
> So, what would be the best way to tell Varnish that I don't want to cache an ESI fragment?
> 
> Regards,
> Aanchal
> 
> On Mon, Oct 8, 2012 at 2:08 PM, Andreas Plesner Jacobsen <apj at mutt.dk> wrote:
>> On Mon, Oct 08, 2012 at 02:03:55PM +0530, Aanchal Gera wrote:
>> >
>> > I have set the Cache-Control directives as "no-cache" for an ESI fragment.
>> > Even then, I receive a HIT from Varnish. How is this possible?
>> 
>> You'll have to implement no-cache in VCL yourself, it's not honored by default.
>> 
>> --
>> Andreas
>> 
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121008/042d01aa/attachment.html>

From aanchal.gera at gmail.com  Mon Oct  8 10:24:01 2012
From: aanchal.gera at gmail.com (Aanchal Gera)
Date: Mon, 8 Oct 2012 15:54:01 +0530
Subject: No-cache but still a HIT
In-Reply-To: <50522E9D-771F-48C5-8727-FCE42B55B535@gmail.com>
References: <CAFf4ghZU=dLk972RDWz0ve_Ktv4X6ivm6EORc-0gWk0sZYRq1w@mail.gmail.com>
 <20121008083831.GK4805@nerd.dk>
 <CAFf4ghYxNbv3j6_ER7qZDG-CE59sABLGdgnW4JzK5_q=Ybxn=g@mail.gmail.com>
 <50522E9D-771F-48C5-8727-FCE42B55B535@gmail.com>
Message-ID: <CAFf4ghab0AcmDttGL=F95Tbrcav_OsP6krPpvaMFPxR1DD+tpA@mail.gmail.com>

Before trying no-cache header, I used s-maxage=0. Even then, I had a HIT at
times :(

On Mon, Oct 8, 2012 at 3:50 PM, Omega BK <omegabk at gmail.com> wrote:

> Varnish takes care of the s-maxage, you have to send it on your header to
> specify the duration of the TTL
>
> Best.
>
>
> Sent from my iPhone
>
> On 8 oct. 2012, at 10:45, Aanchal Gera <aanchal.gera at gmail.com> wrote:
>
> Ohh, Is it!!!
>
> So, what would be the best way to tell Varnish that I don't want to cache
> an ESI fragment?
>
> Regards,
> Aanchal
>
> On Mon, Oct 8, 2012 at 2:08 PM, Andreas Plesner Jacobsen <apj at mutt.dk>wrote:
>
>> On Mon, Oct 08, 2012 at 02:03:55PM +0530, Aanchal Gera wrote:
>> >
>> > I have set the Cache-Control directives as "no-cache" for an ESI
>> fragment.
>> > Even then, I receive a HIT from Varnish. How is this possible?
>>
>> You'll have to implement no-cache in VCL yourself, it's not honored by
>> default.
>>
>> --
>> Andreas
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121008/dabbd7da/attachment.html>

From hugo.cisneiros at gmail.com  Mon Oct  8 23:28:03 2012
From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch))
Date: Mon, 8 Oct 2012 20:28:03 -0300
Subject: No-cache but still a HIT
In-Reply-To: <CAFf4ghYxNbv3j6_ER7qZDG-CE59sABLGdgnW4JzK5_q=Ybxn=g@mail.gmail.com>
References: <CAFf4ghZU=dLk972RDWz0ve_Ktv4X6ivm6EORc-0gWk0sZYRq1w@mail.gmail.com>
 <20121008083831.GK4805@nerd.dk>
 <CAFf4ghYxNbv3j6_ER7qZDG-CE59sABLGdgnW4JzK5_q=Ybxn=g@mail.gmail.com>
Message-ID: <CA+KACL=CQQf6g7U67G_-PT+ENe2RO4VjZMPcHBZg-HdXWTym3g@mail.gmail.com>

On Mon, Oct 8, 2012 at 5:45 AM, Aanchal Gera <aanchal.gera at gmail.com> wrote:
> Ohh, Is it!!!
>
> So, what would be the best way to tell Varnish that I don't want to cache an
> ESI fragment?

In your case, something like this on vcl_recv:

if (req.http.cache-control ~ "no") { return (pass); }

If you have other pages that returns the http header cache-control:
no, you have to be more specific at the if statement (req.url or
something like that).

-- 
[]'s
Hugo
www.devin.com.br



From aanchal.gera at gmail.com  Tue Oct  9 07:02:03 2012
From: aanchal.gera at gmail.com (Aanchal Gera)
Date: Tue, 9 Oct 2012 12:32:03 +0530
Subject: No-cache but still a HIT
In-Reply-To: <CA+KACL=CQQf6g7U67G_-PT+ENe2RO4VjZMPcHBZg-HdXWTym3g@mail.gmail.com>
References: <CAFf4ghZU=dLk972RDWz0ve_Ktv4X6ivm6EORc-0gWk0sZYRq1w@mail.gmail.com>
 <20121008083831.GK4805@nerd.dk>
 <CAFf4ghYxNbv3j6_ER7qZDG-CE59sABLGdgnW4JzK5_q=Ybxn=g@mail.gmail.com>
 <CA+KACL=CQQf6g7U67G_-PT+ENe2RO4VjZMPcHBZg-HdXWTym3g@mail.gmail.com>
Message-ID: <CAFf4ghb4w9aLC9_fYV7fh22LeTa6H21NpcM8OtP-drZ+NBFFvg@mail.gmail.com>

I am sorry, but I am very confused here. no-cache is an attribute of
response. Why should I check it in req?

Anyway, this should not be so complicated. My simple requirement is that I
don't want varnish to cache my esi URL. Why can't I simply set something in
the response headers which tells varnish not to store it in cache.

I can write specific statements in varnish for that particular url, but I
don't want to do that as there will be more such urls with time. This logic
should be handled in my code, not in varnish.

On Tue, Oct 9, 2012 at 4:58 AM, Hugo Cisneiros (Eitch) <
hugo.cisneiros at gmail.com> wrote:

> On Mon, Oct 8, 2012 at 5:45 AM, Aanchal Gera <aanchal.gera at gmail.com>
> wrote:
> > Ohh, Is it!!!
> >
> > So, what would be the best way to tell Varnish that I don't want to
> cache an
> > ESI fragment?
>
> In your case, something like this on vcl_recv:
>
> if (req.http.cache-control ~ "no") { return (pass); }
>
> If you have other pages that returns the http header cache-control:
> no, you have to be more specific at the if statement (req.url or
> something like that).
>
> --
> []'s
> Hugo
> www.devin.com.br
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121009/74e9f10a/attachment.html>

From apj at mutt.dk  Tue Oct  9 07:10:12 2012
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Tue, 9 Oct 2012 09:10:12 +0200
Subject: No-cache but still a HIT
In-Reply-To: <CAFf4ghb4w9aLC9_fYV7fh22LeTa6H21NpcM8OtP-drZ+NBFFvg@mail.gmail.com>
References: <CAFf4ghZU=dLk972RDWz0ve_Ktv4X6ivm6EORc-0gWk0sZYRq1w@mail.gmail.com>
 <20121008083831.GK4805@nerd.dk>
 <CAFf4ghYxNbv3j6_ER7qZDG-CE59sABLGdgnW4JzK5_q=Ybxn=g@mail.gmail.com>
 <CA+KACL=CQQf6g7U67G_-PT+ENe2RO4VjZMPcHBZg-HdXWTym3g@mail.gmail.com>
 <CAFf4ghb4w9aLC9_fYV7fh22LeTa6H21NpcM8OtP-drZ+NBFFvg@mail.gmail.com>
Message-ID: <20121009071012.GM4805@nerd.dk>

On Tue, Oct 09, 2012 at 12:32:03PM +0530, Aanchal Gera wrote:

> I am sorry, but I am very confused here. no-cache is an attribute of
> response. Why should I check it in req?

You shouldn't. You should inspect beresp.http.Cache-Control in vcl_fetch.

> Anyway, this should not be so complicated. My simple requirement is that I
> don't want varnish to cache my esi URL. Why can't I simply set something in
> the response headers which tells varnish not to store it in cache.

You can. Varnish will set TTL based on max-age, s-maxage or Expires by default
(but you're free to override it in your VCL). For any other header you will
have to write the VCL to do that.

> I can write specific statements in varnish for that particular url, but I
> don't want to do that as there will be more such urls with time. This logic
> should be handled in my code, not in varnish.

VCL is a powerful configuration language that allows you to specify your own
policies. Perhaps you should read the documentation provided with Varnish.

-- 
Andreas



From lafraia at gmail.com  Tue Oct  9 15:08:19 2012
From: lafraia at gmail.com (Daniel Lafraia)
Date: Tue, 9 Oct 2012 12:08:19 -0300
Subject: vcl_error check backend
In-Reply-To: <CABHsmAApV+WTSnrokaWwMKEE8UUPQ_-8TBfYFAMA-3dCXb0uiw@mail.gmail.com>
References: <CABHsmAB2-PK2+r7Utk-gkiJwVOnR0JvgoV14NFGhEtRBbMxV+g@mail.gmail.com>
 <CABHsmAApV+WTSnrokaWwMKEE8UUPQ_-8TBfYFAMA-3dCXb0uiw@mail.gmail.com>
Message-ID: <CABHsmABb4K-NFtphV5r6d8EoWszxiv_aZRatpNoBw5GGgCkbKg@mail.gmail.com>

Is there a way to check which backend was used under vcl_error()?

Something like:

sub vcl_error {
     if (obj.backend.host == "backend1") {
          synthetic "error on backend1";
     } elsif (obj.backend.director == "dir1") {
          synthetic "error on director dir1";
     } else {
          synthetic "";
     }
}

thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121009/c4104186/attachment.html>

From james at ifixit.com  Tue Oct  9 19:03:49 2012
From: james at ifixit.com (James Pearson)
Date: Tue, 09 Oct 2012 12:03:49 -0700
Subject: invalidating (ban) a cache problematically, real life maximum?
In-Reply-To: <CAG2rwuNDxvHWFXC_Y3sn2hEd6y9=tSHJqsqU+3uRhJhSdgkphg@mail.gmail.com>
References: <CAG2rwuNDxvHWFXC_Y3sn2hEd6y9=tSHJqsqU+3uRhJhSdgkphg@mail.gmail.com>
Message-ID: <1349809075-sup-1801@geror.local>

Excerpts from S Ahmed's message of 2012-10-03 12:31:05 -0700:
> Maybe I have misunderstood this, but I'm guessing their are ruby/java
> client libraries that when content gets updated in a web application, I
> could then send a request using the ruby/java client library to ban a
> specific item from cache right?

I don't think there are /libraries/; you just use your favorite asynchronous
HTTP library to make a request to Varnish such that it will hit the VCL rules
that purge the object.  At the end of the page you linked to, there's an
example in PHP using the overly-verbose builtin curl library; even in Java it
shouldn't be that bad.  It'll pretty much be the same thing, though, just in
another language.
 - P




From ross at trademe.co.nz  Wed Oct 10 00:42:22 2012
From: ross at trademe.co.nz (Ross Brown)
Date: Wed, 10 Oct 2012 00:42:22 +0000
Subject: Backend selection based on integer value in request?
Message-ID: <6ED006BA952A6E4F802D61A21A5D4F0522F16C@TMMAIL1.trademe.local>

Hi all

We use Varnish for accelerating images. They are uploaded by our users and given a numeric ID, which is allocated sequentially. We have a lot of them.

I'm about to start storing newly-uploaded in a different place, and ideally would like to select a backend by doing a compare against the image ID, eg if the ID > a number, use a different backend. Otherwise the backend will have to check both storage locations to find the image.

I could do this with a regex, but ideally I'd like to do a proper integer comparison. I have read about the integer function in vmod_std, but I don't see how I use that when the integer is part of the request URL.
https://www.varnish-cache.org/docs/3.0/reference/vmod_std.html?highlight=vmod_std#integer

Request URLs look similar to this: /photoserver/size/1234567.jpg

Any ideas?




From pprocacci at datapipe.com  Wed Oct 10 00:58:40 2012
From: pprocacci at datapipe.com (Paul A. Procacci)
Date: Tue, 9 Oct 2012 19:58:40 -0500
Subject: Backend selection based on integer value in request?
In-Reply-To: <6ED006BA952A6E4F802D61A21A5D4F0522F16C@TMMAIL1.trademe.local>
References: <6ED006BA952A6E4F802D61A21A5D4F0522F16C@TMMAIL1.trademe.local>
Message-ID: <20121010005840.GX56954@nat.myhome>

You probably want something like the following (untested):

######################################
if(req.url ~ "/\d+\.jpg$"){
  set req.http.X-Image-Int = req.url;
  regsub( req.http.X-Image-Int, "/(\d+)\.jpg$", "\1");
  if(std.integer(req.http.X-Image-Int, 0) > 10) {
    set req.backend = foo;
  }
  elseif(std.integer(req.http.X-Image-Int, 0) > 5) {
    set req.backend = foo2;
  }
  remove req.http.X-Image-Int;
}
######################################

~Paul

On Wed, Oct 10, 2012 at 12:42:22AM +0000, Ross Brown wrote:
> Hi all
>
> We use Varnish for accelerating images. They are uploaded by our users and given a numeric ID, which is allocated sequentially. We have a lot of them.
>
> I'm about to start storing newly-uploaded in a different place, and ideally would like to select a backend by doing a compare against the image ID, eg if the ID > a number, use a different backend. Otherwise the backend will have to check both storage locations to find the image.
>
> I could do this with a regex, but ideally I'd like to do a proper integer comparison. I have read about the integer function in vmod_std, but I don't see how I use that when the integer is part of the request URL.
> https://www.varnish-cache.org/docs/3.0/reference/vmod_std.html?highlight=vmod_std#integer
>
> Request URLs look similar to this: /photoserver/size/1234567.jpg
>
> Any ideas?
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

________________________________

This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you.



From pprocacci at datapipe.com  Wed Oct 10 01:43:21 2012
From: pprocacci at datapipe.com (Paul A. Procacci)
Date: Tue, 9 Oct 2012 20:43:21 -0500
Subject: Backend selection based on integer value in request?
In-Reply-To: <20121010005840.GX56954@nat.myhome>
References: <6ED006BA952A6E4F802D61A21A5D4F0522F16C@TMMAIL1.trademe.local>
 <20121010005840.GX56954@nat.myhome>
Message-ID: <20121010014321.GY56954@nat.myhome>

On Tue, Oct 09, 2012 at 07:58:40PM -0500, Paul A. Procacci wrote:
> You probably want something like the following (untested):
>
> ######################################
> if(req.url ~ "/\d+\.jpg$"){
>   set req.http.X-Image-Int = req.url;
>   regsub( req.http.X-Image-Int, "/(\d+)\.jpg$", "\1");
>   if(std.integer(req.http.X-Image-Int, 0) > 10) {
>     set req.backend = foo;
>   }
>   elseif(std.integer(req.http.X-Image-Int, 0) > 5) {
>     set req.backend = foo2;
>   }
>   remove req.http.X-Image-Int;
> }
> ######################################
>
> ~Paul

Apologies for the posting to myself.  My regex is poor.

>   regsub( req.http.X-Image-Int, "/(\d+)\.jpg$", "\1");

Probably should be:

regsub( req.http.X-Image-Int, ".*/(\d+)\.jpg$", "\1");

~Paul

________________________________

This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you.



From ross at trademe.co.nz  Wed Oct 10 02:45:54 2012
From: ross at trademe.co.nz (Ross Brown)
Date: Wed, 10 Oct 2012 02:45:54 +0000
Subject: Backend selection based on integer value in request?
In-Reply-To: <20121010014321.GY56954@nat.myhome>
References: <6ED006BA952A6E4F802D61A21A5D4F0522F16C@TMMAIL1.trademe.local>
 <20121010005840.GX56954@nat.myhome> <20121010014321.GY56954@nat.myhome>
Message-ID: <6ED006BA952A6E4F802D61A21A5D4F052301B1@TMMAIL1.trademe.local>

-----Original Message-----
From: Paul A. Procacci [mailto:pprocacci at datapipe.com] 
Sent: Wednesday, 10 October 2012 2:43 p.m.
To: Ross Brown
Cc: varnish-misc at varnish-cache.org
Subject: Re: Backend selection based on integer value in request?

On Tue, Oct 09, 2012 at 07:58:40PM -0500, Paul A. Procacci wrote:
> You probably want something like the following (untested):
>
> ######################################
> if(req.url ~ "/\d+\.jpg$"){
>   set req.http.X-Image-Int = req.url;
>   regsub( req.http.X-Image-Int, "/(\d+)\.jpg$", "\1");
>   if(std.integer(req.http.X-Image-Int, 0) > 10) {
>     set req.backend = foo;
>   }
>   elseif(std.integer(req.http.X-Image-Int, 0) > 5) {
>     set req.backend = foo2;
>   }
>   remove req.http.X-Image-Int;
> }
> ######################################
>
> ~Paul

Thanks Paul,  that got me on the right track.

// Set header containing image ID
   if (req.url ~ "^/photoserver/thumb/") {
      set req.http.X-Image-ID = regsub(req.url, "^/photoserver/thumb/(\d+)\.jpg$", "\1");
      std.log("ImageID: " + req.http.X-Image-ID);
   }

Then in my backend selection  -

...
   } else if (req.url ~ "^/photoserver/thumb") {
      if (std.integer(req.http.X-Image-ID, 0) > 1000) { // UPDATE WITH REAL VALUE
         set req.backend = b-new;
         std.log("Using new backend based on image ID: " + req.http.X-Image-ID);
      } else {
         set req.backend = b-old;
         std.log("Using old backend based on image ID: " + req.http.X-Image-ID);
      }

Not in Production yet, but it works like a charm in my dev environment :)



From pprocacci at datapipe.com  Wed Oct 10 06:08:41 2012
From: pprocacci at datapipe.com (Paul A. Procacci)
Date: Wed, 10 Oct 2012 01:08:41 -0500
Subject: Backend selection based on integer value in request?
In-Reply-To: <6ED006BA952A6E4F802D61A21A5D4F052301B1@TMMAIL1.trademe.local>
References: <6ED006BA952A6E4F802D61A21A5D4F0522F16C@TMMAIL1.trademe.local>
 <20121010005840.GX56954@nat.myhome>
 <20121010014321.GY56954@nat.myhome>
 <6ED006BA952A6E4F802D61A21A5D4F052301B1@TMMAIL1.trademe.local>
Message-ID: <20121010060841.GA56954@nat.myhome>

> Thanks Paul,  that got me on the right track.
>
> // Set header containing image ID
>    if (req.url ~ "^/photoserver/thumb/") {
>       set req.http.X-Image-ID = regsub(req.url, "^/photoserver/thumb/(\d+)\.jpg$", "\1");
>       std.log("ImageID: " + req.http.X-Image-ID);
>    }
>
> Then in my backend selection  -
>
> ...
>    } else if (req.url ~ "^/photoserver/thumb") {
>       if (std.integer(req.http.X-Image-ID, 0) > 1000) { // UPDATE WITH REAL VALUE
>          set req.backend = b-new;
>          std.log("Using new backend based on image ID: " + req.http.X-Image-ID);
>       } else {
>          set req.backend = b-old;
>          std.log("Using old backend based on image ID: " + req.http.X-Image-ID);
>       }
>
> Not in Production yet, but it works like a charm in my dev environment :)

Glad that was helpful.  Just be careful to ensure your req.url matches
what you expect it to match.

For instance:

'/photoserver/thumb/test.jpg'

The above matches your regex.  You probably want to refine your regex to
the following:

if(req.url ~ "^/photoserver/thumb/(?:\d+)\.jpg$"){
  set req.http.X-Image-ID = regsub(req.url, "^/photoserver/thumb/(\d+)\.jpg$", "\1");
  std.log("ImageID: " + req.http.X-Image-ID);
}

...

} else if (req.url ~ "^/photoserver/thumb/(?:\d+)\.jpg$") {
  if (std.integer(req.http.X-Image-ID, 0) > 1000) { // UPDATE WITH REAL VALUE
    set req.backend = b-new;
    std.log("Using new backend based on image ID: " + req.http.X-Image-ID);
  } else {
    set req.backend = b-old;
    std.log("Using old backend based on image ID: " + req.http.X-Image-ID);
  }
}

Naturally, you know your application best, so the above may not actually matter,
but I figured I would throw it out there just in case.

~Paul

________________________________

This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you.



From perbu at varnish-software.com  Wed Oct 10 12:59:04 2012
From: perbu at varnish-software.com (Per Buer)
Date: Wed, 10 Oct 2012 14:59:04 +0200
Subject: invalidating (ban) a cache problematically, real life maximum?
In-Reply-To: <CAG2rwuNDxvHWFXC_Y3sn2hEd6y9=tSHJqsqU+3uRhJhSdgkphg@mail.gmail.com>
References: <CAG2rwuNDxvHWFXC_Y3sn2hEd6y9=tSHJqsqU+3uRhJhSdgkphg@mail.gmail.com>
Message-ID: <CAOXZevCRZ-FyAh8+SQo-8FrwZti49LxBD=y6JZwYdY_Q88+q3g@mail.gmail.com>

Hi,

On Wed, Oct 3, 2012 at 9:31 PM, S Ahmed <sahmed1020 at gmail.com> wrote:

> I was reading:
> https://www.varnish-software.com/static/book/Cache_invalidation.html
>
> My main question is, what sort of # of ban requests can one realistically
> send to varnish to invalid caches? (individual items, not ranges etc.)
>

I would limit bans to maybe 10-30 per second, no more. Make sure they are
lurker-friendly.

Say a site like tumblr is using varnish, which has thousands and thousands
> of blogs, and so many blog entries are being updated via comments etc.  How
> realistic would sending 1K ban's be to varnish?
>

Sending 1k purges to Varnish would be no problem at all. Bans - I'm not
sure that is a good idea.


> Maybe I have misunderstood this, but I'm guessing their are ruby/java
> client libraries that when content gets updated in a web application, I
> could then send a request using the ruby/java client library to ban a
> specific item from cache right?
>
> If there are client libraries, where can they be found?  I'm looking for
> both ruby and java.
>

It might make sense to implement it yourself. It is dead simple and you
don't need the complexities of a generic library.

-- 
Per Buer
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121010/583b4079/attachment.html>

From jhalfmoon at milksnot.com  Wed Oct 10 16:09:44 2012
From: jhalfmoon at milksnot.com (Johnny Halfmoon)
Date: Wed, 10 Oct 2012 18:09:44 +0200
Subject: Monitor per site in Varnish
In-Reply-To: <C1C0937CADC64B49A0B1F5C83E284FE50A3816AF@EXCH2.bds.internal>
References: <C1C0937CADC64B49A0B1F5C83E284FE50A3816AF@EXCH2.bds.internal>
Message-ID: <50759DC8.2030306@milksnot.com>


On 09/28/2012 12:02 AM, Johan Olsson wrote:
>
> Hi
>
> I've been looking on how to monitor varnish. I've found that there
> exists a snmp for varnish which gives some info that is good to have.
> I've found it and looked at it
> (http://sourceforge.net/projects/varnishsnmp/), but it dosen't give
> all that I need (I think). What I'm missing is to be able to monitor
> how much traffic one site is using. So if I have two sites like
> www.example1.com <http://www.example1.com> and www.example2.com
> <http://www.example2.com>, I would like to be able to get how many
> connections each one gets and how much Mbps each one is using.
>
> Is this possible to do?
>
Hi Johan,

Maybe I can help. I've got about 35 sites running on a 4 node Varnish
cluster here and monitor throughput, request rate and http status codes
per-site using Cacti and Nagios via SNMP . The way it works is like this:

Each server is running the exact same varnish config. In this config
there's a VCL chunk that defines a bunch of macro's for gathering site info:

STATS_NODE - Define a new node. This generates a structure at compile
time, where the statistics will be stored.
STATS_INIT - Initialize a node. Unfortunately this gets called each time
a site is accessed. But the code is only a few lines and very lightweight.
STATS_SET_BACKEND - Defines the current backend to use. This is called
each time a site is accessed.
STATS_UPDATE - Update the site's statistics. This is called each time a
site is accessed.
STATS_DUMP - This gets called periodically to dump the entire statistics
linked-list to a syslog.

The flow is roughly as follows:

- Each site references the macros at certain points to generate the
statistics;
- The main configuration calls STATS_DUMP periodically, which sends
statistics info to syslog;
- Syslog then sends it to a dedicated FIFO;
- A script (called varnish-snmp-stats-prep-backends.sh) is listening to
the FIFO and parses the stats;
- The stats are then parsed and written to a per-site textfile;
- SNMPD is configured to access the per-site stats files.

Each server also generates varnishd-specific data every 5 minutes using
a script (varnish-snmp-stats-prep-srv.sh) that calls varnishstat. The
parsed output of varnish is dumped to a text file and made available to
SNMPD.

One varnish server is appointed the main statistics server. On that
server a cronjob calls "varnish-snmp-summarize-backends.py" every 5
minutes, which gathers and summarizes the statistics of all 4 servers,
using SNMP. This data is then dumped to per-site text files again, but
then containing the aggregate per-site counts. Cacti can the query this
server for the combined per-site and varnishd statistics.

Another approach to generate the per-site statistics could be to pipe
the varnishlog output to a script that parses this data. Though I fear
this method might cause quite a heavy load on the machine doing the
parsing, so this may have to be offloaded to another machine. But this
is not the path I chose.

Note: We're still running varnish 2 in production. Version 3 is in test.
But the conversion is trivial. I've prepared a tarball of this setup for
sharing, but I have to get permission to release this (anonimized)
configuration to the public. I'll get back to you on this subject
tomorrow or the day after to hopefully supply you with the entire setup
(varnish, cron, support scripts, syslog). Just let me know the best way
to share this on this list.

And somewhat unrelated to your question, but interresting nonetheless:
Another bit of VCL code dumps each request tot syslog in a modified NSCA
format, for debugging, traceability and such. But because the machines
sometimes generate beyond 2MB of logdata per second per server, and I
like to keep the logs for a few weeks, the logs need to be rotateted
fairly often to prevent gigantic files and they need to be compressed to
minimize storage requirements.There are two seperate scripts to handle
logrotation:

- varnish-log-rotate.sh - Checks the size of the log and rotates it if
it excceeds 2GB.
- varnish-log-compress.sh - Waits for rotated logs and compresses and
archives them using idle-priority, to minimize CPU impact.

This allows you to store 2.5TB of logs on 250GB of storage and minimize
the log compression load on the servers.

Cheers,

Johnny

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121010/fd63e542/attachment.html>

From pprocacci at datapipe.com  Wed Oct 10 17:55:00 2012
From: pprocacci at datapipe.com (Paul A. Procacci)
Date: Wed, 10 Oct 2012 12:55:00 -0500
Subject: Monitor per site in Varnish
In-Reply-To: <50759DC8.2030306@milksnot.com>
References: <C1C0937CADC64B49A0B1F5C83E284FE50A3816AF@EXCH2.bds.internal>
 <50759DC8.2030306@milksnot.com>
Message-ID: <20121010175500.GA15033@nat.myhome>

On Wed, Oct 10, 2012 at 06:09:44PM +0200, Johnny Halfmoon wrote:
>    On 09/28/2012 12:02 AM, Johan Olsson wrote:
>
>    Hi
>
>    I've been looking on how to monitor varnish. I've found that there
>    exists a snmp for varnish which gives some info that is good to have.
>    I've found it and looked at it
>    ([1]http://sourceforge.net/projects/varnishsnmp/), but it dosen't give
>    all that I need (I think). What I'm missing is to be able to monitor
>    how much traffic one site is using. So if I have two sites like
>    [2]www.example1.com and [3]www.example2.com, I would like to be able to
>    get how many connections each one gets and how much Mbps each one is
>    using.
>
>    Is this possible to do?

<snip>

>    Hi Johan,
>    Maybe I can help. I've got about 35 sites running on a 4 node Varnish
>    cluster here and monitor throughput, request rate and http status codes
>    per-site using Cacti and Nagios via SNMP . The way it works is like
>    this:
>    Each server is running the exact same varnish config. In this config
>    there's a VCL chunk that defines a bunch of macro's for gathering site
>    info:

<snip>

The snmp features are very useful no doubt.  I understand the OP opened his
question in relation to snmp, but later asked two very simple questions.
The first was regarding how many connections and the second was how much
Mbps _per site_.

For a simple task like this, I'd suggest awstats.  In fact, I do this
currrently for client of mine whom has currently 104 sites sitting behind
his varnish instance.  Here is how we tackled it:

- I've got varnishncsa running with the following:
varnishncsa -D -P /var/run/varnishncsa.pid -a -c -w /var/log/varnishncsa.log -F '%{Host}i %h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"'

- newsyslog running to rotate at a specific time:
/var/log/varnishncsa.log       644 10       * $D3  BR /usr/local/scripts/run_awstats.sh

- The following shell script that newsyslog invokes as per the above:
########################################
LOGFILE=/var/log/varnishncsa.log.0

kill -HUP `cat /var/run/varnishncsa.pid`

count=0
for i in `ls /usr/local/www/awstats/cgi-bin/*www\.*`; do
  i=${i##*/awstats\.};
  i=${i%%\.conf};

  /usr/bin/nice -n 5 su -m www -c "/usr/local/www/awstats/cgi-bin/awstats.pl -update -config=${i} -configdir=/usr/local/www/awstats/cgi-bin/ -LogFile=${LOGFILE}" &
  count=`expr ${count} + 1`;
  [ $((count%4)) -eq 0 ] && wait
done;

wait

bzip2 ${LOGFILE}
########################################

- With the above, the awstats configuration simply has to be updated with
  the correct log format:
LogFormat="%virtualname %host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot"

-----------------
Again, just one way to tackle this.  You can probably use any other log
analyzer to grab the same information.  Hopefully this helps the OP.

~Paul

________________________________

This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you.



From hugues.alary at gmail.com  Wed Oct 10 21:41:00 2012
From: hugues.alary at gmail.com (Hugues Alary)
Date: Wed, 10 Oct 2012 14:41:00 -0700
Subject: Varnish regular expression evaluated at compile time.
Message-ID: <CAN-YAk_n1vJQsmAQA+7K-Es4TEVviS6s8veLep7Sfe5=7WCPcg@mail.gmail.com>

Hi there,

I have been struggling for the past few days with this problem:

Basically, I want to send to a client browser a cookie of the form
foo[sha1oftheurl]=[randomvalue]. If and only if the cookie has not already
been sent.

e.g. If a client browser requests "/page.html", the HTTP response will be
like:
resp.http.Set-Cookie = "foo4c9ae249e9e061dd6e30893e03dc10a58cc40ee6=ABCD;"

then, if the same client request "/index.html", the HTTP response will
contain a header:
resp.http.Set-Cookie = "foo14fe4559026d4c5b5eb530ee70300c52d99e70d7=QWERTY;"

In the end, the client browser will have 2 cookies:
`foo4c9ae249e9e061dd6e30893e03dc10a58cc40ee6=ABCD`
`foo14fe4559026d4c5b5eb530ee70300c52d99e70d7=QWERTY`


Now, that, is not complicated in itself. The following code does it:

import digest;
        import random;

 sub vcl_recv()
{
 ## We compute the sha1 of the requested URL and store it in
req.http.Url-Sha1
set req.http.Url-Sha1 = digest.hash_sha1(req.url);
                set req.http.random-value = random.get_rand();
}

sub vcl_deliver()
{
 ## We create a cookie on the client browser by creating a "Set-Cookie"
header
## In our case the cookie we create is of the form foo[sha1]=[randomvalue]
 ## e.g for a URL "/page.html" the cookie will be
foo4c9ae249e9e061dd6e30893e03dc10a58cc40ee6=[randomvalue]
 set resp.http.Set-Cookie = {""} + resp.http.Set-Cookie +
"foo"+req.http.Url-Sha1+"="+req.http.random-value;
 }


However, this code does not take into account the case where the Cookie
already exists. I need to check that the Cookie does not exists before
generating a random value. So I thought about this code:

import digest;
        import random;

 sub vcl_recv()
 {
## We compute the sha1 of the requested URL and store it in
req.http.Url-Sha1
 set req.http.Url-Sha1 = digest.hash_sha1(req.url);
                set req.http.random-value = random.get_rand();

                set req.http.regex = "abtest"+req.http.Url-Sha1;

 if(!req.http.Cookie ~ req.http.regex)
{
 set req.http.random-value = random.get_rand();
}
 }


The problem is that Varnish does not compute Regular expression at run
time. Which leads to this error when I try to compile:

Message from VCC-compiler:
Expected CSTR got 'req.http.regex'
(program line 940), at
('input' Line 42 Pos 31)
        if(req.http.Cookie !~ req.http.regex) {
------------------------------##############---

Running VCC-compiler failed, exit 1

VCL compilation failed


One could propose to solve my problem by matching on the "abtest" part of
the cookie or even "abtest[a-fA-F0-9]{40}":

                if(!req.http.Cookie ~ "abtest[a-fA-F0-9]{40}")
{
 set req.http.random-value = random.get_rand();
}

But this code matches any cookie starting by 'abtest' and containing an
hexadecimal string of 40 characters. Which means that if a client requests
"/page.html" first, then "/index.html", the condition will evaluate to true
even if the cookie for the "/index.html" has not been set.

I found in bug report phk or someone else stating that computing regular
expressions was extremely expensive which is why they are evaluated during
compilation. Considering this, I believe that there is no way of achieving
what I want the way I've been trying to.

Is there any way of solving this problem, other than writting a vmod?

Thanks for your help!

-Hugues
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121010/2d46712b/attachment.html>

From pprocacci at datapipe.com  Wed Oct 10 22:59:25 2012
From: pprocacci at datapipe.com (Paul A. Procacci)
Date: Wed, 10 Oct 2012 17:59:25 -0500
Subject: Varnish regular expression evaluated at compile time.
In-Reply-To: <CAN-YAk_n1vJQsmAQA+7K-Es4TEVviS6s8veLep7Sfe5=7WCPcg@mail.gmail.com>
References: <CAN-YAk_n1vJQsmAQA+7K-Es4TEVviS6s8veLep7Sfe5=7WCPcg@mail.gmail.com>
Message-ID: <20121010225925.GE15033@nat.myhome>

Everything you stated is true.  Without inline C or a vmod
the only alternative I know of to accomplish this would be
to dynamically generate the parts of your config that
depend on the sha1 of a url.

For instance, you could use something similar to the following:

#################################################
#!/usr/bin/perl -w

use strict;
use Digest::SHA qw(sha1_hex);

my @files = qw!/index.html /homepage.html!;

my $output;
my $seen = 0;
foreach(@files){
  if($seen++){
    $output .= "else";
  }
  $output .= "if(req.url == \"$_\"){\n";
  $output .= "  set resp.http.Set-Cookie = \"foo" . sha1_hex($_) . "=\" + random.get_rand();\n";
  $output .= "}\n"
}

print $output;
#################################################

Modified to your liking obviously, which results in the following
output:

#################################################
if(req.url == "/index.html"){
  set resp.http.Set-Cookie = "foo14fe4559026d4c5b5eb530ee70300c52d99e70d7=" + random.get_rand();
}
elseif(req.url == "/homepage.html"){
  set resp.http.Set-Cookie = "foo6593125bb8fade312b1081d4ee1998f316aa4081=" + random.get_rand();
}
#################################################

This has the benefit that everything with exception to the random
number is computed ahead of time, but has the drawback of having
to use in this case, another script to maintain a list of
accessible files to generate their hash.

Honestly not sure if this is going to help or not, but
maybe it'll give you other ideas to explore.

~Paul

________________________________

This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you.



From hugues.alary at gmail.com  Wed Oct 10 23:14:03 2012
From: hugues.alary at gmail.com (Hugues Alary)
Date: Wed, 10 Oct 2012 16:14:03 -0700
Subject: Varnish regular expression evaluated at compile time.
In-Reply-To: <20121010225925.GE15033@nat.myhome>
References: <CAN-YAk_n1vJQsmAQA+7K-Es4TEVviS6s8veLep7Sfe5=7WCPcg@mail.gmail.com>
 <20121010225925.GE15033@nat.myhome>
Message-ID: <CAN-YAk9KG1F+6-hKjvgi6G1x26QohfDBryr_D=cazMy1GWn_gg@mail.gmail.com>

Hi Paul,

Thank you for your answer, and your idea. It is a very good idea I didn't
think of.

I don't know if it's gonna be the solution I'll adopt, but at this very
moment I only have that or coding a vmod. However I don't think that coding
a vmod is a solution as I would just implement something (regular
expression evaluation at run time) that phk says being too expensive.

-Hugues

On Wed, Oct 10, 2012 at 3:59 PM, Paul A. Procacci <pprocacci at datapipe.com>wrote:

> Everything you stated is true.  Without inline C or a vmod
> the only alternative I know of to accomplish this would be
> to dynamically generate the parts of your config that
> depend on the sha1 of a url.
>
> For instance, you could use something similar to the following:
>
> #################################################
> #!/usr/bin/perl -w
>
> use strict;
> use Digest::SHA qw(sha1_hex);
>
> my @files = qw!/index.html /homepage.html!;
>
> my $output;
> my $seen = 0;
> foreach(@files){
>   if($seen++){
>     $output .= "else";
>   }
>   $output .= "if(req.url == \"$_\"){\n";
>   $output .= "  set resp.http.Set-Cookie = \"foo" . sha1_hex($_) . "=\" +
> random.get_rand();\n";
>   $output .= "}\n"
> }
>
> print $output;
> #################################################
>
> Modified to your liking obviously, which results in the following
> output:
>
> #################################################
> if(req.url == "/index.html"){
>   set resp.http.Set-Cookie =
> "foo14fe4559026d4c5b5eb530ee70300c52d99e70d7=" + random.get_rand();
> }
> elseif(req.url == "/homepage.html"){
>   set resp.http.Set-Cookie =
> "foo6593125bb8fade312b1081d4ee1998f316aa4081=" + random.get_rand();
> }
> #################################################
>
> This has the benefit that everything with exception to the random
> number is computed ahead of time, but has the drawback of having
> to use in this case, another script to maintain a list of
> accessible files to generate their hash.
>
> Honestly not sure if this is going to help or not, but
> maybe it'll give you other ideas to explore.
>
> ~Paul
>
> ________________________________
>
> This message may contain confidential or privileged information. If you
> are not the intended recipient, please advise us immediately and delete
> this message. See http://www.datapipe.com/legal/email_disclaimer/ for
> further information on confidentiality and the risks of non-secure
> electronic communication. If you cannot access these links, please notify
> us by reply message and we will send the contents to you.
>



-- 
Hugues ALARY
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121010/97d89652/attachment.html>

From hugo.cisneiros at gmail.com  Thu Oct 11 00:12:07 2012
From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch))
Date: Wed, 10 Oct 2012 21:12:07 -0300
Subject: vcl_error check backend
In-Reply-To: <CABHsmABb4K-NFtphV5r6d8EoWszxiv_aZRatpNoBw5GGgCkbKg@mail.gmail.com>
References: <CABHsmAB2-PK2+r7Utk-gkiJwVOnR0JvgoV14NFGhEtRBbMxV+g@mail.gmail.com>
 <CABHsmAApV+WTSnrokaWwMKEE8UUPQ_-8TBfYFAMA-3dCXb0uiw@mail.gmail.com>
 <CABHsmABb4K-NFtphV5r6d8EoWszxiv_aZRatpNoBw5GGgCkbKg@mail.gmail.com>
Message-ID: <CA+KACLmbGbBNKzNJs==eaLd-UDz4oHnJE8GP2toU24Vtz1stkQ@mail.gmail.com>

On Tue, Oct 9, 2012 at 12:08 PM, Daniel Lafraia <lafraia at gmail.com> wrote:
> Is there a way to check which backend was used under vcl_error()?

In my tests, req.backend couldn't be tested on vcl_error (it is null),
so you could set a req.http.header and check it at vcl_error. Like
this:

sub vcl_recv {
  if (req.http.host == "example1.com") {
    set req.backend = be_1;
  } elsif (req.http.host == "example2.com") {
    set req.backend = be_2;
  } else {
    req.backend = be_default;
  }

  set req.http.X-Varnish-Backend = req.backend; # this will convert
the req.backend to string
}

sub vcl_error {
  set obj.http.Content-Type = "text/html; charset=utf-8";
  set obj.http.Retry-After = "5";

  synthetic {"<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
  <head>
    <title>"} + obj.status + " " + obj.response + {"</title>
  </head>
  <body>
    <h1>Error "} + obj.status + " " + obj.response + {"</h1>
    <p>"} + obj.response + {"</p>
    <h3>Guru Meditation:</h3>
    <p>XID: "} + req.xid + {"</p>
    <p>Backend: "} + req.http.X-Varnish-Backend + {"</p>
    <hr>
    <p>Varnish cache server</p>
  </body>
</html>
"};

  return (deliver);
}

-- 
[]'s
Hugo
www.devin.com.br



From tfheen at varnish-software.com  Thu Oct 11 10:46:28 2012
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Thu, 11 Oct 2012 12:46:28 +0200
Subject: Varnish regular expression evaluated at compile time.
In-Reply-To: <CAN-YAk9KG1F+6-hKjvgi6G1x26QohfDBryr_D=cazMy1GWn_gg@mail.gmail.com>
References: <CAN-YAk_n1vJQsmAQA+7K-Es4TEVviS6s8veLep7Sfe5=7WCPcg@mail.gmail.com>
 <20121010225925.GE15033@nat.myhome>
 <CAN-YAk9KG1F+6-hKjvgi6G1x26QohfDBryr_D=cazMy1GWn_gg@mail.gmail.com>
Message-ID: <20121011104628.GE8602@err.no>

]] Hugues Alary

> I don't know if it's gonna be the solution I'll adopt, but at this very
> moment I only have that or coding a vmod. However I don't think that coding
> a vmod is a solution as I would just implement something (regular
> expression evaluation at run time) that phk says being too expensive.

Yes, compiling it is expensive, but if you can afford it or not is
obviously going to depend on the amount of CPU you have and your request
rates.

We currently don't support doing runtime compilation of regexes (as you
discovered), so a vmod would be a way to go for now.

-- 
Tollef Fog Heen
Technical lead, Varnish Software
t: +47 21 98 92 64



From aanchal.gera at gmail.com  Fri Oct  5 08:12:23 2012
From: aanchal.gera at gmail.com (Aanchal Gera)
Date: Fri, 5 Oct 2012 13:42:23 +0530
Subject: Stats of Hit/Miss for a particular URL
Message-ID: <CAFf4ghYN-upo0EVqXH8MQjKiOiGnjoZW3fa=A5GdA6v7iSeRdw@mail.gmail.com>

Hi All

I am using ESI Tags with varnish.

How can I check with varnishstat/varnishlog if a particular URL ever hits
the cache. And it if it does, how can I see the Hit/Miss ratio. My
requirement is that it should never hit the cache.

Regards,
Aanchal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121005/7b522643/attachment.html>

From tfheen at varnish-software.com  Thu Oct 11 10:49:31 2012
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Thu, 11 Oct 2012 12:49:31 +0200
Subject: Installation Prerequisites related query
In-Reply-To: <CABfdyPYCRAXjLVNWTNyVUdQ0poNc1Pv_htKp3en_MV8RWYM2Og@mail.gmail.com>
References: <CABfdyPYCRAXjLVNWTNyVUdQ0poNc1Pv_htKp3en_MV8RWYM2Og@mail.gmail.com>
Message-ID: <20121011104931.GF8602@err.no>

]] samir devaliya

> Is it Possible to install varnish on debian 3.1 (sarge)
> 
> if yes kindly provide the version of varnish which can be installed

We currently don't support sarge, no.  It's quite old and it's likely
Varnish won't work correctly there.  (It was released in 2005.)

I'd really recommend you upgrade to at Debian 6.0 squeeze.

Cheers,
-- 
Tollef Fog Heen
Technical lead, Varnish Software
t: +47 21 98 92 64



From perbu at varnish-software.com  Thu Oct 11 12:05:41 2012
From: perbu at varnish-software.com (Per Buer)
Date: Thu, 11 Oct 2012 14:05:41 +0200
Subject: Error 413 Request Entity Too Large
In-Reply-To: <002001cda53d$5b81e540$1285afc0$@co.uk>
References: <CABfdyPYCRAXjLVNWTNyVUdQ0poNc1Pv_htKp3en_MV8RWYM2Og@mail.gmail.com>
 <002001cda53d$5b81e540$1285afc0$@co.uk>
Message-ID: <CAOXZevAOiG0r2OTMs3-+gOm3caTQ6gVUnwdM7PVkFQ5D2uFotQ@mail.gmail.com>

On Mon, Oct 8, 2012 at 12:12 PM, Rowan @ Jetboy <rowan at jetboy.co.uk> wrote:

> I've had a handful of complaints from my website users that they're seeing
> the error below when trying to access an installation of the MyBB forum.**
> **
>
> ** **
>
> Error 413****
>
> Request Entity Too Large****
>
> ****
>
> Guru Meditation:****
>
> XID: 190197240****
>
> ****
>
> Varnish cache server
>

I'm not 100% sure but if memory serves me well you need to tweak this one:

       http_req_size

              ? Units: bytes

              ? Default: 32768

              Maximum  number  of  bytes  of  HTTP client request we will
deal
              with.  This is a limit on all bytes up to the double blank
 line
              which  ends  the  HTTP  request.   The memory for the request
is
              allocated from the session workspace (param: sess_workspace)
and
              this parameter limits how much of that the request is allowed
to
              take up.


-- 
Per Buer
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121011/930dec21/attachment.html>

From hugues.alary at gmail.com  Thu Oct 11 21:45:03 2012
From: hugues.alary at gmail.com (Hugues Alary)
Date: Thu, 11 Oct 2012 14:45:03 -0700
Subject: Varnish regular expression evaluated at compile time.
In-Reply-To: <20121011104628.GE8602@err.no>
References: <CAN-YAk_n1vJQsmAQA+7K-Es4TEVviS6s8veLep7Sfe5=7WCPcg@mail.gmail.com>
 <20121010225925.GE15033@nat.myhome>
 <CAN-YAk9KG1F+6-hKjvgi6G1x26QohfDBryr_D=cazMy1GWn_gg@mail.gmail.com>
 <20121011104628.GE8602@err.no>
Message-ID: <CAN-YAk_yhk0cvOwbSSwpXiSkumRhFmnNzJC2jzJV9n1dVwp9LQ@mail.gmail.com>

Thanks for the clarification. I guess I'll have dig up my C knowledges now!

-Hugues

On Thu, Oct 11, 2012 at 3:46 AM, Tollef Fog Heen <
tfheen at varnish-software.com> wrote:

> ]] Hugues Alary
>
> > I don't know if it's gonna be the solution I'll adopt, but at this very
> > moment I only have that or coding a vmod. However I don't think that
> coding
> > a vmod is a solution as I would just implement something (regular
> > expression evaluation at run time) that phk says being too expensive.
>
> Yes, compiling it is expensive, but if you can afford it or not is
> obviously going to depend on the amount of CPU you have and your request
> rates.
>
> We currently don't support doing runtime compilation of regexes (as you
> discovered), so a vmod would be a way to go for now.
>
> --
> Tollef Fog Heen
> Technical lead, Varnish Software
> t: +47 21 98 92 64
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>



-- 
Hugues ALARY
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121011/94493043/attachment.html>

From lin.support at gmail.com  Fri Oct 12 09:52:07 2012
From: lin.support at gmail.com (linuxsupport)
Date: Fri, 12 Oct 2012 15:22:07 +0530
Subject: ESI and Caching
Message-ID: <CAPpMyBFejpnrGNmYiMPwAq0bU8NncqY9Ai0EP9yVvxxUN7Hm7w@mail.gmail.com>

Hi All,

I have an application which sends cookie, home page has 75% static contents
and 25% dynamic.
is there a way to cache the static part of the page while get the dynamic
contents using ESI.

I know if application does not send cookie then there is no problem, but
cookie is important in our case.

Please suggest what possible way it can be achieve.

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121012/dc7582b7/attachment.html>

From pprocacci at datapipe.com  Sat Oct 13 02:08:28 2012
From: pprocacci at datapipe.com (Paul A. Procacci)
Date: Fri, 12 Oct 2012 21:08:28 -0500
Subject: ESI and Caching
In-Reply-To: <CAPpMyBFejpnrGNmYiMPwAq0bU8NncqY9Ai0EP9yVvxxUN7Hm7w@mail.gmail.com>
References: <CAPpMyBFejpnrGNmYiMPwAq0bU8NncqY9Ai0EP9yVvxxUN7Hm7w@mail.gmail.com>
Message-ID: <20121013020828.GG7125@nat.myhome>

On Fri, Oct 12, 2012 at 03:22:07PM +0530, linuxsupport wrote:
>    Hi All,
>    I have an application which sends cookie, home page has 75% static
>    contents and 25% dynamic.
>    is there a way to cache the static part of the page while get the
>    dynamic contents using ESI.
>    I know if application does not send cookie then there is no problem,
>    but cookie is important in our case.
>    Please suggest what possible way it can be achieve.
>    Thanks

If a cookie exists, don't pass it, and `return (lookup)`?

The vcl is pretty clear about passing requests with cookies to the
backend, but your vcl can be modified quite easily change this function-
ality.

Perhaps I'm not completely understanding the question.

~Paul

________________________________

This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you.



From lin.support at gmail.com  Sat Oct 13 05:50:24 2012
From: lin.support at gmail.com (linuxsupport)
Date: Sat, 13 Oct 2012 11:20:24 +0530
Subject: ESI and Caching
In-Reply-To: <20121013020828.GG7125@nat.myhome>
References: <CAPpMyBFejpnrGNmYiMPwAq0bU8NncqY9Ai0EP9yVvxxUN7Hm7w@mail.gmail.com>
 <20121013020828.GG7125@nat.myhome>
Message-ID: <CAPpMyBE5SMq5qfj9bh6Zaai2=jcCF6o=WPZmKi-urn=L=dYDag@mail.gmail.com>

I think I did not make it very clear, let me explain

I have a file named abc.html as below
<html>
<head>
<title>Test</title?
<body>
<p>Test Page</p>
<p><esi:include src="/cgi-bin/abc.php"/></p>
</body>
</html>

This abc.php sends cookie, what I want is to cache all the contents of
abc.html and when doing esi processing (abc.php) include the Set-Cookie
header sent by abc.php with the response.

right now, abc.html is being cached and esi processing for abc.php is also
working but Set-Cookie header sent by abc.php is not being sent to client.

possible?

On Sat, Oct 13, 2012 at 7:38 AM, Paul A. Procacci <pprocacci at datapipe.com>wrote:

> On Fri, Oct 12, 2012 at 03:22:07PM +0530, linuxsupport wrote:
> >    Hi All,
> >    I have an application which sends cookie, home page has 75% static
> >    contents and 25% dynamic.
> >    is there a way to cache the static part of the page while get the
> >    dynamic contents using ESI.
> >    I know if application does not send cookie then there is no problem,
> >    but cookie is important in our case.
> >    Please suggest what possible way it can be achieve.
> >    Thanks
>
> If a cookie exists, don't pass it, and `return (lookup)`?
>
> The vcl is pretty clear about passing requests with cookies to the
> backend, but your vcl can be modified quite easily change this function-
> ality.
>
> Perhaps I'm not completely understanding the question.
>
> ~Paul
>
> ________________________________
>
> This message may contain confidential or privileged information. If you
> are not the intended recipient, please advise us immediately and delete
> this message. See http://www.datapipe.com/legal/email_disclaimer/ for
> further information on confidentiality and the risks of non-secure
> electronic communication. If you cannot access these links, please notify
> us by reply message and we will send the contents to you.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121013/4ff56f14/attachment.html>

From pprocacci at datapipe.com  Sat Oct 13 06:05:46 2012
From: pprocacci at datapipe.com (Paul A. Procacci)
Date: Sat, 13 Oct 2012 01:05:46 -0500
Subject: ESI and Caching
In-Reply-To: <CAPpMyBE5SMq5qfj9bh6Zaai2=jcCF6o=WPZmKi-urn=L=dYDag@mail.gmail.com>
References: <CAPpMyBFejpnrGNmYiMPwAq0bU8NncqY9Ai0EP9yVvxxUN7Hm7w@mail.gmail.com>
 <20121013020828.GG7125@nat.myhome>
 <CAPpMyBE5SMq5qfj9bh6Zaai2=jcCF6o=WPZmKi-urn=L=dYDag@mail.gmail.com>
Message-ID: <20121013060546.GH7125@nat.myhome>

On Sat, Oct 13, 2012 at 11:20:24AM +0530, linuxsupport wrote:
>    I think I did not make it very clear, let me explain
>    I have a file named abc.html as below
>    <html>
>    <head>
>    <title>Test</title?
>    <body>
>    <p>Test Page</p>
>    <p><esi:include src="/cgi-bin/abc.php"/></p>
>    </body>
>    </html>
>    This abc.php sends cookie, what I want is to cache all the contents of
>    abc.html and when doing esi processing (abc.php) include the Set-Cookie
>    header sent by abc.php with the response.
>    right now, abc.html is being cached and esi processing for abc.php is
>    also working but Set-Cookie header sent by abc.php is not being sent to
>    client.
>    possible?

See chapter six of the ESI spec: http://www.w3.org/TR/esi-lang

When an ESI template is processed, a separate request will need to be made for each include encountered. Implementations may use the original request's headers (e.g., Cookie, User-Agent, etc.) when doing so. Additionally, response headers from fragments (e.g., Set-Cookie, Server, Cache-Control, Last-Modified) may be ignored, and should not influence the assembled page.

Given the above, and given your explanation of your app disgarding cookies, then I'm going to go out on a limb and say i`not possible`.

~Paul

________________________________

This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you.



From lin.support at gmail.com  Sat Oct 13 06:50:22 2012
From: lin.support at gmail.com (linuxsupport)
Date: Sat, 13 Oct 2012 12:20:22 +0530
Subject: ESI and Caching
In-Reply-To: <20121013060546.GH7125@nat.myhome>
References: <CAPpMyBFejpnrGNmYiMPwAq0bU8NncqY9Ai0EP9yVvxxUN7Hm7w@mail.gmail.com>
 <20121013020828.GG7125@nat.myhome>
 <CAPpMyBE5SMq5qfj9bh6Zaai2=jcCF6o=WPZmKi-urn=L=dYDag@mail.gmail.com>
 <20121013060546.GH7125@nat.myhome>
Message-ID: <CAPpMyBGcfe7zgnre8pAZMbYDoeay654TuHzR-yFNMdzZ-9X6bQ@mail.gmail.com>

Understood.

any possibility of storing the header sent by esi processed script
(abc.php) into a temparoty variable and then including it to response from
vcl_deliver?

On Sat, Oct 13, 2012 at 11:35 AM, Paul A. Procacci
<pprocacci at datapipe.com>wrote:

> On Sat, Oct 13, 2012 at 11:20:24AM +0530, linuxsupport wrote:
> >    I think I did not make it very clear, let me explain
> >    I have a file named abc.html as below
> >    <html>
> >    <head>
> >    <title>Test</title?
> >    <body>
> >    <p>Test Page</p>
> >    <p><esi:include src="/cgi-bin/abc.php"/></p>
> >    </body>
> >    </html>
> >    This abc.php sends cookie, what I want is to cache all the contents of
> >    abc.html and when doing esi processing (abc.php) include the
> Set-Cookie
> >    header sent by abc.php with the response.
> >    right now, abc.html is being cached and esi processing for abc.php is
> >    also working but Set-Cookie header sent by abc.php is not being sent
> to
> >    client.
> >    possible?
>
> See chapter six of the ESI spec: http://www.w3.org/TR/esi-lang
>
> When an ESI template is processed, a separate request will need to be made
> for each include encountered. Implementations may use the original
> request's headers (e.g., Cookie, User-Agent, etc.) when doing so.
> Additionally, response headers from fragments (e.g., Set-Cookie, Server,
> Cache-Control, Last-Modified) may be ignored, and should not influence the
> assembled page.
>
> Given the above, and given your explanation of your app disgarding
> cookies, then I'm going to go out on a limb and say i`not possible`.
>
> ~Paul
>
> ________________________________
>
> This message may contain confidential or privileged information. If you
> are not the intended recipient, please advise us immediately and delete
> this message. See http://www.datapipe.com/legal/email_disclaimer/ for
> further information on confidentiality and the risks of non-secure
> electronic communication. If you cannot access these links, please notify
> us by reply message and we will send the contents to you.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121013/6aa6a890/attachment.html>

From vstavrinov at gmail.com  Sat Oct 13 20:11:19 2012
From: vstavrinov at gmail.com (Vladimir Stavrinov)
Date: Sun, 14 Oct 2012 00:11:19 +0400
Subject: cache object stub
Message-ID: <20121013201119.GA5119@terra.00>

Hi All.

While testing varnish 3.0.3 with full stream support, I've faced this
problem. When storage space exhausted the new file fetched from backend
partially overrides some of the existing cache object. But this
truncated object still can be delivered to client, that is wrong,
because actually it is broken file. It looks like bug, because I don't
see any case where this "feature" may be useful.


***************************
###  Vladimir Stavrinov
###  vstavrinov at gmail.com
***************************




From tfheen at varnish-software.com  Mon Oct 15 07:15:25 2012
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Mon, 15 Oct 2012 09:15:25 +0200
Subject: cache object stub
In-Reply-To: <20121013201119.GA5119@terra.00>
References: <20121013201119.GA5119@terra.00>
Message-ID: <20121015071524.GA21705@err.no>

]] Vladimir Stavrinov

> While testing varnish 3.0.3 with full stream support, I've faced this
> problem. When storage space exhausted the new file fetched from backend
> partially overrides some of the existing cache object. But this
> truncated object still can be delivered to client, that is wrong,
> because actually it is broken file. It looks like bug, because I don't
> see any case where this "feature" may be useful.

This sounds like a bug, please file it in trac so we keep track of it.
https://www.varnish-cache.org/trac

Cheers,
-- 
Tollef Fog Heen
Technical lead, Varnish Software
t: +47 21 98 92 64



From vstavrinov at gmail.com  Tue Oct 16 09:32:46 2012
From: vstavrinov at gmail.com (Vladimir Stavrinov)
Date: Tue, 16 Oct 2012 13:32:46 +0400
Subject: cache object stub
In-Reply-To: <20121015071524.GA21705@err.no>
References: <20121013201119.GA5119@terra.00>
 <20121015071524.GA21705@err.no>
Message-ID: <20121016093245.GR23544@mana.playfast.ru>

On Mon, Oct 15, 2012 at 09:15:25AM +0200, Tollef Fog Heen wrote:

> > While testing varnish 3.0.3 with full stream support, I've faced this
> > problem. When storage space exhausted the new file fetched from backend

Further investigation shows it is not stream related problem, but it is
exists with permanent storage only. So it marked right as experimental.

-- 

***************************
##  Vladimir Stavrinov
##  vstavrinov at gmail.com
***************************




From sahmed1020 at gmail.com  Wed Oct 17 16:19:31 2012
From: sahmed1020 at gmail.com (S Ahmed)
Date: Wed, 17 Oct 2012 12:19:31 -0400
Subject: invalidating (ban) a cache problematically, real life maximum?
In-Reply-To: <CAOXZevCRZ-FyAh8+SQo-8FrwZti49LxBD=y6JZwYdY_Q88+q3g@mail.gmail.com>
References: <CAG2rwuNDxvHWFXC_Y3sn2hEd6y9=tSHJqsqU+3uRhJhSdgkphg@mail.gmail.com>
 <CAOXZevCRZ-FyAh8+SQo-8FrwZti49LxBD=y6JZwYdY_Q88+q3g@mail.gmail.com>
Message-ID: <CAG2rwuNcMP6cXBAbvLKe8EC-zVVgUzxww_L686MEP9UEYruNrg@mail.gmail.com>

Per,

Confused, I thought ban and purge are the same thing now in varnish 3?

Varnish 3 uses the term ban and banning for what was known as purge and
purging in Varnish 2. They are the same.

   - Varnish 3 has a new function called purge; that did not exist in VCL
   in Varnish 2.
   - purge; is a much improved way of doing what was in Varnish 2 done
   using set obj.ttl = 0s;.


So when you said limiting ban's to 10-30 per second, you were referring to
a regex based way of removing an item from cache?

And a purge is a specific url?


On Wed, Oct 10, 2012 at 8:59 AM, Per Buer <perbu at varnish-software.com>wrote:

> Hi,
>
> On Wed, Oct 3, 2012 at 9:31 PM, S Ahmed <sahmed1020 at gmail.com> wrote:
>
>> I was reading:
>> https://www.varnish-software.com/static/book/Cache_invalidation.html
>>
>> My main question is, what sort of # of ban requests can one realistically
>> send to varnish to invalid caches? (individual items, not ranges etc.)
>>
>
> I would limit bans to maybe 10-30 per second, no more. Make sure they are
> lurker-friendly.
>
> Say a site like tumblr is using varnish, which has thousands and thousands
>> of blogs, and so many blog entries are being updated via comments etc.  How
>> realistic would sending 1K ban's be to varnish?
>>
>
> Sending 1k purges to Varnish would be no problem at all. Bans - I'm not
> sure that is a good idea.
>
>
>> Maybe I have misunderstood this, but I'm guessing their are ruby/java
>> client libraries that when content gets updated in a web application, I
>> could then send a request using the ruby/java client library to ban a
>> specific item from cache right?
>>
>> If there are client libraries, where can they be found?  I'm looking for
>> both ruby and java.
>>
>
> It might make sense to implement it yourself. It is dead simple and you
> don't need the complexities of a generic library.
>
> --
> Per Buer
> Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
> *Varnish makes websites fly!*
> Whitepapers <http://www.varnish-software.com/whitepapers> | Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
> Twitter <https://twitter.com/varnishsoftware>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121017/2d25f32c/attachment.html>

From perbu at varnish-software.com  Wed Oct 17 17:01:34 2012
From: perbu at varnish-software.com (Per Buer)
Date: Wed, 17 Oct 2012 19:01:34 +0200
Subject: invalidating (ban) a cache problematically, real life maximum?
In-Reply-To: <CAG2rwuNcMP6cXBAbvLKe8EC-zVVgUzxww_L686MEP9UEYruNrg@mail.gmail.com>
References: <CAG2rwuNDxvHWFXC_Y3sn2hEd6y9=tSHJqsqU+3uRhJhSdgkphg@mail.gmail.com>
 <CAOXZevCRZ-FyAh8+SQo-8FrwZti49LxBD=y6JZwYdY_Q88+q3g@mail.gmail.com>
 <CAG2rwuNcMP6cXBAbvLKe8EC-zVVgUzxww_L686MEP9UEYruNrg@mail.gmail.com>
Message-ID: <CAOXZevCObB=nN6wUCP9n-q4Z8M2dT9M3+ejo-3U10uD2iC-E0Q@mail.gmail.com>

Hi,

On Wed, Oct 17, 2012 at 6:19 PM, S Ahmed <sahmed1020 at gmail.com> wrote:

> Per,
>
> Confused, I thought ban and purge are the same thing now in varnish 3?
>

No.


> Varnish 3 uses the term ban and banning for what was known as purge and
> purging in Varnish 2. They are the same.
>

No.

>
>    - Varnish 3 has a new function called purge; that did not exist in VCL
>    in Varnish 2.
>    - purge; is a much improved way of doing what was in Varnish 2 done
>    using set obj.ttl = 0s;.
>
>
> So when you said limiting ban's to 10-30 per second, you were referring to
> a regex based way of removing an item from cache?
>

No. I meant you should be careful when doing to many bans. You can easily
purge 10000 objects per second.

And a purge is a specific url?
>

Please see the documentation for the explanation of what purges and bans
are. If you have problems understanding the documentation then raise an
issue and I'll make sure it becomes easier to understand.


-- 
Per Buer
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121017/6ec55f3f/attachment.html>

From lin.support at gmail.com  Thu Oct 18 11:16:51 2012
From: lin.support at gmail.com (linuxsupport)
Date: Thu, 18 Oct 2012 16:46:51 +0530
Subject: ESI and Gzip
Message-ID: <CAPpMyBHvYHQ1h61qqZmXrpcbb=BuP+yjj0yuMKzBgGsE4ORLhA@mail.gmail.com>

Hi All,

I was testing ESI and Gzip in varnish

Backend - Apache 2.2.17
Varnish - 3.03

It works well, however, I found that varnish is not able to compress the
contents as much as Apache deflate does.

Total size of the contents was 68KB, when served directly from Apache it
shows 16KB on the google chrome (checked through Chrome developer tools),
same contents when served from Varnish the size was 28KB

Anyone of you have experienced this? is it normal or there is still scope
to improve this?

Here is the configuration which I used

sub vcl_fetch {
       if(req.url == "/test.html"){
            set beresp.do_esi = true;
            set beresp.do_gzip = true;
       }
}

Also, tried removing Accept-Encoding header from vcl_miss as below

sub vcl_miss {
        if(req.url == "/test.html"){
           unset bereq.http.accept-encoding;
        }
}

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121018/df324b7d/attachment.html>

From phk at phk.freebsd.dk  Thu Oct 18 11:46:23 2012
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 18 Oct 2012 11:46:23 +0000
Subject: ESI and Gzip
In-Reply-To: <CAPpMyBHvYHQ1h61qqZmXrpcbb=BuP+yjj0yuMKzBgGsE4ORLhA@mail.gmail.com>
References: <CAPpMyBHvYHQ1h61qqZmXrpcbb=BuP+yjj0yuMKzBgGsE4ORLhA@mail.gmail.com>
Message-ID: <20062.1350560783@critter.freebsd.dk>

--------
In message <CAPpMyBHvYHQ1h61qqZmXrpcbb=BuP+yjj0yuMKzBgGsE4ORLhA at mail.gmail.com>, linuxsupport writes:

>I was testing ESI and Gzip in varnish

ESI & Gzip works in a pretty sneaky way in Varnish:  We stitch together
gzip'ed files on the fly.

The advantage of this is very high speed delivery, since there is no
gzip'ing to be done at delivery time, the disadvantage is a lower
compression ration, because we need to flush & align the gzip streams
to make the stitching possible.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From lin.support at gmail.com  Thu Oct 18 13:42:46 2012
From: lin.support at gmail.com (linuxsupport)
Date: Thu, 18 Oct 2012 19:12:46 +0530
Subject: ESI and Gzip
In-Reply-To: <20062.1350560783@critter.freebsd.dk>
References: <CAPpMyBHvYHQ1h61qqZmXrpcbb=BuP+yjj0yuMKzBgGsE4ORLhA@mail.gmail.com>
 <20062.1350560783@critter.freebsd.dk>
Message-ID: <CAPpMyBGJ3pDzvTg9rtHX+iWnwwuvrEk1EWk5pEVRDkZtBMi4yQ@mail.gmail.com>

Thanks Poul for the information.

So it should be faster but total download time is higher when using varnish
compared to when served directly from Apache.

On Thu, Oct 18, 2012 at 5:16 PM, Poul-Henning Kamp <phk at phk.freebsd.dk>wrote:

> --------
> In message <CAPpMyBHvYHQ1h61qqZmXrpcbb=
> BuP+yjj0yuMKzBgGsE4ORLhA at mail.gmail.com>, linuxsupport writes:
>
> >I was testing ESI and Gzip in varnish
>
> ESI & Gzip works in a pretty sneaky way in Varnish:  We stitch together
> gzip'ed files on the fly.
>
> The advantage of this is very high speed delivery, since there is no
> gzip'ing to be done at delivery time, the disadvantage is a lower
> compression ration, because we need to flush & align the gzip streams
> to make the stitching possible.
>
> --
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121018/9d2e9dcd/attachment.html>

From phk at phk.freebsd.dk  Thu Oct 18 14:09:04 2012
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 18 Oct 2012 14:09:04 +0000
Subject: ESI and Gzip
In-Reply-To: <CAPpMyBGJ3pDzvTg9rtHX+iWnwwuvrEk1EWk5pEVRDkZtBMi4yQ@mail.gmail.com>
References: <CAPpMyBHvYHQ1h61qqZmXrpcbb=BuP+yjj0yuMKzBgGsE4ORLhA@mail.gmail.com>
 <20062.1350560783@critter.freebsd.dk>
 <CAPpMyBGJ3pDzvTg9rtHX+iWnwwuvrEk1EWk5pEVRDkZtBMi4yQ@mail.gmail.com>
Message-ID: <20533.1350569344@critter.freebsd.dk>

--------
In message <CAPpMyBGJ3pDzvTg9rtHX+iWnwwuvrEk1EWk5pEVRDkZtBMi4yQ at mail.gmail.com>, linuxsupport writes:

>Thanks Poul for the information.
>
>So it should be faster but total download time is higher when using varnish
>compared to when served directly from Apache.

That may be the case, yes.

btw:  Do check what the "gzip_level" parameter is set to in varnish,
you can probably easily afford 9, the default I belive is 6.
(Feedback on any effect would be most welcome)

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From benjamindadams at gmail.com  Thu Oct 18 16:11:02 2012
From: benjamindadams at gmail.com (Benjamin Adams)
Date: Thu, 18 Oct 2012 12:11:02 -0400
Subject: Stunnl to Varnish to Nginx with websockets
Message-ID: <CADb0XWL8THeJw-aoC3r45Uq01rG+PdHVb0ppzCLV83ZKPSWr2w@mail.gmail.com>

hello,

I'm trying to setup websockets with varnish to a python tornadojs server.
on connection I just have:
"Request URL: wss://......."
then nothing else.


Next like should be Request : Get
Then "101 Switch"


Network setup:
Stunnel: 8443 -> Varnish(80)
Varnish -> Nginx (8080)
or
Varish -> Tornadojs (87) Websocket

Thanks
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vcl_ex.vcl
Type: application/octet-stream
Size: 2865 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121018/fb821624/attachment.obj>

From lin.support at gmail.com  Thu Oct 18 16:32:28 2012
From: lin.support at gmail.com (linuxsupport)
Date: Thu, 18 Oct 2012 22:02:28 +0530
Subject: ESI and Gzip
In-Reply-To: <20533.1350569344@critter.freebsd.dk>
References: <CAPpMyBHvYHQ1h61qqZmXrpcbb=BuP+yjj0yuMKzBgGsE4ORLhA@mail.gmail.com>
 <20062.1350560783@critter.freebsd.dk>
 <CAPpMyBGJ3pDzvTg9rtHX+iWnwwuvrEk1EWk5pEVRDkZtBMi4yQ@mail.gmail.com>
 <20533.1350569344@critter.freebsd.dk>
Message-ID: <CAPpMyBH+o7fgwjUrwb=FapT6K_DaMqAaObLcGnNt2SSFro2EfA@mail.gmail.com>

I had already set gzip_level to 9 (that is what I could thought of that
time) but no effect.

On Thu, Oct 18, 2012 at 7:39 PM, Poul-Henning Kamp <phk at phk.freebsd.dk>wrote:

> --------
> In message <
> CAPpMyBGJ3pDzvTg9rtHX+iWnwwuvrEk1EWk5pEVRDkZtBMi4yQ at mail.gmail.com>,
> linuxsupport writes:
>
> >Thanks Poul for the information.
> >
> >So it should be faster but total download time is higher when using
> varnish
> >compared to when served directly from Apache.
>
> That may be the case, yes.
>
> btw:  Do check what the "gzip_level" parameter is set to in varnish,
> you can probably easily afford 9, the default I belive is 6.
> (Feedback on any effect would be most welcome)
>
> --
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121018/b64f260e/attachment.html>

From benjamindadams at gmail.com  Thu Oct 18 17:56:06 2012
From: benjamindadams at gmail.com (Benjamin Adams)
Date: Thu, 18 Oct 2012 13:56:06 -0400
Subject: Stunnl to Varnish to Nginx with websockets
In-Reply-To: <CADb0XWL8THeJw-aoC3r45Uq01rG+PdHVb0ppzCLV83ZKPSWr2w@mail.gmail.com>
References: <CADb0XWL8THeJw-aoC3r45Uq01rG+PdHVb0ppzCLV83ZKPSWr2w@mail.gmail.com>
Message-ID: <CADb0XWLWp-BoowhExPH4uLZbfyjCGtz+4FvYDjn9qufD0fPezg@mail.gmail.com>

Also... backend tornado is https.

Thanks
On Oct 18, 2012 12:11 PM, "Benjamin Adams" <benjamindadams at gmail.com> wrote:

> hello,
>
> I'm trying to setup websockets with varnish to a python tornadojs server.
> on connection I just have:
> "Request URL: wss://......."
> then nothing else.
>
>
> Next like should be Request : Get
> Then "101 Switch"
>
>
> Network setup:
> Stunnel: 8443 -> Varnish(80)
> Varnish -> Nginx (8080)
> or
> Varish -> Tornadojs (87) Websocket
>
> Thanks
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121018/66efe427/attachment.html>

From list at sprymed.com  Mon Oct 22 14:17:08 2012
From: list at sprymed.com (list, mailing)
Date: Mon, 22 Oct 2012 10:17:08 -0400
Subject: Multi sites to one Apache
Message-ID: <CAKy=mtCyn0pPNcwnm8JgGPyt7-qUvfC=mK-y=izy2so=M=5A6Q@mail.gmail.com>

I'm trying to setup varnish on a multi domains to go to one apache server
then let apache handle everything after caching.

When I search online People breakout the backend's by domain name.

I do see to add:
sub vcl_hash {
    set req.hash += req.url;
    if (req.http.Accept-Encoding) {
        set req.hash += req.http.Accept-Encoding;
    }
    #set req.hash += req.http.cookie;
    return (hash);
}

for hashing per site.
is that all I need to do and leave backend default to re to server and no
probe?

Thanks

Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121022/f6f33ec9/attachment.html>

From rowan at jetboy.co.uk  Mon Oct 22 16:22:35 2012
From: rowan at jetboy.co.uk (Rowan @ Jetboy)
Date: Mon, 22 Oct 2012 17:22:35 +0100
Subject: Logging the client IP with Nginx/Varnish/Apache
In-Reply-To: <CAKy=mtCyn0pPNcwnm8JgGPyt7-qUvfC=mK-y=izy2so=M=5A6Q@mail.gmail.com>
References: <CAKy=mtCyn0pPNcwnm8JgGPyt7-qUvfC=mK-y=izy2so=M=5A6Q@mail.gmail.com>
Message-ID: <001401cdb071$7233e8d0$569bba70$@co.uk>

I have Nginx listening on port 443 as an SSL terminator, and proxying
unencrypted traffic to Varnish on the same server. Varnish is handling this
traffic, and traffic coming in directly on port 80. All traffic is passed,
unencrypted, to Apache instances on other servers in the cluster. The Apache
instances use mod_rpaf to replace the logged client IP with the contents of
the X-Forwarded-For header.

 

My problem is that if the traffic is coming via Nginx, while the 'correct'
client IP is getting logged in the VarnishNCSA logs, it looks as if Varnish
is (understandably) replacing Nginx's X-Forwarded-For header with 127.0.0.1
downstream, and this is getting logged with Apache. Is there a nice simple
way to stop Varnish rewriting X-Forwarded-For if it's already populated?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121022/db7689d0/attachment.html>

From drais at icantclick.org  Mon Oct 22 16:34:55 2012
From: drais at icantclick.org (david raistrick)
Date: Mon, 22 Oct 2012 12:34:55 -0400 (EDT)
Subject: Logging the client IP with Nginx/Varnish/Apache
In-Reply-To: <001401cdb071$7233e8d0$569bba70$@co.uk>
References: <CAKy=mtCyn0pPNcwnm8JgGPyt7-qUvfC=mK-y=izy2so=M=5A6Q@mail.gmail.com>
 <001401cdb071$7233e8d0$569bba70$@co.uk>
Message-ID: <alpine.BSF.2.00.1210221232250.75038@murf.icantclick.org>

On Mon, 22 Oct 2012, Rowan @ Jetboy wrote:

> My problem is that if the traffic is coming via Nginx, while the 'correct'
> client IP is getting logged in the VarnishNCSA logs, it looks as if Varnish
> is (understandably) replacing Nginx's X-Forwarded-For header with 127.0.0.1
> downstream, and this is getting logged with Apache. Is there a nice simple
> way to stop Varnish rewriting X-Forwarded-For if it's already populated?


choose your own adventure.

in vcl_recv:

         if (req.http.x-forwarded-for) {
             # remove req.http.X-Forwarded-For;
             # set req.http.X-Forwarded-For = client.ip;
             #retain x-forwarded-for data and append
             set req.http.X-Forwarded-For = req.http.X-Forwarded-For "," client.ip;
         } else {
             set req.http.X-Forwarded-For = client.ip;
         }


(noting that I use 2.1...)

--
david raistrick        http://www.netmeister.org/news/learn2quote.html
drais at icantclick.org       ascii ribbon campaign - stop html mail
                                 http://www.asciiribbon.org/






From me at ibotty.net  Mon Oct 22 16:39:57 2012
From: me at ibotty.net (me at ibotty.net)
Date: Mon, 22 Oct 2012 18:39:57 +0200
Subject: Logging the client IP with Nginx/Varnish/Apache
In-Reply-To: <001401cdb071$7233e8d0$569bba70$@co.uk>
References: <CAKy=mtCyn0pPNcwnm8JgGPyt7-qUvfC=mK-y=izy2so=M=5A6Q@mail.gmail.com>
 <001401cdb071$7233e8d0$569bba70$@co.uk>
Message-ID: <508576DD.2030608@ibotty.net>

hi,

> Is there a nice simple way to stop Varnish rewriting X-Forwarded-For
> if it's already populated?

you should not unconditionally trust any http header. so the best way
might be to check for presence of the header and rewrite it unless the
client's ip is 127.0.0.1. two if clauses should do just fine. (if it
does not work for you, i can get a snippet i once wrote. but that won't
be tomorrow.)

tobi



From rowan at jetboy.co.uk  Mon Oct 22 16:45:54 2012
From: rowan at jetboy.co.uk (Rowan @ Jetboy)
Date: Mon, 22 Oct 2012 17:45:54 +0100
Subject: Error 413 Request Entity Too Large
In-Reply-To: <CAOXZevAOiG0r2OTMs3-+gOm3caTQ6gVUnwdM7PVkFQ5D2uFotQ@mail.gmail.com>
References: <CABfdyPYCRAXjLVNWTNyVUdQ0poNc1Pv_htKp3en_MV8RWYM2Og@mail.gmail.com>
 <002001cda53d$5b81e540$1285afc0$@co.uk>
 <CAOXZevAOiG0r2OTMs3-+gOm3caTQ6gVUnwdM7PVkFQ5D2uFotQ@mail.gmail.com>
Message-ID: <001901cdb074$b4056600$1c103200$@co.uk>

Cheers Per. Nick Tailor's been helping out off-list as well, and I've ended
up with:

 

DAEMON_OPTS="-a :80 \

             -f /etc/varnish/varnish.vcl \

             -p http_req_size=65536 \

             -p http_resp_size=65536 \

             -p sess_workspace=262144 \

             -p thread_pool_min=250 \

             -p thread_pool_max=2000 \

             -s malloc,4G \

             -S /etc/varnish/secret \

             -T localhost:6082"

 

I tried increasing http_req_size and http_resp_size first, together with
http_req_hdr_len and http_resp_hdr_len, but I was still getting reports of
problems. Bumping up sess_workspace as well *seems* to have fixed things,
but time will tell. Another fix was to get the client to clear their
cookies, but I don't know whether this is a temporary or permanent solution.
The problem seemed to be isolated to IE/Windows, but included IE8/Windows XP
and IE9/Windows 7. I can't replicate the problem use virgin Windows installs
on VMs or other Windows boxes I have access to. The user with IE8/Windows
had no problem accessing the forum on a non-Windows tablet via the same ISP.
While having a (tentative) fix is great, I still don't know why these users
were having the problem in the first place.

 

Thanks to both.

 

 

From: Per Buer [mailto:perbu at varnish-software.com] 
Sent: 11 October 2012 13:06
To: rowan at jetboy.co.uk
Cc: varnish-misc at varnish-cache.org
Subject: Re: Error 413 Request Entity Too Large

 

On Mon, Oct 8, 2012 at 12:12 PM, Rowan @ Jetboy <rowan at jetboy.co.uk> wrote:

I've had a handful of complaints from my website users that they're seeing
the error below when trying to access an installation of the MyBB forum.

 

Error 413

Request Entity Too Large

Guru Meditation:

XID: 190197240

Varnish cache server

 

I'm not 100% sure but if memory serves me well you need to tweak this one:

 

       http_req_size

 

              . Units: bytes

 

              . Default: 32768

 

              Maximum  number  of  bytes  of  HTTP client request we will
deal

              with.  This is a limit on all bytes up to the double blank
line

              which  ends  the  HTTP  request.   The memory for the request
is

              allocated from the session workspace (param: sess_workspace)
and

              this parameter limits how much of that the request is allowed
to

              take up.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121022/43f0ebc3/attachment.html>

From oddur at ccpgames.com  Mon Oct 22 17:49:21 2012
From: oddur at ccpgames.com (=?iso-8859-1?Q?Oddur_Sn=E6r_Magn=FAsson?=)
Date: Mon, 22 Oct 2012 17:49:21 +0000
Subject: Sticky Session with an assignable backend
Message-ID: <E49910F11F025B4990FA56AD8158EB570634731A@RKV-IT-EXCH103.ccp.ad.local>

I have a problem where I need to be able to do "sticky session" based on some identity, but be able to tell varnish which identity belongs to which backend.

In my case, I have a value in the request header(oauth token) which should always go to the same backend. I begin by sending the request to a random backend, and if the users session does not live on that backend I get X-Accel-Redirect header back telling me which backend that request should go to.  I can easily do that part using a restart and send the request to the correct one.

But  from then on I want to remember that that specific identity should always go to that given backend.

I'm not aware of being able to store any such mapping inside varnish, so I thought about using the memcached vmod to store the mapping and then looking it up on each request.  I have not tried it yet so I'm not sure what the performance hit will be on that extra memcached lookup per request.   But in fact, I think it's an overkill since all I need is way to access a in memory hashtable within my vcl.

Would it be possible to "control" the client backend director, mapping a specific identity to a backend ?
Is there a known way to access a hashtable which is shared by the workee threads within a varnish instance, possibly using inline c?


Best regards,

Oddur Sn?r Magn?sson
Senior Programmer | CCP Games | Reykjav?k | Iceland

:wq

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121022/f644435e/attachment.html>

From perbu at varnish-software.com  Tue Oct 23 07:03:40 2012
From: perbu at varnish-software.com (Per Buer)
Date: Tue, 23 Oct 2012 09:03:40 +0200
Subject: Multi sites to one Apache
In-Reply-To: <CAKy=mtCyn0pPNcwnm8JgGPyt7-qUvfC=mK-y=izy2so=M=5A6Q@mail.gmail.com>
References: <CAKy=mtCyn0pPNcwnm8JgGPyt7-qUvfC=mK-y=izy2so=M=5A6Q@mail.gmail.com>
Message-ID: <CAOXZevBY0bTX16C3RkLFO9jCAgNL2ShM9DtGd8JiW_KHWTxk7A@mail.gmail.com>

Hi Ben,

On Mon, Oct 22, 2012 at 4:17 PM, list, mailing <list at sprymed.com> wrote:

> I'm trying to setup varnish on a multi domains to go to one apache server
> then let apache handle everything after caching.


You don't need to do anything. Varnish does this out of the box. Just
define the backend and if there is only one Varnish will automatically send
all traffic to it.



> When I search online People breakout the backend's by domain name.
>
> I do see to add:
> sub vcl_hash {
>     set req.hash += req.url;
>     if (req.http.Accept-Encoding) {
>         set req.hash += req.http.Accept-Encoding;
>     }
>     #set req.hash += req.http.cookie;
>     return (hash);
> }
>
> for hashing per site.
>

What you're doing here is totally different and it is not related to what
you are trying to achieve.



-- 
 <http://www.varnish-software.com/> *Per Buer*
Varnish Software AS
Phone: +958 39 117 | Skype: per.buer
We Make Websites Fly!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121023/c46cfda8/attachment.html>

From rowan at jetboy.co.uk  Tue Oct 23 11:03:00 2012
From: rowan at jetboy.co.uk (Rowan @ Jetboy)
Date: Tue, 23 Oct 2012 12:03:00 +0100
Subject: Logging the client IP with Nginx/Varnish/Apache
In-Reply-To: <alpine.BSF.2.00.1210221232250.75038@murf.icantclick.org>
References: <CAKy=mtCyn0pPNcwnm8JgGPyt7-qUvfC=mK-y=izy2so=M=5A6Q@mail.gmail.com>
 <001401cdb071$7233e8d0$569bba70$@co.uk>
 <alpine.BSF.2.00.1210221232250.75038@murf.icantclick.org>
Message-ID: <000601cdb10d$f726f8b0$e574ea10$@co.uk>

David,

I'm a little confused. As I understand it, the default Varnish vcl_recv
behaviour will be appended to my custom rules. In Varnish 3, this includes:

# sub vcl_recv {
#     if (req.restarts == 0) {
#       if (req.http.x-forwarded-for) {
#           set req.http.X-Forwarded-For =
#               req.http.X-Forwarded-For + ", " + client.ip;
#       } else {
#           set req.http.X-Forwarded-For = client.ip;
#       }
#     }

which seems to be pretty much the same as you've posted. Can I assume that
this already being parsed (as my custom vcl_recv isn't terminated), and not
doing what I need?

<snip>

choose your own adventure.

in vcl_recv:

         if (req.http.x-forwarded-for) {
             # remove req.http.X-Forwarded-For;
             # set req.http.X-Forwarded-For = client.ip;
             #retain x-forwarded-for data and append
             set req.http.X-Forwarded-For = req.http.X-Forwarded-For ","
client.ip;
         } else {
             set req.http.X-Forwarded-For = client.ip;
         }


(noting that I use 2.1...)

--
david raistrick        http://www.netmeister.org/news/learn2quote.html
drais at icantclick.org       ascii ribbon campaign - stop html mail
                                 http://www.asciiribbon.org/






From lasse.karstensen at gmail.com  Tue Oct 23 11:18:52 2012
From: lasse.karstensen at gmail.com (Lasse Karstensen)
Date: Tue, 23 Oct 2012 13:18:52 +0200
Subject: Sticky Session with an assignable backend
In-Reply-To: <E49910F11F025B4990FA56AD8158EB570634731A@RKV-IT-EXCH103.ccp.ad.local>
References: <E49910F11F025B4990FA56AD8158EB570634731A@RKV-IT-EXCH103.ccp.ad.local>
Message-ID: <20121023111845.GC30742@sierra.hyse.org>

Oddur Sn?r Magn?sson:
> I have a problem where I need to be able to do "sticky session" based on some identity, but be able to tell varnish which identity belongs to which backend.
> In my case, I have a value in the request header(oauth token) which should always go to the same backend. I begin by sending the request to a random backend, and if the users session does not live on that backend I get X-Accel-Redirect header back telling me which backend that request should go to.  I can easily do that part using a restart and send the request to the correct one.
> But  from then on I want to remember that that specific identity should always go to that given backend.

Given that the HTTP client supports it, I'd just put the backend information
into a client side cookie. 

Send a set-cookie in deliver, and extract/remove it in recv on the next
request.

-- 
Lasse Karstensen
Varnish Software AS
http://www.varnish-software.com/



From vstavrinov at gmail.com  Tue Oct 23 13:33:58 2012
From: vstavrinov at gmail.com (Vladimir Stavrinov)
Date: Tue, 23 Oct 2012 17:33:58 +0400
Subject: logging request time
Message-ID: <20121023133358.GA31642@mana.playfast.ru>


Is subj possible? i.e. if there available variables like $request_time
in nginx or %T, or %D in apache? I do believe that std.log(key:value)
may be used to output this value into ncsa log via %{VCL_Log:key)
formatter, but question is where to get this value from?

-- 

***************************
##  Vladimir Stavrinov
##  vstavrinov at gmail.com
***************************




From mi+thun at aldan.algebra.com  Mon Oct 22 20:54:54 2012
From: mi+thun at aldan.algebra.com (Mikhail T.)
Date: Mon, 22 Oct 2012 16:54:54 -0400
Subject: Can vmodules provide functions with optional arguments?
Message-ID: <5085B29E.70802@aldan.algebra.com>

The subject says it: can functions provided by a loaded module have optional 
arguments? So that one could, for example, say:

    g2o_verify(req.http.X-Akamai-G2O-Auth-Data, req.http.x-akamai-g2o-auth-sign,
    req.path)

or:

    g2o_verify(req.http.X-Akamai-G2O-Auth-Data, req.http.x-akamai-g2o-auth-sign,
    req.path, *600*)

I'd like the last (optional) argument in this example to be filled-out by 
Varnish if omitted by the caller. It does not seem like vmod.py currently 
supports anything like it, but I wanted to check... Thanks!

    -mi

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121022/0aced211/attachment.html>

From tfheen at varnish-software.com  Wed Oct 24 08:13:24 2012
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Wed, 24 Oct 2012 10:13:24 +0200
Subject: Multi sites to one Apache
In-Reply-To: <CAKy=mtCyn0pPNcwnm8JgGPyt7-qUvfC=mK-y=izy2so=M=5A6Q@mail.gmail.com>
References: <CAKy=mtCyn0pPNcwnm8JgGPyt7-qUvfC=mK-y=izy2so=M=5A6Q@mail.gmail.com>
Message-ID: <20121024081324.GB24585@err.no>

]] list, mailing

> I'm trying to setup varnish on a multi domains to go to one apache server
> then let apache handle everything after caching.
> 
> When I search online People breakout the backend's by domain name.
> 
> I do see to add:
> sub vcl_hash {
>     set req.hash += req.url;
>     if (req.http.Accept-Encoding) {
>         set req.hash += req.http.Accept-Encoding;
>     }
>     #set req.hash += req.http.cookie;
>     return (hash);
> }
> 
> for hashing per site.

This won't hash per site.  It will put the request URL and the
Accept-Encoding into the hash.  This seems like a bad idea, I'd
recommend you rather hash on req.url and req.http.host (as is done in
the default vcl_hash) and leave Accept-Encoding out of the mix.

-- 
Tollef Fog Heen
Technical lead | Varnish Software AS
t: +47 21 98 92 64
We Make Websites Fly!



From tfheen at varnish-software.com  Wed Oct 24 08:15:45 2012
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Wed, 24 Oct 2012 10:15:45 +0200
Subject: Can vmodules provide functions with optional arguments?
In-Reply-To: <5085B29E.70802@aldan.algebra.com>
References: <5085B29E.70802@aldan.algebra.com>
Message-ID: <20121024081545.GC24585@err.no>

]] Mikhail T.

> The subject says it: can functions provided by a loaded module have
> optional arguments?

No, they can't.  You could have it take a magic value like -1 or similar
to signify "unset", though.

-- 
Tollef Fog Heen
Technical lead | Varnish Software AS
t: +47 21 98 92 64
We Make Websites Fly!



From tfheen at varnish-software.com  Wed Oct 24 08:17:46 2012
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Wed, 24 Oct 2012 10:17:46 +0200
Subject: logging request time
In-Reply-To: <20121023133358.GA31642@mana.playfast.ru>
References: <20121023133358.GA31642@mana.playfast.ru>
Message-ID: <20121024081746.GD24585@err.no>

]] Vladimir Stavrinov

> Is subj possible? i.e. if there available variables like $request_time
> in nginx or %T, or %D in apache? I do believe that std.log(key:value)
> may be used to output this value into ncsa log via %{VCL_Log:key)
> formatter, but question is where to get this value from?

There's currently no support for the time for the full request.  You can
use %{Varnish:time_firstbyte}x in the format string for the time to
first byte.

-- 
Tollef Fog Heen
Technical lead | Varnish Software AS
t: +47 21 98 92 64
We Make Websites Fly!



From oddur at ccpgames.com  Wed Oct 24 17:50:44 2012
From: oddur at ccpgames.com (=?utf-8?B?T2RkdXIgU27DpnIgTWFnbsO6c3Nvbg==?=)
Date: Wed, 24 Oct 2012 17:50:44 +0000
Subject: Sticky Session with an assignable backend
In-Reply-To: <20121023111845.GC30742@sierra.hyse.org>
References: <E49910F11F025B4990FA56AD8158EB570634731A@RKV-IT-EXCH103.ccp.ad.local>
 <20121023111845.GC30742@sierra.hyse.org>
Message-ID: <E49910F11F025B4990FA56AD8158EB570634C092@RKV-IT-EXCH103.ccp.ad.local>

Unfortunately its 'a REST APi which does not support cookies. 

I'm attempting to get it working as a proof of concept using this vmod https://github.com/v	arnish/libvmod-var 

I might extend it to support a hashtable for efficient lookups. 

-----Original Message-----
From: Lasse Karstensen [mailto:lasse.karstensen at gmail.com] 
Sent: Tuesday, October 23, 2012 11:19 AM
To: Oddur Sn?r Magn?sson
Cc: varnish-misc at varnish-cache.org
Subject: Re: Sticky Session with an assignable backend

Oddur Sn?r Magn?sson:
> I have a problem where I need to be able to do "sticky session" based on some identity, but be able to tell varnish which identity belongs to which backend.
> In my case, I have a value in the request header(oauth token) which should always go to the same backend. I begin by sending the request to a random backend, and if the users session does not live on that backend I get X-Accel-Redirect header back telling me which backend that request should go to.  I can easily do that part using a restart and send the request to the correct one.
> But  from then on I want to remember that that specific identity should always go to that given backend.

Given that the HTTP client supports it, I'd just put the backend information into a client side cookie. 

Send a set-cookie in deliver, and extract/remove it in recv on the next request.

--
Lasse Karstensen
Varnish Software AS
http://www.varnish-software.com/


From vstavrinov at gmail.com  Thu Oct 25 15:01:15 2012
From: vstavrinov at gmail.com (Vladimir Stavrinov)
Date: Thu, 25 Oct 2012 19:01:15 +0400
Subject: logging request time
In-Reply-To: <20121024081746.GD24585@err.no>
References: <20121023133358.GA31642@mana.playfast.ru>
 <20121024081746.GD24585@err.no>
Message-ID: <20121025150115.GA24645@mana.playfast.ru>

On Wed, Oct 24, 2012 at 10:17:46AM +0200, Tollef Fog Heen wrote:

> There's currently no support for the time for the full request.  You
> can

There are some times data in ReqStart & ReqEnd tags of shared memory
log. May be there some variables exists to get those data?


> use %{Varnish:time_firstbyte}x in the format string for the time to
> first byte.

It doesn't help as we need calculate download speed of clients.

-- 

***************************
##  Vladimir Stavrinov
##  vstavrinov at gmail.com
***************************




From hugues.alary at gmail.com  Thu Oct 25 18:49:16 2012
From: hugues.alary at gmail.com (Hugues Alary)
Date: Thu, 25 Oct 2012 11:49:16 -0700
Subject: Last-Modified, IMS, 304 and ESI includes
Message-ID: <CAN-YAk9Koy+TL-g_-8dBkCf-XDhnoWBFXqD0AmjuLrJXim4pLw@mail.gmail.com>

Hi there,

I've been reading a lot about the Last-Modified header, IMS, 304 responses
and ESI, on various websites and mostly on the Varnish trac because I'm
trying to solve the following problem:

All the pages on my website contain ESI includes. I'd like the pages to
have their Last-Modified header set to the Last-Modified of
the youngest ESI include contained in the page.

All the documentation I read seem to indicate that it has been more or less
on the roadmap for the last few years, I even saw some patches floating
around, but all that stuff seems pretty old and I'm using Varnish 3.0.2 and
could even use 3.0.3.

Is there any way to achieve that? Does the experimental-IMS branch solves
this "problem"? If yes, is it too experimental for production use?

I have the feeling that it is not possible at the moment, but I'd really
like to have this feature as it would fairly improve the performances of my
website.

Thanks for your help,

-Hugues
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121025/4d82e075/attachment.html>

From perbu at varnish-software.com  Fri Oct 26 07:15:14 2012
From: perbu at varnish-software.com (Per Buer)
Date: Fri, 26 Oct 2012 09:15:14 +0200
Subject: Last-Modified, IMS, 304 and ESI includes
In-Reply-To: <CAN-YAk9Koy+TL-g_-8dBkCf-XDhnoWBFXqD0AmjuLrJXim4pLw@mail.gmail.com>
References: <CAN-YAk9Koy+TL-g_-8dBkCf-XDhnoWBFXqD0AmjuLrJXim4pLw@mail.gmail.com>
Message-ID: <CAOXZevAUj9bz7+_mytfaO7rMpW-Vy+mKLTQQwizrteG22d9rMQ@mail.gmail.com>

Hi Hughes.

On Thu, Oct 25, 2012 at 8:49 PM, Hugues Alary <hugues.alary at gmail.com>wrote:

> Hi there,
>
> I've been reading a lot about the Last-Modified header, IMS, 304 responses
> and ESI, on various websites and mostly on the Varnish trac because I'm
> trying to solve the following problem:
>
> All the pages on my website contain ESI includes. I'd like the pages to
> have their Last-Modified header set to the Last-Modified of
> the youngest ESI include contained in the page.
>

This is well known weakness, I'm afraid.


> All the documentation I read seem to indicate that it has been more or
> less on the roadmap for the last few years, I even saw some patches
> floating around, but all that stuff seems pretty old and I'm using Varnish
> 3.0.2 and could even use 3.0.3.
>
> Is there any way to achieve that? Does the experimental-IMS branch solves
> this "problem"? If yes, is it too experimental for production use?
>

Currently I don't know any reliable way of doing it. The IMS branch is
based of trunk code so whether it is stable enough is dependent on your
usage. Also, I don't know if it actually addresses the ESI/IMS issue.


I have the feeling that it is not possible at the moment, but I'd really
> like to have this feature as it would fairly improve the performances of my
> website.
>

It would have to implemented first.


-- 
 <http://www.varnish-software.com/> *Per Buer*
Varnish Software
Phone: +47 958 39 117 | Skype: per.buer
We Make Websites Fly!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121026/6792042a/attachment.html>

From lasse.karstensen at gmail.com  Fri Oct 26 10:46:30 2012
From: lasse.karstensen at gmail.com (Lasse Karstensen)
Date: Fri, 26 Oct 2012 12:46:30 +0200
Subject: Sticky Session with an assignable backend
In-Reply-To: <E49910F11F025B4990FA56AD8158EB570634C092@RKV-IT-EXCH103.ccp.ad.local>
References: <E49910F11F025B4990FA56AD8158EB570634731A@RKV-IT-EXCH103.ccp.ad.local>
 <20121023111845.GC30742@sierra.hyse.org>
 <E49910F11F025B4990FA56AD8158EB570634C092@RKV-IT-EXCH103.ccp.ad.local>
Message-ID: <20121026104629.GD30742@sierra.hyse.org>

Oddur Sn?r Magn?sson:
> Unfortunately its 'a REST APi which does not support cookies. 

I see.

> I'm attempting to get it working as a proof of concept using this vmod https://github.com/v	arnish/libvmod-var 
> I might extend it to support a hashtable for efficient lookups. 

Please do, we've had that on our wishlist for a while.

-- 
Lasse Karstensen
Varnish Software AS
http://www.varnish-software.com/



From hugues.alary at gmail.com  Fri Oct 26 18:18:10 2012
From: hugues.alary at gmail.com (Hugues Alary)
Date: Fri, 26 Oct 2012 11:18:10 -0700
Subject: Last-Modified, IMS, 304 and ESI includes
In-Reply-To: <CAOXZevAUj9bz7+_mytfaO7rMpW-Vy+mKLTQQwizrteG22d9rMQ@mail.gmail.com>
References: <CAN-YAk9Koy+TL-g_-8dBkCf-XDhnoWBFXqD0AmjuLrJXim4pLw@mail.gmail.com>
 <CAOXZevAUj9bz7+_mytfaO7rMpW-Vy+mKLTQQwizrteG22d9rMQ@mail.gmail.com>
Message-ID: <CAN-YAk8qS6WbBP7W94qWFk0uQmQD9jtz5zLXPaEB3TSsnQDCbA@mail.gmail.com>

Hi Per,

Thanks for the information!

-H

On Fri, Oct 26, 2012 at 12:15 AM, Per Buer <perbu at varnish-software.com>wrote:

> Hi Hughes.
>
> On Thu, Oct 25, 2012 at 8:49 PM, Hugues Alary <hugues.alary at gmail.com>wrote:
>
>> Hi there,
>>
>> I've been reading a lot about the Last-Modified header, IMS, 304
>> responses and ESI, on various websites and mostly on the Varnish trac
>> because I'm trying to solve the following problem:
>>
>> All the pages on my website contain ESI includes. I'd like the pages to
>> have their Last-Modified header set to the Last-Modified of
>> the youngest ESI include contained in the page.
>>
>
> This is well known weakness, I'm afraid.
>
>
>> All the documentation I read seem to indicate that it has been more or
>> less on the roadmap for the last few years, I even saw some patches
>> floating around, but all that stuff seems pretty old and I'm using Varnish
>> 3.0.2 and could even use 3.0.3.
>>
>> Is there any way to achieve that? Does the experimental-IMS branch solves
>> this "problem"? If yes, is it too experimental for production use?
>>
>
> Currently I don't know any reliable way of doing it. The IMS branch is
> based of trunk code so whether it is stable enough is dependent on your
> usage. Also, I don't know if it actually addresses the ESI/IMS issue.
>
>
> I have the feeling that it is not possible at the moment, but I'd really
>> like to have this feature as it would fairly improve the performances of my
>> website.
>>
>
> It would have to implemented first.
>
>
> --
>  <http://www.varnish-software.com/> *Per Buer*
> Varnish Software
> Phone: +47 958 39 117 | Skype: per.buer
> We Make Websites Fly!
>
>
>


-- 
Hugues ALARY
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121026/818d5f18/attachment.html>

From trystanleftwich at gmail.com  Mon Oct 29 23:11:04 2012
From: trystanleftwich at gmail.com (Trystan Leftwich)
Date: Mon, 29 Oct 2012 16:11:04 -0700
Subject: Dynamically creating strings in inline c for use in the VRT_SetHdr
 function
Message-ID: <CADAQ0t7Vmq-m=T72W_GtuN4HxZkXNvC4m7oMJr8R6RrWtAf5mA@mail.gmail.com>

Hi all,

I have a few questions about using VRT_SetHdr function. I'm sure some
of this is due to lack of c experience so I apologize if this is the
case.

Firstly, Is there a way to create the header string dynamically. I
know the header string has to be in the format "\020X-Forwarded-For:",
it has to start with the length of the string in octal, But if I'm
creating the header name dynamically, Say if i wanted to create
separate headers based of GeoIP data or something similar, How could I
go about creating the string so It matches the format?

I've tried to overwrite the first byte of the header string to be the
length of the rest of the header string as an unsigned int, which will
work to the extent that Varnish passes all of the asserts it and will
display the page correctly, But the header itself will be incorrect,
it seems like the browser is trying to  convert the integer to a char
and ends up just displaying and an empty box and the string itself.

And secondly, purely out of interest, Could someone explain why the
header is prefixed with its length in octal? Does it have something to
do with correctly setting the amount of memory needed?

Thanks!

Trystan



From pprocacci at datapipe.com  Tue Oct 30 01:05:51 2012
From: pprocacci at datapipe.com (Paul A. Procacci)
Date: Mon, 29 Oct 2012 20:05:51 -0500
Subject: Dynamically creating strings in inline c for use in the
 VRT_SetHdr function
In-Reply-To: <CADAQ0t7Vmq-m=T72W_GtuN4HxZkXNvC4m7oMJr8R6RrWtAf5mA@mail.gmail.com>
References: <CADAQ0t7Vmq-m=T72W_GtuN4HxZkXNvC4m7oMJr8R6RrWtAf5mA@mail.gmail.com>
Message-ID: <20121030010551.GB8791@nat.myhome>

> Hi all,

Hi.

> Firstly, Is there a way to create the header string dynamically. I
> know the header string has to be in the format "\020X-Forwarded-For:",
> it has to start with the length of the string in octal, But if I'm
> creating the header name dynamically, Say if i wanted to create
> separate headers based of GeoIP data or something similar, How could I
> go about creating the string so It matches the format?

Here is one way:
###############################################
  char *str = "X-Header: test";
  char *header;

  asprintf(&header, "\\0%o%s", strlen(str), str);
  # Make sure to check return value, etc.
###############################################

> And secondly, purely out of interest, Could someone explain why the
> header is prefixed with its length in octal? Does it have something to
> do with correctly setting the amount of memory needed?

I do not know the reasoning.

~Paul

________________________________

This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you.



From yao.yang at autonavi.com  Tue Oct 30 07:04:16 2012
From: yao.yang at autonavi.com (=?gb2312?B?0e7Sog==?=)
Date: Tue, 30 Oct 2012 07:04:16 +0000
Subject: [For Help] Varnish error information
Message-ID: <B77F7DEDAE19BA4BBAC66F99002545EF1AE87CCA@DAGDH01.autonavi.com>


Hi, All,
I used Varnish for half a year, these days some errors puzzled me.
And I find some error informations from  /var/log/message file.
Is my fault or the software's bug?

Thanks.
Yangyao
________________________________
Oct 28 03:47:01 kernel: imklog 4.6.2, log source = /proc/kmsg started.
Oct 28 03:47:01 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1721" x-info="http://www.rsyslog.com"] (re)start
Oct 28 03:47:39 rhsm-complianced: This system is missing one or more valid entitlement certificates. Please run subscription-manager for more information.
Oct 28 20:20:01 kernel: varnishncsa[17385]: segfault at 7f2a06670f58 ip 00007f2a06687005 sp 00007fff242d0720 error 4 in libvarnishapi.so.1.0.0[7f2a0667f000+11000]
Oct 28 20:20:01 abrt[17387]: abrt daemon is not running. If it crashed, /proc/sys/kernel/core_pattern contains a stale value, consider resetting it to 'core'
Oct 29 03:13:03 varnishd[2591]: Child (2592) not responding to CLI, killing it.
Oct 29 03:13:13 varnishd[2591]: Child (2592) not responding to CLI, killing it.
Oct 29 03:13:15 varnishd[2591]: Child (2592) not responding to CLI, killing it.
Oct 29 03:13:15 varnishd[2591]: Child (2592) died signal=3 (core dumped)
Oct 29 03:13:15 varnishd[2591]: child (28563) Started
Oct 29 03:13:15 varnishd[2591]: Child (28563) said Child starts
Oct 29 03:45:46 rhsm-complianced: This system is missing one or more valid entitlement certificates. Please run subscription-manager for more information.
Oct 29 17:23:16 varnishd[2591]: Child (28563) not responding to CLI, killing it.
Oct 29 17:23:26 varnishd[2591]: Child (28563) not responding to CLI, killing it.
Oct 29 17:23:34 varnishd[2591]: Child (28563) not responding to CLI, killing it.
Oct 29 17:23:34 varnishd[2591]: Child (28563) not responding to CLI, killing it.
Oct 29 17:23:34 varnishd[2591]: Child (28563) died signal=3 (core dumped)
Oct 29 17:23:34 varnishd[2591]: child (15919) Started
Oct 29 17:23:34 varnishd[2591]: Child (15919) said Child starts
Oct 29 20:56:39 varnishd[2591]: Child (15919) not responding to CLI, killing it.
Oct 29 20:56:43 varnishd[2591]: Child (15919) not responding to CLI, killing it.
Oct 29 20:56:43 varnishd[2591]: Child (15919) not responding to CLI, killing it.
Oct 29 20:56:43 varnishd[2591]: Child (15919) died signal=3 (core dumped)
Oct 29 20:56:43 varnishd[2591]: child (22868) Started
Oct 29 20:56:43 varnishd[2591]: Child (22868) said Child starts
Oct 30 03:32:46 rhsm-complianced: This system is missing one or more valid entitlement certificates. Please run subscription-manager for more information.
Oct 30 10:30:40 kernel: varnishncsa invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
Oct 30 10:30:40 kernel: varnishncsa cpuset=/ mems_allowed=0-1
Oct 30 10:30:40 kernel: Pid: 15396, comm: varnishncsa Not tainted 2.6.32-131.0.15.el6.x86_64 #1
Oct 30 10:30:40 kernel: Call Trace:
Oct 30 10:30:40 kernel: [<ffffffff810c0101>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Oct 30 10:30:40 kernel: [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0
Oct 30 10:30:40 kernel: [<ffffffff81110880>] ? select_bad_process+0xd0/0x110
Oct 30 10:30:40 kernel: [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0
Oct 30 10:30:40 kernel: [<ffffffff81110b19>] ? out_of_memory+0x199/0x210
Oct 30 10:30:40 kernel: [<ffffffff811202dd>] ? __alloc_pages_nodemask+0x80d/0x8b0
Oct 30 10:30:40 kernel: [<ffffffff8115464a>] ? alloc_pages_current+0xaa/0x110
Oct 30 10:30:40 kernel: [<ffffffff8110d717>] ? __page_cache_alloc+0x87/0x90
Oct 30 10:30:40 kernel: [<ffffffff81122c4b>] ? __do_page_cache_readahead+0xdb/0x210
Oct 30 10:30:40 kernel: [<ffffffff81122da1>] ? ra_submit+0x21/0x30
Oct 30 10:30:40 kernel: [<ffffffff8110e9e3>] ? filemap_fault+0x4c3/0x500
Oct 30 10:30:40 kernel: [<ffffffff812628a9>] ? cpumask_next_and+0x29/0x50
Oct 30 10:30:40 kernel: [<ffffffff81137114>] ? __do_fault+0x54/0x510
Oct 30 10:30:40 kernel: [<ffffffff811376c7>] ? handle_pte_fault+0xf7/0xb50
Oct 30 10:30:40 kernel: [<ffffffff8100987e>] ? __switch_to+0x26e/0x320
Oct 30 10:30:40 kernel: [<ffffffff81057f99>] ? find_busiest_queue+0x69/0x150
Oct 30 10:30:40 kernel: [<ffffffff811382f8>] ? handle_mm_fault+0x1d8/0x2a0
Oct 30 10:30:40 kernel: [<ffffffff81092ff3>] ? __hrtimer_start_range_ns+0x1a3/0x460
Oct 30 10:30:40 kernel: [<ffffffff81041529>] ? __do_page_fault+0x139/0x480
Oct 30 10:30:40 kernel: [<ffffffff8109332f>] ? hrtimer_try_to_cancel+0x3f/0xd0
Oct 30 10:30:40 kernel: [<ffffffff810933e2>] ? hrtimer_cancel+0x22/0x30
Oct 30 10:30:40 kernel: [<ffffffff814dccc3>] ? do_nanosleep+0x93/0xc0
Oct 30 10:30:40 kernel: [<ffffffff810934b4>] ? hrtimer_nanosleep+0xc4/0x180
Oct 30 10:30:40 kernel: [<ffffffff810922a0>] ? hrtimer_wakeup+0x0/0x30
Oct 30 10:30:40 kernel: [<ffffffff814e0c3e>] ? do_page_fault+0x3e/0xa0
Oct 30 10:30:40 kernel: [<ffffffff814ddfe5>] ? page_fault+0x25/0x30
Oct 30 10:30:40 kernel: Mem-Info:
Oct 30 10:30:40 kernel: Node 0 DMA per-cpu:
Oct 30 10:30:40 kernel: CPU    0: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    1: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    2: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    3: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    4: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    5: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    6: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    7: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    8: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    9: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU   10: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU   11: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU   12: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU   13: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU   14: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU   15: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: Node 0 DMA32 per-cpu:
Oct 30 10:30:40 kernel: CPU    0: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    1: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    2: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    3: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    4: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    5: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    6: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    7: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    8: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    9: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   10: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   11: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   12: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   13: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   14: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   15: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: Node 0 Normal per-cpu:
Oct 30 10:30:40 kernel: CPU    0: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    1: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    2: hi:  186, btch:  31 usd:  11
Oct 30 10:30:40 kernel: CPU    3: hi:  186, btch:  31 usd:   2
Oct 30 10:30:40 kernel: CPU    4: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    5: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    6: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    7: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    8: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    9: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   10: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   11: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   12: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   13: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   14: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   15: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: Node 1 Normal per-cpu:
Oct 30 10:30:40 kernel: CPU    0: hi:  186, btch:  31 usd:   5
Oct 30 10:30:40 kernel: CPU    1: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    2: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    3: hi:  186, btch:  31 usd:   1
Oct 30 10:30:40 kernel: CPU    4: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    5: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    6: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    7: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    8: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    9: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   10: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   11: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   12: hi:  186, btch:  31 usd:   8
Oct 30 10:30:40 kernel: CPU   13: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   14: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   15: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: active_anon:15180189 inactive_anon:868880 isolated_anon:0
Oct 30 10:30:40 kernel: active_file:496 inactive_file:603 isolated_file:0
Oct 30 10:30:40 kernel: unevictable:0 dirty:22 writeback:20 unstable:0
Oct 30 10:30:40 kernel: free:55664 slab_reclaimable:10223 slab_unreclaimable:284822
Oct 30 10:30:40 kernel: mapped:386 shmem:0 pagetables:54263 bounce:0
Oct 30 10:30:40 kernel: Node 0 DMA free:15616kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15224kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 30 10:30:40 kernel: lowmem_reserve[]: 0 2991 32281 32281
Oct 30 10:30:40 kernel: Node 0 DMA32 free:121196kB min:4168kB low:5208kB high:6252kB active_anon:1856648kB inactive_anon:457828kB active_file:124kB inactive_file:372kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3063392kB mlocked:0kB dirty:12kB writeback:16kB mapped:156kB shmem:0kB slab_reclaimable:3732kB slab_unreclaimable:2508kB kernel_stack:0kB pagetables:5060kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:352 all_unreclaimable? no
Oct 30 10:30:40 kernel: lowmem_reserve[]: 0 0 29290 29290
Oct 30 10:30:40 kernel: Node 0 Normal free:40808kB min:40844kB low:51052kB high:61264kB active_anon:27839988kB inactive_anon:1739808kB active_file:1104kB inactive_file:1240kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:29992960kB mlocked:0kB dirty:52kB writeback:40kB mapped:332kB shmem:0kB slab_reclaimable:19552kB slab_unreclaimable:584592kB kernel_stack:10656kB pagetables:107120kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1528 all_unreclaimable? no
Oct 30 10:30:40 kernel: lowmem_reserve[]: 0 0 0 0
Oct 30 10:30:40 kernel: Node 1 Normal free:45036kB min:45072kB low:56340kB high:67608kB active_anon:31024120kB inactive_anon:1277884kB active_file:756kB inactive_file:800kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:33095680kB mlocked:0kB dirty:24kB writeback:24kB mapped:1056kB shmem:0kB slab_reclaimable:17608kB slab_unreclaimable:552188kB kernel_stack:5456kB pagetables:104872kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1267 all_unreclaimable? no
Oct 30 10:30:40 kernel: lowmem_reserve[]: 0 0 0 0
Oct 30 10:30:40 kernel: Node 0 DMA: 2*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15616kB
Oct 30 10:30:40 kernel: Node 0 DMA32: 1357*4kB 979*8kB 689*16kB 1413*32kB 447*64kB 66*128kB 15*256kB 10*512kB 7*1024kB 0*2048kB 0*4096kB = 122684kB
Oct 30 10:30:40 kernel: Node 0 Normal: 10120*4kB 45*8kB 7*16kB 3*32kB 1*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 42776kB
Oct 30 10:30:40 kernel: Node 1 Normal: 2241*4kB 1460*8kB 766*16kB 273*32kB 51*64kB 2*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 46692kB
Oct 30 10:30:40 kernel: 10994 total pagecache pages
Oct 30 10:30:40 kernel: 9447 pages in swap cache
Oct 30 10:30:40 kernel: Swap cache stats: add 10177329, delete 10167882, find 6787957/7512667
Oct 30 10:30:40 kernel: Free swap  = 0kB
Oct 30 10:30:40 kernel: Total swap = 4194296kB
Oct 30 10:30:40 kernel: 16777215 pages RAM
Oct 30 10:30:40 kernel: 285063 pages reserved
Oct 30 10:30:40 kernel: 2192 pages shared
Oct 30 10:30:40 kernel: 16161719 pages non-shared
Oct 30 10:30:40 kernel: Out of memory: kill process 2591 (varnishd) score 164119553 or a child
Oct 30 10:30:40 kernel: Killed process 22868 (varnishd) vsz:147678356kB, anon-rss:63571052kB, file-rss:616kB
Oct 30 10:30:50 varnishd[2591]: Child (22868) not responding to CLI, killing it.
Oct 30 10:30:53 varnishd[2591]: Child (22868) not responding to CLI, killing it.
Oct 30 10:30:53 varnishd[2591]: Child (22868) died signal=9
Oct 30 10:30:53 varnishd[2591]: child (15405) Started
Oct 30 10:30:53 varnishd[2591]: Child (15405) said Child starts
Oct 30 14:44:27 varnishd[21828]: Platform: Linux,2.6.32-131.0.15.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit
Oct 30 14:44:27 varnishd[21828]: Child start failed: could not open sockets
Oct 30 14:44:32 varnishd[21855]: Platform: Linux,2.6.32-131.0.15.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit
Oct 30 14:44:32 varnishd[21855]: child (21856) Started
Oct 30 14:44:32 varnishd[21855]: Child (21856) said Child starts
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121030/7ffde25f/attachment.html>

From yao.yang at autonavi.com  Tue Oct 30 07:18:47 2012
From: yao.yang at autonavi.com (=?gb2312?B?0e7Sog==?=)
Date: Tue, 30 Oct 2012 07:18:47 +0000
Subject: =?gb2312?B?u9i4tDogW0ZvciBIZWxwXSBWYXJuaXNoIGVycm9yIGluZm9ybWF0aW9u?=
References: <B77F7DEDAE19BA4BBAC66F99002545EF1AE87CCA@DAGDH01.autonavi.com>
Message-ID: <B77F7DEDAE19BA4BBAC66F99002545EF1AE88025@DAGDH01.autonavi.com>

There are some error on another server.

________________________________
Oct 28 03:13:17 kernel: imklog 4.6.2, log source = /proc/kmsg started.
Oct 28 03:13:17 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1811" x-info="http://www.rsyslog.com"] (re)start
Oct 28 03:13:20 rhsm-complianced: This system is missing one or more valid entitlement certificates. Please run subscription-manager for more information.
Oct 29 03:11:23 rhsm-complianced: This system is missing one or more valid entitlement certificates. Please run subscription-manager for more information.
Oct 30 00:26:19 varnishd[29228]: Platform: Linux,2.6.32-131.0.15.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit
Oct 30 00:26:19 varnishd[29228]: child (29229) Started
Oct 30 00:26:19 varnishd[29228]: Child (29229) said Child starts
Oct 30 03:38:26 rhsm-complianced: This system is missing one or more valid entitlement certificates. Please run subscription-manager for more information.
Oct 30 11:22:48 abrt[10036]: abrt daemon is not running. If it crashed, /proc/sys/kernel/core_pattern contains a stale value, consider resetting it to 'core'
Oct 30 11:22:50 varnishd[29228]: Child (29229) died signal=6 (core dumped)
Oct 30 11:22:50 varnishd[29228]: Child (29229) Panic message: Missing errorhandling code in SMS_Makesynth(), storage_synth.c line 88:#012  Condition((sto) != 0) not true.errno = 12 (Cannot allocate memory)#012thread = (cache-worker)#012ident = Linux,2.6.32-131.0.15.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012sp = 0x7f7440300008 {#012  fd = 1194, id = 1194, xid = 844468737,#012  client = 202.98.152.243 59144,#012  step = STP_ERROR,#012  handling = deliver,#012  err_code = 503, err_reason = (null),#012  restarts = 0, esi_level = 0#012  ws = 0x7f7440300080 { #012    id = "sess",#012    {s,f,r,e} = {0x7f7440300cf0,+608,(nil),+5000000},#012  },#012  http[req] = {#012    ws = 0x7f7440300080[sess]#012      "GET",#012      "/mapabc/maptile?v=w2.61&zoom=2&x=26040&y=13438",#012      "HTTP/1.1",#012      "Accept: */*",#012      "Referer: http://www.aibang.com/youbian/a-%E5%8D%97%E5%85%85%E5%B8%82%E5%8C%BA%E9%A1%BA%E5%BA%86%E7%96%BE%E6%8E%A7%E4%B8%AD%E5%BF%83",#012      "Accept-Language: zh-cn",#012      "User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; msn OptimizedIE8;ZHCN)",#012      "Host: emap2.mapabc.com",#012      "Connection: Keep-Alive",#012      "X-Forwarded-For: 202.98.152.243, 202.98.152.243",#012      "Accept-Encoding: gzip",#012  },#012  worker = 0x7f755c3f3b60 {#012    ws = 0x7f755c3f3d00 { #012      id = "wrk",#012      {s,f,r,e} = {0x7f755c3e1af0,+4592,(nil),+65536},#012    },#012    },#012    vcl = {#012      srcname = {#012        "input",#012        "Default",#012      },#012    },#012  obj = 0x7f689ea4f200 {#012    xid = 844468737,#012    ws = 0x7f689ea4f218 { #012      id = "obj",#012      {s,f,r,e} = {0x7f689ea4f770,+152,(nil),+1024},#012    },#012    http[obj] = {#012      ws = 0x7f689ea4f218[obj]#012        "HTTP/1.1",#012        "Service Unavailable",#012        "Date: Tue, 30 Oct 2012 03:22:48 GMT",#012        "Server: Varnish",#012        "Content-Type: text/html; charset=utf-8",#012        "Retry-After: 5",#012    },#012    len = 0,#012    store = {#012
Oct 30 11:22:50 varnishd[29228]: child (10037) Started
Oct 30 11:22:50 varnishd[29228]: Child (10037) said Child starts
Oct 30 12:01:27 wall[11761]: wall: user root broadcasted 1 lines (24 chars)
Oct 30 14:47:55 abrt[14718]: abrt daemon is not running. If it crashed, /proc/sys/kernel/core_pattern contains a stale value, consider resetting it to 'core'
Oct 30 14:47:56 varnishd[29228]: Child (10037) not responding to CLI, killing it.
Oct 30 14:47:56 varnishd[29228]: Child (10037) died signal=6 (core dumped)
Oct 30 14:47:56 varnishd[29228]: Child (10037) Panic message: Missing errorhandling code in HSH_Prealloc(), cache_hash.c line 87:#012  Condition((oc) != 0) not true.errno = 12 (Cannot allocate memory)#012thread = (cache-worker)#012ident = Linux,2.6.32-131.0.15.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012sp = 0x7f7239d00008 {#012  fd = 4444, id = 4444, xid = 167112905,#012  client = 113.140.2.186 2107,#012  step = STP_LOOKUP,#012  handling = hash,#012  restarts = 0, esi_level = 0#012  ws = 0x7f7239d00080 { #012    id = "sess",#012    {s,f,r,e} = {0x7f7239d00cf0,+528,(nil),+5000000},#012  },#012  http[req] = {#012    ws = 0x7f7239d00080[sess]#012      "GET",#012      "/mapabc/maptile?v=w2.61&zoom=1&x=52603&y=26105",#012      "HTTP/1.1",#012      "Accept: */*",#012      "Referer: http://bus.aibang.com/xian/busxid-1231359865-423759482",#012      "Accept-Language: zh-cn",#012      "User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6; QQDownload 687; .NET CLR 2.0.50727)",#012      "Host: emap2.mapabc.com",#012      "Connection: Keep-Alive",#012      "X-Forwarded-For: 113.140.2.186, 113.140.2.186",#012      "Accept-Encoding: gzip",#012  },#012  worker = 0x7f74ce7feb60 {#012    ws = 0x7f74ce7fed00 { #012      id = "wrk",#012      {s,f,r,e} = {0x7f74ce7ecaf0,0x7f74ce7ecaf0,(nil),+65536},#012    },#012    },#012    vcl = {#012      srcname = {#012        "input",#012        "Default",#012      },#012    },#012},#012
Oct 30 14:47:56 varnishd[29228]: child (14729) Started
Oct 30 14:47:56 varnishd[29228]: Child (14729) said Child starts

???? ??<mailto:yao.yang at autonavi.com>
????? 2012-10-30 15:04
???? varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
??? ??<mailto:yao.yang at autonavi.com>
??? [For Help] Varnish error information

Hi, All,
I used Varnish for half a year, these days some errors puzzled me.
And I find some error informations from  /var/log/message file.
Is my fault or the software's bug?

Thanks.
Yangyao
________________________________
Oct 28 03:47:01 kernel: imklog 4.6.2, log source = /proc/kmsg started.
Oct 28 03:47:01 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1721" x-info="http://www.rsyslog.com"] (re)start
Oct 28 03:47:39 rhsm-complianced: This system is missing one or more valid entitlement certificates. Please run subscription-manager for more information.
Oct 28 20:20:01 kernel: varnishncsa[17385]: segfault at 7f2a06670f58 ip 00007f2a06687005 sp 00007fff242d0720 error 4 in libvarnishapi.so.1.0.0[7f2a0667f000+11000]
Oct 28 20:20:01 abrt[17387]: abrt daemon is not running. If it crashed, /proc/sys/kernel/core_pattern contains a stale value, consider resetting it to 'core'
Oct 29 03:13:03 varnishd[2591]: Child (2592) not responding to CLI, killing it.
Oct 29 03:13:13 varnishd[2591]: Child (2592) not responding to CLI, killing it.
Oct 29 03:13:15 varnishd[2591]: Child (2592) not responding to CLI, killing it.
Oct 29 03:13:15 varnishd[2591]: Child (2592) died signal=3 (core dumped)
Oct 29 03:13:15 varnishd[2591]: child (28563) Started
Oct 29 03:13:15 varnishd[2591]: Child (28563) said Child starts
Oct 29 03:45:46 rhsm-complianced: This system is missing one or more valid entitlement certificates. Please run subscription-manager for more information.
Oct 29 17:23:16 varnishd[2591]: Child (28563) not responding to CLI, killing it.
Oct 29 17:23:26 varnishd[2591]: Child (28563) not responding to CLI, killing it.
Oct 29 17:23:34 varnishd[2591]: Child (28563) not responding to CLI, killing it.
Oct 29 17:23:34 varnishd[2591]: Child (28563) not responding to CLI, killing it.
Oct 29 17:23:34 varnishd[2591]: Child (28563) died signal=3 (core dumped)
Oct 29 17:23:34 varnishd[2591]: child (15919) Started
Oct 29 17:23:34 varnishd[2591]: Child (15919) said Child starts
Oct 29 20:56:39 varnishd[2591]: Child (15919) not responding to CLI, killing it.
Oct 29 20:56:43 varnishd[2591]: Child (15919) not responding to CLI, killing it.
Oct 29 20:56:43 varnishd[2591]: Child (15919) not responding to CLI, killing it.
Oct 29 20:56:43 varnishd[2591]: Child (15919) died signal=3 (core dumped)
Oct 29 20:56:43 varnishd[2591]: child (22868) Started
Oct 29 20:56:43 varnishd[2591]: Child (22868) said Child starts
Oct 30 03:32:46 rhsm-complianced: This system is missing one or more valid entitlement certificates. Please run subscription-manager for more information.
Oct 30 10:30:40 kernel: varnishncsa invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
Oct 30 10:30:40 kernel: varnishncsa cpuset=/ mems_allowed=0-1
Oct 30 10:30:40 kernel: Pid: 15396, comm: varnishncsa Not tainted 2.6.32-131.0.15.el6.x86_64 #1
Oct 30 10:30:40 kernel: Call Trace:
Oct 30 10:30:40 kernel: [<ffffffff810c0101>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Oct 30 10:30:40 kernel: [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0
Oct 30 10:30:40 kernel: [<ffffffff81110880>] ? select_bad_process+0xd0/0x110
Oct 30 10:30:40 kernel: [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0
Oct 30 10:30:40 kernel: [<ffffffff81110b19>] ? out_of_memory+0x199/0x210
Oct 30 10:30:40 kernel: [<ffffffff811202dd>] ? __alloc_pages_nodemask+0x80d/0x8b0
Oct 30 10:30:40 kernel: [<ffffffff8115464a>] ? alloc_pages_current+0xaa/0x110
Oct 30 10:30:40 kernel: [<ffffffff8110d717>] ? __page_cache_alloc+0x87/0x90
Oct 30 10:30:40 kernel: [<ffffffff81122c4b>] ? __do_page_cache_readahead+0xdb/0x210
Oct 30 10:30:40 kernel: [<ffffffff81122da1>] ? ra_submit+0x21/0x30
Oct 30 10:30:40 kernel: [<ffffffff8110e9e3>] ? filemap_fault+0x4c3/0x500
Oct 30 10:30:40 kernel: [<ffffffff812628a9>] ? cpumask_next_and+0x29/0x50
Oct 30 10:30:40 kernel: [<ffffffff81137114>] ? __do_fault+0x54/0x510
Oct 30 10:30:40 kernel: [<ffffffff811376c7>] ? handle_pte_fault+0xf7/0xb50
Oct 30 10:30:40 kernel: [<ffffffff8100987e>] ? __switch_to+0x26e/0x320
Oct 30 10:30:40 kernel: [<ffffffff81057f99>] ? find_busiest_queue+0x69/0x150
Oct 30 10:30:40 kernel: [<ffffffff811382f8>] ? handle_mm_fault+0x1d8/0x2a0
Oct 30 10:30:40 kernel: [<ffffffff81092ff3>] ? __hrtimer_start_range_ns+0x1a3/0x460
Oct 30 10:30:40 kernel: [<ffffffff81041529>] ? __do_page_fault+0x139/0x480
Oct 30 10:30:40 kernel: [<ffffffff8109332f>] ? hrtimer_try_to_cancel+0x3f/0xd0
Oct 30 10:30:40 kernel: [<ffffffff810933e2>] ? hrtimer_cancel+0x22/0x30
Oct 30 10:30:40 kernel: [<ffffffff814dccc3>] ? do_nanosleep+0x93/0xc0
Oct 30 10:30:40 kernel: [<ffffffff810934b4>] ? hrtimer_nanosleep+0xc4/0x180
Oct 30 10:30:40 kernel: [<ffffffff810922a0>] ? hrtimer_wakeup+0x0/0x30
Oct 30 10:30:40 kernel: [<ffffffff814e0c3e>] ? do_page_fault+0x3e/0xa0
Oct 30 10:30:40 kernel: [<ffffffff814ddfe5>] ? page_fault+0x25/0x30
Oct 30 10:30:40 kernel: Mem-Info:
Oct 30 10:30:40 kernel: Node 0 DMA per-cpu:
Oct 30 10:30:40 kernel: CPU    0: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    1: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    2: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    3: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    4: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    5: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    6: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    7: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    8: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    9: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU   10: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU   11: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU   12: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU   13: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU   14: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU   15: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: Node 0 DMA32 per-cpu:
Oct 30 10:30:40 kernel: CPU    0: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    1: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    2: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    3: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    4: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    5: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    6: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    7: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    8: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    9: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   10: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   11: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   12: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   13: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   14: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   15: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: Node 0 Normal per-cpu:
Oct 30 10:30:40 kernel: CPU    0: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    1: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    2: hi:  186, btch:  31 usd:  11
Oct 30 10:30:40 kernel: CPU    3: hi:  186, btch:  31 usd:   2
Oct 30 10:30:40 kernel: CPU    4: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    5: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    6: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    7: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    8: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    9: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   10: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   11: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   12: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   13: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   14: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   15: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: Node 1 Normal per-cpu:
Oct 30 10:30:40 kernel: CPU    0: hi:  186, btch:  31 usd:   5
Oct 30 10:30:40 kernel: CPU    1: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    2: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    3: hi:  186, btch:  31 usd:   1
Oct 30 10:30:40 kernel: CPU    4: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    5: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    6: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    7: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    8: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    9: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   10: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   11: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   12: hi:  186, btch:  31 usd:   8
Oct 30 10:30:40 kernel: CPU   13: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   14: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   15: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: active_anon:15180189 inactive_anon:868880 isolated_anon:0
Oct 30 10:30:40 kernel: active_file:496 inactive_file:603 isolated_file:0
Oct 30 10:30:40 kernel: unevictable:0 dirty:22 writeback:20 unstable:0
Oct 30 10:30:40 kernel: free:55664 slab_reclaimable:10223 slab_unreclaimable:284822
Oct 30 10:30:40 kernel: mapped:386 shmem:0 pagetables:54263 bounce:0
Oct 30 10:30:40 kernel: Node 0 DMA free:15616kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15224kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 30 10:30:40 kernel: lowmem_reserve[]: 0 2991 32281 32281
Oct 30 10:30:40 kernel: Node 0 DMA32 free:121196kB min:4168kB low:5208kB high:6252kB active_anon:1856648kB inactive_anon:457828kB active_file:124kB inactive_file:372kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3063392kB mlocked:0kB dirty:12kB writeback:16kB mapped:156kB shmem:0kB slab_reclaimable:3732kB slab_unreclaimable:2508kB kernel_stack:0kB pagetables:5060kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:352 all_unreclaimable? no
Oct 30 10:30:40 kernel: lowmem_reserve[]: 0 0 29290 29290
Oct 30 10:30:40 kernel: Node 0 Normal free:40808kB min:40844kB low:51052kB high:61264kB active_anon:27839988kB inactive_anon:1739808kB active_file:1104kB inactive_file:1240kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:29992960kB mlocked:0kB dirty:52kB writeback:40kB mapped:332kB shmem:0kB slab_reclaimable:19552kB slab_unreclaimable:584592kB kernel_stack:10656kB pagetables:107120kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1528 all_unreclaimable? no
Oct 30 10:30:40 kernel: lowmem_reserve[]: 0 0 0 0
Oct 30 10:30:40 kernel: Node 1 Normal free:45036kB min:45072kB low:56340kB high:67608kB active_anon:31024120kB inactive_anon:1277884kB active_file:756kB inactive_file:800kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:33095680kB mlocked:0kB dirty:24kB writeback:24kB mapped:1056kB shmem:0kB slab_reclaimable:17608kB slab_unreclaimable:552188kB kernel_stack:5456kB pagetables:104872kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1267 all_unreclaimable? no
Oct 30 10:30:40 kernel: lowmem_reserve[]: 0 0 0 0
Oct 30 10:30:40 kernel: Node 0 DMA: 2*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15616kB
Oct 30 10:30:40 kernel: Node 0 DMA32: 1357*4kB 979*8kB 689*16kB 1413*32kB 447*64kB 66*128kB 15*256kB 10*512kB 7*1024kB 0*2048kB 0*4096kB = 122684kB
Oct 30 10:30:40 kernel: Node 0 Normal: 10120*4kB 45*8kB 7*16kB 3*32kB 1*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 42776kB
Oct 30 10:30:40 kernel: Node 1 Normal: 2241*4kB 1460*8kB 766*16kB 273*32kB 51*64kB 2*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 46692kB
Oct 30 10:30:40 kernel: 10994 total pagecache pages
Oct 30 10:30:40 kernel: 9447 pages in swap cache
Oct 30 10:30:40 kernel: Swap cache stats: add 10177329, delete 10167882, find 6787957/7512667
Oct 30 10:30:40 kernel: Free swap  = 0kB
Oct 30 10:30:40 kernel: Total swap = 4194296kB
Oct 30 10:30:40 kernel: 16777215 pages RAM
Oct 30 10:30:40 kernel: 285063 pages reserved
Oct 30 10:30:40 kernel: 2192 pages shared
Oct 30 10:30:40 kernel: 16161719 pages non-shared
Oct 30 10:30:40 kernel: Out of memory: kill process 2591 (varnishd) score 164119553 or a child
Oct 30 10:30:40 kernel: Killed process 22868 (varnishd) vsz:147678356kB, anon-rss:63571052kB, file-rss:616kB
Oct 30 10:30:50 varnishd[2591]: Child (22868) not responding to CLI, killing it.
Oct 30 10:30:53 varnishd[2591]: Child (22868) not responding to CLI, killing it.
Oct 30 10:30:53 varnishd[2591]: Child (22868) died signal=9
Oct 30 10:30:53 varnishd[2591]: child (15405) Started
Oct 30 10:30:53 varnishd[2591]: Child (15405) said Child starts
Oct 30 14:44:27 varnishd[21828]: Platform: Linux,2.6.32-131.0.15.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit
Oct 30 14:44:27 varnishd[21828]: Child start failed: could not open sockets
Oct 30 14:44:32 varnishd[21855]: Platform: Linux,2.6.32-131.0.15.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit
Oct 30 14:44:32 varnishd[21855]: child (21856) Started
Oct 30 14:44:32 varnishd[21855]: Child (21856) said Child starts
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121030/48d37c56/attachment.html>

From perbu at varnish-software.com  Tue Oct 30 09:34:20 2012
From: perbu at varnish-software.com (Per Buer)
Date: Tue, 30 Oct 2012 10:34:20 +0100
Subject: [For Help] Varnish error information
In-Reply-To: <B77F7DEDAE19BA4BBAC66F99002545EF1AE88025@DAGDH01.autonavi.com>
References: <B77F7DEDAE19BA4BBAC66F99002545EF1AE87CCA@DAGDH01.autonavi.com>
 <B77F7DEDAE19BA4BBAC66F99002545EF1AE88025@DAGDH01.autonavi.com>
Message-ID: <CAOXZevDhTwJiD8_bY0fgfy_j-HxC7_36h0HgQ+4jm2Ho7ZetnA@mail.gmail.com>

Hi Yao,

You are seriously low on virtual memory. And probably low on physical
memory as well.

Per.

On Tue, Oct 30, 2012 at 8:18 AM, ?? <yao.yang at autonavi.com> wrote:

> **
> There are some error on another server.
>
>  ------------------------------
>  Oct 28 03:13:17 kernel: imklog 4.6.2, log source = /proc/kmsg started.
>
> Oct 28 03:13:17 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1811" x-info="
> http://www.rsyslog.com"] (re)start
>
> Oct 28 03:13:20 rhsm-complianced: This system is missing one or more valid entitlement certificates. Please run subscription-manager for more information.
>
> Oct 29 03:11:23 rhsm-complianced: This system is missing one or more valid entitlement certificates. Please run subscription-manager for more information.
>
> Oct 30 00:26:19 varnishd[29228]: Platform: Linux,2.6.32-131.0.15.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit
> Oct 30 00:26:19 varnishd[29228]: child (29229) Started
> Oct 30 00:26:19 varnishd[29228]: Child (29229) said Child starts
>
> Oct 30 03:38:26 rhsm-complianced: This system is missing one or more valid entitlement certificates. Please run subscription-manager for more information.
>
> Oct 30 11:22:48 abrt[10036]: abrt daemon is not running. If it crashed, /proc/sys/kernel/core_pattern contains a stale value, consider resetting it to 'core'
> Oct 30 11:22:50 varnishd[29228]: Child (29229) died signal=6 (core dumped)
> Oct 30 11:22:50 varnishd[29228]:
> Child (29229) Panic message: Missing errorhandling code in SMS_Makesynth(), storage_synth.c line 88:#012  Condition((sto) != 0) not true.errno = 12 (Cannot allocate memory)#012thread = (cache-worker)#012ident = Linux,2.6.32-131.0.15.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012sp = 0x7f7440300008
> {#012  fd = 1194, id = 1194, xid = 844468737,#012  client = 202.98.152.243 59144,#012  step = STP_ERROR,#012  handling = deliver,#012  err_code = 503, err_reason = (null),#012  restarts = 0, esi_level = 0#012  ws = 0x7f7440300080 { #012    id = "sess",#012    {s,f,r,e} = {0x7f7440300cf0,+608,(nil),+5000000},#012  },#012  http[req] = {#012    ws = 0x7f7440300080[sess]#012      "GET",#012      "/mapabc/maptile?v=w2.61&zoom=2&x=26040&y=13438",#012      "HTTP/1.1",#012      "Accept: */*",#012      "Referer:
> http://www.aibang.com/youbian/a-%E5%8D%97%E5%85%85%E5%B8%82%E5%8C%BA%E9%A1%BA%E5%BA%86%E7%96%BE%E6%8E%A7%E4%B8%AD%E5%BF%83
> ",#012      "Accept-Language: zh-cn",#012      "User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; msn OptimizedIE8;ZHCN)",#012      "Host:
> emap2.mapabc.com
> ",#012      "Connection: Keep-Alive",#012      "X-Forwarded-For: 202.98.152.243, 202.98.152.243",#012      "Accept-Encoding: gzip",#012  },#012  worker = 0x7f755c3f3b60 {#012    ws = 0x7f755c3f3d00 { #012      id = "wrk",#012      {s,f,r,e} = {0x7f755c3e1af0,+4592,(nil),+65536},#012    },#012    },#012    vcl = {#012      srcname = {#012        "input",#012        "Default",#012      },#012    },#012  obj = 0x7f689ea4f200 {#012    xid = 844468737,#012    ws = 0x7f689ea4f218 { #012      id = "obj",#012      {s,f,r,e} = {0x7f689ea4f770,+152,(nil),+1024},#012    },#012    http[obj] = {#012      ws = 0x7f689ea4f218[obj]#012        "HTTP/1.1",#012        "Service Unavailable",#012        "Date: Tue, 30 Oct 2012 03:22:48 GMT",#012        "Server: Varnish",#012        "Content-Type: text/html; charset=utf-8",#012        "Retry-After: 5",#012    },#012    len = 0,#012    store = {#012
> Oct 30 11:22:50 varnishd[29228]: child (10037) Started
> Oct 30 11:22:50 varnishd[29228]: Child (10037) said Child starts
> Oct 30 12:01:27 wall[11761]: wall: user root broadcasted 1 lines (24 chars)
>
> Oct 30 14:47:55 abrt[14718]: abrt daemon is not running. If it crashed, /proc/sys/kernel/core_pattern contains a stale value, consider resetting it to 'core'
>
> Oct 30 14:47:56 varnishd[29228]: Child (10037) not responding to CLI, killing it.
> Oct 30 14:47:56 varnishd[29228]: Child (10037) died signal=6 (core dumped)
> Oct 30 14:47:56 varnishd[29228]:
>  Child (10037) Panic message: Missing errorhandling code in HSH_Prealloc(), cache_hash.c line 87:#012  Condition((oc) != 0) not true.errno = 12 (Cannot allocate memory)#012thread = (cache-worker)#012ident = Linux,2.6.32-131.0.15.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012sp = 0x7f7239d00008
> {#012  fd = 4444, id = 4444, xid = 167112905,#012  client = 113.140.2.186 2107,#012  step = STP_LOOKUP,#012  handling = hash,#012  restarts = 0, esi_level = 0#012  ws = 0x7f7239d00080 { #012    id = "sess",#012    {s,f,r,e} = {0x7f7239d00cf0,+528,(nil),+5000000},#012  },#012  http[req] = {#012    ws = 0x7f7239d00080[sess]#012      "GET",#012      "/mapabc/maptile?v=w2.61&zoom=1&x=52603&y=26105",#012      "HTTP/1.1",#012      "Accept: */*",#012      "Referer:
> http://bus.aibang.com/xian/busxid-1231359865-423759482
> ",#012      "Accept-Language: zh-cn",#012      "User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6; QQDownload 687; .NET CLR 2.0.50727)",#012      "Host:
> emap2.mapabc.com
> ",#012      "Connection: Keep-Alive",#012      "X-Forwarded-For: 113.140.2.186, 113.140.2.186",#012      "Accept-Encoding: gzip",#012  },#012  worker = 0x7f74ce7feb60 {#012    ws = 0x7f74ce7fed00 { #012      id = "wrk",#012      {s,f,r,e} = {0x7f74ce7ecaf0,0x7f74ce7ecaf0,(nil),+65536},#012    },#012    },#012    vcl = {#012      srcname = {#012        "input",#012        "Default",#012      },#012    },#012},#012
> Oct 30 14:47:56 varnishd[29228]: child (14729) Started
> Oct 30 14:47:56 varnishd[29228]: Child (14729) said Child starts
>
>  *????* ?? <yao.yang at autonavi.com>
> *?????* 2012-10-30 15:04
> *????* varnish-misc at varnish-cache.org
> *???* ?? <yao.yang at autonavi.com>
> *???* [For Help] Varnish error information
>   ****
>
> Hi, All,
> I used Varnish for half a year, these days some errors puzzled me.
> And I find some error informations from  /var/log/message file.
> Is my fault or the software's bug?
>
> Thanks.
> Yangyao
>  ------------------------------
>
> Oct 28 03:47:01 kernel: imklog 4.6.2, log source = /proc/kmsg started.
>
> Oct 28 03:47:01 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1721" x-info="
> http://www.rsyslog.com"] (re)start
>
> Oct 28 03:47:39 rhsm-complianced: This system is missing one or more valid entitlement certificates. Please run subscription-manager for more information.
> Oct 28 20:20:01
>  kernel: varnishncsa[17385]: segfault at 7f2a06670f58 ip 00007f2a06687005 sp 00007fff242d0720 error 4 in libvarnishapi.so.1.0.0[7f2a0667f000+11000]
>
> Oct 28 20:20:01 abrt[17387]: abrt daemon is not running. If it crashed, /proc/sys/kernel/core_pattern contains a stale value, consider resetting it to 'core'
>
> Oct 29 03:13:03 varnishd[2591]: Child (2592) not responding to CLI, killing it.
>
> Oct 29 03:13:13 varnishd[2591]: Child (2592) not responding to CLI, killing it.
>
> Oct 29 03:13:15 varnishd[2591]: Child (2592) not responding to CLI, killing it.
> Oct 29 03:13:15 varnishd[2591]: Child (2592) died signal=3 (core dumped)
> Oct 29 03:13:15 varnishd[2591]: child (28563) Started
> Oct 29 03:13:15 varnishd[2591]: Child (28563) said Child starts
>
> Oct 29 03:45:46 rhsm-complianced: This system is missing one or more valid entitlement certificates. Please run subscription-manager for more information.
>
> Oct 29 17:23:16 varnishd[2591]: Child (28563) not responding to CLI, killing it.
>
> Oct 29 17:23:26 varnishd[2591]: Child (28563) not responding to CLI, killing it.
>
> Oct 29 17:23:34 varnishd[2591]: Child (28563) not responding to CLI, killing it.
>
> Oct 29 17:23:34 varnishd[2591]: Child (28563) not responding to CLI, killing it.
> Oct 29 17:23:34 varnishd[2591]: Child (28563) died signal=3 (core dumped)
> Oct 29 17:23:34 varnishd[2591]: child (15919) Started
> Oct 29 17:23:34 varnishd[2591]: Child (15919) said Child starts
>
> Oct 29 20:56:39 varnishd[2591]: Child (15919) not responding to CLI, killing it.
>
> Oct 29 20:56:43 varnishd[2591]: Child (15919) not responding to CLI, killing it.
>
> Oct 29 20:56:43 varnishd[2591]: Child (15919) not responding to CLI, killing it.
> Oct 29 20:56:43 varnishd[2591]: Child (15919) died signal=3 (core dumped)
> Oct 29 20:56:43 varnishd[2591]: child (22868) Started
> Oct 29 20:56:43 varnishd[2591]: Child (22868) said Child starts
>
> Oct 30 03:32:46 rhsm-complianced: This system is missing one or more valid entitlement certificates. Please run subscription-manager for more information.
>
> Oct 30 10:30:40 kernel: varnishncsa invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
> Oct 30 10:30:40 kernel: varnishncsa cpuset=/ mems_allowed=0-1
>
> Oct 30 10:30:40 kernel: Pid: 15396, comm: varnishncsa Not tainted 2.6.32-131.0.15.el6.x86_64 #1
> Oct 30 10:30:40 kernel: Call Trace:
>
> Oct 30 10:30:40 kernel: [<ffffffff810c0101>] ? cpuset_print_task_mems_allowed+0x91/0xb0
> Oct 30 10:30:40 kernel: [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0
>
> Oct 30 10:30:40 kernel: [<ffffffff81110880>] ? select_bad_process+0xd0/0x110
> Oct 30 10:30:40 kernel: [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0
> Oct 30 10:30:40 kernel: [<ffffffff81110b19>] ? out_of_memory+0x199/0x210
>
> Oct 30 10:30:40 kernel: [<ffffffff811202dd>] ? __alloc_pages_nodemask+0x80d/0x8b0
>
> Oct 30 10:30:40 kernel: [<ffffffff8115464a>] ? alloc_pages_current+0xaa/0x110
> Oct 30 10:30:40 kernel: [<ffffffff8110d717>] ? __page_cache_alloc+0x87/0x90
>
> Oct 30 10:30:40 kernel: [<ffffffff81122c4b>] ? __do_page_cache_readahead+0xdb/0x210
> Oct 30 10:30:40 kernel: [<ffffffff81122da1>] ? ra_submit+0x21/0x30
> Oct 30 10:30:40 kernel: [<ffffffff8110e9e3>] ? filemap_fault+0x4c3/0x500
> Oct 30 10:30:40 kernel: [<ffffffff812628a9>] ? cpumask_next_and+0x29/0x50
> Oct 30 10:30:40 kernel: [<ffffffff81137114>] ? __do_fault+0x54/0x510
> Oct 30 10:30:40 kernel: [<ffffffff811376c7>] ? handle_pte_fault+0xf7/0xb50
> Oct 30 10:30:40 kernel: [<ffffffff8100987e>] ? __switch_to+0x26e/0x320
>
> Oct 30 10:30:40 kernel: [<ffffffff81057f99>] ? find_busiest_queue+0x69/0x150
> Oct 30 10:30:40 kernel: [<ffffffff811382f8>] ? handle_mm_fault+0x1d8/0x2a0
>
> Oct 30 10:30:40 kernel: [<ffffffff81092ff3>] ? __hrtimer_start_range_ns+0x1a3/0x460
> Oct 30 10:30:40 kernel: [<ffffffff81041529>] ? __do_page_fault+0x139/0x480
>
> Oct 30 10:30:40 kernel: [<ffffffff8109332f>] ? hrtimer_try_to_cancel+0x3f/0xd0
> Oct 30 10:30:40 kernel: [<ffffffff810933e2>] ? hrtimer_cancel+0x22/0x30
> Oct 30 10:30:40 kernel: [<ffffffff814dccc3>] ? do_nanosleep+0x93/0xc0
> Oct 30 10:30:40 kernel: [<ffffffff810934b4>] ? hrtimer_nanosleep+0xc4/0x180
> Oct 30 10:30:40 kernel: [<ffffffff810922a0>] ? hrtimer_wakeup+0x0/0x30
> Oct 30 10:30:40 kernel: [<ffffffff814e0c3e>] ? do_page_fault+0x3e/0xa0
> Oct 30 10:30:40 kernel: [<ffffffff814ddfe5>] ? page_fault+0x25/0x30
> Oct 30 10:30:40 kernel: Mem-Info:
> Oct 30 10:30:40 kernel: Node 0 DMA per-cpu:
> Oct 30 10:30:40 kernel: CPU    0: hi:    0, btch:   1 usd:   0
> Oct 30 10:30:40 kernel: CPU    1: hi:    0, btch:   1 usd:   0
> Oct 30 10:30:40 kernel: CPU    2: hi:    0, btch:   1 usd:   0
> Oct 30 10:30:40 kernel: CPU    3: hi:    0, btch:   1 usd:   0
> Oct 30 10:30:40 kernel: CPU    4: hi:    0, btch:   1 usd:   0
> Oct 30 10:30:40 kernel: CPU    5: hi:    0, btch:   1 usd:   0
> Oct 30 10:30:40 kernel: CPU    6: hi:    0, btch:   1 usd:   0
> Oct 30 10:30:40 kernel: CPU    7: hi:    0, btch:   1 usd:   0
> Oct 30 10:30:40 kernel: CPU    8: hi:    0, btch:   1 usd:   0
> Oct 30 10:30:40 kernel: CPU    9: hi:    0, btch:   1 usd:   0
> Oct 30 10:30:40 kernel: CPU   10: hi:    0, btch:   1 usd:   0
> Oct 30 10:30:40 kernel: CPU   11: hi:    0, btch:   1 usd:   0
> Oct 30 10:30:40 kernel: CPU   12: hi:    0, btch:   1 usd:   0
> Oct 30 10:30:40 kernel: CPU   13: hi:    0, btch:   1 usd:   0
> Oct 30 10:30:40 kernel: CPU   14: hi:    0, btch:   1 usd:   0
> Oct 30 10:30:40 kernel: CPU   15: hi:    0, btch:   1 usd:   0
> Oct 30 10:30:40 kernel: Node 0 DMA32 per-cpu:
> Oct 30 10:30:40 kernel: CPU    0: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    1: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    2: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    3: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    4: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    5: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    6: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    7: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    8: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    9: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU   10: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU   11: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU   12: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU   13: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU   14: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU   15: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: Node 0 Normal per-cpu:
> Oct 30 10:30:40 kernel: CPU    0: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    1: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    2: hi:  186, btch:  31 usd:  11
> Oct 30 10:30:40 kernel: CPU    3: hi:  186, btch:  31 usd:   2
> Oct 30 10:30:40 kernel: CPU    4: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    5: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    6: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    7: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    8: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    9: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU   10: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU   11: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU   12: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU   13: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU   14: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU   15: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: Node 1 Normal per-cpu:
> Oct 30 10:30:40 kernel: CPU    0: hi:  186, btch:  31 usd:   5
> Oct 30 10:30:40 kernel: CPU    1: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    2: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    3: hi:  186, btch:  31 usd:   1
> Oct 30 10:30:40 kernel: CPU    4: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    5: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    6: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    7: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    8: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU    9: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU   10: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU   11: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU   12: hi:  186, btch:  31 usd:   8
> Oct 30 10:30:40 kernel: CPU   13: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU   14: hi:  186, btch:  31 usd:   0
> Oct 30 10:30:40 kernel: CPU   15: hi:  186, btch:  31 usd:   0
>
> Oct 30 10:30:40 kernel: active_anon:15180189 inactive_anon:868880 isolated_anon:0
> Oct 30 10:30:40 kernel: active_file:496 inactive_file:603 isolated_file:0
> Oct 30 10:30:40 kernel: unevictable:0 dirty:22 writeback:20 unstable:0
>
> Oct 30 10:30:40 kernel: free:55664 slab_reclaimable:10223 slab_unreclaimable:284822
> Oct 30 10:30:40 kernel: mapped:386 shmem:0 pagetables:54263 bounce:0
>
> Oct 30 10:30:40 kernel: Node 0 DMA free:15616kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15224kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Oct 30 10:30:40 kernel: lowmem_reserve[]: 0 2991 32281 32281
>
> Oct 30 10:30:40 kernel: Node 0 DMA32 free:121196kB min:4168kB low:5208kB high:6252kB active_anon:1856648kB inactive_anon:457828kB active_file:124kB inactive_file:372kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3063392kB mlocked:0kB dirty:12kB writeback:16kB mapped:156kB shmem:0kB slab_reclaimable:3732kB slab_unreclaimable:2508kB kernel_stack:0kB pagetables:5060kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:352 all_unreclaimable? no
> Oct 30 10:30:40 kernel: lowmem_reserve[]: 0 0 29290 29290
>
> Oct 30 10:30:40 kernel: Node 0 Normal free:40808kB min:40844kB low:51052kB high:61264kB active_anon:27839988kB inactive_anon:1739808kB active_file:1104kB inactive_file:1240kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:29992960kB mlocked:0kB dirty:52kB writeback:40kB mapped:332kB shmem:0kB slab_reclaimable:19552kB slab_unreclaimable:584592kB kernel_stack:10656kB pagetables:107120kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1528 all_unreclaimable? no
> Oct 30 10:30:40 kernel: lowmem_reserve[]: 0 0 0 0
>
> Oct 30 10:30:40 kernel: Node 1 Normal free:45036kB min:45072kB low:56340kB high:67608kB active_anon:31024120kB inactive_anon:1277884kB active_file:756kB inactive_file:800kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:33095680kB mlocked:0kB dirty:24kB writeback:24kB mapped:1056kB shmem:0kB slab_reclaimable:17608kB slab_unreclaimable:552188kB kernel_stack:5456kB pagetables:104872kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1267 all_unreclaimable? no
> Oct 30 10:30:40 kernel: lowmem_reserve[]: 0 0 0 0
>
> Oct 30 10:30:40 kernel: Node 0 DMA: 2*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15616kB
>
> Oct 30 10:30:40 kernel: Node 0 DMA32: 1357*4kB 979*8kB 689*16kB 1413*32kB 447*64kB 66*128kB 15*256kB 10*512kB 7*1024kB 0*2048kB 0*4096kB = 122684kB
>
> Oct 30 10:30:40 kernel: Node 0 Normal: 10120*4kB 45*8kB 7*16kB 3*32kB 1*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 42776kB
>
> Oct 30 10:30:40 kernel: Node 1 Normal: 2241*4kB 1460*8kB 766*16kB 273*32kB 51*64kB 2*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 46692kB
> Oct 30 10:30:40 kernel: 10994 total pagecache pages
> Oct 30 10:30:40 kernel: 9447 pages in swap cache
>
> Oct 30 10:30:40 kernel: Swap cache stats: add 10177329, delete 10167882, find 6787957/7512667
> Oct 30 10:30:40 kernel: Free swap  = 0kB
> Oct 30 10:30:40 kernel: Total swap = 4194296kB
> Oct 30 10:30:40 kernel: 16777215 pages RAM
> Oct 30 10:30:40 kernel: 285063 pages reserved
> Oct 30 10:30:40 kernel: 2192 pages shared
> Oct 30 10:30:40 kernel: 16161719 pages non-shared
> Oct 30 10:30:40 kernel:
> Out of memory: kill process 2591 (varnishd) score 164119553 or a child
>
> Oct 30 10:30:40 kernel: Killed process 22868 (varnishd) vsz:147678356kB, anon-rss:63571052kB, file-rss:616kB
>
> Oct 30 10:30:50 varnishd[2591]: Child (22868) not responding to CLI, killing it.
>
> Oct 30 10:30:53 varnishd[2591]: Child (22868) not responding to CLI, killing it.
> Oct 30 10:30:53 varnishd[2591]: Child (22868) died signal=9
> Oct 30 10:30:53 varnishd[2591]: child (15405) Started
> Oct 30 10:30:53 varnishd[2591]: Child (15405) said Child starts
>
> Oct 30 14:44:27 varnishd[21828]: Platform: Linux,2.6.32-131.0.15.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit
> Oct 30 14:44:27 varnishd[21828]: Child start failed: could not open sockets
>
> Oct 30 14:44:32 varnishd[21855]: Platform: Linux,2.6.32-131.0.15.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit
> Oct 30 14:44:32 varnishd[21855]: child (21856) Started
> Oct 30 14:44:32 varnishd[21855]: Child (21856) said Child starts
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>



-- 
 <http://www.varnish-software.com/> *Per Buer*
CEO | Varnish Software AS
Phone: +47 958 39 117 | Skype: per.buer
We Make Websites Fly!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121030/d9d08d03/attachment.html>

From yao.yang at autonavi.com  Tue Oct 30 09:53:54 2012
From: yao.yang at autonavi.com (=?gb2312?B?0e7Sog==?=)
Date: Tue, 30 Oct 2012 09:53:54 +0000
Subject: =?gb2312?B?u9i4tDogUmU6IFtGb3IgSGVscF0gVmFybmlzaCBlcnJvciBpbmZvcm1hdGlv?=
 =?gb2312?Q?n?=
References: <B77F7DEDAE19BA4BBAC66F99002545EF1AE87CCA@DAGDH01.autonavi.com>,
 <B77F7DEDAE19BA4BBAC66F99002545EF1AE88025@DAGDH01.autonavi.com>,
 <CAOXZevDhTwJiD8_bY0fgfy_j-HxC7_36h0HgQ+4jm2Ho7ZetnA@mail.gmail.com>
Message-ID: <B77F7DEDAE19BA4BBAC66F99002545EF1AE8B28A@DAGDH01.autonavi.com>

hi Per,

is virtual memory referring to swap partition? currently swap partition is 4G.
we are using physical server now, the physical memory is 48G and the OS is Centos6.2 x86_64.
my boot command line is as following.
is there any optimization I should make?

/usr/local/varnish/sbin/varnishd -f /usr/local/varnish/etc/varnish/default.vcl -s malloc,36G -T 127.0.0.1:2000 -a 0.0.0.0:80 -p thread_pool_max=4000 -p thread_pools=14 -p thread_pool_min=40 -p thread_pool_add_delay=2 -p session_linger=50 -p sess_workspace=5000000

________________________________
??

???? Per Buer<mailto:perbu at varnish-software.com>
????? 2012-10-30 17:34
???? ??<mailto:yao.yang at autonavi.com>
??? varnish-misc<mailto:varnish-misc at varnish-cache.org>
??? Re: [For Help] Varnish error information
Hi Yao,

You are seriously low on virtual memory. And probably low on physical memory as well.

Per.

On Tue, Oct 30, 2012 at 8:18 AM, ?? <yao.yang at autonavi.com<mailto:yao.yang at autonavi.com>> wrote:
There are some error on another server.

________________________________
Oct 28 03:13:17 kernel: imklog 4.6.2, log source = /proc/kmsg started.
Oct 28 03:13:17 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1811" x-info="http://www.rsyslog.com"] (re)start
Oct 28 03:13:20 rhsm-complianced: This system is missing one or more valid entitlement certificates. Please run subscription-manager for more information.
Oct 29 03:11:23 rhsm-complianced: This system is missing one or more valid entitlement certificates. Please run subscription-manager for more information.
Oct 30 00:26:19 varnishd[29228]: Platform: Linux,2.6.32-131.0.15.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit
Oct 30 00:26:19 varnishd[29228]: child (29229) Started
Oct 30 00:26:19 varnishd[29228]: Child (29229) said Child starts
Oct 30 03:38:26 rhsm-complianced: This system is missing one or more valid entitlement certificates. Please run subscription-manager for more information.
Oct 30 11:22:48 abrt[10036]: abrt daemon is not running. If it crashed, /proc/sys/kernel/core_pattern contains a stale value, consider resetting it to 'core'
Oct 30 11:22:50 varnishd[29228]: Child (29229) died signal=6 (core dumped)
Oct 30 11:22:50 varnishd[29228]: Child (29229) Panic message: Missing errorhandling code in SMS_Makesynth(), storage_synth.c line 88:#012  Condition((sto) != 0) not true.errno = 12 (Cannot allocate memory)#012thread = (cache-worker)#012ident = Linux,2.6.32-131.0.15.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012sp = 0x7f7440300008 {#012  fd = 1194, id = 1194, xid = 844468737,#012  client = 202.98.152.243 59144,#012  step = STP_ERROR,#012  handling = deliver,#012  err_code = 503, err_reason = (null),#012  restarts = 0, esi_level = 0#012  ws = 0x7f7440300080 { #012    id = "sess",#012    {s,f,r,e} = {0x7f7440300cf0,+608,(nil),+5000000},#012  },#012  http[req] = {#012    ws = 0x7f7440300080[sess]#012      "GET",#012      "/mapabc/maptile?v=w2.61&zoom=2&x=26040&y=13438",#012      "HTTP/1.1",#012      "Accept: */*",#012      "Referer: http://www.aibang.com/youbian/a-%E5%8D%97%E5%85%85%E5%B8%82%E5%8C%BA%E9%A1%BA%E5%BA%86%E7%96%BE%E6%8E%A7%E4%B8%AD%E5%BF%83",#012      "Accept-Language: zh-cn",#012      "User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; msn OptimizedIE8;ZHCN)",#012      "Host: emap2.mapabc.com<http://emap2.mapabc.com>",#012      "Connection: Keep-Alive",#012      "X-Forwarded-For: 202.98.152.243, 202.98.152.243",#012      "Accept-Encoding: gzip",#012  },#012  worker = 0x7f755c3f3b60 {#012    ws = 0x7f755c3f3d00 { #012      id = "wrk",#012      {s,f,r,e} = {0x7f755c3e1af0,+4592,(nil),+65536},#012    },#012    },#012    vcl = {#012      srcname = {#012        "input",#012        "Default",#012      },#012    },#012  obj = 0x7f689ea4f200 {#012    xid = 844468737,#012    ws = 0x7f689ea4f218 { #012      id = "obj",#012      {s,f,r,e} = {0x7f689ea4f770,+152,(nil),+1024},#012    },#012    http[obj] = {#012      ws = 0x7f689ea4f218[obj]#012        "HTTP/1.1",#012        "Service Unavailable",#012        "Date: Tue, 30 Oct 2012 03:22:48 GMT",#012        "Server: Varnish",#012        "Content-Type: text/html; charset=utf-8",#012        "Retry-After: 5",#012    },#012    len = 0,#012    store = {#012
Oct 30 11:22:50 varnishd[29228]: child (10037) Started
Oct 30 11:22:50 varnishd[29228]: Child (10037) said Child starts
Oct 30 12:01:27 wall[11761]: wall: user root broadcasted 1 lines (24 chars)
Oct 30 14:47:55 abrt[14718]: abrt daemon is not running. If it crashed, /proc/sys/kernel/core_pattern contains a stale value, consider resetting it to 'core'
Oct 30 14:47:56 varnishd[29228]: Child (10037) not responding to CLI, killing it.
Oct 30 14:47:56 varnishd[29228]: Child (10037) died signal=6 (core dumped)
Oct 30 14:47:56 varnishd[29228]: Child (10037) Panic message: Missing errorhandling code in HSH_Prealloc(), cache_hash.c line 87:#012  Condition((oc) != 0) not true.errno = 12 (Cannot allocate memory)#012thread = (cache-worker)#012ident = Linux,2.6.32-131.0.15.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012sp = 0x7f7239d00008 {#012  fd = 4444, id = 4444, xid = 167112905,#012  client = 113.140.2.186 2107,#012  step = STP_LOOKUP,#012  handling = hash,#012  restarts = 0, esi_level = 0#012  ws = 0x7f7239d00080 { #012    id = "sess",#012    {s,f,r,e} = {0x7f7239d00cf0,+528,(nil),+5000000},#012  },#012  http[req] = {#012    ws = 0x7f7239d00080[sess]#012      "GET",#012      "/mapabc/maptile?v=w2.61&zoom=1&x=52603&y=26105",#012      "HTTP/1.1",#012      "Accept: */*",#012      "Referer: http://bus.aibang.com/xian/busxid-1231359865-423759482",#012      "Accept-Language: zh-cn",#012      "User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6; QQDownload 687; .NET CLR 2.0.50727)",#012      "Host: emap2.mapabc.com<http://emap2.mapabc.com>",#012      "Connection: Keep-Alive",#012      "X-Forwarded-For: 113.140.2.186, 113.140.2.186",#012      "Accept-Encoding: gzip",#012  },#012  worker = 0x7f74ce7feb60 {#012    ws = 0x7f74ce7fed00 { #012      id = "wrk",#012      {s,f,r,e} = {0x7f74ce7ecaf0,0x7f74ce7ecaf0,(nil),+65536},#012    },#012    },#012    vcl = {#012      srcname = {#012        "input",#012        "Default",#012      },#012    },#012},#012
Oct 30 14:47:56 varnishd[29228]: child (14729) Started
Oct 30 14:47:56 varnishd[29228]: Child (14729) said Child starts

???? ??<mailto:yao.yang at autonavi.com>
????? 2012-10-30 15<tel:2012-10-30%C2%A015>:04
???? varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
??? ??<mailto:yao.yang at autonavi.com>
??? [For Help] Varnish error information

Hi, All,
I used Varnish for half a year, these days some errors puzzled me.
And I find some error informations from  /var/log/message file.
Is my fault or the software's bug?

Thanks.
Yangyao
________________________________
Oct 28 03:47:01 kernel: imklog 4.6.2, log source = /proc/kmsg started.
Oct 28 03:47:01 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1721" x-info="http://www.rsyslog.com"] (re)start
Oct 28 03:47:39 rhsm-complianced: This system is missing one or more valid entitlement certificates. Please run subscription-manager for more information.
Oct 28 20:20:01 kernel: varnishncsa[17385]: segfault at 7f2a06670f58 ip 00007f2a06687005 sp 00007fff242d0720 error 4 in libvarnishapi.so.1.0.0[7f2a0667f000+11000]
Oct 28 20:20:01 abrt[17387]: abrt daemon is not running. If it crashed, /proc/sys/kernel/core_pattern contains a stale value, consider resetting it to 'core'
Oct 29 03:13:03 varnishd[2591]: Child (2592) not responding to CLI, killing it.
Oct 29 03:13:13 varnishd[2591]: Child (2592) not responding to CLI, killing it.
Oct 29 03:13:15 varnishd[2591]: Child (2592) not responding to CLI, killing it.
Oct 29 03:13:15 varnishd[2591]: Child (2592) died signal=3 (core dumped)
Oct 29 03:13:15 varnishd[2591]: child (28563) Started
Oct 29 03:13:15 varnishd[2591]: Child (28563) said Child starts
Oct 29 03:45:46 rhsm-complianced: This system is missing one or more valid entitlement certificates. Please run subscription-manager for more information.
Oct 29 17:23:16 varnishd[2591]: Child (28563) not responding to CLI, killing it.
Oct 29 17:23:26 varnishd[2591]: Child (28563) not responding to CLI, killing it.
Oct 29 17:23:34 varnishd[2591]: Child (28563) not responding to CLI, killing it.
Oct 29 17:23:34 varnishd[2591]: Child (28563) not responding to CLI, killing it.
Oct 29 17:23:34 varnishd[2591]: Child (28563) died signal=3 (core dumped)
Oct 29 17:23:34 varnishd[2591]: child (15919) Started
Oct 29 17:23:34 varnishd[2591]: Child (15919) said Child starts
Oct 29 20:56:39 varnishd[2591]: Child (15919) not responding to CLI, killing it.
Oct 29 20:56:43 varnishd[2591]: Child (15919) not responding to CLI, killing it.
Oct 29 20:56:43 varnishd[2591]: Child (15919) not responding to CLI, killing it.
Oct 29 20:56:43 varnishd[2591]: Child (15919) died signal=3 (core dumped)
Oct 29 20:56:43 varnishd[2591]: child (22868) Started
Oct 29 20:56:43 varnishd[2591]: Child (22868) said Child starts
Oct 30 03:32:46 rhsm-complianced: This system is missing one or more valid entitlement certificates. Please run subscription-manager for more information.
Oct 30 10:30:40 kernel: varnishncsa invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
Oct 30 10:30:40 kernel: varnishncsa cpuset=/ mems_allowed=0-1
Oct 30 10:30:40 kernel: Pid: 15396, comm: varnishncsa Not tainted 2.6.32-131.0.15.el6.x86_64 #1
Oct 30 10:30:40 kernel: Call Trace:
Oct 30 10:30:40 kernel: [<ffffffff810c0101>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Oct 30 10:30:40 kernel: [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0
Oct 30 10:30:40 kernel: [<ffffffff81110880>] ? select_bad_process+0xd0/0x110
Oct 30 10:30:40 kernel: [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0
Oct 30 10:30:40 kernel: [<ffffffff81110b19>] ? out_of_memory+0x199/0x210
Oct 30 10:30:40 kernel: [<ffffffff811202dd>] ? __alloc_pages_nodemask+0x80d/0x8b0
Oct 30 10:30:40 kernel: [<ffffffff8115464a>] ? alloc_pages_current+0xaa/0x110
Oct 30 10:30:40 kernel: [<ffffffff8110d717>] ? __page_cache_alloc+0x87/0x90
Oct 30 10:30:40 kernel: [<ffffffff81122c4b>] ? __do_page_cache_readahead+0xdb/0x210
Oct 30 10:30:40 kernel: [<ffffffff81122da1>] ? ra_submit+0x21/0x30
Oct 30 10:30:40 kernel: [<ffffffff8110e9e3>] ? filemap_fault+0x4c3/0x500
Oct 30 10:30:40 kernel: [<ffffffff812628a9>] ? cpumask_next_and+0x29/0x50
Oct 30 10:30:40 kernel: [<ffffffff81137114>] ? __do_fault+0x54/0x510
Oct 30 10:30:40 kernel: [<ffffffff811376c7>] ? handle_pte_fault+0xf7/0xb50
Oct 30 10:30:40 kernel: [<ffffffff8100987e>] ? __switch_to+0x26e/0x320
Oct 30 10:30:40 kernel: [<ffffffff81057f99>] ? find_busiest_queue+0x69/0x150
Oct 30 10:30:40 kernel: [<ffffffff811382f8>] ? handle_mm_fault+0x1d8/0x2a0
Oct 30 10:30:40 kernel: [<ffffffff81092ff3>] ? __hrtimer_start_range_ns+0x1a3/0x460
Oct 30 10:30:40 kernel: [<ffffffff81041529>] ? __do_page_fault+0x139/0x480
Oct 30 10:30:40 kernel: [<ffffffff8109332f>] ? hrtimer_try_to_cancel+0x3f/0xd0
Oct 30 10:30:40 kernel: [<ffffffff810933e2>] ? hrtimer_cancel+0x22/0x30
Oct 30 10:30:40 kernel: [<ffffffff814dccc3>] ? do_nanosleep+0x93/0xc0
Oct 30 10:30:40 kernel: [<ffffffff810934b4>] ? hrtimer_nanosleep+0xc4/0x180
Oct 30 10:30:40 kernel: [<ffffffff810922a0>] ? hrtimer_wakeup+0x0/0x30
Oct 30 10:30:40 kernel: [<ffffffff814e0c3e>] ? do_page_fault+0x3e/0xa0
Oct 30 10:30:40 kernel: [<ffffffff814ddfe5>] ? page_fault+0x25/0x30
Oct 30 10:30:40 kernel: Mem-Info:
Oct 30 10:30:40 kernel: Node 0 DMA per-cpu:
Oct 30 10:30:40 kernel: CPU    0: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    1: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    2: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    3: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    4: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    5: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    6: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    7: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    8: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU    9: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU   10: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU   11: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU   12: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU   13: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU   14: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: CPU   15: hi:    0, btch:   1 usd:   0
Oct 30 10:30:40 kernel: Node 0 DMA32 per-cpu:
Oct 30 10:30:40 kernel: CPU    0: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    1: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    2: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    3: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    4: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    5: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    6: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    7: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    8: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    9: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   10: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   11: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   12: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   13: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   14: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   15: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: Node 0 Normal per-cpu:
Oct 30 10:30:40 kernel: CPU    0: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    1: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    2: hi:  186, btch:  31 usd:  11
Oct 30 10:30:40 kernel: CPU    3: hi:  186, btch:  31 usd:   2
Oct 30 10:30:40 kernel: CPU    4: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    5: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    6: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    7: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    8: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    9: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   10: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   11: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   12: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   13: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   14: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   15: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: Node 1 Normal per-cpu:
Oct 30 10:30:40 kernel: CPU    0: hi:  186, btch:  31 usd:   5
Oct 30 10:30:40 kernel: CPU    1: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    2: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    3: hi:  186, btch:  31 usd:   1
Oct 30 10:30:40 kernel: CPU    4: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    5: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    6: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    7: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    8: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU    9: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   10: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   11: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   12: hi:  186, btch:  31 usd:   8
Oct 30 10:30:40 kernel: CPU   13: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   14: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: CPU   15: hi:  186, btch:  31 usd:   0
Oct 30 10:30:40 kernel: active_anon:15180189 inactive_anon:868880 isolated_anon:0
Oct 30 10:30:40 kernel: active_file:496 inactive_file:603 isolated_file:0
Oct 30 10:30:40 kernel: unevictable:0 dirty:22 writeback:20 unstable:0
Oct 30 10:30:40 kernel: free:55664 slab_reclaimable:10223 slab_unreclaimable:284822
Oct 30 10:30:40 kernel: mapped:386 shmem:0 pagetables:54263 bounce:0
Oct 30 10:30:40 kernel: Node 0 DMA free:15616kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15224kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 30 10:30:40 kernel: lowmem_reserve[]: 0 2991 32281 32281
Oct 30 10:30:40 kernel: Node 0 DMA32 free:121196kB min:4168kB low:5208kB high:6252kB active_anon:1856648kB inactive_anon:457828kB active_file:124kB inactive_file:372kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3063392kB mlocked:0kB dirty:12kB writeback:16kB mapped:156kB shmem:0kB slab_reclaimable:3732kB slab_unreclaimable:2508kB kernel_stack:0kB pagetables:5060kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:352 all_unreclaimable? no
Oct 30 10:30:40 kernel: lowmem_reserve[]: 0 0 29290 29290
Oct 30 10:30:40 kernel: Node 0 Normal free:40808kB min:40844kB low:51052kB high:61264kB active_anon:27839988kB inactive_anon:1739808kB active_file:1104kB inactive_file:1240kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:29992960kB mlocked:0kB dirty:52kB writeback:40kB mapped:332kB shmem:0kB slab_reclaimable:19552kB slab_unreclaimable:584592kB kernel_stack:10656kB pagetables:107120kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1528 all_unreclaimable? no
Oct 30 10:30:40 kernel: lowmem_reserve[]: 0 0 0 0
Oct 30 10:30:40 kernel: Node 1 Normal free:45036kB min:45072kB low:56340kB high:67608kB active_anon:31024120kB inactive_anon:1277884kB active_file:756kB inactive_file:800kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:33095680kB mlocked:0kB dirty:24kB writeback:24kB mapped:1056kB shmem:0kB slab_reclaimable:17608kB slab_unreclaimable:552188kB kernel_stack:5456kB pagetables:104872kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1267 all_unreclaimable? no
Oct 30 10:30:40 kernel: lowmem_reserve[]: 0 0 0 0
Oct 30 10:30:40 kernel: Node 0 DMA: 2*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15616kB
Oct 30 10:30:40 kernel: Node 0 DMA32: 1357*4kB 979*8kB 689*16kB 1413*32kB 447*64kB 66*128kB 15*256kB 10*512kB 7*1024kB 0*2048kB 0*4096kB = 122684kB
Oct 30 10:30:40 kernel: Node 0 Normal: 10120*4kB 45*8kB 7*16kB 3*32kB 1*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 42776kB
Oct 30 10:30:40 kernel: Node 1 Normal: 2241*4kB 1460*8kB 766*16kB 273*32kB 51*64kB 2*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 46692kB
Oct 30 10:30:40 kernel: 10994 total pagecache pages
Oct 30 10:30:40 kernel: 9447 pages in swap cache
Oct 30 10:30:40 kernel: Swap cache stats: add 10177329, delete 10167882, find 6787957/7512667
Oct 30 10:30:40 kernel: Free swap  = 0kB
Oct 30 10:30:40 kernel: Total swap = 4194296kB
Oct 30 10:30:40 kernel: 16777215 pages RAM
Oct 30 10:30:40 kernel: 285063 pages reserved
Oct 30 10:30:40 kernel: 2192 pages shared
Oct 30 10:30:40 kernel: 16161719 pages non-shared
Oct 30 10:30:40 kernel: Out of memory: kill process 2591 (varnishd) score 164119553 or a child
Oct 30 10:30:40 kernel: Killed process 22868 (varnishd) vsz:147678356kB, anon-rss:63571052kB, file-rss:616kB
Oct 30 10:30:50 varnishd[2591]: Child (22868) not responding to CLI, killing it.
Oct 30 10:30:53 varnishd[2591]: Child (22868) not responding to CLI, killing it.
Oct 30 10:30:53 varnishd[2591]: Child (22868) died signal=9
Oct 30 10:30:53 varnishd[2591]: child (15405) Started
Oct 30 10:30:53 varnishd[2591]: Child (15405) said Child starts
Oct 30 14:44:27 varnishd[21828]: Platform: Linux,2.6.32-131.0.15.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit
Oct 30 14:44:27 varnishd[21828]: Child start failed: could not open sockets
Oct 30 14:44:32 varnishd[21855]: Platform: Linux,2.6.32-131.0.15.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit
Oct 30 14:44:32 varnishd[21855]: child (21856) Started
Oct 30 14:44:32 varnishd[21855]: Child (21856) said Child starts

_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



--
[http://www.varnish-software.com/static/media/logo-email.png]<http://www.varnish-software.com/> Per Buer
CEO | Varnish Software AS
Phone: +47 958 39 117 | Skype: per.buer
We Make Websites Fly!



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121030/489e8993/attachment.html>

From mi+thun at aldan.algebra.com  Wed Oct 24 13:13:24 2012
From: mi+thun at aldan.algebra.com (Mikhail T.)
Date: Wed, 24 Oct 2012 09:13:24 -0400
Subject: Can vmodules provide functions with optional arguments?
In-Reply-To: <20121024081545.GC24585@err.no>
References: <5085B29E.70802@aldan.algebra.com> <20121024081545.GC24585@err.no>
Message-ID: <5087E974.40706@aldan.algebra.com>

On 24.10.2012 04:15, Tollef Fog Heen wrote:
>> The subject says it: can functions provided by a loaded module have
>> optional arguments?
> No, they can't.  You could have it take a magic value like -1 or similar
> to signify "unset", though.
Thanks for the reply, Tollef. My goal was to make the vcl_recv be 
smaller and easier to read, so, instead of using a magic value, I 
renamed the actual function (which takes full set of arguments) into 
foo_ex() and created a separate foo() -- which passes all the mandatory 
arguments plus the default values for optional ones to foo_ex().

Still, I'd say, module-creation framework should some day support 
optional arguments (supplying default values)...

    -mi

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121024/a24f3ede/attachment.html>

From mi+thun at aldan.algebra.com  Wed Oct 24 13:26:19 2012
From: mi+thun at aldan.algebra.com (Mikhail T.)
Date: Wed, 24 Oct 2012 09:26:19 -0400
Subject: vmodule data, that would be private per thread
Message-ID: <5087EC7B.1010602@aldan.algebra.com>

I need to cache and reuse some data in each thread -- to avoid 
lock-contention. It does not seem like PRIV_VCL and PRIV_CALL offer 
that, so, what I am doing, is this:

 1. init_function creates static version of the data once and
    initializes thread key with pthread_key_create()
 2. each worker functions attempts to extract its thread's copy of the
    data with pthread_getspecific()
 3. if the pthread_getspecific() returned NULL (first call in this
    thread), new memory is allocated and recorded with pthread_setspecific()

Once loaded, the memory is not -- and needs not -- be freed until 
varnishd is restarted. This seems to work, however:

  * I see twice as many new-memory allocations as the top limit on the
    number of worker-threads: I run varnishd with -w 4,7 argument, but,
    when I hit it with ab, 14 lines indicating creation of a new copy of
    data are logged (with 14 different thread-IDs).
  * I'm wondering, if there is already Varnish API, that would hide
    these pthread_* manipulations...

Any ideas? Thank you!

    -mi


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121024/5f58160f/attachment.html>

From mi+thun at aldan.algebra.com  Wed Oct 24 13:33:48 2012
From: mi+thun at aldan.algebra.com (Mikhail T.)
Date: Wed, 24 Oct 2012 09:33:48 -0400
Subject: Returning an error (exception) from vmod's function
Message-ID: <5087EE3C.4080601@aldan.algebra.com>

I can't find an example of a vmod's function indicating an error -- 
other than returning NULL in std.readfile().

Besides normal results, my vmod's functions can fail due to a number of 
reasons and I'd like to be able to, at least, indicate them to the 
caller (vcl_recv) so an intelligent status can be returned to the 
client: some times the error ought to be 403 (my module checks 
credentials), and some times it must be 500 (internal error, an 
exception), for example.

Currently I log such exceptions with syslog(3) and return a static 
string to vcl_recv. But then there is no (easy) way to distinguish the 
kinds of failures...

Is there a proper way to do this? Something like SetStatus(sp, int 
status, const char *format, ...)? Thanks!

    -mi

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121024/0e959fd2/attachment.html>

From jnerin+varnish at gmail.com  Wed Oct 31 21:05:52 2012
From: jnerin+varnish at gmail.com (=?UTF-8?B?Sm9yZ2UgTmVyw61u?=)
Date: Wed, 31 Oct 2012 22:05:52 +0100
Subject: logging request time
In-Reply-To: <20121025150115.GA24645@mana.playfast.ru>
References: <20121023133358.GA31642@mana.playfast.ru>
 <20121024081746.GD24585@err.no>
 <20121025150115.GA24645@mana.playfast.ru>
Message-ID: <CAF-dNMaL9OMu3kW92z3E42oSJHckEdz8MHCTGkeZGhVqLpmh6w@mail.gmail.com>

Well, currently I use an indirect method, the ss utility & a quick'n'dirty
perl one liner:

ss -i -t 'sport = :80' |perl -e 'while (<>) { if (m/send (.*?)([M|K|]bps)
/){ ($s,$u)=($1,$2); $s=$s*1000000 if ("Mbps" eq "$u"); $s=$s*1000 if
("Kbps" eq "$u"); $t=$t+$s; $c=$c+1}} print "$t total bps, $c clients: ",
($t/$c) , " avg. bps\n" '

The "output" looks like:
1352182000 total bps, 539 clients: 2508686.45640074 avg. bps

I'm working on a munin plugin with this idea, as I'm testing different tcp
congestion avoidance algorithms and playing with initcwnd & initrwnd and I
like to keep track of the changes.

P.S. Remember to load the tcp_diag module in busy servers to speed up "ss"
so it doesn't have to resort to parsing /proc/net/tcp.


On Thu, Oct 25, 2012 at 5:01 PM, Vladimir Stavrinov <vstavrinov at gmail.com>wrote:

> On Wed, Oct 24, 2012 at 10:17:46AM +0200, Tollef Fog Heen wrote:
>
> > There's currently no support for the time for the full request.  You
> > can
>
> There are some times data in ReqStart & ReqEnd tags of shared memory
> log. May be there some variables exists to get those data?
>
>
> > use %{Varnish:time_firstbyte}x in the format string for the time to
> > first byte.
>
> It doesn't help as we need calculate download speed of clients.
>
> --
>
> ***************************
> ##  Vladimir Stavrinov
> ##  vstavrinov at gmail.com
> ***************************
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>



-- 
Jorge Ner?n
<jnerin at gmail.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20121031/c33d4448/attachment.html>