*/5 --> Could not get Let's Encrypt certificate

[snoopotic]

Hi.

I went from 0.10.2 over 0.10.3

by missing 0.10.4 over to 0.10.5

since the upgrade from 0.10.3 to 0.10.5 every 5 minutes the lets encrypt cron runs and throws:

Could not get Let's Encrypt certificate for <domain>: [Fri 08 Nov 2019 10:50:06 PM CET] Renew: '<domain>' [Fri 08 Nov 2019 10:50:06 PM CET] Skip, Next renewal time is: Wed 01 Jan 2020 11:05:25 PM UTC [Fri 08 Nov 2019 10:50:06 PM CET] Add '--force' to force to renew.

for every domain.

This also causes continuously:

"Neuerstellung der Webserver-Konfiguration" task in froxlor and also every 5 minutes my apache config gets renewed... meh

I am on git  @b162324f#0.10.5

 

running "php -q /var/www/froxlor/scripts/froxlor_master_cronjob.php --letsencrypt --force --debug"

is weird as it does not look bad but still keeps:

[error] Could not get Let's Encrypt certificate for <domain>:
[Sat 09 Nov 2019 07:21:15 PM CET] Renew: '<domain>'
[Sat 09 Nov 2019 07:21:15 PM CET] Skip, Next renewal time is: Thu 02 Jan 2020 11:03:17 PM UTC
[Sat 09 Nov 2019 07:21:15 PM CET] Add '--force' to force to renew.

...

Any ideas how to look deeper?

I walked through 450 pages of the system log in froxlor and found that this messages appear since the upgrade to 0.10.5

[d00p]

10 minutes ago, snoopotic said:

running "php -q /var/www/froxlor/scripts/froxlor_master_cronjob.php --letsencrypt --force --debug"

it's either --module (e.g. letsencrypt) OR --force (which just combines a few tasks and runs them, like regenerating config files)

 

12 minutes ago, snoopotic said:

since the upgrade from 0.10.3 to 0.10.5 every 5 minutes the lets encrypt cron runs and throws:

I'm pretty sure it's the tasks cronjob, not the letsencrypt cronjob (which ONLY checks whether there is a renew required or not, rest is done by tasks/webserver cronjob)

And I'm also pretty sure, that there is way more output when using --debug when running the cron manually

[snoopotic]

Hmm. 

I now ran "php -q /var/www/froxlor/scripts/froxlor_master_cronjob.php --force --debug" two times as it now always changes between 2 states.

I replaced my IPs and the domains via sed but kept the rest. I can't read anything abnormal...

need more?

froxlor2.log froxlor.log

[d00p]

So, obviously, the domain is known by acme.sh and tells froxlor "yo man, it's not time yet" but somehow froxlor does not have that information anymore (maually deleted certificate?) and just thinks: ah geez, lets encrypt activated but no certificate, let's get some.

Try to remove the domain from acme.sh so froxlor can start fresh.

/root/.acme.sh/acme.sh --remove -d domain1.tld

Best, if you know you'll need more than 5 minutes - just stop the cron-daemon for that time so it does not run while you adjust things

[snoopotic]

Hmm.

now it throws:

[information] Creating certificate for domain1.tld
[information] Adding SAN entry: domain1.tld
[information] Adding SAN entry: www.domain1.tld
[debug] [Sun 10 Nov 2019 08:55:25 AM CET] Renew: 'domain1.tld'
[Sun 10 Nov 2019 08:55:25 AM CET] 'domain1.tld' is not a issued domain, skip.
[error] Could not get Let's Encrypt certificate for domain1.tld:
[Sun 10 Nov 2019 08:55:25 AM CET] Renew: 'domain1.tld'
[Sun 10 Nov 2019 08:55:25 AM CET] 'domain1.tld' is not a issued domain, skip.
[information] No new certificates or certificates due for renewal found

But

./acme.sh --issue -d domain1.tld -w /var/www/froxlor

works if you need to know that. Sadly froxlor does not recognize 😕

[d00p]

You need to remove any certificate in froxlor too of course (admin -> ssl-certificates -> delete the corresponding certificates)

5 minutes ago, snoopotic said:

./acme.sh --issue -d domain1.tld -w /var/www/froxlor

works if you need to know that. Sadly froxlor does not recognize 😕

It' not intended to issue certificates manually, froxlor cannot know about these this way

[snoopotic]

Hi.

Yes, of course it was just to tell you that it's not an issue of acme.sh - it CAN get the certs. but froxlor doesn't

what I did:

1) delete cert od domain1.tld (not the others) in froxlor

2) unchecked https-ips and let's encrypt settings in froxlor.

3) deleted certs by 

/root/.acme.sh/acme.sh --remove -d domain1.tld

and also removed the folder

4) ran php -q /var/www/froxlor/scripts/froxlor_master_cronjob.php --debug --tasks twice

last run: 

[information] TasksCron: Searching for tasks to do
[notice] Checking system's last guid

5) enabled ssl-ips and checked the let's encrypt checkbox in froxlor of domain (assuming this re-eanables reissuing of certsI.

6) ran php -q /var/www/froxlor/scripts/froxlor_master_cronjob.php --debug --tasks twice again

first run throws the one above (Could not get Let's Encrypt certificate for domain1.tld).

The second run is the same like in 4).

5) to make sure ran php -q /var/www/froxlor/scripts/froxlor_master_cronjob.php --debug --letsencrypt

This also throws the "Could not get Let's Encrypt certificate for domain1.tld" from above...

 

expectation: this workflow regenerates certs in first run of 6)

 

 

[d00p]

7 minutes ago, snoopotic said:

first run throws the one above (Could not get Let's Encrypt certificate for domain1.tld).

and? There definetly should be MORE output...way more

8 minutes ago, snoopotic said:

it CAN get the certs. but froxlor doesn't

well, works for me and many others just fine...no idea what's up there with you. Maybe your manually-ip-changing-stuff and whatsoever...no idea what you did there

[d00p]

Also, do not manually call --tasks or --letsencrypt, just use --force --debug to be sure that regeneration tasks are inserted (because --tasks does nothing without tasks and --letsencrypt does nothing else but check for renew and inserts a tasks for --tasks)

[snoopotic]

16 minutes ago, d00p said:

and? There definetly should be MORE output...way more

Hmm. The rest of the output looks the same as the attached log files I posted initially.

So you mean that whole output of "--force --debug" shall have way more information? oO

I am on

PHP 7.3.11-1+0~20191026.48+debian10~1.gbpf71ca0 (cli) (built: Oct 26 2019 14:20:37) ( NTS )

and the git version tagged with 0.10.5: b162324f#0.10.5

 

[d00p]

Just reproduced that: removed ssl-certificate in froxlor, removed acme.sh certificate, removed acme.sh directory of that domain:

root@froxsrv:~# php /var/www/html/froxlor/scripts/froxlor_master_cronjob.php --force --debug
[information] TasksCron: Searching for tasks to do
[information] Running Let's Encrypt cronjob prior to regenerating webserver config files
[information] Requesting/renewing Let's Encrypt certificates
[information] Creating certificate for demo.froxlor.org
[information] Adding SAN entry: demo.froxlor.org
[information] Adding SAN entry: www.demo.froxlor.org
[information] Checking for LetsEncrypt client upgrades before renewing certificates:
[So 10. Nov 10:03:13 CET 2019] Installing from online archive.
[So 10. Nov 10:03:13 CET 2019] Downloading https://github.com/Neilpang/acme.sh/archive/master.tar.gz
[So 10. Nov 10:03:14 CET 2019] Extracting master.tar.gz
[So 10. Nov 10:03:14 CET 2019] Installing to /root/.acme.sh
[So 10. Nov 10:03:14 CET 2019] Installed to /root/.acme.sh/acme.sh
[So 10. Nov 10:03:14 CET 2019] Good, bash is found, so change the shebang to use bash as preferred.
[So 10. Nov 10:03:14 CET 2019] OK
[So 10. Nov 10:03:14 CET 2019] Install success!
[So 10. Nov 10:03:14 CET 2019] Upgrade success!
[So 10. Nov 10:03:15 CET 2019] Removing cron job
[debug] [So 10. Nov 10:03:15 CET 2019] Creating domain key
[So 10. Nov 10:03:16 CET 2019] The domain key is here: /root/.acme.sh/demo.froxlor.org/demo.froxlor.org.key
[So 10. Nov 10:03:16 CET 2019] Multi domain='DNS:demo.froxlor.org,DNS:www.demo.froxlor.org'
[So 10. Nov 10:03:16 CET 2019] Getting domain auth token for each domain
[So 10. Nov 10:03:19 CET 2019] Getting webroot for domain='demo.froxlor.org'
[So 10. Nov 10:03:19 CET 2019] Getting webroot for domain='www.demo.froxlor.org'
[So 10. Nov 10:03:19 CET 2019] demo.froxlor.org is already verified, skip http-01.
[So 10. Nov 10:03:19 CET 2019] www.demo.froxlor.org is already verified, skip http-01.
[So 10. Nov 10:03:20 CET 2019] Verify finished, start to sign.
[So 10. Nov 10:03:20 CET 2019] Lets finalize the order, Le_OrderFinalize: https://acme-v02.api.letsencrypt.org/acme/finalize/xxx/yyy
[So 10. Nov 10:03:21 CET 2019] Download cert, Le_LinkCert: https://acme-v02.api.letsencrypt.org/acme/cert/asdasdasd
[So 10. Nov 10:03:22 CET 2019] Cert success.
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
[So 10. Nov 10:03:22 CET 2019] Your cert is in  /root/.acme.sh/demo.froxlor.org/demo.froxlor.org.cer
[So 10. Nov 10:03:22 CET 2019] Your cert key is in  /root/.acme.sh/demo.froxlor.org/demo.froxlor.org.key
[So 10. Nov 10:03:22 CET 2019] The intermediate CA cert is in  /root/.acme.sh/demo.froxlor.org/ca.cer
[So 10. Nov 10:03:22 CET 2019] And the full chain certs is there:  /root/.acme.sh/demo.froxlor.org/fullchain.cer
[information] Updated Let's Encrypt certificate for demo.froxlor.org
[information] Let's Encrypt certificates have been updated

works just fine

[snoopotic]

Hmm.

although I deleted in froxlor the cert,

I don't get: 

[debug] [So 10. Nov 10:03:15 CET 2019] Creating domain key

It's starting with:

[debug] [Sun 10 Nov 2019 10:27:55 AM CET] Renew: 'domain.tld'

although now I unchecked: "Let's Encrypt Schlüssel wiederverwenden" in ssl settings.(was checked before)  but still same behaviour...

[d00p]

Well Looks like it still wants to renew, so you did not delete completely, check the froxlor SSL certificates page (or manually check the domain_ssl_settings table looking for entries with the corresponding domain id)

[snoopotic]

Hmm, in DB it looks good.

in AcmeSh.php

I placed in Line 320: (

https://github.com/Froxlor/Froxlor/blob/b162324ff02a3279d2728b722d70acd3ccd59059/lib/Froxlor/Cron/Http/LetsEncrypt/AcmeSh.php#L320)

for testing:

//FELIX
echo "GIVE CERTMODE: $cert_mode <-thanks ";

and got

[information] Creating certificate for domain.tld
[information] Adding SAN entry: domain.tld
[information] Adding SAN entry: www.domain.tld
GIVE CERTMODE: renew <-thanks [debug] [Sun 10 Nov 2019 11:59:56 AM CET] Renew: 'domain.tld'
[Sun 10 Nov 2019 11:59:56 AM CET] 'domain.tld' is not a issued domain, skip.
[error] Could not get Let's Encrypt certificate for domain.tld:
[Sun 10 Nov 2019 11:59:56 AM CET] Renew: 'domain.tld'
[Sun 10 Nov 2019 11:59:56 AM CET] 'domain.tld' is not a issued domain, skip.

then I looked further:

placed around 

https://github.com/Froxlor/Froxlor/blob/b162324ff02a3279d2728b722d70acd3ccd59059/lib/Froxlor/Cron/Http/LetsEncrypt/AcmeSh.php#L267

} else {
                                // FELIX
                                $cert_mode = 'issue';
                                        $cronlog->logAction(FroxlorLogger::CRON_ACTION, LOG_INFO, "Creating certificate for " . $certrow['domain']);
                                //FELIX
                                echo "IS IT ISSUE OR NOT $cert_mode <-- ";
                                }

then it worked! 

(as you see I explicitly set cert_mode = issue again - but afaik it's not necessary but you may see it in your code - I wasn't able yet.)

I now have 2 domains that have new certs...

Well, after re-running, there domains are also NOT marked to be renewed so there must be a check that sets the renew-cert_mode because of some DB data...!?

Is there a DB query I might check?

I guess we're going further.

 

Is it maybe I skipped the 0.10.3 --> 0.10.4 update and did a 0.10.3 --> 0.10.5 update?

EDIT: AH! 
as we're in a loop here: 

https://github.com/Froxlor/Froxlor/blob/b162324ff02a3279d2728b722d70acd3ccd59059/lib/Froxlor/Cron/Http/LetsEncrypt/AcmeSh.php#L247

$cert_mode may change (and does in my case) and thus the default in L246 is overridden... So the workaround may be to set it explicitly like I did for recreating... what do you think?

[d00p]

16 minutes ago, snoopotic said:

Hmm, in DB it looks good

Meaning? Is there an entry for that domain or not?

18 minutes ago, snoopotic said:

(as you see I explicitly set cert_mode = issue again - but afaik it's not necessary but you may see it in your code - I wasn't able yet.)

Might be correct after all, need to check later when back home. 

 

19 minutes ago, snoopotic said:

Well, after re-running, there domains are also NOT marked to be renewed so there must be a check that sets the renew-cert_mode because of some DB data...!?

Yes, Froxlor checks the date from its entries in the database.

20 minutes ago, snoopotic said:

Is it maybe I skipped the 0.10.3 --> 0.10.4 update and did a 0.10.3 --> 0.10.5 update?

No, even if updating from 0.9 to 0.10.5 all required changes are being executed.

 

20 minutes ago, snoopotic said:

$cert_mode may change (and does in my case) and thus the default in L246 is overridden... So the workaround may be to set it explicitly like I did for recreating... what do you think?

Yes that what I meant earlier with "might be correct after all"

[snoopotic]

Hi.

With this little Workaround to set $cert_mode = issue above L267

I was able to delete all certs under admin_domains.php?page=sslcertificates and let them regenerate by running the master_cron.

Now the log is empty of this warnings!

[information] TasksCron: Searching for tasks to do
[information] Running Let's Encrypt cronjob prior to regenerating webserver config files
[information] Requesting/renewing Let's Encrypt certificates
[information] No new certificates or certificates due for renewal found

I will keep this variable setting until the next update.

 

Also: The issue that my Webserver config is rewritten every 5 minutes the cert-job runs is also over.

But what really the cause was - I have no idea. Thank you for your effort.

[d00p]

Yup, looks about right, the $cert_mode variable is not re-initialized to its initial state 'issue' in the foreach-loop, following patch should fix that:

diff --git a/lib/Froxlor/Cron/Http/LetsEncrypt/AcmeSh.php b/lib/Froxlor/Cron/Http/LetsEncrypt/AcmeSh.php
index 545ad323..eb620603 100644
--- a/lib/Froxlor/Cron/Http/LetsEncrypt/AcmeSh.php
+++ b/lib/Froxlor/Cron/Http/LetsEncrypt/AcmeSh.php
@@ -243,9 +243,11 @@ class AcmeSh extends \Froxlor\Cron\FroxlorCron
 
                // customer domains
                $certrows = $needRenew ? $needRenew['customer_ssl'] : array();
-               $cert_mode = 'issue';
                foreach ($certrows as $certrow) {
 
+                       // initialize mode to 'issue'
+                       $cert_mode = 'issue';
+
                        // set logger to corresponding loginname for the log to appear in the users system-log
                        $cronlog = FroxlorLogger::getInstanceOf(array(
                                'loginname' => $certrow['loginname'],

Will be part of next release (0.10.6)