Remove external OTP dependency from deploy hook Synology_DSM.sh

[Eagle3386]

Also adapt to DSM 7's API improvements.

See #2727 for further details.

[wyxls]

@Neilpang Hello, Thank you so much for this project.

Could this PR be merged into master branch and be released in recent days because it's a problem that syno's dsm OS doesn't have oathtool and related dependencies in it.

Syno user really need this to auto deploy the certs for the accounts that have enabled two-step authtication.

[Neilpang]

the SYNO_TOTP_SECRET is totally removed in this pr, how does the exising users renew the cert?

if you make change to the code, keep compatible with the existing users.

Imagine that the existing users are just using a cronjob to renew the cert, we must make sure the existing certs can be renewed automatically without human involved.

There is only one exception that: the api changed, and the exising code can not work anymore, so we have to change the code.

[Eagle3386]

the SYNO_TOTP_SECRET is totally removed in this pr, how does the exising users renew the cert?

This is intentionally, because it's one of two reasons for the whole change: getting rid of storing the OTP secret for good - just like you can via the web UI by ticking "Remember this device", but via Synology's official API instead of the UI.

if you make change to the code, keep compatible with the existing users.

Due to change in login procedure for enhanced comfort - as well as security due to removal of the previous requirement to store the secret in plain text on disk - that's simply impossible.
We fundamentally switch from one login method that can be considered "legacy" to a modern one that supports OTP as well as non-OTP for API scenarios like ours.

Imagine that the existing users are just using a cronjob to renew the cert, we must make sure the existing certs can be renewed automatically without human involved.

Again, this is impossible in this case.
Even more so, the breaking change is deliberately & intentional as it simply adapts to Synology's modernized API & its improved security.

There is only one exception that: the api changed, and the exising code can not work anymore, so we have to change the code.

Yes, Synology changed the API upon release of DSM 7. They only included the legacy login method for backward compatibility.
However, security is more important than supporting insecure, legacy login methods - which is why I explicitly refrained from supporting the old behavior.

Nonetheless, I happily update the FAQ / Wiki as soon as this PR got merged - just as I updated the deploy API plug-in's file header with improved docs regarding its usage:

# Usage:
# 1. export SYNO_Username="adminUser"
# 2. export SYNO_Password="adminPassword"
# Optional exports (shown values are the defaults):
# - export SYNO_Certificate="" to replace a specific certificate via description
# - export SYNO_Scheme="http"
# - export SYNO_Hostname="localhost"
# - export SYNO_Port="5000"
# - export SYNO_Device_Name="CertRenewal" - required for skipping 2FA-OTP
# - export SYNO_Device_ID=""              - required for skipping 2FA-OTP
# 3. acme.sh --deploy --deploy-hook synology_dsm -d example.com

[evilhamsterman]

I agree with @Eagle3386, this is a breaking change, but it's for good reason. The legacy method is problematic to setup and maintain, frequently breaking. There's no guarantee that it will continue to be support, in fact it will likely be removed in a future major update of DSM. The new API is the way forward and will require breaking changes to support.

[wyxls]

I have merged the new 'deploy/synology_dsm.sh' manually and tried using it. I think it's acceptable for those existing synology DSM users. The old method to deploy certs via SYNO_DID or SYNO_TOTP_SECRET with two-step authentication enabled account is too far from easy.

It's much more simple than before:

1. export SYNO_Username and SYNO_Password
2. fill in the OTP once when it prompts

Just like login via DSM WebUI and only run it once.

[Eagle3386]

(…)
It's much more simple than before:

1. export SYNO_Username and SYNO_Password
2. fill in the OTP once when it prompts

Just like login via DSM WebUI and only run it once.

I'd like to clarify that the mentioned OTP code prompt needs to be answered only once - any future execution of the deploy script then utilizes the received device ID for the actual deployment of the certificates.

This fulfills 3 requirements at once:

1. Synology's requirement regarding secure logins of any 3rd party API consumer
2. Provide the maximum comfort to any user of this script - especially regarding automation, i. e. via cron
3. Remove dependencies on external libraries which required even more interference with the system, up to the point of "restoring" them upon Synology DSM updates/-grades, or installation of otherwise unused packages, e. g. Docker package which surely isn't used by every ACME.sh & Synology user.

[wyxls]

I see. The trusted device ID is stored and it's convenient for automatic deploy after the certs is renewed.

[Eagle3386]

Exactly! 😎

@Neilpang Anything further to address from my side or can you, due to explanations/reasoning given above, accept this PR?
Because I'd then update the Wiki right away in order to explain the changed usage, why it's better (especially regarding security) & how any legacy user can easily adopt to the new workflow within less than 2 minutes. 🥳

[Neilpang]

if the cert was first issued by the old credentials, it MUST be renewed with the same credentials too.
This is the compatiblity. Unless the old credentials/api was shutdown, it can not be renewed anymore.

I understand you want to get rid of the old api, but compatiblity is the FIRST rule.

Imagine you configured a server with acme.sh, and it has been run well for years. but one day the ssl cert is not renewed in time, what do you think?

please make your code:

1. check if the old credentials exist, use the old methods to renew. You can print a warning message to notfify the user to upgrade.
2. otherwise, check and use the new api.

[Eagle3386]

While I disagree with your compatibility point (if Synology changes the API tomorrow, user complaints then would be pretty much the same as those caused by this PR), I consider your suggestion of issuing a warning a valid alternative.

I'll edit my PR accordingly within the next couple of days.

However, let me be clear regarding support of your mentioned "old methods": I will not support any additional Docker or goathtool approach - only the initial one, i.e. the one using oathtool will be supported.
This is 100% in line with the old deploy script & therefore the only officially supported TOTP method thus far.

[evilhamsterman]

@Neilpang We never know when Synology might pull the legacy API as far as I can tell they've never made a commitment to when they'll EOL that. Like @Eagle3386 said if Synology breaks it users would be in the same mess, so it would be better to do it now. I don't think there'd be very many production Synology systems using the current method as it is so weird and fragile, I know I wouldn't. Most likely it's homelab stuff, If someone can figure out the existing script they shouldn't have much difficulty figuring out the new one.

[Neilpang]

If the legacy API will ever be closed, then it's not our fault.
Our goal is to keep our code as compatible as long as we can.

You can modify the usage wiki to only describe your new credentials and usages, so that anyone from now on would use the new methods.

But we must support to renew the cert which was ever issued by the old methods.

So, I think I'm clear:

1. edit the wiki to only introduce the new usage.
2. when issuing/renewing a cert, if new credentials exists, try use the new methods, otherwise, if the old credentials exist, try to use the old methods as before and show a warning message to tell the user to upgrade to use the new method. Otherwise, no credentials found, just return error code.
3. if one day the old api is closed, we can clean the code to remove all the old methods.

[Eagle3386]

@Neilpang Finally had some free time to complete this, but found that even the old method using OTP was calling the Synology API wrong:

acme.sh/deploy/synology_dsm.sh

Lines 105 to 111 in 41b6aeb

	if [ -n "$SYNO_DID" ]; then
	_H1="Cookie: did=$SYNO_DID"
	export _H1
	_debug3 H1 "${_H1}"
	fi
	response=$(_post "method=login&account=$encoded_username&passwd=$encoded_password&api=SYNO.API.Auth&version=$api_version&enable_syno_token=yes&otp_code=$otp_code&device_name=certrenewal&device_id=$SYNO_DID" "$_base_url/webapi/auth.cgi?enable_syno_token=yes")

It's wrong, because

1. it puts &device_id=$SYNO_DID at the POST request's URL end - which the API only allows for API calls omitting the TOTP code after using a different login API call with an TOTP code, see Example 3 & Example 4 over at the API docs for further details.
2. It actually defines enable_syno_token=yes twice: in the middle of the URL query string & right after the ? as part of the "base" URL - while it should've define enable_device_token=yes for that second time instead.

However, since you requested that old setups should continue "as is", I reintroduced these misuses of the API - complaints due to errors will be completely redirected to you! 🤪

Also, there's neither a _warn nor __yellow function built-in, hence I can't show a warning.
I can either use _err or the pretty easily ignored _info, but since the former would fail any deployment attempt for those old setups, I opted for the latter instead - much against my own creed regarding good coding!

[alidaf]

Any idea when this is likely to see the light of day? The latest version I see is 9th June. How are the instructions likely to change - will it still be shell based? I found the instructions for determining SYNO_DID and SYNO-TOTP_SECRET completely confusing, the latter being possibly browser/OS dependent.

[alidaf]

I would be very happy if the wiki will also reflect existing installations.
Probably @alidaf would agree. ;-)

I would indeed!

[Eagle3386]

@alidaf Please see Deploy the certificate to Synology DSM for updated instructions on how to automate deployment with 2FA enabled.

@jaydee73 Same as for @alidaf, however, regarding already running setups: yes, they're covered as well, but instructions for those are intentionally dropped completely.
Migration is pretty simple: delete the lines containing SAVED_SYNO_DID & SAVED_SYNO_TOTP_SECRET from your account's configuration file (located at acme.sh/example.net[_ecc]/example.net.conf) & then follow the update wiki's entry as described there.

Regarding the command line parameters: --renewal only renews the certificate, it does not deploy the renewed certificate's files. You have to use --renewal --deploy --deploy-hook synology_dsm to automatically deploy any renewed certificate.

Regarding your linked guide: it uses a Docker container while the wiki page - the old just as the new one - talk about ACME.sh running on your actual system, i. e. inside /usr/local/share/acme.sh

[jaydee73]

@Eagle3386 Thanks for the input. I'll try to work through it the coming days. One thing already popped up: You wrote, that after deleting the two lines (DID und TOTP), we should follow the wiki instructions. You mean: From top to bottom? Point 3 says "acme.sh --deploy --deploy-hook synology_dsm -d example.com". Do we really have to go through this deploy-stuff again, even if we already have an existing certificate? If so, how can me make sure that the script does not create a new (additional!) certificate to the existing one?

And on top of that, we have a new issue (#4721) saying that 3.0.7 synology hook is broken. Are you aware of this? Is this because of your updated stuff?

[Eagle3386]

@jaydee73

1. Yes, from top to bottom, but one-time only.
2. Stop complaining about setting up deployment for subsequent automated deployment - I just fixed insecure, outdated stuff from the original deploy hook author & made it like it was supposed to be done from the beginning!
3. If 3-4 little steps are too much to ask for, then go, look for help elsewhere. I'm done with users of such kind.
4. You don't seem to understand even the very basics of ACME.sh - point 3 is deployment only, hence it doesn't even care, if issuance or renewal happened before.
5. The issue you mentioned clearly has an ECC certificate problem: the user runs on DSM 6 (reported API version is 6 instead of 7) & 6 can't do ECC certificates.
6. The hook is not broken, it just needs to be used correctly - I'm using the exact same script code for an ECC wildcard certificate & neither issuance nor renewal had any problem succeeding.

[jaydee73]

Holy moly, this was not what I wanted to achieve. I'm sorry. That probably happens when one (=me!) is not using his native language. Daher einmal ganz kurz: Ich wollte dir definitiv nicht auf den Schlips treten. Ganz im Gegenteil, ich bin dankbar für deine Hilfe.

It doesn't matter for me if there are 2 steps, 20 steps or 200 steps to be done. It's simply that I want to make sure (in advance!) to do the right steps. And maybe also help others with the same questions. This wasn't meant to be a complain at all.

You are totally right, step 3 ist only about deploying. My bad, I should have paid better attention.

Also thanks for clarifying the other issue. Wasn't clear for me that he still sits on DSM6.

Sorry again! I hope pulse is slowly getting back to normal... :)

[Eagle3386]

Yeah, that helped, clarify a lot & settle things back to normal. - "Schwamm drüber!" 😉

Also, I'm here to help with further questions, if any new arise.

[jaydee73]

Strange...I made a new comment some hours ago, but for whatever reason it isn't visible....ok, I'll try again...:

Yesterday I tried to implement the new method on my already running system. Unfortunately I failed. I deleted the two old conf-file lines (SYNO DID and SYNO TOTP) and added the two new lines (Device name & device ID) to my conf file. Small info: I am using the official acme.he docker container. Every time (I have tried several times...) I executed the script with the deploy hook (directly on the shell), it still used the old method and stated, that is is deprecated. No asking for an otp code. Restart of the container also didn't change anything.

Then I started from scratch and installed a new container. Issuing (again directly on the shell) went without problems. Deploy also acted as expected. I was asked for an otp code, entered it and the certificate was successfully deployed to the Syno. So far so good.

Nevertheless two questions arised:
a)
I defined 'export SYNO_Device_Name="CertRenewal"' in my conf file as proposed. Regardless of that, I was asked by the script to enter a device name. I assume this is expected behaviour?
b)
I also added the line 'export SYNO_Device_ID=""' to my conf file. The wiki says, the script requests the Device ID from the API and saves it to the conf file. But unfortunately nothing was added to my conf file. I would expect a line like 'export SYNO_Device_ID="value"'. I assume, this missing line would cause trouble when renewal time comes?

In general, my conf file indeed does "work". Yesterday I also added Gotify and executed some "export" commands directly in the shell. They were correctly added to my conf file.

So, yes, some new help would be appreciated. Big thanks!

[wyxls]

@jaydee73
The new script will first check the conf file for SYNO_* envs needed, then check&load current running shell's SYNO_* envs, throw errors if there's no such envs. SYNO_Username and SYNO_Password are required and others are optional. So.....

a. I think it's an expected behaviour. Now the script will only check if there's defined "SYNO_Device_ID" even there's already a "SYNO_Device_Name" exist. It doesn't matter anything unless you manually change "SYNO_Device_Name" which doesn't match "SYNO_Device_ID".
Related code shoule be below:
[ -z "${SYNO_Device_ID:-}" ]; then printf "Enter OTP code for user '%s': " "$SYNO_Username" read -r otp_code

b. Yes, it might be trouble when redeployment happens because the script will ask for "SYNO_Device_ID".

Run "export" command will add env variable into current running shell to make sure the new script could load it for deployment. Basically, Device ID is generated by Syno's API while the script successfully pass 2FA authentication unless you already have one.
Related code should be below:
response=$(_get "$_base_url/webapi/entry.cgi?api=SYNO.API.Auth&version=$api_version&method=login&format=sid&account=$encoded_username&passwd=$encoded_password&otp_code=$otp_code&enable_syno_token=yes&enable_device_token=yes&device_name=$SYNO_Device_Name") _debug3 response "$response" SYNO_Device_ID=$(echo "$response" | grep "device_id" | sed -n 's/.*"device_id" *: *"\([^"]*\).*/\1/p') _secure_debug2 SYNO_Device_ID "$SYNO_Device_ID"

If "SYNO_Device_ID" is not defined in advance, the script will add SAVED_SYNO_Device_ID='blahblahblah' into conf file for future redeployment once the deployment succeed.
After that, you will no longer be required to enter anything when redeployment happens.

[Eagle3386]

Thanks, @wyxls for explaining most of the stuff already. 👍🏻

@jaydee73, to make it absolutely clear:

1. Don't add anything manually into your configuration file, but do delete those 2 lines mentioned.
2. The deploy script will add those 2 lines for you, because it must login with your admin's credentials & TOTP code in order to do any subsequent login without TOTP code.
3. That first deployment call must be done manually, i.e. run acme.sh --deploy --deploy-hook synology_dsm (…), input the TOTP code (& optionally a device name, if you don't like the default CertRenewal) & wait for the deployment to succeed.
4. Any subsequent deployment, i.e. when running via Synology DSM's Task Scheduler as the NAS guide explains, will then use a token (retrieved via previously mentioned manual run) & hence neither require another TOTP code nor the TOTP secret to generate a new TOTP code via some 3rd party tool.
5. You don't define export SYNO_Device_Name="CertRenewal" inside the configuration file, but execute that very statement as a command (export is a shell command, not a shell language keyword 😉).
6. Again, the device ID is written to the configuration file only after a successful certificate deployment.

[jaydee73]

Thanks for all your support & patience, guys. But in the end, it didn't work out for me...again...

I really followed all your steps....nothing added to conf file, deleted the two lines, manually run acme.sh for deploy, entered TOTP code and device name.

The script successfully deployed the certificate [Wed Aug 9 20:37:52 UTC 2023] Success
The certificate is visible in Syno GUI, so I do assume that everything went the right way.

But again, no device ID was written to my conf file after deployment.

This seems to be rocket science, or at least my very basic knowledge isn't nearly enough to get it going... :-(

I'll probably wait for some other guys who are also using this script in docker. Maybe they'll have the same problem or maybe it's working for them and they have an additional hint for me. Until then I'll stick with the "old" and less secure "DID method".

Thanks again, especially to Eagle3386.

[Eagle3386]

(…)
The script successfully deployed the certificate [Wed Aug 9 20:37:52 UTC 2023] Success The certificate is visible in Syno GUI, so I do assume that everything went the right way.

But again, no device ID was written to my conf file after deployment.

Hold on here for a second, please.
If the deployment succeeds, yet no update to the configuration file occurs, then it might be a simple permission problem.

This seems to be rocket science, or at least my very basic knowledge isn't nearly enough to get it going... :-(

They put humans onto the moon with far less code, let alone computational power - if certificate deployment / its configuration file update is more complex, then we're officially outta space here! 🥳🤣

I'll probably wait for some other guys who are also using this script in docker. Maybe they'll have the same problem or maybe it's working for them and they have an additional hint for me. Until then I'll stick with the "old" and less secure "DID method".

Please, provide us with a link to the container info page as well as the debug log snippet after deployment into DSM succeeded, but prior to the deploy script finishing its work (use --debug 3 and remove every line that's outside said scope).

I'm thinking of either a permission problem (as stated above) or maybe a container configuration problem & want to analyze this further.

Thanks again, especially to Eagle3386.

You're welcome & thanks for your acknowledgement. 😇

[jaydee73]

I will provide the informations you need this evening.

One info in advance. Several other "hooks" (is this the right expression?) can write (an do so) into the conf file:

--set-default-ca
--issue
--notify-hook
=> They all write into my conf file. All the commands are executed in the same shell. And of course --deploy --deploy-hook... is also executed in this shell. So my basic knowledge wouldn't think of a general permission problem of the container. But of course I do not know how the syno deploy script works...

Hopefully we can figure this out together.

[Eagle3386]

That's strange, because as you can see here:

acme.sh/deploy/synology_dsm.sh

Lines 160 to 163 in 56cf93d

	_savedeployconf SYNO_Username "$SYNO_Username"
	_savedeployconf SYNO_Password "$SYNO_Password"
	_savedeployconf SYNO_Device_Name "$SYNO_Device_Name"
	_savedeployconf SYNO_Device_ID "$SYNO_Device_ID"

device name & also ID are saved just like username & password - which all use ACME.sh's _savedeployconf function.

However, the script not only works on my DSM 7.2, but on many others as well: there are/were 3 users with issues thus far - 2 were solved within a day or so (1 was due to outdated DSM, the other calling the wrong API path on DSM 6), hence your issue is the only remaining one.

Can you - just for testing, of course - remove the lines for username & password as well, re-run the deployment script (you might need to re-run export SYNO_Username="YourUserName" & export SYNO_Password="YourPassword" in advance, too) & report back if username & password do get written into the file?

Because if they reappear within the file, it might be a value issue, but if not, then there might be a more serious issue here…

[jaydee73]

Please, provide us with a link to the container info page

I'm not exactly sure, which page you mean by this? This one? https://github.com/acmesh-official/acme.sh/wiki/Run-acme.sh-in-docker ?

as well as the debug log snippet after deployment into DSM succeeded, but prior to the deploy script finishing its work

I hope, this is the right part. Of course I have the full debug 3 log, if you need more to analyze:

-----END CERTIFICATE-----

----------------------------20230810163611
Content-Disposition: form-data; name="id"


----------------------------20230810163611
Content-Disposition: form-data; name="desc"

test3
----------------------------20230810163611--
'
[Thu Aug 10 16:36:11 UTC 2023] _postContentType='multipart/form-data; boundary=--------------------------20230810163611'
[Thu Aug 10 16:36:11 UTC 2023] curl exists=0
[Thu Aug 10 16:36:11 UTC 2023] mktemp exists=0
[Thu Aug 10 16:36:11 UTC 2023] wget exists=0
[Thu Aug 10 16:36:11 UTC 2023] _CURL='curl --silent --dump-header /acme.sh/http.header  -L  --trace-ascii /tmp/tmp.QuYo83qCOz  -g '
[Thu Aug 10 16:36:11 UTC 2023] _ret='0'
[Thu Aug 10 16:36:11 UTC 2023] response='{"data":{"id":"FwXyK5"},"success":true}'
[Thu Aug 10 16:36:11 UTC 2023] Restarting HTTP services failed
[Thu Aug 10 16:36:11 UTC 2023] GET
[Thu Aug 10 16:36:11 UTC 2023] url='http://localhost:5000/webapi/entry.cgi?api=SYNO.API.Auth&version=7&method=logout'
[Thu Aug 10 16:36:11 UTC 2023] timeout=
[Thu Aug 10 16:36:11 UTC 2023] curl exists=0
[Thu Aug 10 16:36:11 UTC 2023] mktemp exists=0
[Thu Aug 10 16:36:11 UTC 2023] wget exists=0
[Thu Aug 10 16:36:11 UTC 2023] _CURL='curl --silent --dump-header /acme.sh/http.header  -L  --trace-ascii /tmp/tmp.NBa0gwRw7y  -g '
[Thu Aug 10 16:36:11 UTC 2023] ret='0'
[Thu Aug 10 16:36:11 UTC 2023] response='{"success":true}'
[Thu Aug 10 16:36:12 UTC 2023] Success

This "http restart failed" message I'll do get every time. Don't know if this is relevant. Nevertheless the certificate is deployed into Syno and is visible in the GUI. Also the script says "success" at the end.

Can you - just for testing, of course - remove the lines for username & password as well, re-run the deployment script

Done. Deleted user & password line from file and re-ran the script. Username & Password line were NOT written into the file. In fact (in both deploy runs I've done) NO SYNO_ related line has been added to the conf file at all.

[Eagle3386]

I'm not exactly sure, which page you mean by this? This one? https://github.com/acmesh-official/acme.sh/wiki/Run-acme.sh-in-docker ?

I meant something like https://registry.hub.docker.com/_/nextcloud.
However, I read the Wiki you've linked to & there's nowhere any hint regarding the --home parameter - so where's the actual example.com[_ecc] folder & the account.conf file located?

Or is it done - though not mentioned anywhere - like with any other Docker container & you have to use environment variables for properly configuring our environment variables, i.e. SYNO_* ones?

I hope, this is the right part. Of course I have the full debug 3 log, if you need more to analyze:

Yes & no. 😅

No, because I also need to look at the part prior to END CERTIFICATE (though I don't care about the actual certificate data, I mean everything after the log line Logging into [Host]:[Port].

Yes, because:

[Thu Aug 10 16:36:11 UTC 2023] Restarting HTTP services failed

Is nothing to immediately bother yourself with - it might just be that restart_httpd returned with false, although the actual deployment succeeded.

[Thu Aug 10 16:36:11 UTC 2023] url='http://localhost:5000/webapi/entry.cgi?api=SYNO.API.Auth&version=7&method=logout'

So, you're on DSM 7. Good to know, thanks!

[Thu Aug 10 16:36:11 UTC 2023] response='{"success":true}'
[Thu Aug 10 16:36:12 UTC 2023] Success

This "http restart failed" message I'll do get every time. Don't know if this is relevant. Nevertheless the certificate is deployed into Syno and is visible in the GUI. Also the script says "success" at the end.

As already stated above, that one failed message is nothing to worry - the Docker container simply can't restart DSM's Nginx instance, so nevermind.
However, that's at least 1 reason to think about the whole Docker thing for DSM. There's not a real benefit compared to a local install onto the NAS itself - quite the opposite is actually true: a local install could reload Nginx, MailPlus & several services more directly upon deployment, hence the certificate would be used immediately.

Besides, that Success log line at the end is for the logout call, so that probably always returns like this. 😉

Done. Deleted user & password line from file and re-ran the script. Username & Password line were NOT written into the file. In fact (in both deploy runs I've done) NO SYNO_ related line has been added to the conf file at all.

This clearly proves that it's not a bug of my code changes, but a general issue of how the "Docker way" acts/handles the configuration file.
I suspect the container either has no directory/file to write such information to & instead requires environment variables added to the container (the typical Docker way…), you simply might have forgotten to set this up (as the Wiki page doesn't tell you to - probably because it was never meant to be executed on a Synology NAS with Docker that way) or, at the very least, it's simply not possible for Synology DSM to be provided with renewed certificates via Docker containers & saved configuration settings.

[jaydee73]

so where's the actual example.com[_ecc] folder & the account.conf file located?

Both are located in a folder called acme.sh in the root directory of the container. The folder is mounted from a local directory on my NAS into the container.
The acme.sh script itself (and the sub-directories) are located at /root/.acme.sh in the docker container.
I can see the acme-script searching for the folders when it is executed (using debug).

In general (I probably hinted to this already anywhere...), I mainly used a tutorial to get this whole thing up and running: https://www.christosgeo.com/2022/02/03/renew-lets-encrypt-certificates-on-synology-using-acme-sh/ (please don't sentence me for that ;-) ).
Why I am using a container instead of a direct installation? I like it to have separated micro services in a dedicated environment instead of installing stuff "bare metal" on my Syno-Linux.

This clearly proves that it's not a bug of my code changes

You are probably right.

I mean everything after the log line Logging into [Host]:[Port].

Here we go (I deleted my password and changed the domain name stuff in the log. Hopefully everything else is not security sensitive...):

[Thu Aug 10 16:35:44 UTC 2023] api_version='7'
[Thu Aug 10 16:35:44 UTC 2023] Logging into localhost:5000
[Thu Aug 10 16:35:44 UTC 2023] od exists=0
[Thu Aug 10 16:35:44 UTC 2023] _url_encode
[Thu Aug 10 16:35:44 UTC 2023] _hex_str=' 63 65 72 74 61 64 6d 69 6e'
[Thu Aug 10 16:35:44 UTC 2023] od exists=0
[Thu Aug 10 16:35:44 UTC 2023] _url_encode
[Thu Aug 10 16:35:44 UTC 2023] _hex_str=' 35 71 79 79 6a 50 4d 61 4a 36 32 64 35 47 74 42'
Enter OTP code for user 'certadmin': 502625
Enter device name or leave empty for default (CertRenewal): 
[Thu Aug 10 16:36:10 UTC 2023] GET
[Thu Aug 10 16:36:10 UTC 2023] url='http://localhost:5000/webapi/entry.cgi?api=SYNO.API.Auth&version=7&method=login&format=sid&account=certadmin&passwd=*********&otp_code=502625&enable_syno_token=yes&enable_device_token=yes&device_name=CertRenewal'
[Thu Aug 10 16:36:10 UTC 2023] timeout=
[Thu Aug 10 16:36:10 UTC 2023] curl exists=0
[Thu Aug 10 16:36:10 UTC 2023] mktemp exists=0
[Thu Aug 10 16:36:10 UTC 2023] wget exists=0
[Thu Aug 10 16:36:10 UTC 2023] _CURL='curl --silent --dump-header /acme.sh/http.header  -L  --trace-ascii /tmp/tmp.JiwSBvd2oX  -g '
[Thu Aug 10 16:36:11 UTC 2023] ret='0'
[Thu Aug 10 16:36:11 UTC 2023] response='{"data":{"account":"certadmin","device_id":"ABCDEF","ik_message":"","is_portal_port":false,"sid":"H1TJ_8I24WqrDs5H5QwjgOG5QhcHjzJEomzWtQc_l3U-VsN8KCBRLRuXkO_5XbPsKGMg-zZ-ayzijfnNJ2hrpk","synotoken":"xvzufzhkioFZQ"},"success":true}'
[Thu Aug 10 16:36:11 UTC 2023] SYNO_Device_ID='[hidden](please add '--output-insecure' to see this value)'
[Thu Aug 10 16:36:11 UTC 2023] Session ID='H1TJ_8I24WqrDs5H5QwjgOG5QhcHjzJEomzWtQc_l3U-VsN8KCBRLRuXkO_5XbPsKGMg-zZ-ayzijfnNJ2hrpk'
[Thu Aug 10 16:36:11 UTC 2023] SynoToken='xvzufzhkioFZQ'
[Thu Aug 10 16:36:11 UTC 2023] H1='X-SYNO-TOKEN: xvzufzhkioFZQ'
[Thu Aug 10 16:36:11 UTC 2023] APP
[Thu Aug 10 16:36:11 UTC 2023] 21:SAVED_SYNO_Username='certadmin'
[Thu Aug 10 16:36:11 UTC 2023] APP
[Thu Aug 10 16:36:11 UTC 2023] 22:SAVED_SYNO_Password='***deleted***'
[Thu Aug 10 16:36:11 UTC 2023] APP
[Thu Aug 10 16:36:11 UTC 2023] 23:SAVED_SYNO_Device_Name='CertRenewal'
[Thu Aug 10 16:36:11 UTC 2023] APP
[Thu Aug 10 16:36:11 UTC 2023] 24:SAVED_SYNO_Device_ID='ABCDEF'
[Thu Aug 10 16:36:11 UTC 2023] Getting certificates in Synology DSM
[Thu Aug 10 16:36:11 UTC 2023] POST
[Thu Aug 10 16:36:11 UTC 2023] _post_url='http://localhost:5000/webapi/entry.cgi'
[Thu Aug 10 16:36:11 UTC 2023] body='api=SYNO.Core.Certificate.CRT&method=list&version=1&_sid=H1TJ_8I24WqrDs5H5QwjgOG5QhcHjzJEomzWtQc_l3U-VsN8KCBRLRuXkO_5XbPsKGMg-zZ-ayzijfnNJ2hrpk'
[Thu Aug 10 16:36:11 UTC 2023] _postContentType
[Thu Aug 10 16:36:11 UTC 2023] curl exists=0
[Thu Aug 10 16:36:11 UTC 2023] mktemp exists=0
[Thu Aug 10 16:36:11 UTC 2023] wget exists=0
[Thu Aug 10 16:36:11 UTC 2023] _CURL='curl --silent --dump-header /acme.sh/http.header  -L  --trace-ascii /tmp/tmp.Flsaomwz4f  -g '
[Thu Aug 10 16:36:11 UTC 2023] _ret='0'
[Thu Aug 10 16:36:11 UTC 2023] response='{"data":{"certificates":[{"desc":"","id":"S2b81m","is_broken":false,"is_default":true,"issuer":{"city":"Taipel","common_name":"Synology Inc. CA","country":"TW","organization":"Synology Inc."},"key_types":"","renewable":false,"self_signed_cacrt_info":{"issuer":{"city":"Taipel","common_name":"Synology Inc. CA","country":"TW","organization":"Synology Inc."},"subject":{"city":"Taipel","common_name":"Synology Inc. CA","country":"TW","organization":"Synology Inc."}},"services":[{"display_name":"KMIP","display_name_i18n":"remote_key:kmip_tab_title","isPkg":false,"owner":"root","service":"kmip","subscriber":"kmip"},{"display_name":"FTPS","isPkg":false,"owner":"root","service":"ftpd","subscriber":"smbftpd"},{"display_name":"Synology Storage Console Server","display_name_i18n":"SYNO.SDS.iSCSI.Application:service:storage_console_server","isPkg":true,"owner":"root","service":"pkg-scsi-plugin-server","subscriber":"ScsiTarget"},{"display_name":"WebDAVServer","display_name_i18n":"SYNO.SDS.WebDAVServer.Instance:app:app_name","isPkg":true,"owner":"root","service":"webdav","subscriber":"WebDAVServer"},{"display_name":"Synology Drive Server","display_name_i18n":"SYNO.SDS.Drive.Application:app:pkg_name","isPkg":true,"owner":"SynologyDrive","service":"SynologyDrive","subscriber":"SynologyDrive"},{"display_name":"DSM Desktop Service","display_name_i18n":"common:web_desktop","isPkg":false,"multiple_cert":true,"owner":"root","service":"default","subscriber":"system","user_setable":true}],"signature_algorithm":"sha256WithRSAEncryption","subject":{"city":"Taipel","common_name":"synology","country":"TW","organization":"Synology Inc.","sub_alt_name":["synology"]},"user_deletable":true,"valid_from":"Dec 20 19:20:55 2022 GMT","valid_till":"Dec 21 19:20:55 2023 GMT"},{"desc":"mydomain.de","id":"Ssk3SL","is_broken":false,"is_default":false,"issuer":{"common_name":"R3","country":"US","organization":"Let's Encrypt"},"key_types":"ECC","renewable":false,"services":[{"display_name":"gotify.mydomain.de","isPkg":false,"multiple_cert":true,"owner":"root","service":"80fa6d74-71d9-4648-aae8-f2f7cb36f324","subscriber":"ReverseProxy","user_setable":true},{"display_name":"vault.mydomain.de:5555","isPkg":false,"multiple_cert":true,"owner":"root","service":"da1568fd-2ba4-471f-beee-a7ecce1e7d4b","subscriber":"ReverseProxy","user_setable":true},{"display_name":"nas.mydomain.de:4433","isPkg":false,"multiple_cert":true,"owner":"root","service":"e771a6f3-fd31-4791-86a3-53ace43923cb","subscriber":"ReverseProxy","user_setable":true}],"signature_algorithm":"sha256WithRSAEncryption","subject":{"common_name":"mydomain.de","sub_alt_name":["*.mydomain.de","mydomain.de"]},"user_deletable":true,"valid_from":"Aug  3 16:38:21 2023 GMT","valid_till":"Nov  1 16:38:20 2023 GMT"},{"desc":"vaultwarden","id":"X4SswQ","is_broken":false,"is_default":false,"issuer":{"common_name":"R3","country":"US","organization":"Let's Encrypt"},"key_types":"","renewable":false,"services":[],"signature_algorithm":"sha256WithRSAEncryption","subject":{"common_name":"*.mydomain.de","sub_alt_name":["*.mydomain.de"]},"user_deletable":true,"valid_from":"Jun 25 06:22:33 2023 GMT","valid_till":"Sep 23 06:22:32 2023 GMT"},{"desc":"Synology QuickConnect Certificate","id":"lTnIL8","is_broken":false,"is_default":false,"issuer":{"common_name":"R3","country":"US","organization":"Let's Encrypt"},"key_types":"RSA/ECC","renewable":true,"services":[],"signature_algorithm":"sha256WithRSAEncryption","subject":{"common_name":"nas-*****-ds720.direct.quickconnect.to","sub_alt_name":["*.nas-name-ds720.direct.quickconnect.to","nas-****-ds720.direct.quickconnect.to"]},"user_deletable":false,"valid_from":"Aug  2 02:33:52 2023 GMT","valid_till":"Oct 31 02:33:51 2023 GMT"}]},"success":true}'
[Thu Aug 10 16:36:11 UTC 2023] escaped_certificate='test3'
[Thu Aug 10 16:36:11 UTC 2023] id
[Thu Aug 10 16:36:11 UTC 2023] base64 single line.
[Thu Aug 10 16:36:11 UTC 2023] APP
[Thu Aug 10 16:36:11 UTC 2023] 25:SAVED_SYNO_Certificate='__ACME_BASE64__START_dGVzdDM=__ACME_BASE64__END_'
[Thu Aug 10 16:36:11 UTC 2023] Generate form POST request
[Thu Aug 10 16:36:11 UTC 2023] default='This is NOT the default certificate'
[Thu Aug 10 16:36:11 UTC 2023] Upload certificate to the Synology DSM
[Thu Aug 10 16:36:11 UTC 2023] POST
[Thu Aug 10 16:36:11 UTC 2023] _post_url='http://localhost:5000/webapi/entry.cgi?api=SYNO.Core.Certificate&method=import&version=1&SynoToken=xvzufzhkioFZQ&_sid=H1TJ_8I24WqrDs5H5QwjgOG5QhcHjzJEomzWtQc_l3U-VsN8KCBRLRuXkO_5XbPsKGMg-zZ-ayzijfnNJ2hrpk'
[Thu Aug 10 16:36:11 UTC 2023] body='----------------------------20230810163611
Content-Disposition: form-data; name="key"; filename="test3.mydomain.de.key"
Content-Type: application/octet-stream

[Eagle3386]

Will reply more extensively tomorrow, but for now: remove your real domain & especially device ID from the log.
And 1 URL still contains your password in plaintext, so that should be removed ASAP, too.

[jaydee73]

Done. But couldn't find any line where my pw is still readable...

[jaydee73]

I've thought a little bit about the whole thing and I had an idea...

As mentioned, I do have a conf file in the acme.sh folder. In this file, I added all the stuff previously and the scripts (acme, dns script, notify script) also wrote into this file. I thought "maybe the syno-script writes into another conf file" (as there weren't any errors while executing the "_savedeployconf" command.

So I went into the shell and searched for *.conf files anywhere else....tadaaa....there is another conf file in the certificate folder (/acme.sh/test3.mydomain.de_ecc) named test3.mydomain.de.conf.

Opening this conf file showed several SYNO_ lines, and also the device ID line (and username, and password....)

Until now, I have never looked into the certificate folder and for that reason, I've never seen the second conf file... :-(

What do I learn about this? I don't exactly know...is this expected behaviour? Can acme.sh use both conf files? If so, my problem wouldn't be a problem at all as the device id line is there, but simply in another conf file.

Maybe this is helpful for you...

[Eagle3386]

First of all: Of course, there's a second, separate configuration file apart from ACME.sh's one - how else could you deploy several certificates to several hosts with crucial information differing between at least 2 such hosts?!
That's why I already mentioned the example.net[_ecc] folder in my previous message(s)…

Regarding the rest of your previous message(s):

1. Password was part of the URL in line 6 when I looked at it, but it's redacted now, so no worries. 😉
2. Looking at the latest snippet, I'm spotting several issues - not related to ACME.sh in any way, but with your actual certificates:
   a) Synology's self-signed, hence invalid certificate is used for several services - why is that?
   b) As stated in the fourth last log line (very long line, had to split it several times):

{
  "desc":"mydomain.de",
  # … Left out for brevity…
  "is_default":false,
  "issuer":{"common_name":"R3","country":"US","organization":"Let's Encrypt"},
  "key_types":"ECC",
  # … Left out for brevity…
  "signature_algorithm":"sha256WithRSAEncryption",
  "subject":{"common_name":"mydomain.de","sub_alt_name":["*.mydomain.de","mydomain.de"]},
  # … Left out for brevity…
},
{
  "desc":"vaultwarden",
  # … Left out for brevity…
  "is_default":false,
  "issuer":{"common_name":"R3","country":"US","organization":"Let's Encrypt"},
  "key_types":"",
  # … Left out for brevity…
  "signature_algorithm":"sha256WithRSAEncryption",
  "subject":{"common_name":"*.mydomain.de","sub_alt_name":["*.mydomain.de"]},
  # … Left out for brevity…
},
{
  "desc":"Synology QuickConnect Certificate",
  # … Left out for brevity…
  "is_default":false,
  "issuer":{"common_name":"R3","country":"US","organization":"Let's Encrypt"},
  "key_types":"RSA/ECC",
  # … Left out for brevity…
  "signature_algorithm":"sha256WithRSAEncryption",
  "subject":
  {
    "common_name":"nas-*****-ds720.direct.quickconnect.to",
    "sub_alt_name":["*.nas-name-ds720.direct.quickconnect.to","nas-****-ds720.direct.quickconnect.to"]
  },
  # … Left out for brevity…
}

which is basically just wrong, because Let's Encrypt's R3 instance can't issue any ECC certificate as is well-explained here: https://letsencrypt.org/certificates/

But apart from that, as everything works as expected, I'm finally closing this (already closed) PR as there's nothing wrong, just 2 files & that's how it's supposed to be.

[jeffxt]

@Eagle3386 Regarding your point:

4. Any subsequent deployment, i.e. when running via Synology DSM's Task Scheduler as the NAS guide explains, will then use a token (retrieved via previously mentioned manual run) & hence neither require another TOTP code nor the TOTP secret to generate a new TOTP code via some 3rd party tool.

Just to clarify, I will NOT need to enter a new TOTP code in e.g., ~2 months from now when the renewal script runs?

The reason I ask is because I assumed that the login would expire after some time and I would need to manually re-enter another TOTP code (like how DSM prompts if you haven't logged in, in a while). But please correct me if I'm wrong -- I just don't want to be surprised later and find out my Task Scheduler failed during the renewal!

[Eagle3386]

No, you're not re-entering a TOTP code any time after once running manual deployment with entering a TOTP code.

However, as mentioned in the deploy hook script's issue: currently, there's a bug in Synology's API (they're aware of it, but "can't" give an ETA for a fix) which breaks this & hence you've got to manually deploy every time.

[jeffxt]

No, you're not re-entering a TOTP code any time after once running manual deployment with entering a TOTP code.

However, as mentioned in the deploy hook script's issue: currently, there's a bug in Synology's API (they're aware of it, but "can't" give an ETA for a fix) which breaks this & hence you've got to manually deploy every time.

@Eagle3386 Thanks for clarifying. It's unfortunate to hear that a bug on Synology's side is preventing us from being able to automatically renew our certs. I found your comment in this thread, which is the one I'm assuming you're referring to?

And just to make sure I understand for the next renewal -- I'm assuming I can re-run this renew script, it'll prompt for the TOTP code, I enter it (like I did the first time to issue the cert), and then it should push into DSM -- correct?

/usr/local/share/acme.sh/acme.sh --renew -d "*.example.com" --home /usr/local/share/acme.sh --server letsencrypt

Btw, also wanted to say thanks for all your clear and direct communication in these threads, especially when you contacted Synology's support directly and shared their responses. You've really helped me understand how all this works much better!

[Eagle3386]

And just to make sure I understand for the next renewal -- I'm assuming I can re-run this renew script, it'll prompt for the TOTP code, I enter it (like I did the first time to issue the cert), and then it should push into DSM -- correct?

Nope. --renew is for renewing. Executing it right after a successful run will simply skip any renewing at all.
You've got to execute --deploy for the second attempt.

Besides, the script is designed to work with a working Synology API, not a broken / bugged one. Hence, you've got 3 options:

1. Make sure that SYNO_Device_Id is empty prior to the deployment. Otherwise, the script assumes that TOTP-less login works.
2. Simply copy/move the certificate files to some DSM accessible folder, e.g., your home folder, log into your DSM with an admin user & import it via DSM Control Center manually - that's pretty much the "UI way" of doing what the DSM API normally does…

(…)
Btw, also wanted to say thanks for all your clear and direct communication in these threads, especially when you contacted Synology's support directly and shared their responses. You've really helped me understand how all this works much better!

You're welcome. I'm just glad, it was helpful to other people than me, too.

[jaydee73]

No, you're not re-entering a TOTP code any time after once running manual deployment with entering a TOTP code.

However, as mentioned in the deploy hook script's issue: currently, there's a bug in Synology's API (they're aware of it, but "can't" give an ETA for a fix) which breaks this & hence you've got to manually deploy every time.

As there was a new DSM release some days ago (DSM 7.2.1-69057 Update 5), do you have any informations if the bug was fixed with this version? Or has anyone tested this already?

[Eagle3386]

As the release notes state, that update isn't even targeting that bug.
TBH, I don't expect any update for this prior to release of DSM 7.3 or something like that.

[alidaf]

I had this working for months and now my certificates have expired. The script runs, apparently successfully, but the certificates are still out of date. There was a recent DSM update - did this break something?